Transposable elements (TEs) are ubiquitously present in plant genomes and often account for significant fractions of the nuclear DNA. For example, roughly 40% of the rice genome consists of TEs, many of which are retrotransposons, including 14% LTR- and ～1% non-LTR retrotransposons. Despite their wide distribution and abundance, very few TEs have been found to be transpositional, indicating that TE activities may be tightly controlled by the host genome to minimize the potentially mutagenic effects associated with active transposition. Consistent with this notion, a growing body of evidence suggests that epigenetic silencing pathways such as DNA methylation, RNA interference, and H3K9me2 function collectively to repress TE activity at the transcriptional and posttranscriptional levels. It is not yet clear, however, whether the removal of histone modifications associated with active transcription is also involved in TE silencing. Here, we show that the rice protein JMJ703 is an active H3K4-specific demethylase required for TEs silencing. Impaired JMJ703 activity led to elevated levels of H3K4me3, the misregulation of numerous endogenous genes, and the transpositional reactivation of two families of non-LTR retrotransposons. Interestingly, loss of JMJ703 did not affect TEs (such as Tos17) previously found to be silenced by other epigenetic pathways. These results indicate that the removal of active histone modifications is involved in TE silencing and that different subsets of TEs may be regulated by distinct epigenetic pathways.