Phased small interfering RNAs (phasiRNAs) are abundantly expressed in anthers and linked to environment-related male fertility in grasses, yet how they function under different environmental conditions remains unclear. Here, we identified a rice (Oryza sativa) low temperature-induced Argonaute (AGO) protein, OsAGO1d, that is responsible for generating phasiRNAs and preserving male fertility at low temperature. Loss of OsAGO1d function causes low-temperature male sterility associated with delayed programmed cell death of tapetal cells during anther development. OsAGO1d binds miR2118 and miR2275 family members and triggers phasiRNA biogenesis; it also binds 21-nt phasiRNAs with a 5′ terminal U. In total, phasiRNAs from 972 loci are OsAGO1d-dependent. OsAGO1d protein moves from anther wall cells into meiocytes, where it loads miR2275 to produce 24-nt phasiRNAs. Together, our results show that OsAGO1d acts as a mobile signal to fine-tune phasiRNA production and this function is important for male fertility at low temperature.