Grain number per panicle (GNP) is an important agronomic trait that contributes to rice grain yield. Despite its importance in rice breeding, the molecular mechanism underlying GNP regulation remains largely unknown. Here, we report that genetic variation in the CArG box-containing inverted repeat (IR) sequence of OsREM20 promoter alters its expression level and contributes to GNP variation among rice varieties through genetic analysis and transgenic validation. We revealed that the IR sequence regulates OsREM20 expression by affecting the direct binding of OsMADS34 to the CArG box within the IR sequence. The divergent pOsREM20^IR and pOsREM20^ΔIR alleles originated from different O. rufipogon accessions, and were independently inherited into the japonica and indica subspecies respectively during domestication. We further determined that IR sequence variations in the OsREM20 promoter can be utilized for germplasm improvement through either genome editing or traditional breeding. Our findings highlight exciting implications not only for characterizing the novel genetic variation that responsible for GNP diversity in rice, but also for revealing the molecular mechanism and potential application of the cis-regulatory element-containing IR sequence in the regulation of agronomically important gene expression.