The utilization of natural genetic variation greatly contributes to improvement of important agronomic traits in crops. Understanding the genetic basis for natural variation of grain size can help breeders develop high-yield rice varieties. In this study, we identify a previously unrecognized gene, named GSE5, in the qSW5/GW5 locus controlling rice grain size by combining the genome-wide association study (GWAS) with functional analyses. GSE5 encodes a plasma membrane-associated protein with IQ domains (IQD), which interacts with the rice calmodulin protein, OsCaM1-1. We found that loss of GSE5 function caused wide and heavy grains, while overexpression of GSE5 resulted in narrow grains. We showed that GSE5 regulates grain size predominantly by influencing cell proliferation in spikelet hulls. Three major haplotypes of GSE5 (GSE5, GSE5^DEL1+IN1 and GSE5^DEL2) in cultivated rice were identified based on the deletion/insertion type in its promoter region. We demonstrated that a 950-bp deletion (DEL1) in indica varieties carrying the GSE5^DEL1+IN1 haplotype and a 1212-bp deletion (DEL2) in japonica varieties carrying the GSE5^DEL2 haplotype associated with decreased expression of GSE5, resulting in wide grains. Further analyses indicate that wild rice accessions contain all the three haplotypes of GSE5, suggesting that the GSE5 haplotypes present in cultivated rice are likely to have originated from different wild rice accessions during rice domestication. Taken together, our results indicate that the previously unrecognized GSE5 gene in the qSW5/GW5 locus that is widely utilized by rice breeders controls grain size and reveal that natural variation in the promoter region of GSE5 contributes to grain size diversity in rice.