Seed size is controlled not only by intrinsic genetic factors but also by external environmental signals. Here, we report a major quantitative trait locus (QTL) gene for seed size and weight on chromosome 1 (SSW1) in Arabidopsis, and we found SSW1 acts maternally to positively regulate seed size. Natural variation in SSW1 contains three types of alleles. The SSW1Cvi allele produces larger seeds with more amino acid and storage protein contents than the SSW1Ler allele. SSW1Cvi displays higher capacity for amino acid transport than SSW1Ler due to the differences in transport efficiency. Under low nitrogen supply, the SSW1Cvi allele exhibits increased seed yield and nitrogen use efficiency (NUE). Locations of natural variation alleles of SSW1are associated with local soil nitrogen contents, suggesting that SSW1 might contribute to geographical adaptation in Arabidopsis. Thus, our findings reveal a mechanism that coordinates seed growth and NUE, suggesting a potential target for improving seed yield and NUE in crops.