Application of chemical fertilizers, especially nitrogen (N), to crops has increased dramatically in the last half century and therefore developing crop varieties with improved N use efficiency (NUE) is urgent for sustainable agriculture. N utilization procedures generally can be divided into uptake, transport, and assimilation. Transporters for nitrate or ammonium acquisition and enzymes for assimilation are among the essential components determining NUE, and many transcription factors also play a pivotal role in regulating N use-associated genes, thereby contributing to NUE. Although some efforts in improving NUE have been made in various plants, the regulatory mechanisms underlying NUE are still elusive, and NUE improvement in crop breeding is very limited. In this review, the crucial components involved in N utilization and the candidates with the potential for NUE improvement in dicot Arabidopsis and monocot rice are summarized. In addition, strategies based on new techniques which can be used for dissecting regulatory mechanisms of NUE and also the possible ways in which NUE can be improved in crops are discussed.