During meiotic prophase I, chromatin undergoes dynamic changes to establish a structural basis for essential meiotic events. However, the mechanism that coordinates chromosome structure and meiotic progression remains poorly understood in plants. Here, we characterized a spontaneous sterile mutant MS5bMS5b in oilseed rape (Brassica napus L) and found its meiotic chromosomes were arrested at leptotene. MS5 is preferentially expressed in reproductive organs and encodes a Brassica-specific protein carrying conserved coiled-coil and DUF626 domains with unknown function. MS5 is essential for pairing of homologs in meiosis, but not necessary for the initiation of DNA double-strand breaks. The distribution of the axis element-associated protein ASY1 occurs independently of MS5, but localization of the meiotic cohesion subunit SYN1 requires functional MS5. Furthermore, both the central element of the synaptonemal complex and the recombination element do not properly form in MS5bMS5b mutants. Our results demonstrate that MS5 participates in progression of meiosis during early prophase I and its allelic variants lead to differences in fertility, which may provide a promising strategy for pollination control for heterosis breeding.