The molecular mechanisms for the retrograde motor dynein–dynactin to unload its cargoes at their final destination remain to be elucidated. In this study, we have investigated the regulatory mechanism underlying release of retromer-associated cargoes at the trans-Golgi network (TGN). We report that phosphotidylinositol-4-phosphate (PtdIns(4)P), a Golgi-enriched phosphoinositide, negatively regulates the protein–protein interaction between the p150^Glued subunit of dynein–dynactin and the retromer component SNX6. We show that PtdIns(4)P specifically facilitates dissociation of retromer-mediated membranous cargoes from the motor at the TGN and uncover an important function for PtdIns(4)P in the spatial control of retrograde vesicular trafficking to the TGN membrane. PtdIns(4)P also regulates SNX4-mediated retrograde vesicular trafficking to the endocytic recycling compartment by modulating its interaction with dynein. These results establish organelle-specific phosphoinositide regulation of motor–cargo interaction as a mechanism for cargo release by molecular motors at target membrane.