With the advocacy of green development in agriculture and rural areas in China, the policy direction of fertilizer input has gradually changed from increasing production to improving efficiency. And how to improve the efficiency of fertilizer application has become a key issue. Based on a micro-survey data of rice growers in Hubei Province, this paper evaluated the fertilizer application efficiency by the stochastic frontier method and further analyzed the impacts of land fragmentation and land quality on fertilizer application efficiency by the CLAD model. Results show that the current fertilizer application efficiency of rice farmers was only 63.85%. If the efficiency loss was eliminated completely the fertilizer application efficiency could be improved by 36.15%. The degree of farmland fragmentation significantly reduces the efficiency of fertilizer application, which may be related to the hedge behavior of farmers. Cognition of farmland quality improves fertilizer application efficiency significantly, indicating that farmers’ scientific understanding of farmland quality could improve the fertilizer application efficiency. Therefore, to improve the efficiency of fertilizer application, this paper provides the following policy suggestions: speeding up the construction of laws and regulations on the integration of farmland and contiguity operation of farmland, accelerating the combination of farmland fragmentation management experience and innovation, strengthening the guidance of farmers’ correct cognition of fertilizer application, and promoting scientific cultivation and production mode.