The phenomenon of farmland “non-grain use” in Hunan Province is becoming more and more prominent. To explore its spatiotemporal differentiation and driving mechanism, and to propose mitigation strategies is of great significance for local agricultural economic development and national food security. Based on the geospatial attribute data of Hunan Province from 2003 to 2020, this study adopts the comprehensive evaluation method, the spatial autocorrelation analysis model and the geographical probe model to investigate the spatiotemporal evolution patterns and driving mechanisms of farmland “non-grain use” at the county level, and put forward the alleviation strategies. Results indicate that: 1) The level of farmland “non-grain use” in Hunan Province fluctuated, posing a concerning situation for grain security. The “non-grain use” level of farmland decreased slightly from 2007 to 2017 compared with the previous period, but increased significantly from 2018 to 2020; 2) The spatial differentiation characteristics of “non-grain use” at the county level are significant, displaying a “northwest high-southeast low” distribution pattern, with an annual trend of aggregation from the periphery towards the center. The level of farmland “non-grain use” in Hunan Province has a significant positive spatial agglomeration effect. The high-high agglomeration areas are mainly distributed in Fenghuang County, Luxi County, Mayang Miao Autonomous County, Zhijiang Dong Autonomous County; and 3) Grain unit yield is the primary driving factor for farmland “non-grain use”, while the lowest driving degree attributed to terrain undulation. The interactive driving force of these factors accounts for 93.2% of the explanatory power for “non-grain use”. Based on these findings, Hunan Province should carry out differentiated regional “non-grain use” control countermeasures, improve grain output level, and increase farmers’ economic income to effectively alleviate the trend of “non-grain use” of farmland.