Based on the provincial production cost data of rice, wheat, and corn and applying a stochastic frontier cost function, this paper investigated the driving forces of rising production costs of main grains in China. Results show that: 1) rising grain production costs are mainly attributed to increased input prices and deteriorated scale effects. On the contrary, both technological progress and efficiency change, particularly the former one, can lower the production costs; 2) increasing prices of labor, fertilizer and other inputs contribute to the soaring production costs, of which rising labor cost has the largest impact; 3) the scale effect of rice declined slightly, while that of wheat and corn increased slightly; and 4) technological progress can reduce main grain production costs, especially for rice; while efficiency change only has limited influence on reducing production costs. This study implicates that increasing production scale and promoting technology and mechanization in wheat and corn production can reduce production costs. In addition, developing value-added agricultural production system (e.g., interplanting) and extending the value chain of agriculture products can serve as an alternative way to increase land yield, and stimulate farmers’ interests and improve China’s food security.