While rural tourism plays an important role in developing rural economy, improving rural living environment, and optimizing rural industrial structure, rural cultural ecology can be considered as perfect combination of rural culture and natural environment, which give great momentum to the stabilization of rural society. Based on the cultural ecology theory, this paper analyzes the cultural and ecological imbalance in the development of rural tourism, and discusses the ways to solve the imbalance issues. Result shows that during the rapid development of rural tourism in China, the rural cultural ecology system faces two imminent problems demanding solutions from two perspectives: 1) from time perspective, rural traditional culture has been losing its “authenticity” as time goes by; and 2) from space perspective, tourism developers have been destroying the environment of cultural ecology, which is the foundation of rural traditional culture. The intrinsic nature of rural cultural ecology system, the concept of misunderstanding and different techniques during rural tourism development, and the inadequate engagement of villagers in tourism development are major causes for the ecological imbalance of rural culture. Therefore, this paper suggests that the best ways to build the rural cultural ecology in rural tourism development include: to formulate a sound mechanism of tourism development to work out a reasonable tourism development pattern; to enhance the planning and designing level to provide technical support for rural tourism development, and to establish an effective community engagement mechanism to encourage the villagers’ participation.