Abstract: From the perspective of coupled social-ecological system (SES), the rural tourism destination was considered as the local SES, the “tourism” as a disturbance to the system, and the “participation in tourism” as farmers’ adaptive livelihood strategy, and with the method of subjective and objective weighting, this paper examined the development history of rural tourism in the five selected villages of Fujian Province, evaluated the vulnerability of farmers’ livelihood, and analyzed the influencing factors. The results show that the farmers’ livelihood capital has been improved by the development of rural tourism but the local SES also suffered the disturbance and shocks from rural tourism. The farmers’ livelihood vulnerability is at middle-high level and it depends on different villages and different livelihood strategies. In addition, farmers’ livelihood capital is positively correlated with the cooperation among the sample villages and travel agencies, the capacity of village officials and the tourists’ civilization, and mutual exchange among different farmers. The adverse disturbance to the social system became the largest obstacle for farmers to reduce their livelihood vulnerability, and the high uncertainty of social capital and cognitive capital is the key driving force. In order to reduce farmers’ livelihood vulnerability and to achieve the sustainable development of rural tourism, the government should ensure the land increment income of farmers, guide farmers to organize themselves, guarantee farmers’ decision-making on the community public affairs through the supply of institutions, improve their natural and physical capital, establish higher coverage of social security system, and enhance farmers’ capability of resisting livelihood risks.