It is of great practical significance to guide rural middle-aged and elderly people to seek medical treatment rationally and to improve the efficiency of their medical service utilization to achieve rural revitalization and “healthy China”. From the perspective of health differences and based on a data of China Health and Retirement Longitudinal Study (CHARLS) from 2013 to 2018, this paper analyzed the reality and the characteristics of patients’ medical choices and discussed the impacts of the Integrated Medical Insurance System (IMIS) on the primary health care visit behaviors of rural middle-aged and elderly people from both aspects of outpatients and inpatients by the cluster analysis, the time-varying DID and the instrumental variable methods. Results show that the proportion of outpatient and inpatient primary care for the middle-aged and elderly in rural areas was 63.2% and 44.3%, respectively. The proportion of patients in the integrated areas who went to the nearest hospital was higher than that in the non-integrated areas, and with the passage of time, the medical choice of patients with different health conditions tended to be rational. The IMIS can effectively guide the outpatients and inpatients of rural middle-aged and elderly to seek medical treatment nearby, which has a greater significant impact on outpatients’ choice. The IMIS can significantly improve the probability of outpatients with good health level to seek medical treatment nearby and guide inpatients with medium health conditions to seek medical treatment nearby. However, the willingness of unhealthy patients to select primary health care services is significantly weakened. The policy effect of the IMIS on rural middle-aged and elderly people’s access to medical care nearby has been increasing over time with the implementation of the reform. In addition, the “single system, single-standard” model can more effectively guide patients to select primary health care services than “single system, multi-standard” model. Therefore, this paper suggests: refining the compensation policy for medical insurance for urban and rural residents based on health differences, improving the quality of rural primary care services, and gradually promoting the integration process of medical insurance for urban and rural residents.