Objectives: The effect of 12 weeks of muscle strength training on the physical performance of elderly people was investigated. Methods: This study recruited 60 people older than 65 years and without a regular exercise habit. On the basis of their upper and lower limb muscle strength and walking speed, they were divided into a control group (15 participants) and three training groups, namely high physical fitness (18 participants), general fitness (16 participants), and physical weakness (11 participants) groups. The training groups trained twice weekly for 90 min for 12 weeks, whereas the control group kept their normal routine. Upper and lower limb muscle strength, one-leg stance, 10-m walking speed, and short physical performance battery (SPPB) score were assessed before and after the exercise intervention. Results: A total of 56 subjects completed this research. Except for the prevalence of chronic diseases, no significant differences were discovered in the basic data of the groups. After 12 weeks of incremental muscle training, the high physical fitness and general fitness groups achieved significant improvements in upper and lower limb muscle strength, one-leg stance, and 10-m walking speed. By contrast, the poor physical fitness group significantly improved their upper and lower limb muscle strength and 10-m walking speed. Upon further analysis, we discovered that the improvements in upper limb muscle strength in the high physical fitness and general fitness groups were significantly greater than those in the control group. Moreover, the improvements in lower limb muscle strength and one-leg stance in the high physical fitness and general fitness groups were significantly greater than those in the poor physical weakness and control groups. No clear differences in 10-m walking speed and SPPB score were found (p ＞ .05). Conclusions: The 12-week incremental muscle training benefitted the elderly participants with different physical fitness levels in every aspect of physical performance. This research can be provided as a reference to every community care center in Taiwan to demonstrate an effective exercise intervention and the benefits for elderly people with varying levels of physical fitness.