This study integrates wearable augmented reality and wearable virtual reality into the STEM model and examines the effect of the sources of science learning self-efficacy on high school students’ science learning self-efficacy and performance. A total of 98 high school students from three basic physical experiment courses participated in this study. They were divided into a traditional teaching group, a wearable augmented reality group, and a mixed group with both wearable augmented reality and virtual reality. After learning, participants completed a sources of science learning self-efficacy instrument, a science learning self-efficacy scale, and a learning performance test. The results of the multiple regression analysis indicated that mastery experience positively predicted science learning self-efficacy in the traditional group. As for the mixed group, the vicarious experience positively predicted science learning self-efficacy; the somatic and emotional states negatively predicted learning performance. The results showed that traditional teaching still has benefits for predicting learners’ science learning self-efficacy by mastery experience. In comparison to traditional teaching, wearable augmented reality mixed with wearable virtual reality can predict not only science learning self-efficacy by vicarious experience but also learning performance by somatic and emotional states. In other words, wearable augmented reality mixed with wearable virtual reality can predict two directions. The study suggests that instructors can use the strategies of vicarious experience as well as somatic and emotional states with wearable augmented reality mix wearable virtual reality to help learners in the science experiment courses, which improves learners’ science learning self-efficacy and performance.