The so-called defective coffee beans refer to coffee beans of poor quality. These blemish beans can give a cup of coffee an uncomfortable taste. At present, most of selecting defective beans is still artificially, and the selection is made through eye examination. However, this manual classification method not only consumes manpower and time, but also often causes the problem of unstable screening quality due to human negligence and inconsistent judgment standards of different people. These problems can be overcome by automatic detection mechanism. Therefore, this research proposes a model for automatically classifying coffee beans. This model uses VGG16 as the main architecture to improve the classification accuracy of coffee beans through transfer learning. Experimental results show that this model can achieve 99% accuracy.