To improve the recognition efficiency of surface defects in castings, this article first uses median filtering algorithm to denoise the defect image to distinguish between defects and background. Then, gray threshold method is used to segment the image, and the processed image is sent to the improved RefineDet network structure. Improving the RefineDet network structure mainly improves the network depth and incorporates dataset augmentation algorithms. Finally, an experimental platform was built to train, recognize, and compare the collected image dataset. The results show that the accuracy of detecting porosity, blowhole, and flaw defects is 95.6% and 97.3% and 98.15%, the method proposed in this article is accurate and efficient.