In contrast enhancement of fuzzy defect image, details loss and noise expansion are east to occur, which brings difficulties to subsequent image analysis and defect recognition. Therefore, a fuzzy defect image restoration and enhancement method based on neural network is proposed. A double fusion neural network composed of a depth generation network and a discrimination network is designed. The residual of the denoised fuzzy image and the real image is output by the network, which is input into the discrimination network together with the real image, and the difference between the two is judged by the total loss function. To solve the problem of pixel coordinate value of fuzzy defect image, neural network is used to build a fast correction algorithm. Therefore, a fuzzy image restoration and enhancement method based on neural network is proposed to improve the image quality. By reconstructing the resolution of fuzzy defect image, a hierarchical enhancement method of fuzzy defect image region is constructed to achieve fuzzy defect image restoration and enhancement. The results show that the proposed method has high image processing ability in restoration and enhancement of fuzzy defect images. The fitting value of neural network is 0.92, which is significantly higher than that of the other two methods, indicating that the image restoration and enhancement method based on neural network has higher accuracy. Therefore, the restoration and enhancement method of fuzzy defect image based on neural network has a good restoration and enhancement effect, and can effectively meet the actual needs of people for high-quality images.