We develop a more efficient lightweight network based on SE-ShuffleNet V2 to address the issues of large parameter sizes and sluggish feature extraction rates in large networks in the field of face recognition. First, to increase the network’s accuracy and inference speed, the ReLU activation function of the original ShuffleNet V2 basic unit is swapped out for a segmented linear activation function. Second, the SE attention mechanism is added to the lightweight network ShuffleNet V2, which may improve the effective feature weights and decrease the invalid feature weights, and the SE attention causes the network to focus on more helpful features. Finally, the addition of the Arcface loss function enhances the face recognition network’s capacity for categorization. Experiments indicate that the SE-ShuffleNet V2 network that we created achieves superior performance under the parameters of position and age. Particularly, the LFW accuracy is 99.38%. The algorithm presented in this research significantly increases face recognition accuracy when compared to the original ShuffleNet V2 network, therefore the additional parameters and longer inference times can be disregarded. To match the accuracy of substantial convolutional networks, we developed the lightweight SE-ShuffleNet V2.