To address the problems of poor robustness and high complexity of current physical layer user authentication algorithms, a lightweight convolutional neural network (CNN) channel feature extraction algorithm was proposed to reduce the channel state response required for training by changing the form of network input, and a multi-user physical layer channel feature-assisted authentication mechanism was established based on this algorithm to design a detailed process from user registration to authentication, and multi-user authentication and network parameter update online were completed.Simulation results show that the proposed algorithm can complete multi-user authentication, obtain good detection performance with smaller training rounds, and require fewer training samples than existing multi-user authentication algorithms.