Research on Lightweight Deployment Technology of Large Language Model for Edge Intelligent Perception Scenarios

With the significant progress made by large language model（LLM）in the field of natural language processing，deploying them in edge intelligent perception scenarios to achieve real-time semantic understanding and intelligent interaction has become an important research trend.However，the inherent computational power，memory，and power consumption constraints of edge platforms，combined with the massive parameters of LLM，present a significant contradiction，severely restricting the large-scale deployment of LLM.Firstly，using the NVIDIA Jetson Orin NX 16GB module as the hardware deployment platform and ERNIE-4.5-0.3B as the benchmark model，a prediction model for inference performance based on Roofline is established.Secondly，the sensitivity mechanism of the word table mapping layer in small models to quantization errors is revealed，and a differentiated quantization strategy is proposed accordingly.Finally，the logical reasoning ability of the large model is transferred to the small model using the offline knowledge distillation method.Experimental results show that the theoretical performance prediction error is less than 5% compared to the actual evaluation，the PTQ W8A16 scheme achieves a 30.43% inference acceleration with a compression ratio of 1.55 while maintaining semantic accuracy，and the accuracy of the Countdown inference task increases from 2.20% to 15.40%，providing a beneficial engineering reference for the construction of edge semantic intelligent perception systems in resource-constrained environments.