Designing Machine Learning Solutions for AI Based Embedded Systems

Machine Learning is moving to On-Device or Edge for reasons of latency, power, security. ML Algorithms that were once designed to work on high end GPUs & not on embedded systems. This brings in new sets of challenges in designing ML solutions for embedded involving small memory footprints, limited to no float point operations, rich instruction set of fixed-point operations, proprietary software tool chains, libraries. This paper talks about addressing these ML challenges for Embedded Systems and challenges required to support wide range of ML frameworks, network types and layers, performance, accuracy loss. Our proposed solution consists of: €¢ converting pre-trained floating-point network models of some of the popular ML frameworks into a common proprietary model format €¢ custom quantization techniques to quantize model €¢ selection of most optimum library function, kernel fusion, kernel rejection €¢ tiling and DMA management of data €¢ generate highly optimized platform specific executable