Neural Network Optimizations for On-Device AI

On-device AI brings unprecedented capabilities and opportunities to endpoint devices with improved privacy, security, and reliability. Deploying on-device AI must consider various constraints of memory, power, latency, and cost. Therefore, neural network optimizations become critical for exploiting hardware features, reducing memory footprint and improving computation efficiency. In this talk, we present techniques for model optimizations such as pruning, clustering and quantization for enabling energy-efficient AI on resource-constrained devices. Optimization cascading is achieved where each technique preserves the preceding attributes. Under different optimization objectives, exhaustive search for finding the optimal sparsity, quantization and clustering levels for each layer is compute intensive and impractical. To counter this, we have developed efficient techniques to navigate through the large search space, thus enabling trade-offs between accuracy, latency and compressibility.