Paper Accepted to DATE 2027

DATE 2027

We are happy to share that our work on running large language models efficiently on edge hardware will appear at DATE 2027, the Design, Automation and Test in Europe Conference. The paper, titled “Optimizing LLM Inference on Heterogeneous Edge SoCs with Operator-Level Controls”, will be presented at the conference’s 30th edition in Dresden, Germany, from March 22 to 24, 2027.

Modern edge SoCs pair CPU cores with NPUs, but popular inference engines only let users split a model between them layer by layer. Our work shows that a matrix multiplication’s speed on each processor depends strongly on its shape. The NPU can also address only a limited amount of memory. We first profile every distinct matrix shape on the CPU and on each NPU configuration. We then choose which operations to run on the NPU within its memory limit. To make this practical, we built a Rockchip NPU backend for llama.cpp. On 18 combinations of models and quantization formats, this shape-aware approach was faster than both shape-unaware NPU offloading and CPU-only execution in every case. On average it raised prefill throughput by 32.9% and decode throughput by 20.2% over shape-unaware offloading.

Here is the abstract of the paper:

On-device large language model (LLM) inference on heterogeneous edge systems-on-chip (SoCs) can improve privacy, latency, and energy efficiency. However, efficiently exploiting CPU-NPU heterogeneity is challenging because matrix multiplication (MM) performance varies substantially across tensor geometries and accelerator configurations, while NPU execution may also face limited accelerator-addressable memory. We present a two-stage framework for shape-aware MM placement in edge LLM inference. Stage 1 profiles each unique MM geometry across CPU and NPU configurations, while Stage 2 selects MM instances for NPU execution under the memory budget, accounting for activation buffers shared by equal-geometry MMs. We realize this framework in a 3,000-LOC Rockchip NPU backend for llama.cpp supporting fine-grained placement, multi-core NPU execution, and efficient memory and layout management. Across 18 model-quantization pairs, shape-aware placement improves average prefill/decode throughput by 32.9%/20.2% over the shape-unaware first-come, first-served (FCFS) strategy and 253.8%/12.6% over the CPU-only strategy, outperforming both in every pair.