Cloud-Edge Speculative Decoding for Multimodal LLMs
- Level
- Master
- Supervisor
- Lin Wang
- Status
- Completed
- Date of Completion
- 2026.10
Description
Multimodal LLMs (e.g., LLaVA) are served almost exclusively on datacenter GPUs, which is costly and adds latency for end users, while increasingly capable edge devices such as the NVIDIA Jetson AGX Orin sit idle. Speculative decoding lets a small draft model propose tokens that a large target model verifies in parallel without changing the output. SpecEdge [1] showed that drafting on an edge GPU and verifying on a server works for text-only LLMs; DREAM [2], MSD [3], and EAGLE-2 [4] brought speculative decoding to vision-language models. Combining the two, namely multimodal drafting on the edge, verification in the cloud, is still unexplored.
The goal of this thesis is to explore how multimodal LLM inference can be split between an edge device and a cloud server using speculative decoding, so that latency and serving cost go down without changing the model’s outputs. Key questions include:
- How should the multimodal inference pipeline (vision encoding, drafting, verification) be partitioned across edge and server?
- Which drafting strategies are well suited to resource-constrained edge devices, and how can the network round-trip between edge and server be hidden?
- What are the resulting trade-offs in latency, throughput, and resource usage compared to server-only serving?
The student will build a prototype system on real edge and server hardware and evaluate it on standard multimodal benchmarks.
Requirements
- Good Python/PyTorch skills
- Experience with an inference engine (vLLM, TensorRT-LLM) is a plus
- Comfortable with Linux, CUDA environments, and Docker
- Interest in systems research for ML
- Prior knowledge of speculative decoding or VLMs is helpful but not required
References
- SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs. arXiv:2505.17052
- DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding. arXiv:2505.19201
- Speculative Decoding Reimagined for Multimodal Large Language Models (MSD). arXiv:2505.14260
- EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv:2406.16858