Cloud-Edge Speculative Decoding for Multimodal LLMs

Level
Master
Supervisor
Lin Wang
Status
Completed
Date of Completion
2026.10

Description

Multimodal LLMs (e.g., LLaVA) are served almost exclusively on datacenter GPUs, which is costly and adds latency for end users, while increasingly capable edge devices such as the NVIDIA Jetson AGX Orin sit idle. Speculative decoding lets a small draft model propose tokens that a large target model verifies in parallel without changing the output. SpecEdge [1] showed that drafting on an edge GPU and verifying on a server works for text-only LLMs; DREAM [2], MSD [3], and EAGLE-2 [4] brought speculative decoding to vision-language models. Combining the two, namely multimodal drafting on the edge, verification in the cloud, is still unexplored.

The goal of this thesis is to explore how multimodal LLM inference can be split between an edge device and a cloud server using speculative decoding, so that latency and serving cost go down without changing the model’s outputs. Key questions include:

The student will build a prototype system on real edge and server hardware and evaluate it on standard multimodal benchmarks.

Requirements

References

  1. SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs. arXiv:2505.17052
  2. DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding. arXiv:2505.19201
  3. Speculative Decoding Reimagined for Multimodal Large Language Models (MSD). arXiv:2505.14260
  4. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv:2406.16858