Analyzing Cold Start Latency in SGLang

Level

Bachelor

Supervisor

Huzaifa Shaaban Kabakibo, Lin Wang

Status

Completed

Date of Completion

2026.09

Student

Tim Willeke

Description

Serverless computing is an attractive paradigm for LLM serving, as it scales GPU resources automatically with demand. Its critical limitation is cold start latency: under bursty workloads, new instances must be initialized, directly inflating time-to-first-token (TTFT). Our recent work [1] provided the first end-to-end characterization of the vLLM engine’s startup process, but no comparable study exists for SGLang [2]. Because SGLang differs fundamentally from vLLM, e.g., RadixAttention with a radix-tree KV cache, its initialization bottlenecks cannot simply be assumed to be the same.

Research question: What are the foundational initialization steps that govern cold start latency in the SGLang inference engine, and how do their durations scale across different models and configurations?

The thesis covers: (1) instrumenting the SGLang runtime to decompose startup into measurable phases such as framework bootstrapping, tokenizer initialization, weight loading and CUDA graph capture, and (2) benchmarking cold start latency across LLM families such as Llama-3 and Qwen on L40S and H100 GPUs to analyze how model architecture and size influence each phase.

Requirements

• Solid Python skills; experience with PyTorch and running LLMs on GPUs
• Basic understanding of LLM inference engines (vLLM, SGLang) and concepts such as KV cache, CUDA graphs and weight loading
• Familiarity with Linux, Docker and profiling/observability tools (OpenTelemetry, Prometheus) is a plus
• Comfortable with statistical analysis of benchmark data; interest in systems research

References

1. H. S. Kabakibo, A. Trivedi, and L. Wang. Breaking the Ice: Analyzing Cold Start Latency in vLLM. MLSys 2026.
2. L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024.