Data Center ML Inference

Overview

With the proliferation of machine learning (ML) such as LLMs, inference serving systems have become critical for the deployment of ML-powered applications. An inference serving system, typically deployed in a large-scale data center, takes existing pre-trained ML models and creates multiple instances of it to serve incoming inference requests at a high rate. The main challenge consists in minimizing the total resource consumption for serving a given number of inference requests while respecting the latency service-level objectives (SLOs) of the requests. The processing of an inference request can span multiple stages of processing involving both ML model inference and tool call executions, as known as agentic workflows, and can take multi-modal inputs that require complex preprocessing. To address such a challenge, we are exploring auto-scaling mechanisms to enable extreme elasticity for handling ML inference workloads and practical GPU sharing schemes for improving the GPU resource utilization.

People

Publications

  • EuroMLSys 2026
    Before the First Token: Benchmarking Data Preprocessing in Vision-Language Models
    Sepideh Zohdi, Lin Wang
    ACM European Workshop on Machine Learning and Systems (EuroMLSys), 2026
  • MLSys 2026
    Breaking the Ice: Analyzing Cold Start Latency in vLLM
    Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin Wang
    Annual Conference on Machine Learning and Systems (MLSys), 2026
  • IPDPS 2025
    It Takes Two to Tango: Serverless Workflow Serving via Bilaterally Engaged Resource Adaptation
    Jing Wu, Lin Wang, Quanfeng Deng, Chen Yu, Dong Zhang, Bingheng Yan, Fangming Liu
    IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2025
  • ICPE 2025
    IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency
    Saeid Ghafouri, Kamran Razavi, Mehran Salmani, Alireza Sanaee, Tania Lorido-Botran, Lin Wang, Joseph Doyle, Pooyan Jamshidi
    ACM/SPEC International Conference on Performance Engineering (ICPE), 2025
  • TSUSC 2025
    Working Smarter Not Harder: Hybrid Cooling for Deep Learning in Edge Datacenters
    Qiangyu Pei, Yongjie Yuan, Haichuan Hu, Lin Wang, Dong Zhang, Bingheng Yan, Chen Yu, Fangming Liu
    IEEE Transactions on Sustainable Computing (TSUSC), 2025
  • SoCC 2024
    InferCool: Enhancing AI Inference Cooling through Transparent, Non-Intrusive Task Reassignment
    Qiangyu Pei, Lin Wang, Dong Zhang, Bingheng Yan, Chen Yu, Fangming Liu
    ACM Symposium on Cloud Computing (SoCC), 2024
  • EuroMLSys 2024
    Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling
    Kamran Razavi, Saeid Ghafouri, Max Mühlhäuser, Pooyan Jamshidi, Lin Wang
    ACM European Workshop on Machine Learning and Systems (EuroMLSys), 2024
  • WWW 2024
    LambdaGrapher: A Resource-Efficient Serverless System for GNN Serving through Graph Sharing
    Haichuan Hu, Fangming Liu, Qiangyu Pei, Yongjie Yuan, Zichen Xu, Lin Wang
    ACM Web Conference (WWW), 2024
  • RTS 2024
    Inference Serving with End-to-End Latency SLOs over Dynamic Edge Networks
    Vinod Nigade, Pablo Bauszat, Henri E. Bal, Lin Wang
    Real-Time Systems (RTS), 2024
  • RTAS 2022
    FA2: Fast, Accurate Autoscaling for Serving Deep Learning Inference with SLA Guarantees
    Kamran Razavi, Manisha Luthra, Boris Koldehofe, Max Mühlhäuser, Lin Wang
    IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2022