InferenceOps: Building a Reproducible LLM Serving Lab
Introduction
Purpose
InferenceOps is a reproducible lab for studying LLM inference systems under real serving conditions: request replay, routing, GPU workers, controller policies, and Kubernetes deployment. The goal was not just to serve a model. The goal was to build enough infrastructure to compare serving decisions with repeatable workloads, real GPU runs, and inspectable artifacts.
Problem Statement
LLM serving performance depends on workload shape, prompt locality, routing policy, backend health, GPU scheduling, and deployment topology. A useful benchmark needs deterministic traces, real backend runs, raw metrics, and enough operational detail to explain why a system behaves the way it does.
What you'll learn
- How I built a serving benchmark from a single vLLM baseline to a two-node GPU cluster.
- Why deterministic trace replay matters for LLM inference evaluation.
- How SGLang Gateway and vLLM Production Stack fit into an inference operations workflow.
- What heuristic and agentic controllers can change, and what they cannot prove without strong baselines.
- What broke during real Kubernetes GPU deployment and how I fixed it.