Skip to content

2026ΒΆ

InferenceOps: Building a Reproducible LLM Serving Lab

Introduction

Purpose

InferenceOps is a reproducible lab for studying LLM inference systems under real serving conditions: request replay, routing, GPU workers, controller policies, and Kubernetes deployment. The goal was not just to serve a model. The goal was to build enough infrastructure to compare serving decisions with repeatable workloads, real GPU runs, and inspectable artifacts.

Problem Statement

LLM serving performance depends on workload shape, prompt locality, routing policy, backend health, GPU scheduling, and deployment topology. A useful benchmark needs deterministic traces, real backend runs, raw metrics, and enough operational detail to explain why a system behaves the way it does.

What you'll learn

  • How I built a serving benchmark from a single vLLM baseline to a two-node GPU cluster.
  • Why deterministic trace replay matters for LLM inference evaluation.
  • How SGLang Gateway and vLLM Production Stack fit into an inference operations workflow.
  • What heuristic and agentic controllers can change, and what they cannot prove without strong baselines.
  • What broke during real Kubernetes GPU deployment and how I fixed it.

Werewolf Arena Benchmark: An Agentic Social Deduction Benchmark

Introduction

Purpose

Most LLM benchmarks sit in safe territory: code, math, or single-turn QA. Useful, but narrow. Social intelligence - deception, persuasion, coordination under uncertainty - is a different axis. Werewolf (Mafia) is a compact testbed for it, forcing agents to reason with hidden roles, influence votes, and adapt as information unfolds.

Problem Statement

Werewolf is a strong social-deduction benchmark because it forces agents to reason under hidden roles, persuade others in public dialogue, and adapt as partial information accumulates. A single misvote can swing the game, so decision quality, consistency, and role-specific play matter as much as raw win rate. Recent work like Werewolf Arena (2024) and WereWolf-Plus (2025) shows what this benchmark can capture and motivates a more reproducible, community-friendly evaluation stack. (Papers: Werewolf Arena, WereWolf-Plus)

What you'll learn

  • How Werewolf tests social reasoning and why it exposes behavior that win-rate-only metrics miss.
  • What recent papers contribute and where they stop short.
  • Why AgentBeats needs a reproducible harness with controlled baselines and clear submission flow.
  • How evaluator/assessee separation works in this benchmark.
  • Which metrics matter when you care about decision quality, not just outcomes.