DevOps for AI workloads: capacity, failure modes and observability

The outage that takes down an inference service is almost never the one your web-ops instincts prepare you for. It is a driver that drifted out of step with a CUDA toolkit, a model file someone replaced by hand, a queue that grew for ninety seconds before any dashboard noticed, or a capacity plan derived from a single-stream benchmark that had nothing to do with production. GPU serving is a different operational discipline from running stateless web services, and the difference shows up in capacity planning, in failure modes, and in what you have to measure. This is the operating model we hold ourselves to.

Capacity planning: the number you need is not the number vendors quote

Inference capacity is quoted as throughput — requests or tokens or seconds-of-audio per second. Almost no production system is throughput-bound. What users experience is concurrency at a latency budget: how many simultaneous sessions the system can carry while staying inside the delay a human will tolerate. Those two numbers can differ by more than an order of magnitude on identical hardware, because the second one includes everything the first one excludes — queueing, preprocessing, network, TLS, batching delay.

Three rules follow, and they are the ones we see teams skip:

1. Define the budget before you benchmark. For live captioning ours is a median caption delay under 2.5 seconds; for a document pipeline it might be ninety seconds. The budget is a product decision, and without it “capacity” has no meaning. Every capacity figure should be quoted as N concurrent sessions at latency L, or it is marketing.
2. Benchmark the whole path, not the model. In our own live-transcription work the ceilings were repeatedly in transport and unbatched CPU preprocessing, never in the accelerator — and the largest single latency improvement at high concurrency came from changing how audio reached the model rather than anything inside it. A model-only benchmark would have sent us shopping for a second card that would have fixed nothing. The full campaign is written up in scaling live transcription.
3. Plan headroom explicitly, including memory. Co-locating several models on one accelerator is how you make self-hosting economical — speech recognition, translation, synthesis and vision can share a card. The moment you do, VRAM becomes a budget with a named owner, not a number you check when something crashes. Decide the reserve, write it down, and alert on it.

The corollary is uncomfortable and worth stating: a published capacity figure — including ours — tells you very little about your deployment. The same models on the same class of hardware moved by a factor of four to six for us purely through configuration. Treat any vendor number, ours included, as evidence that a range exists, not as a plan.

Squeeze the hardware before you buy more of it

The cheapest GPU is the one you don’t need. Before scaling out, scale in:

• Quantize — then verify. Int8 inference via runtimes like CTranslate2 usually delivers large throughput gains at negligible quality cost for speech and translation models. “Usually” is doing real work in that sentence: on one of the cards we tested, int8 was measurably slower than float16 for one of the two models we tried. Quantization is a property of the model/kernel/card combination, not a law.
• Batch across requests. Independent per-request inference leaves accelerators mostly idle. A batching scheduler with an explicit latency budget changes the throughput class of the same card — in our live-transcription work it was one of the levers behind a several-fold concurrency gain on the same card.
• Share the card deliberately. Running several inference processes on one GPU so their kernels interleave, or partitioning a large card with MIG, both beat leaving a card owned by one idle process. Which mechanism suits you depends on isolation requirements and card generation — this is a decision to benchmark, not to inherit.
• Profile before you procure. Every one of the ceilings above was invisible in a single-stream test. Buying capacity to fix a bottleneck you have not located is the most expensive mistake available in this field.

What actually breaks

GPU servers fail in ways ordinary servers don’t. The operational checklist we hold ourselves to:

1. Pin the driver/CUDA/runtime matrix. Most GPU outages we see are version drift: a driver update that desynchronizes from the CUDA toolkit or the inference runtime. New silicon makes this sharper — each GPU generation typically requires a newer CUDA and math-library baseline than your base image ships, so a container that built and ran a month ago simply will not start on the new card. Containerize the runtime, pin the driver, upgrade on a schedule with a rollback image — never opportunistically.
2. Treat models as deployable artifacts. Version them, checksum them, stage them, roll them back like any release. “Someone updated the model on the server” must not be a sentence anyone can say. Model files are the one part of an AI stack that teams routinely leave outside their release process, and it is where silent quality regressions come from.
3. Expect the failure modes hardware brings with it. ECC errors, thermal throttling under sustained load, a card that drops off the PCIe bus and needs a host reset, memory fragmentation after long-running processes cycle models. None of these look like application errors, and none of them page you unless you decided in advance that they should.
4. Load-test with production-shaped traffic, from more than one machine. Concurrency behaviour is nonlinear. Synthetic single-stream benchmarks predicted almost nothing about behaviour at full concurrency, and our first apparent “GPU ceiling” turned out to be the load generator’s own cores saturating. If your test client and your server are the same box, you are measuring the box.
5. Design for N+0.5 redundancy. Full duplicate GPU capacity is expensive. A pragmatic pattern: dedicated hardware for steady state, with a tested, automated path to spin up cloud GPUs as overflow or failover. You pay hourly rates only during incidents and peaks — which is exactly the load shape hourly pricing is good at. The word doing the work in that sentence is tested: an untested failover path is a story you tell auditors, not a control.

Observability: utilization is the metric that lies

The single most common monitoring mistake in AI serving is treating GPU utilization as the health signal. A card at 40% may be perfectly healthy or may be starving behind a CPU bottleneck; the number cannot tell you which. Ours was starving, for weeks, at a utilization figure that looked reassuring on a dashboard.

What we monitor instead, in roughly this order of usefulness:

• Queue latency in front of the model — the time a request waits before decode starts. This is the earliest and clearest indicator of saturation, and it moves long before utilization or error rates do.
• End-to-end latency percentiles against the stated budget, not averages. The median is a comfort; the tail is the user experience.
• VRAM headroom per card, alerted against the reserve you decided on rather than against “out of memory”.
• Throughput alongside concurrency, so you can tell a demand change from a performance regression.
• Hardware health — temperature, power draw, ECC counters, PCIe link state.
• Model provenance in logs — which version served which request. Without it, “quality dropped last Tuesday” is unanswerable.

Two operational habits make the difference between graphs and observability. Alert on the symptom the user feels (latency budget breached) and use the rest for diagnosis. And keep a saturation test you can re-run on demand, so “is this slow because of load?” is a five-minute question rather than an afternoon.

Where the workload should live

Nothing above argues against the cloud. The load profile decides:

The pattern we recommend and practice: rent by the hour to learn, commit to flat-rate dedicated hardware to run. Benchmark in the cloud where you can switch card types freely; once the model, quantization and concurrency envelope are known, move steady-state serving to dedicated hardware, where a predictable bill and a predictable latency profile arrive together.

And if you keep cloud GPUs for testing: automate their shutdown. An idle-guard that stops instances when utilization flatlines is the single highest-ROI script we have ever deployed. One forgotten GPU instance over a long weekend costs more than the engineer-hours to write it. Every team has that story; make yours short.

Sovereignty is a property of this architecture, not a side effect

For our regulated clients, operational control is only half the argument. Dedicated, self-operated GPU infrastructure means inference data — customer audio, documents, video frames — stays on hardware you control, in a jurisdiction you choose. Under the EU AI Act, operators of AI systems carry logging and record-keeping duties regardless of where the model runs; running it yourself makes those records something you already have rather than something you must request from a vendor. The operational argument and the GDPR/data-residency argument point at the same architecture. That alignment is rare; use it.

This is the operating model behind our cloud, DevOps & managed operations and the AI infrastructure case study; the models themselves are covered under AI & model engineering.

FAQ

How should we capacity-plan a self-hosted inference service? Quote capacity as concurrent sessions at a stated latency budget, never as raw throughput, and measure it end to end — media or request path included. Benchmark from more than one client machine, plan explicit VRAM headroom if you co-locate models, and re-measure on your own traffic shape rather than trusting anyone’s published figure, including ours.

What is the most common cause of GPU downtime in production? Version drift between the GPU driver, the CUDA toolkit and the inference runtime — not hardware failure. Pin all three, ship them in a container, and upgrade deliberately with a rollback image ready. A new card generation is the highest-risk moment, because it raises the minimum toolkit version your image must contain.

Which GPU metrics should we actually alert on? Queue latency in front of the model and end-to-end latency percentiles against your stated budget. Utilization is a diagnostic signal, not a health signal — a card at moderate utilization may be starving behind a CPU bottleneck. Add VRAM headroom against a deliberate reserve, plus hardware health counters for temperature and ECC.

Rajan Jain is the CEO of Vaagmodo and leads our AI and infrastructure engineering, including the GPU serving stack behind our live transcription work. Talk to us about your infrastructure: contact or info@vaagmodo.com.