DevOps for AI/ML platforms — GPU, pipelines, production reliability

What regulates the infrastructure

What actually goes wrong here

• Accelerator capacity simply being unavailable in the required region and instance type — data-centre GPU lead times running roughly 36 to 52 weeks, and HBM memory reported sold out through 2026, mean capacity is procured months ahead and cannot be bought during an incident
• Power rather than silicon becoming the binding constraint: AI racks draw far more per rack than general compute, and grid connection approvals in major US and European markets run into years, so a site can hold installed GPUs it cannot fully energise
• A single node or interconnect fault killing an entire synchronous multi-node training job, where the cost of the failure is every GPU-hour since the last checkpoint multiplied by the whole cluster — which is why checkpoint interval, not MTBF, determines the real loss
• Collective-communication and fabric faults (NCCL errors, InfiniBand or NVLink link flaps) that manifest as a hang rather than a crash, so the job holds the cluster while making no progress
• Uncorrectable HBM/ECC errors and thermal or power capping, which either terminate long runs or silently reduce throughput so a run misses its schedule without ever erroring
• KV-cache exhaustion under long-context or high-concurrency serving: memory per request scales with context length, so a shift in prompt length distribution — not in request rate — causes out-of-memory failures and preemptions
• Queueing collapse when arrival rate exceeds batch throughput, where latency degrades non-linearly and the endpoint remains nominally up while becoming useless
• Cold-start latency for large models, where loading tens or hundreds of gigabytes of weights makes scale-out response times minutes long, defeating conventional autoscaling
• Storage and data-pipeline stalls starving accelerators, leaving very expensive hardware I/O-bound and idle
• Silent quality regressions from a model, quantisation, kernel or serving-framework change that availability and latency monitoring cannot detect at all

How demand behaves

Bimodal, and the two modes have opposite infrastructure requirements. Training is long-running, synchronous and all-or-nothing: a job occupies a fixed block of interconnected accelerators for hours to weeks and either completes or must resume from a checkpoint. Inference is short, latency-sensitive, spiky and priced per token, so it is dimensioned on concurrency and tail latency. Because idle accelerators dominate cost, both modes are managed through queueing, batching and reserved capacity rather than through elastic autoscaling — and in practice scaling out is limited by whether the hardware is physically obtainable, not by a scaling policy.

Data you will be holding

Three distinct classes with different constraints: training corpora, which may contain personal data and raise GDPR lawful-basis, purpose-limitation and copyright questions; inference-time prompts and outputs, which are unbounded in content and frequently carry customer confidential data, making retention and training-on-customer-data policies a contractual issue; and model weights themselves, which are trade secrets and a live subject of export-control policy — threshold-based controls on closed model weights have been proposed and withdrawn rather than durably established, so the applicable regime must be checked against current rules rather than assumed. The EU AI Act adds documentation duties for general-purpose models, including a sufficiently detailed public summary of training content.

Architecture this pushes you toward

Training and serving are effectively different infrastructures sharing a hardware type: training needs dense east-west bandwidth, gang scheduling and fast checkpoint storage, while serving needs weight caching on local NVMe, request batching and tight tail-latency control. Capacity is reserved and scheduled rather than autoscaled, so quota, queue policy and preemption tiers do the work that elasticity does elsewhere. Power density per rack, cooling, and the scheduler's awareness of network topology matter more than in general-purpose compute, and checkpoint frequency is a deliberate cost trade against expected failure rate.

Availability expectation

Hosted inference APIs are commonly offered around 99.9% monthly availability, but the meaningful commitments in this sector are latency-based — time-to-first-token and inter-token latency percentiles — because a technically 'available' endpoint that is queueing behind saturated accelerators is unusable. Training has no equivalent availability metric; it is measured as goodput, the fraction of allocated GPU-hours that produce retained progress after failures and restarts.