Engineering Debt at Scale: Three Structural Failures in Production AI Systems

[Brindal P]

Most of what breaks AI systems in production has nothing to do with the model.

I spend a good chunk of my job reviewing code: production systems and open-source contributions as an engineer, plus the code behind systems and software papers as an academic peer reviewer. No matter the context, the pattern is the same: state-of-the-art math wrapped in software that can't survive a Tuesday afternoon of real traffic. Basic engineering discipline seems to evaporate the moment import torch shows up.

Based on hundreds of these reviews, the failures cluster into three recurring architectural flavors. Here's what they look like, why they happen, and the fixes that actually hold up under load.

Flavor	Root Cause	What You'll See in Prod
Notebook-Driven State	Global model/cache objects, undetached tensors	OOM crashes after N requests
Happy-Path Networking	No timeouts, no circuit breakers, synchronized retries	Thread exhaustion, cascading outages
Dependency Anarchy	Unpinned transitive deps, implicit CUDA coupling	Clean installs that break the moment someone else runs them

1. Notebook-Driven State (Memory Bleed)​

In a notebook, global state is a feature: load a 14 GB checkpoint into cell two, keep it resident, mutate variables downstream forever. Port that habit into a long-running FastAPI worker or Celery consumer and it becomes a liability. Those processes are expected to survive days, not a Restart Kernel.

MODEL = None
METRICS_HISTORY = []  # unbounded growth

def predict(x):
    global MODEL
    if MODEL is None:
        MODEL = torch.load("weights.pt").to(DEVICE)
    out = MODEL(x)
    METRICS_HISTORY.append(out)  # keeps the full autograd graph alive
    return out

Every tensor a forward pass produces carries its grad_fn unless you strip it. Appending that tensor to a global list doesn't store numbers. It pins the entire computation graph in memory, and a few hundred requests later CUDA throws an OOM and takes the worker down with it.

Treat models and caches as scoped, injected resources, never module-level singletons.

• Constructor injection for models, devices, and allocators: nothing hides in module globals
• @torch.inference_mode() (or .detach()) on anything you retain past the request
• Bounded caches: an LRU ring buffer, never an unbounded list or dict

You can see this same discipline win out at the infrastructure layer, too. PagedAttention became the default architecture for open-source LLM serving because vLLM stopped pre-allocating memory for worst-case context length and started paging the KV cache the way an OS pages RAM. It's the same scoped-allocation idea above, just enforced one layer down the stack.

2. The Happy-Path Network Trap​

Modern AI systems are never isolated: vector databases, model routers, remote inference endpoints. A shocking number of repositories still assume the network is local, cheap, and infinite.

def fetch_embeddings(texts):
    r = requests.post(VECTOR_DB_URL, json={"texts": texts})  # no timeout
    return r.json()["vectors"]

One hung connection is a shrug. Multiply it across 32 worker threads and your thread pool is exhausted within seconds. Naive for attempt in range(3): retries without jitter make it worse: when the upstream service recovers, every queued client fires at the same instant and knocks it straight back offline.

The failure boundary matters as much as the retry logic. Earlier this year vLLM patched an advisory (CVE-2026-54236) after unsanitized error messages in its API router were found leaking raw memory addresses back to clients. It's a textbook case of an exception bubbling past a service boundary instead of failing safely. The same blind spot shows up in unattended retry logic: an agent stuck in a loop overnight doesn't crash, it just quietly runs up a bill until someone happens to check the dashboard the next morning. Nothing trips, because nothing is watching.

A network call without a timeout is a bet you didn't know you were making.

• Explicit connect/read/write timeouts on every call, no exceptions
• Pooled, persistent clients (httpx.Client, urllib3.PoolManager) instead of a new connection per request
• Circuit breakers that fail fast and return a cached or degraded response
• Exponential backoff with jitter, so retries scatter instead of synchronizing

3. Dependency Anarchy & Non-Deterministic Builds​

You pull down a repo that topped the benchmarks six months ago, and it will not build. The Python packaging ecosystem is already fragile; layering in CUDA toolkits, native C++ bindings, and platform-specific wheels turns it into a minefield.

The usual sins: a flat requirements.txt with bare torch or transformers, no locked transitive tree, and an assumption that everyone's host runs the exact CUDA minor version the wheels were compiled against. One silent numpy or pydantic point release later, serialization formats shift and the repo is bricked for the next person who tries to run it.

A repo you can't rebuild the same way twice isn't really done.

• Commit a hashed, cross-platform lockfile (uv.lock, poetry.lock), not just a pyproject.toml
• Run CI across a real Python × CUDA/architecture matrix, not just ubuntu-latest
• Degrade gracefully on unmapped hardware (custom kernel → Triton → CPU) instead of segfaulting

uv has become the default choice here for good reason: it resolves and locks a universal, hashed dependency tree far faster than pip ever did. But it doesn't own the non-Python binary layer, which is exactly why teams shipping heavy CUDA dependencies still lean on conda or explicit driver-matrixed CI alongside it, not instead of it.

One Function, Three Fixes​

Here's how all three flavors show up in a single retrieval-augmented inference handler, and the refactor that fixes it.

Before: the antipattern

# anti_pattern_inference.py
import requests
import torch

MODEL = None
GLOBAL_CACHE = {}

def process_and_infer(batch_texts):
    global MODEL
    if MODEL is None:
        MODEL = torch.load("model.pt").cuda()  # hardcoded CUDA, crashes without GPU

    res = requests.post("https://api.internal/embed", json={"texts": batch_texts})  # no timeout
    embeddings = res.json()["embeddings"]

    tensors = torch.tensor(embeddings).cuda()
    predictions = MODEL(tensors)

    for text, pred in zip(batch_texts, predictions):
        GLOBAL_CACHE[text] = pred  # retains the full computation graph
    return predictions

After: production architecture

# robust_inference.py
import logging, random, time
from collections import OrderedDict
from typing import Dict, List, Optional
import httpx
import torch

logger = logging.getLogger(__name__)

class ResilientInferenceEngine:
    def __init__(
        self,
        model: torch.nn.Module,
        device: torch.device,
        embed_endpoint: str,
        http_client: Optional[httpx.Client] = None,
        max_cache_size: int = 10_000,
    ):
        self.device = device
        self.model = model.to(device).eval()
        self.embed_endpoint = embed_endpoint
        self.max_cache_size = max_cache_size
        self._cache: "OrderedDict[str, List[float]]" = OrderedDict()
        self._client = http_client or httpx.Client(
            timeout=httpx.Timeout(connect=2.0, read=5.0, write=5.0, pool=10.0)
        )

    def _fetch_embeddings(self, texts: List[str], max_retries: int = 3) -> List[List[float]]:
        for attempt in range(max_retries):
            try:
                r = self._client.post(self.embed_endpoint, json={"texts": texts})
                r.raise_for_status()
                return r.json()["embeddings"]
            except (httpx.RequestError, httpx.HTTPStatusError) as exc:
                if attempt == max_retries - 1:
                    logger.error("Embedding API failed after %d retries.", max_retries)
                    raise
                sleep_time = (0.5 * (2 ** attempt)) + random.uniform(0.05, 0.2)
                logger.warning("Retrying in %.2fs after %s", sleep_time, exc)
                time.sleep(sleep_time)

    def _cache_get(self, text: str) -> Optional[List[float]]:
        if text not in self._cache:
            return None
        self._cache.move_to_end(text)  # touch: mark as most recently used
        return self._cache[text]

    def _cache_put(self, text: str, vector: List[float]) -> None:
        if text in self._cache:
            self._cache.move_to_end(text)
        elif len(self._cache) >= self.max_cache_size:
            self._cache.popitem(last=False)  # evict the true least-recently-used entry
        self._cache[text] = vector

    def infer(self, batch_texts: List[str]) -> torch.Tensor:
        results: Dict[int, List[float]] = {}
        miss_positions, misses = [], []

        for i, text in enumerate(batch_texts):
            cached = self._cache_get(text)
            if cached is not None:
                results[i] = cached
            else:
                miss_positions.append(i)
                misses.append(text)

        if misses:
            raw = self._fetch_embeddings(misses)
            with torch.inference_mode():
                tensors = torch.tensor(raw, dtype=torch.float32, device=self.device)
                computed = self.model(tensors).detach().cpu().tolist()
            for pos, text, vector in zip(miss_positions, misses, computed):
                self._cache_put(text, vector)
                results[pos] = vector

        return torch.tensor([results[i] for i in range(len(batch_texts))], dtype=torch.float32)

    def close(self):
        self._client.close()

None of this changes what the function does, only how it survives contact with production. The model now comes in through the constructor instead of hiding in a global. The HTTP call runs through a pooled client with real timeouts and jittered backoff instead of a bare requests.post. The cache is actually consulted before any work happens, a repeated text skips the network call and the forward pass entirely, and eviction removes the genuinely least-recently-used entry instead of just whatever was inserted first. inference_mode() stays scoped tightly around the forward pass, so the tensor infer() hands back is a plain, ordinary tensor a caller can safely slice or mutate, not one still tagged by an active inference context. Same functionality, fewer reasons for anyone to get paged at 3 a.m.

Production Readiness Checklist​

Checkpoint	Target State	Failure Indicator
Memory isolation	Forward passes wrapped in @torch.inference_mode(); zero graph allocation in monitoring hooks	Process memory grows monotonically over a 1,000-iteration stress loop
Deterministic lifecycles	Resources passed via constructors; explicit close() or context management	Reliance on __del__ or module-level singletons for GPU contexts
Failure boundaries	Every external call has bounded pools, hard timeouts, jittered retries	An unreachable endpoint hangs the client process indefinitely
Reproducible isolation	pyproject.toml paired with a hashed, platform-pinned lockfile	Fresh install in a clean environment breaks on a sub-dependency bump
Device agnosticism	Explicit device targeting with a working CPU fallback	.cuda() called directly without checking driver availability

Architecture Is the Multiplier​

A breakthrough model wrapped in a leaking, unguarded script is still just an experiment: promising in a demo, unusable in production. None of this asks teams to slow down. It asks that model code get the same engineering hygiene we've expected from databases and network protocols for years. Treat memory as finite, design for the network to fail, and make your build reproducible, and the payoff shows up less in adoption metrics and more in who doesn't get paged at 3 a.m..