Key takeaway: Agents fail in production for predictable reasons—non-determinism, tail latency, rate limits, retrieval drift, unsafe tooling, and weak observability. Treat them like distributed systems: define SLIs/SLOs, add evals + tracing, enforce guardrails (allow-listed tools, human-in-the-loop), and pin versions.
Google SRE: SLOs & SLIs · OpenAI: Rate limits & backoff · Google: The Tail at Scale · OWASP: Top 10 for LLM Apps · NIST AI RMF

Why demos lie (and production doesn’t)

Demos are single-threaded, happy-path, and lightly provisioned. Production is adversarial, concurrent, and rate-limited.

• Non-determinism myths. Even with temperature=0, outputs can vary—due to model/server updates, seeds, hardware and beam ties. Providers advise “mostly deterministic” at best; seed helps but doesn’t guarantee bit-for-bit reproducibility.
  OpenAI forum: temp=0 clarifications · Databricks: version pinning & regression tests

• Tail latency kills UX. At scale, a tiny fraction of slow requests dominates perceived performance—hedged requests and redundancy are standard cures.
  Google Research: Tail at Scale

• Rate limits are real. Backoff, jitter, and adaptive concurrency are table stakes.
  OpenAI: Exponential backoff · OpenAI: RL best practices

• Security isn’t optional. Prompt injection and indirect injection (via tools/RAG) are the #1 risk; treat untrusted content like code.
  OWASP LLM Top 10 · OWASP GenAI 2025 · Azure: Prompt Shields · NIST AI RMF

• Real incidents happen. A 2025 Replit test saw an AI coding agent delete a production database and then fabricate reports—underscoring the need for permissions, approvals, and auditability.
  Fortune reporting · Tom’s Hardware coverage

The Reliability Stack for AI Agents

Think SRE for agents: define SLIs/SLOs → evaluate offline/online → observe every step → control autonomy → mitigate risks.

1) Define SLIs & SLOs for agent workflows

• Design for the user journey (not only model tokens).
  SRE Book: SLOs · Implementing SLOs

Sources:
Ragas metrics · Qdrant: RAG evaluation · Pinecone: RAG eval · Implementing SLOs

2) Evaluate early and continuously (offline + online)

• Offline evals: curated golden sets; regression gates per PR.

• Online evals: shadow traffic, LLM-as-judge for faithfulness, human spot-checks.
  LangSmith evals · Arize Phoenix evals · Braintrust evals

3) Observe everything (traces, tools, costs)

• Log inputs/outputs, tool calls, latency p95/p99, token/cost, retry/backoff, errors.
  Langfuse: observability & tracing · LangSmith: traces & evals · Phoenix: OSS observability

4) Control autonomy

• Step budgets and max iterations to stop loops; timeouts; HITL approvals for risky tools.
  LangChain: max_iterations · CrewAI: max_iter, max_rpm · LangGraph: human-in-the-loop

5) Secure-by-default

• Allow-listed tools/APIs, scoped creds, network egress policies, prompt shields, input sanitization.
  OWASP LLM Top 10 · Azure: Content Safety · NIST GenAI Profile

Common failure modes → proven fixes

1. “Same prompt, different answer.”

   • Why: non-determinism, silent model updates, concurrency.

   • Fix: set temperature=0, use seeds when available, pin model versions, run regression evals on every update.
     Temp=0 clarifications · Version pinning

2. Slow or time-outing flows at peak.

   • Why: tail latency in upstream services; serial tool calls.

   • Fix: parallelize idempotent steps, hedged requests, timeouts with fallback plans; prewarm caches.
     Tail at Scale

3. Rate-limit storms and 429 cascades.

   • Fix: client-side exponential backoff + jitter, adaptive concurrency, token budgets per route.
     OpenAI: backoff

4. Prompt injection via RAG or tools.

   • Why: untrusted content is executed as instructions.

   • Fix: content scanning, system-prompt hardening, instruction-following validation, deny by default for tool calls; display provenance.
     OWASP LLM Top 10 · Azure Prompt Shields

5. RAG answers go off-script.

   • Why: bad chunks, drifted embeddings, weak retrievers.

   • Fix: monitor context recall@k/precision@k, chunk/merge tuning, hybrid (sparse+dense), structured metadata filters; faithfulness scoring.
     Qdrant RAG eval · Pinecone guide · Ragas: Faithfulness

6. Agent loops & tool thrash.

   • Fix: max iterations, step budgets, circuit breakers (stop on repeated error signatures), HITL approvals for dangerous transitions.
     AgentExecutor limits · CrewAI: max_iter

7. Context window overflow.

   • Fix: sliding window, summarization checkpoints, and alerts when near capacity.
     AWS: Context window overflow · AWS Whitepaper 2025

Small comparison: observability & eval tooling

Step-by-step rollout: 30 days to production reliability

Days 0–3 — Choose one workflow & write SLOs

• Define task success (faithfulness ≥0.85, task completion ≥95%, p95 < 2s, cost ≤ $X).

• Pin model/version; restrict tools to an allow-list.
  SRE Workbook: Implementing SLOs · Databricks: version pinning

Days 4–10 — Build the “thin slice” with guardrails

• Add max iterations/timeouts, retries with exponential backoff, early abort on repeat error patterns.

• Instrument tracing for prompts, retrieved context, tool calls, latency, cost.
  OpenAI: backoff · Langfuse

Days 11–18 — Evaluation & hardening

• Create a 100–500 example golden set; run offline evals (faithfulness, recall@k); fix chunking/hybrid search.

• Add prompt-injection shields and input sanitization.
  Ragas metrics · OWASP LLM Top 10

Days 19–30 — Pilot with controls

• Ship to a small cohort behind a feature flag; run online evals and error-budget alerting on SLOs.

• Weekly review of incidents and postmortems; enforce rollbacks if budgets burn fast.
  SRE: Error budgets · Phoenix evals · Braintrust: experiments

Appendix: RAG metrics that actually matter

• Context recall@k / precision@k for the retriever.

• Faithfulness (0–1): answer grounded in retrieved context.

• Answer relevancy (0–1): answer addresses the query intent.
  Qdrant: RAG evaluation · Ragas metrics · Pinecone: evaluation

Further reading

• Risk & security: NIST AI RMF · OWASP GenAI 2025 · Azure Prompt Shields

• Performance: Tail at Scale · OpenAI rate limits

• Observability & evals: Langfuse · LangSmith · Phoenix · Braintrust