How to design agentic AI systems for predictability, not just autonomy
An AI agent logged into a production database at 2:47am. It was supposed to clean up stale records. Instead, it deleted 340,000 rows from a table it wasn’t supposed to touch.
The incident took six engineers 11 hours to diagnose and reverse. Not because recovery was technically hard. Because nobody could tell them, definitively, what the agent had actually decided and why.
That’s agentic AI’s real production problem. Not capability. Visibility.
Quote me on this:
If your engineering team can’t explain exactly what an agent decided, why it decided it, and what it touched downstream, you don’t have an AI system. You have an autonomous liability.
I’ve spent the last two years building and operating AI systems at New Relic, where our platform sits inside production environments for thousands of companies. Here’s what I’ve learned the hard way.
This is the most practical framework I know for building agentic AI you can actually trust:
- Understand the 3 generations of AI risk, and why generation 3 is different
- Face the brutal math behind agentic failures and scope use cases accordingly
- Employ the ILTI framework: observability for non-deterministic systems
- Run the 3-filter ROI test before you deploy
- Use escalation design that actually works in production
- Employ orchestration patterns that scale
- Think hard about what great looks like for you vs. just going with what most teams build
1. The 3 generations of AI risk
Enterprise AI risk has gone through three distinct phases. Most teams are applying generation 1 thinking to a generation 3 problem.
Generation 1: Rule-based systems If X, then Y. Explicit logic. Fully auditable. When something went wrong, you found the rule and fixed it. Risk was contained, traceable, and fixable.
Generation 2: ML and chat AI Models started making predictions. Outputs became probabilistic. But the interaction was still single-step: input in, output out. If the model gave a bad answer, that was the entire blast radius.
Generation 3: Agentic AI This is where we are now. Agents interpret goals, select tools, retrieve context, and execute sequences of actions across live infrastructure. A single run might involve 20 tool calls across 8 systems before a human sees anything.
The problem isn’t that agents make mistakes. It’s that mistakes compound.
A wrong inference in step 3 shapes the context for step 7. By step 12, you’re looking at a failure that traces back to an ambiguous instruction four hours ago, running through six different APIs.
Going from generation 2 to generation 3 isn’t an upgrade. The risk model changes entirely.
2. The brutal math behind agentic failures
Large organizations lose an average of $2 million per hour during unplanned outages, according to the New Relic 2025 Observability Forecast (although customers with full-stack Observability cut those losses in half). Research shows that for financial services the number is higher. For e-commerce at peak it’s basically uncapped.
Now add agents into the mix.
Research from Princeton shows that focused coding agents average 30+ actions per task on standard software engineering benchmarks. (SWE-agent, Yang et al., 2024) That’s a contained, single-system task. An enterprise incident response workflow spans log retrieval, trace correlation, alert deduplication, root cause analysis, and remediation steps across multiple live systems. The action count is higher. Run dozens of these workflows in parallel and you’re looking at thousands of autonomous actions per hour across systems you’re already watching through imperfect telemetry.
The failure mode that scares me most isn’t the dramatic crash. It’s gradual drift. An agent runs correctly for six weeks, then starts behaving slightly differently. Nobody catches it for three more weeks. By the time you investigate, you’re tracing an impact that started 45 days ago and touched 12 systems. Every agentic action that isn’t traced is a liability you’ve deferred, not eliminated.
3. The ILTI framework: observability for non-deterministic systems
Traditional observability is built around MELT: metrics, events, logs, traces. It works for deterministic systems. You know what the system should do, and you watch to see if it does it.
Agentic AI breaks this.
Agents are non-deterministic. The same goal, the same context, can produce different action sequences on different runs. You can’t just monitor outputs. You have to capture the reasoning layer.
That’s what I call the ILTI framework. Every agent action should emit telemetry across four dimensions:
Most teams capture Tools and sometimes Impact. Almost nobody captures Intent and Logic at the moment of the action. That’s the gap that turns a recoverable incident into an 11-hour investigation involving six irritated engineers.
In practice, ILTI means instrumenting at the framework level, not just the API level. If you’re using LangChain, LlamaIndex, or a custom orchestrator, every reasoning step should emit an OpenTelemetry (OTel) span — capturing the agent’s goal, the context it was working with, the tool it invoked, and the result it got. (OpenTelemetry GenAI semantic conventions)
Teams with full ILTI data can ask: “Over the last 30 days, when our incident triage agent escalated versus resolved, what was the reasoning, and were those calls correct?” That’s how you improve agent performance without guesswork, based on traced evidence.
4. The 3-filter ROI test before you deploy
Sometimes we ask “what can we automate?” before asking “what should we automate?”
Here’s the test to run before committing any workflow to agentic automation. All three filters have to pass.
Filter 1: Is human effort actually the bottleneck? Incident triage passes this test easily. A senior engineer might spend 45 minutes pulling logs, correlating traces, and assembling context before diagnosing anything. An agent can do that in 90 seconds with better recall. Things that fail this filter: creative judgment calls, relationship-sensitive communication, any decision where knowing the room matters more than knowing the data.
Filter 2: Can you define “good” in numbers? MTTR reduction. False positive rate on escalations. Cost per resolved ticket. If you can’t define success numerically, you can’t tell whether the agent is getting better or worse. And when someone questions the investment — and they will — you have nothing.
Filter 3: Do you have real-time, authoritative data? An agent working off a metrics pipeline that’s 20 minutes delayed, or log data missing 15% of events due to sampling, will produce confident-sounding decisions that are wrong in hard-to-catch ways.
Quick check: take the last 10 incidents your team resolved. Could an agent have accessed all the data your engineers used to diagnose them? If the answer is no for more than half, you’re not ready.
The irony of Filter 3 is it forces you to fix your data infrastructure first. That’s not a tax on AI. It’s instrumentation you should’ve built years ago.
5. Escalation design that actually works in production
Most HITL implementations land in one of two failure modes.
Too aggressive: agents ask for approval on every significant action, teams rubber-stamp without reading. You end up with the illusion of control, not the substance of it.
Completely absent: agents run autonomously, failures surface hours later, no way to intervene mid-run.
The right model is threshold-based escalation. Three signals drive it:
Confidence. Good agents know when they’re uncertain. If confidence drops below a threshold, the agent pauses and surfaces it to an operator. OpenAI’s structured outputs and Anthropic’s extended thinking modes both give you ways to extract this signal. If you’re not capturing agent confidence, you’re missing the most actionable escalation trigger available.
Blast radius. Every proposed action should carry an estimate of what it could affect. Read query: low. Config change: medium. Database write: high. Set thresholds per category and auto-escalate when the agent proposes something in the high zone.
Novelty. If the agent hits a situation that matches nothing in its historical action log, that’s a flag. An agent that encounters something genuinely new and keeps going is more dangerous than one that stops and asks.
An agent that knows when to stop is more trustworthy than one that always pushes forward.
And here’s the counterintuitive part: good escalation design makes agents more autonomous over time, not less. When operators see intelligent escalation, they start approving faster. Trust accumulates, exactly the way it does with a new hire who asks the right questions.
6. Orchestration patterns that scale
Connecting agents and tools is the easy part. The hard part is designing what happens when things go wrong: dependencies fail, outputs are ambiguous, retries create duplicate states. That’s where most teams underinvest.
Three patterns separate teams running agentic AI reliably from teams constantly firefighting.
- Explicit dependency graphs. The most common orchestration mistake is agents that implicitly depend on each other’s outputs without declaring those dependencies. Agent A produces a summary. Agent B consumes it. Nobody documents the relationship. When Agent A’s behavior shifts subtly, Agent B starts producing garbage. Nobody knows why for two weeks.
- Good orchestration versions of every input an agent consumes and monitors. When upstream output quality changes, downstream agents get an alert. Not a silent degradation two weeks later.
- Idempotency by default. Agentic systems fail and retry. If running the same action twice produces different results, you get data corruption and state drift. Every action touching a stateful system should be idempotent by design. This is standard distributed systems practice, but teams skip it for AI workloads because “the agent handles it.” It doesn’t. You have to design for it.
- Evaluation pipelines tied to business metrics. Without this, you’re relying on production incidents to tell you something went wrong. Too late. Run evaluation pipelines weekly against labeled datasets of known-good and known-bad decisions. Tie metrics to MTTR reduction, incident avoidance rate, false escalation rate.
Microsoft’s AutoGen is worth studying here for multi-agent orchestration, including built-in evaluation tooling. LangGraph is an option, too. But it’s worthwhile having your teams keep an eye on this space because it’s quickly evolving, and the needs for and demands of orchestration frameworks may change drastically soon.
7. What great looks like vs. what most teams build
The engineering choice that defines the next few years
Gartner named AI one of its Top Strategic Technology Trends for 2025, projecting it’ll be embedded in 40% of enterprise applications by the end of 2030, up from less than 2% today. That’s the steepest enterprise adoption curve since cloud.
I keep coming back to this: the teams shipping agentic AI reliably in 2026 aren’t the ones who moved fastest. They’re the ones who built observability from day one, before anything broke.
The difference isn’t intelligence or budget. It’s whether the team decided at the start to treat agentic AI like any other mission-critical system: full tracing, explicit escalation logic, evaluation pipelines tied to real outcomes. I’ve seen both patterns play out. The gap between them, in incident resolution time alone, is measured in days.
Agentic AI is genuinely powerful. The teams that capture it will be the ones who can see what their agents are doing, explain why they did it, and prove it worked.
Build for visibility. Everything else follows.
