\n<\/p>\n
Aruna Veerappan<\/em><\/a> is Senior Director of Engineering at Upwork, leading Developer Enablement to reduce friction and boost team productivity.<\/em><\/p>\n Engineering leaders are discovering that the hardest part of AI agents isn’t the AI\u2014it’s the architecture underneath.<\/p>\n I learned this firsthand when a quarterly budget disappeared in weeks. Nothing was broken, the models worked and the engineers were strong. But the system hadn’t been designed for cost, and the bill arrived before a single workflow reached production.<\/p>\n The root cause: we were pointing expensive models at every task. Verifying file existence. Checking ownership against APIs. Routing logic that could have been a single if-statement. Each call seemed reasonable. The cumulative cost was not.<\/p>\n I’ve come to call this the Agent Cost Spiral\u2014and engineering teams across the industry are running into it right now.<\/p>\n “An Agent Cost Spiral isn’t an AI problem. It’s an architecture problem. And once you see it, you can’t unsee it.”<\/em><\/p>\n This pattern has a precedent. A decade ago, teams migrated to the cloud chasing savings, then watched their bills explode past on-premise costs. The architecture was the problem\u2014not the technology. AI inference costs follow the same arc. The fix: stop treating it like a utility and start treating it like an engineering problem.\u200b<\/p>\n A well-built AI agent isn’t a single model receiving a single prompt. It’s a choreographed system where each task is matched to the minimum level of intelligence required to complete it well.<\/p>\n If your process follows a fixed rule\u2014“if a customer’s order exceeds $5,000, route to a Senior Rep”<\/em>\u2014you don’t need AI. You need a conditional statement. Enterprise teams routinely spend real money asking frontier models to handle basic routing logic, and the cost problem is the smaller concern. AI is probabilistic, which means even a capable model can get a simple rule wrong some percentage of the time. For business logic that must be consistent 100% of the time, probabilistic is another word for broken. Build your guardrails in code. Let AI operate within them.<\/p>\n Summarizing documents. Extracting fields from structured data. Reformatting outputs. These are real, valuable tasks\u2014but they don’t require a frontier model. Smaller “flash” models handle these workloads at roughly 1% of the cost of a premium model. If you’re using a frontier model for this work, you’re not just overpaying\u2014you’re slowing down your pipeline.<\/p>\n Top-tier models are extraordinary at synthesis: taking conflicting information from multiple sources and producing nuanced, well-reasoned output. That’s where the cost is justified. The mistake is giving them everything else too. When you feed a frontier model thousands of lines of raw, unfiltered context, two bad things happen\u2014costs spike and quality drops. The right move is to let Tier 2 do the reading and summarizing, then hand a clean, pre-processed brief to your Tier 3 model. You’re paying for reasoning, not retrieval.\u200b<\/p>\n<\/section>\n One of the most common enterprise headaches is keeping technical documentation current\u2014most teams either let it stale or throw expensive engineering hours at it.<\/p>\n Send the entire codebase to a premium model and ask for documentation. Cost: ~$15 per service. The model is overwhelmed by irrelevant code, hallucinates configuration details, misses security settings and gets version numbers wrong.<\/p>\n This approach can be divided into three tiers:\u200b<\/p>\n Code: automatically identify and extract the relevant configuration files\u2014no AI needed, just pattern matching.<\/p>\n Workhorse Model: summarize those files into a structured brief. Fast, cheap and accurate.<\/p>\n Frontier Model: take the brief and write the final, polished documentation.<\/p>\n “Cost: $0.50 per service. Accuracy: measurably higher. That’s a 30\u00d7 cost reduction with better output\u2014not a trade-off.”<\/em><\/p>\n The quality improvement isn\u2019t incidental\u2014it\u2019s structural. The frontier model performs better because it\u2019s receiving cleaner input. You\u2019ve set it up to succeed.\u200b<\/p>\n<\/section>\n There’s a second failure mode that hits teams who’ve already built something good. The agent tests well, confidence is high and someone makes the call to run it on everything at once.<\/p>\n High-cost failures almost always trace back to under-validated systems running at scale. The fix is Staircase Scaling\u2014earning the right to scale by proving the system at each step before moving to the next.<\/p>\n The Quintet (n=5): run five samples and manually review every output. If the agent fails here, your debugging cost is $2, not $2,000.<\/p>\n The Squad (n=15): run a more diverse batch of fifteen. This is where edge cases surface.<\/p>\n Full Rollout: only when your Squad pass rate is consistently above 90% should you scale to the full dataset.<\/p>\n This sounds slow. It isn’t. Teams that skip this process lose weeks to remediation. Teams that follow it reach production confidently within days.\u200b<\/p>\n<\/section>\n Here’s what separates teams genuinely automating from teams just shifting work around: Cost per Successful Output (CSO)\u2014not cost per API call or tokens consumed, but cost per output that clears your quality bar without human correction.<\/p>\n If a senior engineer spends three hours cleaning up AI-generated documentation that cost $500 to produce, nothing was automated. The work simply moved\u2014with frustration on top. The real test is whether your CSO is lower than the cost of a human doing the same task well. Everything else is theater.<\/p>\n Engineering leaders getting this right share a shift: they stopped asking “Which model is smartest?” and started asking “What does each task need?” Costs start making sense. Failure modes become predictable. The architecture becomes something you can defend to a CFO. <\/p>\n You don’t need the smartest model. You need the right model for each job\u2014and the discipline to know the difference.<\/p>\n The Agent Cost Spiral is real. It isn’t a reason to pull back\u2014it’s a reason to build deliberately. Get the architecture right first. The ROI will follow.\u200b <\/p>\n Forbes Technology Council<\/u><\/a> is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?<\/u><\/em><\/a><\/p>\nTiered Architecture Every Agentic System Needs<\/h2>\n
Tier 1: The Deterministic Skeleton\u2014Just Use Code<\/h3>\n
Tier 2: The Workhorse Models\u2014Cheap, Fast and Good Enough<\/h3>\n
Tier 3: The Frontier Model\u2014Reserve It For What It’s Good At<\/h3>\n
What This Looks Like In Practice<\/h2>\n
The Lazy Approach<\/h3>\n
The Architected Approach<\/h3>\n
Tier 1. <\/h3>\n
Tier 2.<\/h3>\n
Tier 3. <\/h3>\n
The Staircase Scaling Rule<\/h2>\n
Step 1. <\/h3>\n
Step 2. <\/h3>\n
Step 3. <\/h3>\n
The Only Metric That Matters\u200b<\/h2>\n
\n
<\/section>\n<\/div>\n