10 Facts You Wish You Knew Before Building Agentic AI

Most agentic AI projects fail on facts that were knowable before anyone wrote code. Here are the ten I most wish every architect and CIO knew going in. None of them are about how smart the model is. Every one is a property of the model you rent; every fix is a property of the harness you build.

1. You control the harness, not the model

The labs ship a new flagship every few months and retire the old one on their schedule. The best available model changed hands three times in one two-week stretch in November 2025, and Anthropic’s deprecation policy gives as little as 60 days’ notice. So the durable thing you own is the harness: prompts, tools, memory, orchestration, evals, guardrails, routing. I keep the model ID in config, never in code, and a regression suite that can grade any candidate in a day. Longer version in The AI Model Harness.

2. The “same” model drifts

Stanford and Berkeley researchers ran the same tasks against GPT-4 three months apart in 2023 and watched accuracy on a simple test fall from 84 to 51 percent. Same name, same API. I pin dated snapshots, never floating aliases, and run a canary eval whenever a provider announces an update.

3. The context window is smaller than advertised

Chroma tested 18 frontier models and found every one degraded as input grew, even on simple tasks. A million-token window is a marketing number. I saw an income-calculation agent that handled a 40-page tax return start missing Schedule C lines once the file hit 200 pages. The fix was retrieval and chunking, not a bigger model.

4. Nothing you tune for one model ports to the next

One ICLR paper measured up to 76 accuracy points of swing from prompt formatting a human would call cosmetic, and the best format for one model is often wrong for another. Tool calling is just as uneven on the Berkeley leaderboard. One prompt variant per supported model, tool calls tested on my own schemas, a second provider behind a routing layer.

5. The model knows nothing about your business and has no memory

Every model has a training cutoff and zero knowledge of your policies, your customers, or the last conversation. Anything that looks like memory was built in the harness. MIT’s NANDA report on AI in business found context loss among the top reasons pilots stall; one Fortune 500 insurer’s pilot, in the report’s words, collapsed in the field because it could not retain context. Nobody built the memory. I test every agent from a cold start on real files.

6. Errors compound

A 95 percent step run 20 times is a 36 percent agent. METR found the task length models finish at 50 percent reliability doubling every seven months, but read the rest: near-perfect under four minutes, under 10 percent past four hours. I put a deterministic check between steps, and a human at the two or three junctions where a wrong step is expensive.

7. Temperature zero is not deterministic

Thinking Machines sampled one prompt 1,000 times at temperature zero and got 80 different outputs, because inference servers batch requests and the math shifts with batch size. I watched an engineer declare an adverse-action narrative “reproducible” at temperature zero; two runs, two differently worded reasons. No exact-match tests, consistency measured across runs, and the exact prompt and output captured for every consequential decision.

8. The model does not know when it is wrong

OpenAI’s own paper on why models hallucinate says training rewards guessing over admitting uncertainty, so some fabrication is baked in. Reasoning models can be worse: the o3 system card reported o3 hallucinating on 33 percent of PersonQA questions, o4-mini on 48, against 16 for o1. In origination, reason codes come deterministically from the decision engine and the model only phrases them. The model is the writer, never the witness.

9. Prompt injection is unsolved

A model reads instructions and data through the same channel, which is why injection has held the top spot on the OWASP LLM Top 10 two editions running. Simon Willison’s lethal trifecta is the test: private data, untrusted content, and a way out, all in one agent. My loan agent that read uploads, queried the customer database, and emailed the title company was exactly that. It is now three agents.

10. Agents cost 10 to 100 times a chat call, and evals decide who ships

An agent loop resends its context every turn and, on reasoning models, bills thousands of thinking tokens you never see. The pilot was priced on the call; production is priced on the loop. Gartner predicts over 40 percent of agentic projects canceled by 2027, cost first among the reasons. I budget per completed task, cap reasoning on every call, and route cheap steps to small models. And I do not trust a leaderboard: Kapoor and Narayanan’s AI Agents That Matter shows why benchmarks mislead. Your only measure is the eval suite you build from your own ugly files, run on every change. That suite is the moat. More in What Agentic AI Actually Costs.

The one lesson

Rented, perishable, ignorant of you, unreliable over long tasks, non-deterministic, fluent at fabrication, injectable, expensive, unmeasured. All properties of the model. All fixed in the harness.

Instructions in, results out was IT; intent in, outcomes out is agentic AI, and the thing that turns intent into an outcome is the system you build around a component you rent. That is what my Five-Stage Roadmap and my books Agentic AI and Autonomy are about.

Which of these ten did your team find out after the pilot?

Go deeper

Agentic AI Architect: control design for enterprise agents.
Agentic AI Case Studies: deployment evidence, one teardown at a time.
Agentic AI P&L: cost, payback, and risk in a CFO’s voice.
Agentic AI Governance: who owns the harness.
Substack first access: agenticaiarch.substack.com/subscribe

Dr. Harish Kotadia, Ph.D., is an Enterprise AI Architect with 20+ years of IT consulting experience serving Fortune 100 clients, specializing in agentic AI systems built on Anthropic Claude, AWS Bedrock, and Google Vertex AI.

Disclaimer: This blog post is based on publicly available academic publications, vendor documentation, open standards, and news items from reputed media sources linked above. This post is intended for educational purposes, to help the enterprise agentic AI community build a shared vocabulary from public, authoritative sources.

Views and opinions expressed here are my own and do not represent those of any employer or client, past or present. The analysis presented is my independent interpretation of the published sources linked above and does not constitute legal, financial, or consulting advice of any kind.

10 Facts You Wish You Knew Before Building Agentic AI Solution