A production AI agent is not a chatbot with better prompts. It's a system — one that connects to real data, makes decisions under uncertainty, recovers from failure, and hands off cleanly when it hits a wall. Most businesses never get there. They buy a chatbot, watch it hallucinate on day three, and quietly pull the plug. At Nuclear Marmalade we've built enough of these to know exactly where the wheels come off — and it's almost never where people expect.
Why do most AI chatbots fail so fast?
They're stateless, disconnected, and built to impress in a demo rather than survive daily use. A chatbot that answers "what's your return policy" from a PDF is not the same thing as an agent that looks up an order, checks a warehouse, and sends a confirmation email. The first one takes an afternoon. The second takes weeks of integration work. That gap is where most projects die.
The demo problem is real. A founder sees GPT-4 nail a complicated question on a sales call. They greenlight the build. Three weeks later the thing is confidently telling customers the wrong shipping date because it's reading from a document that was updated six months ago. No live data connection. No fallback. Just vibes.
The fix isn't a better model. It's architecture. Before you touch an LLM, you need to know what data it'll read, how fresh that data needs to be, and what happens when the answer isn't in the data at all.
What makes a production agent actually different?
It's designed around failure, not success. That's the counterintuitive part. When we scope an agent build at Nuclear Marmalade, we spend more time on edge cases and fallback paths than on the happy path. What does it do when the API times out? When the user asks something wildly off-scope? When the retrieved context is ambiguous?
The technical pieces that separate a real agent from a toy: tool use (calling external systems), memory (short-term within a session, long-term across sessions), guardrails on both input and output, and a handoff protocol — a clean way to escalate to a human without losing context.
The handoff piece is the one most teams skip. And it's the one that kills trust fastest. A user who gets a bad answer and then has to repeat their entire problem to a support rep doesn't just lose trust in the bot — they lose trust in your company. We spent serious time on exactly this problem building Telehance. The solution wasn't technical. It was about designing the conversation so the handoff felt like a feature, not an admission of failure.
How much does integration complexity actually matter?
More than anything else. More than model choice. More than prompt engineering. The model is the easy part. Connecting it to your actual business — your CRM, your inventory system, your calendar, your billing platform — that's where the real work lives.
Here's a concrete example. A client came to us wanting an AI agent to handle inbound customer service. Simple enough brief. When we mapped their data sources we found three separate CRM systems (two legacy, one modern), a custom order management tool built in 2014, and a ticketing system that only had an email API. No webhooks. No REST endpoints for half of what the agent needed to read.
We cut their average phone handling time from 4 hours per day to 12 minutes. But that result came after eight weeks of integration work before we wrote a single agent prompt. Anyone who tells you otherwise is selling you the demo, not the product.
What should you actually look for when evaluating an AI agent?
Look at the failure modes, not the capabilities. Any vendor can show you a good demo. Ask them what happens when the agent doesn't know the answer. Ask them how it handles a hostile or confused user. Ask what the escalation path looks like and whether context is preserved.
Five things worth checking before you commit:
- Live data vs. static documents. Is the agent reading from a current source or a snapshot? If it's a snapshot, how stale can it get?
- Confidence scoring. Does the agent know when it doesn't know? Can it say "I'm not sure, let me get a human" without being prompted?
- Audit trail. Can you see what the agent retrieved, what it reasoned, and what it output? You need this for debugging and compliance.
- Scope enforcement. What stops a user from asking it to do something it shouldn't? This isn't just a safety question — it's a liability question.
- Cost per conversation. GPT-4 calls aren't free. Does the architecture minimise token usage or just hammer the API on every turn?
If you want to see how these questions play out in a real build, the Forge project is a good example of an agent system where we had to get all five right at once.
Why do so many businesses underestimate what this takes?
Because the entry point is deceptively easy. You can have a working ChatGPT wrapper in an afternoon. That creates a false baseline. The jump from "working prototype" to "production system I'd trust my customers with" isn't a 10x problem — it's closer to 100x, in terms of edge cases, reliability requirements, and integration depth.
Honest opinion: the "we'll start simple and add complexity later" approach fails more often than it succeeds. Not because adding complexity later is impossible — it's that the simple version sets expectations. Customers use it, form opinions, and then you're retrofitting reliability onto a system that was never built for it. It's hell.
The smarter move is a narrower scope, built properly. One workflow. One data source. One user type. Nail that, then expand. That's how we work at Nuclear Marmalade, and it's why Glen Healy spends so much time in scoping conversations before a line of code gets written. Scope discipline is the difference between an agent that compounds in value and one that quietly gets abandoned.
If you're thinking about building an AI agent and want to talk through what it actually takes, reach out. Not to sell you something — just to give you an honest read on where your idea sits and what the real build looks like.
Key Takeaways
- The gap between a chatbot demo and a production agent isn't a polish problem — it's an architecture problem. They're genuinely different systems.
- Integration work — connecting the agent to live data — almost always takes longer and costs more than the AI part. Build that into your timeline.
- Design for failure first. A clean human handoff with context preserved is worth more than a slightly better answer in the happy path.
- "Start simple" is fine. "Start simple with no plan for complexity" is how you end up with an agent nobody trusts.
- Before you evaluate any agent build, ask what happens when it doesn't know the answer. That question tells you everything.