The Hard Part of Agents Isn’t the Model

Why we built AgentForge, what broke, and what we’d tell you before you build your own

If you only read one section

Build the layer, not the engine. We kept open-source orchestration (LangGraph) and built a thin layer on top that owns how every agent gets data, calls tools, asks for approval and gets traced. That gave us control of providers, data and approvals without maintaining a graph runtime.
It paid off quickly. Our last hand-built agent took several sprints. The first service generated by AgentForge reached staging in eight days, and the time went on business logic, not plumbing.
The hard part isn’t the model. Almost every serious problem was identity, permissions, cost ceilings, tool coverage or packaging. Most of them failed silently.
Autonomous agents are a different product from chat. Different identity model, different threat model, different governance. Ours are read-only until that governance is enforced, not just written down.

Short on time? Read Governance and security, Lessons from the systems around the agent, Where we are now and the closing section, If you are weighing build versus buy.

uilding one LLM agent is now easy. A notebook, an API key and an afternoon will get you a demo. Building the tenth agent in a company is a different problem. By then every one of them needs to log the same way, authenticate the same way, account for tokens the same way, refuse risky actions the same way, and survive upgrades to libraries that change monthly.

At Madgicx we build AI for performance marketers: tools that read ad accounts, explain what is happening, and increasingly act on the user’s behalf. In about a year we went from a single chat assistant to a family of agents: conversational copilots, deterministic workflows, and scheduled analysts that run while nobody is watching. AgentForge is the framework that made that possible. It is a code generator plus a small runtime library on top of open-source building blocks: LangGraph (a library for building agents as graphs of steps), LangChain, FastAPI and Pydantic.

This is for engineers who would use something like it, and for CTOs and CIOs deciding whether to build or buy.

When	What happened
Mid to late 2025	Principles set; frameworks evaluated; first agent service on LangGraph
Early 2026	First-gen chat hits its limits; a hand-built campaign agent takes several sprints
Mar to May 2026	AgentForge codebase starts; first generated services; runtime split into its own package
Jun to Sep 2026	Agentic chat rebuilt on AgentForge; reasoning, caching, approvals and widgets land in the runtime; the generator gets its own CI
Oct 2026	Autonomous Super Agents enter an internal staging pilot; governance control plane in design

Why build our own

The prototype that discovered the requirements

Our first-generation chat ran on LangGraph with a plan, execute, validate and compose loop, checkpointed state, versioned prompts and audit logging on every write. It was reasonable engineering. Its post-mortem summed it up in one line: “v1 wasn’t a failed idea. It was the prototype that discovered the requirements.”

It discovered three to eight seconds of latency per message; too much reasoning and too many purposes packed into one node; one token-hungry insights tool doing far too much; tools calling the ad platform’s API directly even though our own data platform already held normalised, enriched data; partial tracing; and test coverage of roughly 40 to 50%, with no regression suite for agent behaviour.

Then we built a second agent, for campaign creation, and watched the same pattern repeat. As the planning epic put it: “Right now building a new agent at Madgicx is a bespoke process. Every agent starts from scratch.” That agent took several sprints, mostly plumbing. The target for the next one was blunt: one day to generate, one sprint to staging.

Build versus buy: we did both

In late 2025 we evaluated LangChain, LangGraph, PydanticAI, Google ADK and CrewAI, and later OpenAI Agents and n8n for integration scenarios. LangGraph won the runtime layer for practical reasons: explicit graph control flow, state checkpointed to Postgres, streaming, async throughout, iteration limits and provider-neutral model classes.

None of them could answer the questions that actually consumed our time. Where does this agent get data, and does it use our platform before the external API? What may it do, for which team, on which accounts? Who approves a write, and where is that recorded? How is every step traced, tested, packaged and deployed?

Those answers are specific to a company. Hosted agent platforms solve the plumbing, but they decide which providers you may use, where your data runs and how approvals work. For agents that can change real advertising spend, we did not want to outsource those decisions. So we bought the orchestration primitives and built the production contract around them.

Two more problems showed up as services multiplied. Where teams hand-wrote a backend per model provider, small rules drifted; the cap on tool-calling rounds quietly became different per provider. And a shared library that supported one provider “for now” passed that limit to every product built on it. We also needed to route each task to a model by cost and capability, which is impossible unless every provider reports the same facts: can it reason, can it cache, what did this turn really cost.

Our first scaffold, built on generic LangGraph templates, scored 5 out of 10 in our own review. It ran, but with none of our data paths, tracing or guardrails, and engineers rewrote most of it. A scaffold is only as good as the production code it is derived from.

What AgentForge is

Three pieces ship from one repository:

The scaffolder (CLI). Describe a service in one project.yaml, or answer a short wizard, and it generates a complete Python project: FastAPI API, LangGraph workflow, agents, tests, CI, Dockerfile, secret-scanning rules and a Makefile. Everything optional is off by default.
The runtime SDK. Generated projects import shared behaviour instead of copying it: the model-provider factory, base agent and registry, logging, tracing, input sanitising, prompt secret scanning, the tool-approval policy and token accounting. A security fix ships once and reaches every service through a package upgrade.
The widget kit. A JSON Schema contract for visualisations, a Python emitter and a React renderer, so a backend with no frontend can still return something a person can read.

The scaffolder runs once when a service is created. The runtime SDK, highlighted, is the shared seam that every service keeps importing.

A team picks one of five execution patterns:

Pattern	Use it when
`react`	One agent reasons, calls tools and repeats in a bounded loop until it has an answer
`orchestrator`	A supervisor (an LLM or fixed rules) routes each request to a specialist agent
`workflow`	Steps run in a fixed order, with optional human approval between them
`fanout`	Several specialists analyse the same input in parallel and their results are merged
`planner`	A request is broken into a task graph, run concurrently, checked for gaps and replanned

The framework ships all five because different services genuinely need them. Our customer-facing products default to a single stateful planner: one lifecycle is easier to guard, policy context stays in one place, traces stay readable, and there is no cross-agent coordination latency. Products adopt the multi-agent patterns only when a spike proves the extra complexity pays for itself.

Four principles, written down before most of the code, shape everything else:

Separation of concerns. Planners reason, tools execute, validators validate. Validation is never baked into a prompt.
Progressive commitment. Intent, proposal, review, confirmation, execution. Every step reversible where possible.
Evidence traceability. Every recommendation records which tools and data informed it, and every model call, tool call and routing decision is traced end to end.
Reusability. New agents supply only domain models, configuration and prompts.

Two concrete consequences are worth calling out. Agents touch the world only through typed tool families (search, CRUD, action, ML, UI and eval), and the family doubles as a permission boundary. And data, agents and UI travel on separate channels: MCP (Model Context Protocol, the open standard for exposing tools to AI models) for tools, A2A (Agent2Agent, an open protocol for agents to message each other) for agent messages, and a typed UI channel for things like “show an approval dialog”. That kept our public MCP servers clean data contracts with no knowledge of our own interface.

The current codebase has 153 commits from March to September 2026. It was built with AI coding agents (AEP – https://www.indikamaligaspe.com/2026/10/16/inside-our-agentic-engineering-playbook-architecture-governance-and-continuous-learning/) from day one, which shaped both its speed and some of its hardest lessons.

Design decisions, and why we made them

Each decision traded some flexibility for consistency. Most were written down as decision records at the time.

1. Wrap open source; do not replace it. We chose a thin factory over LangChain’s own model classes instead of a universal adapter library, which would have added a heavy dependency and hidden native features like tool binding and structured output. A few dozen lines keep generated code readable and auditable. The cost is that we inherit LangChain’s bugs, as you will see.

2. One declarative file, and nothing changes unless you ask.project.yaml is validated by Pydantic before a single file is written. Every new option defaults to off, and a test fails if one byte of output changes for an existing reference configuration. Teams can upgrade the tool without being surprised by it.

3. Split the generator from the runtime. This is the most important structural choice in AgentForge. Until May 2026 the generator copied every shared module into each project, so a template fix never reached services already generated. Over one week we moved shared code into a separately versioned runtime package.

	CLI (scaffolder)	Runtime SDK
Runs	Once, when a service is created	Every request, inside every service
Dependencies	Templating and CLI tooling	Only langchain-core, Pydantic and structlog
Upgrade path	Regenerate, or keep what you have	`pip install -U`, no regeneration

By default a project gets thin “linked” stubs that re-export runtime code. A team that must customise deeply, or deploy where the package cannot be installed, can mark any of eleven modules “vendored” (the module’s full source is copied into the project instead of imported from the package). Each project records the CLI and minimum runtime version it was built with. Releases are cut from commit messages, and that is a coordination tool, not a label: a patch reaches every consumer automatically, while below 1.0 a minor version crosses consumers’ version pins, so each team adopts it deliberately. We have shipped a fix as a separate patch precisely so it would reach everyone at once.

One duplication we accepted: the CLI needs each provider’s capabilities at generation time but cannot import the runtime, so capabilities live in three places that must change in the same commit, with a test that parses the code to prove they agree. The alternative, a CLI that depends on the runtime, would tie every template change to a runtime release.

4. Providers are interchangeable; capabilities are explicit. Every provider declares what it supports: tools, structured output, streaming, vision, JSON mode, reasoning, prompt caching. Generated code checks the flags instead of assuming. Token usage, including reasoning and cached tokens, is normalised into one shape. That is what makes cost-based routing possible.

5. Contracts are schemas, and code is generated from them. For widgets, JSON Schema is the source of truth; Python models and TypeScript types are generated from it and CI fails on any difference. The payload also says which column is the axis. We learned why the hard way: an earlier version shipped bar charts with dollars and ratios on the same axis.

6. Pin dependencies, and watch the ones you pinned. The runtime pins LangChain to a narrow range, and a nightly canary job (an early-warning build) installs the latest release and runs the tests against it.

7. Shared code is earned in production. We do not invent framework features. Product teams patch what they need, and patches are reviewed for reuse and ported back behind opt-in flags, so the next team starts with them. The table under “What teams built on it” shows where they came from.

Governance and security, built in

Our agents can change live advertising budgets, so governance cannot be an add-on. The framework owns the rules and the vocabulary, each product owns how it pauses and asks a human, and the audit record is written at the moment of decision.

Fail-closed approval (when in doubt, block rather than allow). Every tool is classified as a read or a write. Writes need human approval, and a tool nobody classified is treated as a write, so a wiring mistake fails safe. Mutation tests (tests that deliberately break the rule to prove the safety test notices) back it up: flip the rule to fail open and the safety test fails. Approval outcomes and records are defined once in the runtime so every product audits the same way.
Authentication and permissions. API keys compared in constant time, or JWT with a pinned algorithm; a missing secret raises an error instead of letting requests through. One production team found a permissions claim sent as a string would pass a naive check, because “adm” is a substring of “admin”. Permissions must now be a list, for everyone.
Tenancy. One switch turns on middleware that checks every request’s account against the accounts the caller may access. The default credential provider allows no accounts at all.
Bounded autonomy. Tool loops, replanning and retries all have upper limits. Workflows can pause for human review before any named step, and the schema rejects review points for steps that do not exist.
Secrets and logs. Secret scanning is generated into every project, an opt-in scanner blocks prompts containing keys, logs are redacted and stripped of control characters, and blank secrets fail at start-up rather than in production traffic.

Most controls are opt-in, because a read-only prototype and a service that spends customer money need different defaults. A few cannot be configured away: unclassified tools need approval, permissions must be a list, JWT will not verify without a key, and unknown configuration keys are rejected.

We are explicit about the limits. Our first design said no code path could bypass approval. We withdrew that claim as unachievable. The honest claim is narrower: approval is the path of least resistance, and any bypass is visible in the audit trail. Some opt-in defaults still lean permissive, and tightening them is on the hardening list.

What went wrong along the way

The hardest problems were rarely loud. Almost every serious defect in our history failed silently: no exception, no red build, just quietly wrong behaviour.

The model that was never called

Early templates always created an OpenAI client, whatever model the configuration named. A project that asked for Claude sent a Claude model name to the OpenAI client, and the config schema listed Claude as a valid choice. We hotfixed it in May, then replaced the guesswork with an explicit provider per agent.

Telemetry that reported zero

In July we added reasoning-token accounting. For one provider the count was always zero. LangChain’s Anthropic integration discarded the reasoning-token count the provider returns, on every code path and version we tested. The field is optional, so nothing raised an error, and a turn that reasoned heavily looked identical to one that did not reason at all.

Worse, our own test had passed the whole time, because its hand-written fixture asserted a field no real release ever produced. We added a small compensating shim and filed a reproducible report upstream. The shim was done to self-heal when the actual fix was done.

Tests that never ran

In August and September we found holes in several safety nets. The nightly LangChain canary ran zero tests for weeks, because a module-level import failed during collection and aborted the run. A test that imports every generated module had been skipping, never passing; when CI finally ran it, a freshly generated project could not import itself. And the generator had no CI of its own, so templates drifted for months until generated projects failed the very checks their own CI told them to run.

Turning on the type checker found 22 errors that tests had missed, because tests check values, not types. One turned every error response into an HTTP 500. Another had the supervisor calling a registry method that never existed.

Instruments you cannot trust

Two versions of the same linter reported 19 and 71 findings on identical code. Our check script once reported “0 files need formatting” while 27 did, because it read the wrong output stream. An automated clean-up removed “unused” imports that existed for their side effects, and the agent registry stopped registering agents. A template is a family of outputs, not a file. We now pin tool versions, render several reference configurations on every change, and test our checkers against deliberately broken input.

Add the unglamorous rest (the PyPI name was taken, the release workflow listened on the wrong branch, a base image crashed on ARM, a dependency renamed a module our MCP client used) and you have most of the work of making a framework other teams can trust.

Lessons from the systems around the agent

The framework bugs above are engineering defects. The products built on AgentForge taught a different kind of lesson: product and operational, and almost none of them about the model.

Unattended agents change the threat model. A chat has a user watching; a scheduled agent has nobody. With no user session, “whose credentials does this run use?” needs a real answer (ours is under Super Agents, below). Prompt injection also gets worse, because unattended agents read user-controlled text all day: campaign names, ad copy, comments, files. Prompt hardening is not a security boundary; an action layer that refuses out-of-scope actions is. We also rejected a tempting proposal to execute an approval automatically if nobody answered in time. That silently turns a supervised agent into an autonomous one.

A control that is not enforced is not a control. Our scheduled-agent runtime had a per-run token limit in its configuration that nothing checked. One test run consumed about 1.2 million tokens against a typical 66,000. Usage is now checked after every model step, runs stop at a configurable token limit by default, and a model that fails to report usage stops the run. The same investigation found agents analysing paused and archived campaigns by default. Data selection is a cost control.

“Supports model X” means two things. Routing across Anthropic, OpenAI and Gemini works. Production support also means usage, caching, limits and data terms are correct per provider. We have verified token accounting against one provider’s own numbers and are still verifying the others.

Tool coverage beats model quality. Reviewing real conversations, we found users repeatedly asking to target or exclude custom audiences, and the assistant replying that it had no tool for that. No model upgrade fixes a missing tool. Reading transcripts is our highest-leverage product activity.

Packaging is part of the product. Installing the framework into another repository put a generic top-level scripts package on the import path, shadowing the host’s own and breaking every one of its CLI commands, while our validation, which only checked that the framework imported, passed. Namespace everything, and test the real integration surface.

Streaming is infrastructure. Long-lived streaming connections leaked database connections and delayed broadcasts across instances. Streaming needed health checks, distributed pub/sub and explicit P95 and P99 latency budgets.

Approve before the fan-out. An internal agent that edits governance and data definitions originally synced changes downstream before review finished. Now only the merge of an approved pull request triggers the sync.

What teams built on it

Project	How it uses AgentForge	What it gave back
Agentic chat assistant	Scaffold layout; runtime for three providers, token usage and approvals	Reasoning and caching flags, normalised usage, the approval policy
In-product copilot	Contextual Q&A on the shared conversation engine	Proof the seam works for a streaming tool loop
Super Agents (internal pilot)	Each scheduled run is one turn on the shared runtime and approval path	Not yet
Internal operations agents	A governance co-pilot and a documentation co-pilot	The approve-before-sync rule

Where we are now

As of October 2026 the runtime is at version 0.4.1, supports Anthropic, OpenAI and Gemini and five execution patterns, and is used by several teams alongside the conversation engine and widget kit. Generated projects now pass their own lint, format, import and type checks on the core path; opt-in modules still carry type errors, and we keep a list of known-failing tests that is meant to shrink, not grow.

Governance is where we are most careful. The policies are written and the control plane is designed, but parts of enforcement, kill switches in particular, are still being built. That is the main reason autonomous agents are not yet writing to anyone’s ad account.

What comes next

The plan is convergence: every agentic product on one seam, with the provider tuning that products wrote for themselves moved into the framework, so each migration is cheaper than the last. This is a sequence, not a schedule.

Finish hardening. Zero type errors in the opt-in modules; an empty known-failing list.
Richer providers. Explicit prompt caching where providers allow it, since agent loops resend the same prompt and tools every round, and native reasoning settings in the framework instead of in each product.
Products converge. Copilots route each screen to a model by cost and capability; conversational products retire their hand-written provider backends.
An evaluation gate. A regression suite, starting at 50 cases, runs on every prompt or model change, with product and engineering jointly owning the release decision. Live monitoring starts alert-only; we would rather earn automatic rollback than assume it.
A governance control plane. Every agent, model and tool registered with an owner, risk tier, autonomy level, budgets and kill scope. “Unregistered means unshippable”, and no LLM in the enforcement loop: a model can help triage, but never decides whether to stop a run.
Customer-built agents inside an envelope. The builder is governed as a high-risk component, and every customer agent lives inside limits on tools, autonomy, data, budgets and confirmation gates that customers can tighten but never loosen.
Event-driven agents. Agents that react to a spend spike or a fatiguing creative, not just a schedule. Our framing: automations and AI agents are the same thing at different points on an intelligence spectrum, so they should share one engine and one governance model.
Sub-agents in products, once a spike shows they pay for their complexity.

If you are weighing build versus buy

We would make the same call again. Some things we’d tell anyone starting out:

Build the layer, not the engine. A thin layer over open-source orchestration gives you control of providers, data and approvals without maintaining a graph runtime.
Derive the framework from a real agent. Ours became useful when its templates came from production services, not tutorials.
Budget for the boring work. Generator CI, release automation, dependency canaries and packaging took more effort than any agent pattern, and that is what makes teams trust the framework.
Put security in the tool layer, and fail closed. If your authorisation depends on the model behaving, you do not have authorisation.
Make every control executable. A limit in a config file that nothing checks is decoration.
Assume silence, not failure. Our worst bugs raised nothing: a wrong client, a zero in telemetry, a skipped test, an unchecked ceiling. Build fixtures from the real integration, and show every check a known-bad input before you trust it to report zero.
Treat autonomous agents as a different product from chat. Different identity model, threat model and governance.