9 min read
For Father’s Day, my children got me Mouse Trap.
Not a metaphor. The actual board game.
We spent more time assembling the machine than playing the game. A boot kicks a bucket, a ball rolls down a ramp, a diver flies into a tub, and eventually a cage drops over a mouse. The first time it worked, everybody laughed. The machine is ridiculous. That is why it is fun.
The game is honest about the contraption being the entertainment. Your agent platform probably has a more ambitious business case.
I kept looking at that board and thinking about agent harnesses.
You have skills. Agents that call other agents. A planning agent, an implementation agent, a testing agent, a reviewer agent, and a senior reviewer agent in case the first reviewer lacked sufficient gravitas. You have routers, hooks, memory, context compaction, tool wrappers, policy files, and a diagram explaining which primitive owns the retry.
To be precise, I am not criticizing the supported agent your vendor ships—the model loop, native tools, terminal, permissions, and repository access. I am talking about the custom scaffolding your company bolts above it: internal skill files, agent hierarchies, routers, reviewer loops, memory layers, and wrappers.
Building that layer is not bad. A measured failure may justify every piece of it. But building and standardizing the extra machinery before you prove it improves the vendor baseline screams premature optimization.
You built a better thing. Fine.
Where is the proof in the pudding?
Did it complete more accepted work? Did it reduce human attention? Did it lower the total cost of a successful outcome? Or did you build an impressive machine around the mouse and assume the cage must therefore fall faster?
The Mouse Trap box at least promises a laugh. Your architecture deck promises productivity.
Your Custom Scaffolding Is a Hypothesis
A skill is a hypothesis that the model performs better with a specific bundle of instructions, examples, tools, or context. A subagent is a hypothesis that decomposition creates more value than coordination costs. A reviewer agent is a hypothesis that independent review catches enough material defects to pay for the extra run.
Most readers also read: The Engineers Who Can’t Use AI Agents Don’t Have a Tools Problem
Those can all be good hypotheses. They are not findings.
This is where teams fool themselves. The research agent returns research. The reviewer leaves comments. The skill activates. The observability screen fills with tool calls and green checks. Motion is easy to demonstrate. Causality is harder.
Did the skill improve the result, or did the model already know how to do the work? Did the reviewer catch a defect, or rewrite working code into its preferred style? Did the subagent save time, or did the parent spend more tokens briefing it and reconciling a slightly different answer?
If you cannot answer those questions, you do not have an agent architecture yet. You have a collection of beliefs stored in Markdown.
Eight extra minutes sounds harmless. Across five hundred engineers doing two agent-assisted tasks a day, it is not. That is one thousand tasks every workday. At eight extra minutes per task, your custom layer consumes 133 engineering hours a day—roughly 29,000 hours a year. At a $150 loaded hourly rate, you are spending $4.4 million on people waiting for and checking the system. The model bill is not the expensive part.
Maybe your custom layer saves more than that. Good. Prove it.
Your Skills Are Not Harry Potter Spells
There is a belief hiding underneath a lot of harness work:
Our magic skills can turn an early locally hosted Llama model into Opus 27 or Codex 4500.
That is funny until somebody puts it in the CFO deck.
A cheaper model with a task-specific harness can beat a bare frontier model on a bounded workload. Specialization is real. But “can” became “obviously does” without the awkward middle step where somebody runs the test.
The team wants the claim to be true. The inexpensive model costs one-fifth as much. The skill library represents months of engineering work. The platform group has a roadmap. Nobody wants to discover that the new frontier model, given the repository and a clear objective, walks past the contraption and drops the cage itself.
Be kind about that attachment. An engineer who spent four months building a harness is attached for the same reason any good engineer is attached to a system that taught them something. The work is real. The learning is real. The pride is earned.
The conclusion is still not earned.
Run the head-to-head:
- Early local model, vendor-supported agent.
- Early local model, vendor-supported agent plus your custom layer.
- Frontier model, vendor-supported agent.
- Frontier model, vendor-supported agent plus your custom layer.
The vendor-supported baseline still gets the tools, permissions, business rules, and security controls required to do the work. It does not get your company’s capability-compensation skills, agent hierarchy, router, or reviewer loops.
Give each configuration the same representative tasks, permissions, starting state, acceptance criteria, and time limit. Run every configuration more than once. Randomize the order. Decide what counts before the custom-platform team sees the traces.
Then measure completed work, reliability, elapsed time, human attention, escaped defects, and total cost per accepted outcome. Tokens are an input. Accepted work is the unit.
If the cheap model plus your harness wins, congratulations. You have evidence. Keep it.
If the frontier model performs worse with your harness than without it, the model upgrade did not fail. Your old instructions are sitting on its hands.
Markdown Is Not Bytecode
Every performance-oriented skill contains an assumption about what the model cannot do by itself. The model forgets to inspect tests, so the skill tells it to inspect tests. The model loses the plan, so the harness writes the plan to a file. The model routes poorly, so another model chooses the tool.
Then the model changes.
Do your skills and agents behave the same after every model upgrade? Of course not. A stronger model may already inspect the tests, hold the plan, choose the tool, and recognize completion. The primitive that rescued the old model can constrain the new one.
Now move the same skill across the ecosystem. Put it in a CLI agent, an IDE extension, a cloud worker, and a CI job. Change the system prompt, tool schemas, context limits, filesystem, permission model, retry behavior, and network access. Do you expect identical behavior?
I do not.
A skill is not a compiled binary. It is a request whispered into a stack of models, tools, policies, and runtime decisions. The same Markdown can sing in one agent, sulk in another, and fail in CI because nobody gave it a terminal.
That does not make skills useless. It makes portability a claim that needs testing.
Every material model release should expire the performance evidence. So should a move to a different agent host, tool layer, or execution environment. Rerun the suite. Keep security boundaries and business rules in place, but make capability compensation earn its way back.
Skills age like bananas. They can be excellent on Tuesday and compost after Thursday’s model release.
A Green Demo Is Theater
One successful run proves that one run succeeded. The presenter chose the task, environment, prompt, and take. That is a demonstration, not an operating model.
Start with representative work your teams have already completed: narrow bugs, cross-file features, dependency failures, behavior-preserving refactors, production investigations, and requests the agent should refuse. Run repeated trials from clean starting states. Grade the repository, not the agent’s confident closing paragraph.
Keep autonomous and assisted completion separate. A task that passes after an engineer spends forty minutes rescuing it may still be useful. It is not autonomous.
The economics can reverse quickly. Suppose the “cheap” harness spends $6 on inference, needs seventy minutes of engineering attention, and succeeds 60 percent of the time. The bare frontier agent spends $28, needs fifteen minutes, and succeeds 85 percent of the time.
At a $100 loaded hourly rate, the cheap run costs roughly $123 per attempt. The frontier run costs $53. Divide by the success rate and the rough variable cost per accepted outcome becomes $205 for the cheap system and $62 for the expensive one—before harness maintenance, eval infrastructure, security, and rework.
You saved $22 on the model and spent another $143 finishing the work.
Change the numbers to match your environment. The formula will survive.
Evals cost money. Good ones require representative tasks, clean environments, repeated runs, graders, trace storage, and experienced people looking at ambiguous failures. That is why teams skip them.
It is also why they should not standardize first. If you cannot afford to evaluate the harness, you cannot afford to multiply it across five hundred engineers.
Govern Risk. Expire Performance.
Govern identity, permissions, data handling, audit evidence, production changes, and rollback immediately. Those controls protect the company regardless of which model wins.
Performance scaffolding is different. Before a skill or agent pattern becomes an organizational standard, its owner should answer five questions:
- What measured failure does it correct?
- Which tasks, models, and agent ecosystems were tested?
- What was the bare-agent baseline?
- How did it change accepted work, human attention, reliability, and cost?
- What model or ecosystem change forces a retest?
If a leader cannot compare the bare agent and the harness from one page, the scorecard is reporting machinery instead of outcomes.
A mature platform team does not brag about how many reusable skills it has published. It knows which ones are load-bearing, which ones are policy controls, and which ones are waiting for deletion.
Standardizing an untested primitive does not create control. It embalms a guess.
Treat the Library Like a Gallery
My opinion right now is that skill libraries and agent libraries should work like libraries and galleries.
Go look. Get inspired. Study how somebody else solved a similar problem. Take a picture of the idea and repaint it inside your own system.
Do not confuse inspiration with installation.
That piece of modern art may look brilliant under the gallery lights. Hang the same thing in your dentist’s office between the root-canal diagram and the payment-policy sign, and suddenly everybody is uncomfortable. The art did not change. The environment did.
Skills work the same way. A beautiful orchestration pattern from another company may depend on its model, tools, repository, permissions, culture, and risk tolerance. Import the lesson. Test the implementation.
The operating model is not “make everybody use the gallery.” It is hire smart people, equip them with strong models, useful examples, clear boundaries, and real evidence, then trust them to make good decisions.
Do not take the current architecture too seriously. All of this will evolve again within three months. Your twelve-month scaling plan can be eighteen months out of date by month six. That math should be impossible, but the PMO will mark it green and schedule a steering committee to investigate why the model did not follow the roadmap.
Earn Every Primitive Back
There is nothing wrong with a sophisticated harness when the work requires one. Long migrations, broad research, incident response, and changes across multiple systems may benefit from decomposition, durable state, specialized tools, parallel workers, and independent verification.
The issue is not complexity. The issue is complexity without a control group.
Take your hardest representative work. Freeze the environment. Put the current harness on one side and the bare frontier agent on the other. Run both enough times that luck has somewhere to hide. Grade the outcome. Count human minutes. Read the failures. Remove one primitive. Run it again.
Then change the model or move to another ecosystem and start over.
The Mouse Trap machine can still be beautiful. The boot can kick the bucket, the ball can roll, and the diver can make everybody at the table smile. But the machine is not the outcome.
Right now, nothing is going to outwork great engineers with great talent and AI.
Talk to me again in a quarter. It could be different.
