Should Your AI Agent Run In-House? Save Tokens, Keep Your Data In.

1. It’s Not Local vs Cloud. It’s Control vs Capability.

A frontier API, like Claude or GPT, means calling someone else’s model over the wire: maximum capability, zero infrastructure, but the weights and your data sit on their side. An open-weight model is one whose trained weights you download and run yourself, on infrastructure you control.

Crucially, “yourself” doesn’t mean a server in the building. An open-weight model can run in your own cloud account (your VPC, your tenancy) just as easily as on-premise, so “local” here means within your control boundary, not on-site.

That’s why the real axis isn’t local versus cloud at all. It’s control versus capability, and for regulated workloads, how much control you need is the question that matters.

2. Save on Tokens: Specialised Local Agents Do More With a Smaller Model.

An agentic task isn’t one call. It’s a loop of dozens or hundreds of calls, and on a per-token API that bill compounds fast. A specialised local agent, scoped tightly to one job, gets the same result from a far smaller model, so the marginal cost of each loop drops to your own fixed compute.

And the cost shape changes too: per-token billing is spiky and hard to forecast (a runaway agent loop can quietly multiply your bill), whereas self-hosted compute is a steady, predictable cost whether the agent runs ten times or ten thousand.

You stop renting intelligence by the token and start owning a narrow, repeatable capability outright.

3. Keep Your Data In: With local SKILL.md, Confidential Data Never Leaves Your Perimeter.

A SKILL.md is a declarative procedure (the steps, tools, and constraints the agent must follow) which turns an unpredictable model into a bounded, auditable one. Run that locally and the determinism comes from the constraint, not the model’s size: the same procedure, the same behaviour, every time.

And many specialised agents don’t need internet access at all. Often a RAG over your own data is enough, where RAG (Retrieval-Augmented Generation) simply means the agent retrieves relevant chunks from your knowledge base and uses them as grounding before it answers, all within your perimeter, and some tasks need no RAG whatsoever.

That means the agent can run fully air-gapped: the confidential data it touches never crosses your network boundary, so there’s nothing to leak and a clean trail to evidence to a regulator.

Conclusion

The biggest model is rarely the right answer for a regulated workload. The right answer is the narrowest one that does the job.

A specialised local agent, scoped by SKILL.md, gives you lower token costs, repeatable outcomes, and data that never leaves your perimeter, which is exactly what matters in sectors built on confidentiality, like insurance, government, and legal practice, where a solicitor’s files or a citizen’s records simply cannot leak.

And “in-house” stays flexible: the same agent runs on premise or inside your own cloud tenancy (GCP, Azure or even OVH), so you keep control of the data wherever it lives.

This post was written with the help of Claude.