AI Economics, Systems Thinking

Agentic Systems Need Economic Boundaries

A breach at an AI coding tool is the example; the lesson is bigger. Unknown risks are managed by boundaries, not predictions, and AI economics comes down to simple controls built with the right governing intent.

Insight

6 min

26 Oct 2026

On 19 October 2026, TechCrunch reported that hackers had hit OpenCode, an AI coding tool, and run up charges on its users' AI accounts.

The company's own description was that it had suffered "a security breach… related to a large scale distillation attack" that affected 1.4% of its Console users.

The shape of it was simple. An older system behind the company's web Console held API keys for users who had linked their own OpenAI, Anthropic or Google accounts. Attackers reached it. Within hours, developers were reporting spending alerts and charges of hundreds, sometimes more than a thousand dollars. The open-source tool and command line were not affected. The way in has not been disclosed.

Distillation means training a cheaper model on the outputs of a stronger one. Done openly, it is a standard technique. Done with someone else's keys, it is theft at scale: the victim pays for every query, and the thief keeps what the answers teach.

It would be easy to stop there, with a lesson about patching old systems. But the more useful question is not about this tool, or this attack. It is about the next one, which will look different and which nobody can currently describe.

One incident, four patterns

Set aside the mechanism and look at the design the incident reveals.

Concentration. Convenience pulls credentials into one place. Every key added makes that place more valuable to an attacker, which raises the stakes for the next user who stores one. Nobody designs this loop. Each step is simply easier than the alternative.

An edge nobody owned. The compromised system was described as an older one. Retiring a system is not an event, it is a process with a long tail, and the tail is where live connections hide.

A speed mismatch. Spending started within hours. An alert tells a person that something is wrong. It stops nothing. When automation acts at machine speed, any control that waits for a human to notice is slower than the thing it is meant to control.

Authority, not secrecy. We talk about keys as if they were passwords. They are better understood as spending authority. Once granted, the damage is set by what the key may do, not by how well the walls around it are built.

None of these is a flaw in one product. They are properties of a kind of system, and the kind is becoming common.

It is not one incident

Analysts of the OpenCode breach point to earlier cases in 2025 with the same weak spot: stolen or over-scoped credentials, handed to something that acts quickly. The Nx "s1ngularity" supply-chain attack, in which stolen tokens launched AI agents on victims' machines, is one. A compromised Amazon Q extension, shipped through an over-scoped token, is another.

Different doors, same room.

This is where systems thinking earns its place. If every incident is treated as a separate event, the response is a list: patch this, rotate that, add a rule for the last attack. The list grows forever and is always one step behind.

Systems thinking asks something else. Instead of what caused this?, it asks what kind of system produces this outcome, and what would limit the consequence whatever the cause?

That is a different discipline. Prevention tries to predict the attack. Design bounds the damage of attacks nobody has imagined yet.

From prediction to boundaries

You cannot enumerate unknown risks. You can decide, in advance, what any single failure is allowed to cost.

Four questions do most of the work:

What is this for? Every agent, key and integration should have a stated purpose.
What can it do? The authority granted, and the worst it could do in the wrong hands.
What is the most it can cost? A number, set in advance, enforced automatically.
Who answers for it? A named owner who absorbs the consequence.

Notice that none of these asks for a forecast of the threat. They ask about the system's own intent and limits.

Controls without intent are theatre. Intent without controls is a wish.

Simple controls, the right intent

AI economics is often construed as complicated. Token pricing, model selection, usage forecasts, total cost of ownership, vendor tiers. The vocabulary alone can make it feel like a specialist field.

In my experience, the controls that actually protect an organisation are simple.

A hard cap, set at the provider, sized to the worst loss you can accept. The cap acts. An alert only informs.
One purpose per key, with the narrowest scope that does the job and an expiry date.
A budget per agent, so one runaway process cannot spend what its role never needed.
A named owner for every credential and every integration.
Retirement that begins with connections, listing every secret a system holds before it is switched off.

A child could explain every one of these. What makes them work is not sophistication. It is governing intent: a clear decision, made first, about what each system is for, how much it may spend, and who is accountable.

Take away the intent and the same controls decay. A cap is raised "temporarily" and never lowered. A key is scoped broadly "for now". A legacy system is retired in the plan but not in the network. Controls are what intent looks like when it is written down.

That is also why governing intent is not a one-off decision, and not only a periodic review either. A calendar review looks at the controls on a date the world did not choose. Real intent means looking again at the moments that change the risk:

an agent, key or integration is added, or its scope is widened
a model, a provider or a price changes
ownership changes hands, or someone leaves
a system is retired
spending departs from its usual pattern
an incident, anywhere in the industry, shows a weakness you share

At each of those moments, ask the same four questions again. A periodic review catches drift. A trigger catches change.

The lesson beyond the incident

We will keep seeing incidents like this one, with new names and new entry points. The organisations that absorb them well will not be those that predicted each attack. They will be the ones that decided, long before, what no single failure should be able to do.

Value, cost, authority, accountability. Four limits, stated explicitly, enforced automatically, and looked at again whenever the risk changes, not only when the calendar says so.

AI economics is often construed as complicated. In my experience, it is simple controls built with the right governing intent.