For the first phase of enterprise generative AI, token cost was easy to dismiss.
A user asked a question. A model answered it. Even if the prompt contained more context than necessary, the total cost of the interaction was usually small enough not to matter.
Agents change that equation.
An agent does not simply answer once. It plans, searches, retrieves, calls tools, reads the results, reasons again, retries when something fails and potentially delegates work to other agents. A single business task can create dozens of model calls and move hundreds of thousands of tokens through a system.
That changes token efficiency from a technical optimisation into an economic design problem.
Microsoft Research now describes efficiency as a first-class design principle for AI applications, arguing that context should be pruned, compressed and selectively retrieved rather than repeatedly loading everything an agent might need. Its ACON research reports reductions of 26–54% in peak token usage on long-horizon agent tasks while largely preserving performance.
Anthropic has made a similar argument through its work on context engineering: context is finite and valuable. Good agents should retrieve information just in time, use token-efficient tools and rely on structured memory and compaction rather than endlessly expanding prompts.
This matters because the naive enterprise architecture is surprisingly expensive.
Imagine an agent supporting a client services team. Every time it is asked to prepare a briefing, the system injects the client profile, brand guidelines, previous meeting notes, campaign history, current projects, user preferences and a large collection of reference documents.
Most of that information may be irrelevant to the task.
Worse, the same information may be reprocessed again and again.
At small scale this is invisible. At enterprise scale it becomes an operating cost.
Context should behave more like working memory
Humans do not consciously reload everything they know before starting a task. We bring forward the information that appears relevant, then retrieve additional detail when required.
Enterprise agents need the same architecture.
Instead of treating the context window as the memory system, organisations should separate four things:
- Long-term memory — durable information about people, projects, decisions and previous activity.
- Approved knowledge — policies, documents, standards and source material the agent may rely on.
- Working context — the small subset of information required for the task currently being performed.
- Tools — controlled ways to retrieve information or take action when needed.
The important shift is that the model no longer receives the whole organisation every time it thinks.
It receives what it needs.
Bigger context windows do not solve the problem
It is tempting to assume that larger context windows make context engineering unnecessary.
They do not.
A larger window increases capacity. It does not automatically increase relevance.
An agent given 100,000 tokens of loosely related information still has to determine what matters. The organisation also still has to pay to process that information.
The better architecture is selective.
Retrieve relevant memories. Retrieve the required evidence. Compress previous activity. Give the model concise descriptions of available tools. Escalate to more context only when the task demands it.
The economic metric therefore should not simply be cost per token.
It should be useful work per token.
The enterprise advantage is outside the model
This is one of the reasons Iridium is built around a simple idea:
Rent the LLM. Own the intelligence.
Models will continue to improve and token prices will continue to change. Enterprises should be free to benefit from that competition.
But the system that knows what information matters, what happened previously, which knowledge is authoritative and which tools can be used is potentially much more durable.
The agent that processes fewer tokens because it remembers what matters is not merely cheaper.
It is better designed.
Sources
- Microsoft Research, ACON: Optimizing Context Compression for Long-horizon LLM Agents (2026)
- Microsoft Research, Efficient AI applications: context engineering and agents
- Anthropic, Effective context engineering for AI agents