Context Infrastructure Will Be the Next Focus in AI

Hello, I'm Jomy. Over the past two years, when people have discussed AI infrastructure, they have usually meant infrastructure at the model layer. More precisely, most of that discussion has concerned LLM infrastructure.

At the bottom are compute and GPUs, with NVIDIA as a typical example. Above that are models and model services: OpenAI and Anthropic resemble closed, full-stack ecosystems, while GLM, Kimi, DeepSeek, and Qwen represent open-model ecosystems. Above those sit platforms that turn models into APIs, hosting, and inference services, such as Fireworks, Together, and Fal.

LLM Infra = GPUs + Model Training + Model Inference.

This chain is important and attracts enormous attention. It requires significant capital, talent, and engineering, and will remain a major business.

But I believe the center of AI infrastructure is shifting.

Over the long term, models resemble electricity.

Electricity is important, but whether it comes from hydro, coal, or wind matters less to users than reliability, price, and usability. Model APIs are increasingly similar. For many developers, switching models essentially means changing an API URL and a key.

Models will remain important, but margins on model APIs alone will be compressed. They are stateless, switching costs are low, capital requirements are high, and open models and price competition exert continuous pressure.

Model companies will remain powerful, and OpenAI and Anthropic will continue advancing. But the model layer alone cannot explain the broader shift: AI is moving from answering questions to taking over tasks.

With LLMs alone, AI mainly served as a question-answering, writing, and search tool. With agents, it is beginning to perform information work in customer service, sales, development, and operations.

That is why I keep saying agents address payroll budgets. SaaS assists people; agents replace some of the information work people perform. Their potential market is therefore much larger than traditional software.

The transition from AI infrastructure to agent infrastructure follows naturally.

Agent infrastructure

In the agent era, I think infrastructure needs a new formula:

Agent Infra = LLM Infra + Harness Infra + Context Infra

LLM infrastructure answers where models come from. It includes GPUs, model training, and inference, providing AI's foundational capabilities.

Harness infrastructure answers how to turn a model into a reliable, usable agent. It includes tool calls, file operations, task loops, and recovery from failures. Without a harness, a bare LLM struggles to complete complex tasks consistently.

Many teams already work in this area. Codex and Claude Code are typical harnesses. DeepSeek has also recently been reported to be assembling a harness team to build an agent product similar to Claude Code. Open-source projects include OpenClaw and Hermes.

It is natural for model providers to develop harnesses. Models and harnesses can be optimized together: models learn how tools are organized, and harnesses understand how models call them. Together, they can perform much better than a bare model.

Harness infrastructure nevertheless remains engineering-intensive. Over time, implementations will converge around similar tool calls, file operations, task loops, and failure recovery.

Recent changes that made Claude Code less useful for users in China also remind us that we need an effective, controllable domestic agent harness—one that makes full use of model capabilities without those overseas service problems.

We recently open-sourced Zleap-Agent, an agent harness for small local models. The project is early and evolving quickly. A key design is the workspace: instead of stuffing everything into an ever-growing prompt, agents enter different workspaces, each exposing different tools, context, and memory.

I hope Zleap-Agent's workspace concept offers useful ideas to others building agent harnesses.

But the harness is only one part of agent infrastructure. Context infrastructure remains comparatively underdeveloped.

Context infrastructure

An agent's performance depends heavily on the quality of its context.

Anyone using Codex, Claude Code, or Cursor has probably experienced this. If you do not explain a project's background clearly at the start, you have to keep adding information: project history, business goals, code structure, writing style.

More frustratingly, much of that background must be repeated. You explain it today, then again for a new task tomorrow. You explain it to one agent, then repeat it to another.

Many people call this memory. I do not think that term is precise enough.

My view is:

Context = Memory + Know-How

Memory is closer to the recollections and experience an agent accumulates through conversation: a user's preference for direct conclusions, why a project's architecture changed, or where the last customer conversation stalled.

Know-how does not necessarily come from agent conversations. It may come from real-world meetings, customer feedback, and offline discussions, or from documents, code, email, and business systems.

These also determine whether an agent can make sound judgments.

Andrew Ng recently introduced a useful idea: the context advantage.

People often retain an advantage over agents because they know more about the situation, rather than because they reason better.

You attended the meeting, heard the customer's tone, understand the boss's unspoken concerns, know the team's past mistakes, and recognize that a polished document is already out of date.

The model does not know these things.

Agents lack many of the sensors people have and do not automatically inherit human know-how. Often the problem is not intelligence, but making decisions from severely incomplete information.

The current interest in distilling employees' expertise is, in effect, an effort to eliminate this human context advantage.

Only when companies turn experience, organizational knowledge, business status, and historical decisions into context agents can consume will agents be able to replace large numbers of basic roles.

This is the value of context infrastructure: giving agents continuous, complete, updatable context.

LLM infrastructure supplies capabilities and harness infrastructure supplies the operating environment. Context infrastructure determines whom an agent understands, whom it serves, and how it personalizes its work.

Dynamic context

The biggest current obstacle is dynamic context.

A static knowledge base is straightforward: upload documents, split them into chunks, create embeddings, and retrieve them. The real world, however, changes continuously.

Projects, customers, code, and organizational judgments change daily. An agent working inside a company cannot simply read a knowledge base once and finish. Every action changes the situation; every meeting adds information; every customer response can change the assessment.

Context develops through use.

Yang Pan's earlier article, Context Is the Agent, identified three difficulties: collecting context, organizing it, and consuming it.

I strongly agree. Context infrastructure must determine how dynamic data enters continuously, how it is structured and updated, and how it becomes usable context when an agent needs it.

At Zleap, we recognized this problem early.

Our appliance is essentially local enterprise context infrastructure. Last year we began with data connectors, then quickly realized that connectors alone merely move data into the system. They do not solve dynamic context.

That led us to introduce SAG.

Traditional RAG is better suited to static documents. Real enterprise data changes continuously and contains many multi-hop relationships.

SAG proposes a hypergraph as a standard for structured data. It decomposes source data into events and entities, then expands their relationships to locate genuinely relevant evidence.

Dynamic relationships can then be updated continuously, reducing dependence on maintaining heavy static knowledge graphs or guessing relevance through vector similarity. Dynamic data becomes a contextual structure agents can consume.

You can explore our open-source project and paper:

Open-source project: https://github.com/Zleap-AI/SAG

Paper: https://arxiv.org/abs/2606.15971

SAG addresses collection and organization. The next, more important step is consumption.

We will soon introduce SAG-based context plugins for different agents, addressing shared memory and knowledge across agents. Stay tuned.

Network effects

Models and harnesses are relatively easy to replace.

You might use Codex today, Claude Code tomorrow, and Cursor the next day, connecting several models along the way. This is already common.

Context is different. It is data, and it is an asset.

It continuously accumulates, changes, and improves. Longer use brings deeper organizational understanding. More connections reveal more of the actual business. More feedback clarifies which information is useful and which is noise.

As more people and agents use it, context becomes more complete. Better context makes agents more useful, and more useful agents generate additional context.

This is the network effect of context infrastructure: accumulated data and usage feedback improve subsequent use.

I believe consumer and enterprise applications will follow different paths.

For individuals, context can live in the cloud. Consumers prioritize convenience, synchronization, and experience. With appropriate privacy boundaries, cloud context may be the most natural approach.

For enterprises, context must remain under local control. This does not mean all compute must be local; it means companies must retain control of their data.

Context is an enterprise asset. If customer information, code, historical decisions, and business experience accumulate in a large provider's closed format, migration costs may become extremely high.

Enterprise context therefore needs controllable data boundaries, straightforward exports, easy migration, and freedom from proprietary format lock-in.

We support open source to prevent that lock-in. We build enterprise appliances so companies can genuinely control their own context.

Closing thoughts

AI's broad direction is now clear: using agents to replace human work.

Reaching that goal involves three stages.

The first stage, LLM infrastructure, is largely solved. For many real tasks, models are already capable enough, and they will become stronger and cheaper.

The second stage, harness infrastructure, offers a growing range of choices. As products mature, their engineering patterns will increasingly converge.

The third stage, context infrastructure, remains relatively open.

Whether agents can truly replace human work depends on obtaining sufficiently good context: their own memory and know-how, an understanding of how the business operates, and knowledge of what happened before.

The LLM determines an agent's capability ceiling. The harness determines whether it can work reliably.

Context gives the agent its understanding of the world it works in.