Why RAG Is Still Here in 2026: Thoughts on the Thermodynamics of AI

Hi, I am Jomy, founder of Zleap. Recently, while designing algorithms and system architectures, one thing has become increasingly clear to me:

Pure algorithms are unreliable on their own. Pure embeddings are unreliable on their own. Relying exclusively on an LLM is even less reliable.

An intelligent system that works in practice, remains stable and can be maintained needs a combination of algorithms, embeddings and LLMs.

I summarize the underlying logic in one sentence:

The intelligence you get depends on the computation you invest.

I call this the thermodynamics of AI.

“Computation” here is not simply how long one GPU call runs. It is the total across the lifecycle:

• Computation invested in data labeling and preprocessing.
• Compute consumed during training.
• Compute continually used during inference.

From this perspective, algorithms, embeddings and LLMs are three engineering choices at different points in the computation–intelligence trade-off.

1. The Thermodynamics of AI: Intelligence Requires Computation

Consider the analogy with thermodynamics:

• In the physical world, ordered structures cannot appear without energy input.
• In AI, higher-level intelligent structures cannot be built without sufficient computation.

Here is a fuller statement of the principle:

Within comparable technological paradigms, the upper limit of intelligence an AI system can exhibit is constrained by the total computation invested in data, training and inference.

Algorithmic innovation matters, of course. Its main contribution is increasing how much intelligence each unit of computation can produce.

With that principle established, let us return to practical engineering and the lessons I learned in real projects.

2. Engineering Differences Between Algorithms, Embeddings and LLMs

1. Algorithms: Fast, but Not Intelligent

Traditional algorithms—rules, scoring functions, state machines and classical optimization or search—have three major advantages in industrial systems:

• Speed: millisecond or even microsecond responses.
• Low cost: they run on CPUs with little compute demand.
• Determinism: predictable, reproducible and testable results.

The major limitation is a lack of intelligence.

• People must modify rules when the environment changes.
• Slightly complex business logic can become a maze of rules.
• They struggle with ambiguous input and semantic understanding.

Do not expect more if-else statements or a few adjusted weights to add up to intelligence. Without sufficient computation, higher-level intelligence cannot emerge.

2. Embeddings: Some Intelligence, Still Relatively Fast

Embeddings occupy a middle layer between pure algorithms and LLMs.

Their characteristics include:

• Some semantic intelligence. Embedding models require training and greater inference computation, allowing them to recognize similar meanings and contexts more naturally than keyword or rule matching.
• Relatively manageable speed and cost. Embedding and vector retrieval can run in seconds, and are much cheaper and more stable than LLMs.

In my experience, vectors fit situations that need some intelligence without becoming too slow:

• Semantic search and FAQ matching to find potentially relevant content.
• Recommendation and retrieval stages that generate candidates.
• Approximate matching of similar questions or cases.

But their intelligence is closer to approximate intuition than rigorous reasoning.

They should not serve as the final decision-maker or replace complex logic. Their best role is intelligent retrieval and candidate generation.

3. LLMs: Most Intelligent, Slowest and Least Certain

LLMs are currently the most intelligent component available in engineering practice, but their operational characteristics are also the most extreme.

Advantages:

• Understanding complex natural-language instructions.
• Multi-step reasoning and planning.
• Generation, summarization, rewriting, correction and explanation.

The disadvantages are equally clear:

1. Slow responses. Long reasoning chains and large contexts increase latency. Large numbers of synchronous LLM calls can be disastrous for high-concurrency, real-time systems.
2. Randomness and unreliability. The same question may receive different answers. Hallucinations and subtle reasoning errors cannot be eliminated completely. Strict input–output validation is not possible in the same way as with conventional algorithms.
3. High cost. API fees and self-hosted GPU clusters both incur real expenses.

These characteristics suggest two primary uses for LLMs:

1. Key decision points and complex reasoning: final ranking, explanation and answer generation based on context and candidates; choosing business workflows and subsequent tool calls.
2. Stages that can operate offline: precomputing structured knowledge, template answers and labels, moving heavy computation from real-time requests into batch processing.

An important practical principle is to avoid extensive synchronous reliance on LLMs in latency-sensitive request paths.

Instead:

• Call them at a few critical points.
• Have a fallback to algorithms and embeddings for timeouts or failures.
• Cache what can be cached and precompute what can be calculated ahead of time.

3. RAG Follows from the Economics of Computation

We often hear this claim:

“Once LLMs become powerful enough, RAG will eventually be obsolete.”

I think this overlooks costs and physical constraints, much like proposing to build an entire bridge from pure titanium.

RAG combines the strengths of the three layers:

• Embeddings quickly narrow down private knowledge or business data.
• Algorithms and rules filter, constrain, rank and enforce access permissions.
• An LLM performs higher-level understanding and generation within a small, precise context.

Even if future LLMs become more powerful and closer to AGI, these realities remain:

• All private data cannot be incorporated into foundation-model parameters.
• Training and inference compute remain finite and costly.
• Enterprises and products need controllable latency, costs, permissions and explainability.

As long as those conditions hold, the combination of retrieval, rules and LLMs will remain useful.

It may evolve into something more automated and systematic, but the architectural idea persists:

• Use inexpensive computation to reduce the problem space.
• Use deterministic rules to enforce hard boundaries.
• Apply expensive LLM intelligence only to the critical parts that require it.

4. SAG: Making the Three Work Together

Intelligence requires computation, but engineering cannot put the most expensive computation everywhere.

In this arrangement:

• Algorithms handle structured, deterministic work.
• Embeddings provide fast, approximate semantic intelligence.
• LLMs perform valuable higher-level reasoning and generation.

This principle led me to design SAG, a next-generation RAG architecture.

In one sentence: SAG deeply preprocesses data with an LLM and constructs a knowledge graph in real time during queries.

From the perspective of AI thermodynamics, its core idea is to move expensive computation earlier and keep real-time computation lightweight.

The workflow has three stages:

• Data processing: an LLM intelligently parses raw data into a consistent event-and-entity format.
• Initial retrieval: embeddings use semantic similarity to quickly identify an initial set of relevant data.
• Graph construction: algorithms perform multi-hop queries over that data to construct relationships efficiently.

In SAG, LLMs, embeddings and algorithms are complementary building blocks that must be used together.

More specifically:

• LLMs contribute substantial intelligence during offline processing, where latency is less sensitive and deeper computation is appropriate.
• Embeddings provide lightweight intelligence during retrieval, quickly defining a candidate answer space.
• Algorithms provide stable, efficient computation during graph construction, dynamically building knowledge graphs through event–entity connections.

SAG exchanges offline LLM computation for online algorithmic efficiency, and uses graph structure to compensate for the ambiguity of vectors.

Its innovation is a simpler bipartite event–entity graph instead of traditional entity–relationship–entity triples. At query time, similarity propagation finds implicit relationships between events and entities and builds the graph dynamically.

We have open-sourced the SAG framework and welcome discussion:

https://github.com/Zleap-AI/SAG

5. Conclusion

I believe a mature, sustainable approach requires us to:

• Acknowledge physical limits and computation costs.
• Let each of the three components do what it does best and work with the others.
• Find a practical balance between speed, quality and cost.

Accepting that intelligence depends on computation, and recognizing the characteristics and limits of algorithms, embeddings and LLMs, naturally leads to this architectural perspective:

Use the appropriate computation to obtain the intelligence you need within the available resources and the demands of the situation, rather than treating any single technology as universally correct.

That is a more realistic approach to designing AI product systems today.