The agent data problem is not latency
Estuary rebuilt its pipeline runtime in Rust today and put transaction boundaries at the centre of the pitch. The throughput is the smaller story: once the consumer of a pipeline can act, fresh data and correct data stop being the same thing.
We spend most of our time trying to give agents fresher data. Lower latency, real-time pipelines, faster retrieval, a shorter distance between an event happening and an agent seeing it.
Freshness is becoming the wrong property to optimise on its own. An autonomous agent does not only need to know what happened recently. It needs to know that the world it is looking at actually existed — and that is a much harder requirement.
Fresh can still be wrong
Estuary released a new generation of its data-processing runtime today, rebuilt in Rust, reporting up to 200 GB per hour sustained per worker with more workers added as volume grows.
The throughput is not the interesting part. What the runtime is designed to preserve while pipelines execute in parallel is: transaction boundaries. Related changes move through together and commit atomically, so that — in the company's own phrasing — a downstream system never sees half of a transaction. Exactly-once processing is claimed to hold from milliseconds to hours and from kilobytes to hundreds of petabytes. Estuary also describes a pipeline continuing to prepare work while a destination is temporarily unavailable, and completing delivery when it returns.
None of these ideas are new. Transactional processing, recovery logs, durable state and exactly-once semantics have existed in distributed systems for years, and Estuary's own open-source Flow architecture has exposed much of this for a long time. Agents did not invent a new data-engineering problem.
What changed is the consequence of getting an old one wrong.
Imagine a business process producing four related changes, and an agent that looks at the system after the second one becomes visible but before the other two arrive.
The transaction
What the agent saw
A world that never existed.
Everything it sees may be fresh. Nothing may technically be stale. And the world it sees is still wrong, because the transaction is incomplete. That distinction matters enormously more once the consumer can act.
Humans could wait. Agents act.
Most enterprise data infrastructure was built around a loop that ended in a person. A dashboard being inconsistent for a few seconds is annoying; a human waits. An analyst notices a discrepancy and reruns the query. Someone investigating an unusual number opens another system before deciding anything. There is friction between observation and action, and that friction has been doing quiet safety work for years.
Agents remove most of it.
Reporting
A control loop
The data stack is no longer merely reporting what happened. It is becoming part of a control loop — and control loops have very different failure modes.
An agent can observe partial state, interpret it, reason correctly about what it sees, call a tool, and change the world before the missing state ever arrives.
The model does not need to hallucinate. Its reasoning does not even need to be wrong. It can make a perfectly sound decision about a world that never existed. That is a far more uncomfortable failure mode than a bad answer, because replacing the model does not fix it.
State coherence is the layer underneath
We already talk about models, harnesses, tools, memory, provenance, permissions and sandboxes. There is another layer hiding underneath all of them, and it does not have a comfortable name yet.
What did the agent actually observe? Did those observations belong together? Were they valid at the same logical point in time? Was one record from before a transaction and another from after it? Did two systems expose different stages of the same operation? Was one event delivered twice and another delayed? Did the agent build its world model out of states that were each individually valid but collectively impossible?
These sound like distributed-systems questions because they are distributed-systems questions. Agents do not make them disappear. They give them authority, and that changes what they are worth.
Correct pieces do not guarantee a correct state
There is a close parallel in agent memory. Suppose a memory system retrieves three claims and all three are true. That sounds like a good retrieval — until you notice that A belongs to the current version of a project, B describes the architecture it replaced, and C came from a different branch.
Every retrieved piece is factually legitimate. Together they describe a world that does not exist.
This is why memory cannot be reduced to retrieval quality. Applicability matters, scope matters, time matters, and the relationships between claims matter. Transactional data consistency and memory applicability are not the same technical problem, but they expose the same principle:
Correct pieces do not guarantee a correct state.
Giving an agent accurate information is not enough. The information also has to be valid together — and that gets harder with every source you add. A database, a CRM, a ticketing system, a payment provider, a deployment platform, a long-term memory store, a live browser session, an internal API. Each one can tell the truth while the combined context still describes a false world.
Grounding is bigger than RAG
We use "grounding" as shorthand for giving a model access to external information: retrieval, search, databases, tools, memory. Access is only the first layer. The harder question is whether what reaches the model forms a coherent observation of reality.
Once agents participate in operational systems, grounding starts touching machinery that AI infrastructure discussions rarely treat as AI at all — CDC, transaction boundaries, event ordering, snapshot isolation, exactly-once semantics, recovery, idempotency. None of it was designed for language models, and none of it becomes an AI technology because an agent happens to consume its output. But it increasingly sits inside the reliability boundary of autonomous systems.
That distinction is worth keeping sharp. Calling every old infrastructure problem an "AI problem" is marketing. Recognising that AI changes the blast radius of those problems is architecture.
The data stack is moving toward the control plane
This is what makes today's announcement interesting to me, and it is not the Rust rewrite, not the 200 GB per hour, and not the "built for agents" positioning. Those are release details.
The signal is what happens when the consumer of a data pipeline stops being someone looking at a chart and becomes software capable of taking action. Consider an agent allowed to issue a refund, change infrastructure, message a customer, modify inventory, pause an account or deploy code. A few seconds of inconsistent state stops being a data-quality issue and becomes an execution-safety one.
Which means the quality of an agent can no longer be described by the model alone. We already learned that the same model inside different harnesses behaves very differently. Memory changes what the agent knows. Tools change what it can do. Permissions change what it is allowed to do. The environment changes what it can observe. State coherence changes whether any of those observations describe a world it is safe to act on — and unlike the others, it fails silently.
was ever real
These multiply rather than add. A zero in the wrong place dominates everything else in the product.
What a transactional pipeline still cannot do
There is an important limit to this argument, and it cuts against the tidy version of it.
A transactional streaming system does not hand an agent a globally consistent view of the world. The agent still combines information from several independent SaaS platforms. Those platforms expose eventually consistent APIs. External side effects complete at different times. A payment provider, an internal database and a fulfilment system may not share a transaction boundary at all.
Exactly-once processing inside one pipeline cannot manufacture a global snapshot across systems that were never designed to provide one. So this is not a solved problem — and noticing it mostly reveals how large the problem is. As agents reach more systems, the question stops being whether each observation is trustworthy and becomes whether the collection of observations is mutually compatible. That is a considerably stronger definition of grounding than anyone is currently shipping.
Old problems, new consequences
Agents did not invent consistency, transactions, event ordering, recovery logs or exactly-once processing. But autonomous software changes what happens when those mechanisms fail.
The old failure
- the dashboard was briefly wrong
- somebody noticed
- somebody reran the query
The agent-era failure
- the system acted on a world that never existed
- the action was correct given what it saw
- the missing state arrived afterwards
We have spent years making data arrive faster. For autonomous systems, speed alone is not the property that matters; the next question is whether the state arriving at the agent makes sense as a whole. Because the more authority we hand to agents, the more expensive one quiet assumption becomes:
Fresh data means correct reality.
It doesn't. And increasingly, that gap is what separates an agent that knows what is happening from one that can safely decide what happens next.
Sources
- Estuary — Estuary Rewrites Its Runtime for Agents at Enterprise Scale — the primary source, 26 August 2026: the Rust rewrite, 200 GB per hour per worker, "a downstream system never sees half of a transaction", and exactly-once processing from milliseconds to hours.
- Why We Rebuilt the Estuary Runtime — the company's own write-up: transactional correctness, autonomous scaling, simpler operations.
- estuary/flow — the open-source runtime, where the transactional state, derivations and recovery mechanisms referred to above are documented.