By Nigel Freddy, CTO, Dot Group
Most organisations I talk to have already bought real-time. Far fewer have got it. The reason usually isn’t the streaming layer. It’s what happens after it.
The first half is usually solved already. Kafka, and Confluent, now an IBM company, move data the moment it’s created: a payment, a login, a sensor reading, captured as it happens. The problem is the next step. In a typical cloud-warehouse architecture, that live stream is handed through connectors, staging layers, object storage and batch processing before anyone can query it – reshaped into Iceberg, processed with Flink, Spark or dbt along the way. Every hop adds latency. By the time the data is queryable, it’s minutes or hours old. The stream was real-time; the thing you actually query isn’t.
That’s the gap the Accelerate pillar is built to close. And the fix isn’t a faster dashboard, it’s removing the distance between an event arriving and that event being usable. It needs a serving layer that ingests the stream directly and answers queries against it in sub-second time.
In the architecture we build, that layer is SingleStore: a distributed SQL database that takes Kafka straight into distributed tables and runs transactional, analytical, vector and full-text workloads on the same live data, in one engine, without shuttling it between systems first. Confluent keeps the data moving; SingleStore makes it usable the instant it lands. On the IBM data foundation, that’s a real-time stack rather than a real-time claim.
Here’s what it looks like in production.
Live analytics that reflect what happened a second ago, not four hours ago – with thousands of people querying the same live data at once, without standing up replica after replica to cope.
Decisioning in flight. Fraud and risk judged in the time it takes a transaction to clear, rather than in the next overnight batch. The decision lands before the event completes, not after it’s already succeeded.
AI on live context. Vector search and retrieval-augmented generation running over data that includes what a user did moments ago, so a model reasons over the current state rather than last night’s extract. Confluent streams the context; SingleStore serves it back.
Modernisation without a rip-and-replace. For mainframe, Oracle or SQL Server estates, events can be streamed through Kafka into a real-time analytical layer that sits alongside the legacy system – AI-ready, without touching what runs underneath it.
That last one is worth dwelling on. You don’t have to finish modernising the estate before you can start acting in real time. The same holds for organisations already on cloud warehouses that handle reporting well but weren’t built for live, high-concurrency workloads – a real-time serving layer complements what’s there rather than replacing it.
None of it holds up without the layer nobody puts on a slide: most of your data isn’t in the stream to begin with – it’s locked in mainframe, Oracle and SQL Server systems that were never built to share it. Getting it out of there and into motion is the job of IBM StreamSets and watsonx.data integration – cleanly, continuously, shaped for use the moment it lands – with governance through IBM Guardium and watsonx.governance, so a real-time decision is one a regulated business can stand behind.
This is where we work. We don’t start by selling a database. We design the architecture and deliver the complete solution: the streaming backbone, the serving layer, governance, and the unglamorous integration between them. Any number of partners can point at the same IBM portfolio. What’s harder to replicate is thirty years of doing this, and a team that stays when it gets difficult. We’ve never walked away from a client mid-project, and some of our clients have been with us fifteen years and more. In real-time work that matters more than most, because the hard edges don’t show up in the design review. They show up in week six, when it’s live and something upstream quietly changes shape.
Real-time then stops being a feature you switched on and becomes how the business runs. If your “live” data isn’t, bring us the architecture you’ve already got. That’s usually the more interesting conversation.