By Simon Parkinson, Managing Director
Kafka has a habit of becoming critical without anyone deciding it should.
It’s a familiar story. A team built its pipelines on Apache Kafka years ago. They have very good engineers. The platform hasn’t had a serious incident in a long time. So why would they need anything else?
It’s a fair question, and sometimes the honest answer is that they don’t. But a cluster that started as one team’s project often ends up carrying data the whole business depends on. A platform that has never failed is usually one nobody has looked at closely for a while.
What the recent PHP alert tells us
Researchers have recently disclosed serious vulnerabilities in PHP, the open-source language behind a large share of the web, including WordPress. The flaws had gone undetected for more than a decade. Found with the help of AI tools, the PHP Foundation subsequently released emergency patches, and the UK’s National Cyber Security Centre advised organisations to patch immediately or consider taking affected sites offline.
Researchers have recently disclosed serious vulnerabilities in PHP, the open-source language behind a large share of the web, including WordPress. The flaws had gone undetected for more than a decade, and were found with the help of AI tools. The PHP Foundation released emergency patches, and the UK’s National Cyber Security Centre advised organisations to patch immediately or consider taking affected sites offline.
So the question for any self-managed platform is simple. When the next critical patch lands, who applies it, and how quickly?
Running is not the same as healthy
Most self-managed Kafka estates don’t fail loudly. They drift. Configuration set for last year’s volumes stays in place. Workarounds become permanent. Small warning signs build up underneath a platform that still appears to be working.
None of this shows up as an incident until the day it does. These are the five areas I’d look at first.
1. Lag and replication
Consumer lag is one of the earliest signs that something is falling behind. A consumer that is slowly losing ground can look fine on any given day while the gap widens week by week.
Under-replicated partitions matter in the same way. The cluster keeps serving data, but with less resilience than you think it has, right up to the moment a broker goes down.
2. Connectors
Kafka Connect is where a lot of quiet failure happens. A connector that retries silently, drops malformed records or has no proper error handling can feed incomplete data downstream for weeks. Usually nobody notices until a report doesn’t reconcile.
3. Schemas
As more teams produce to and consume from the same topics, schema discipline matters more. One upstream change that isn’t compatible can break consumers that the producing team doesn’t even know exist.
The same applies to stream processing workloads in Kafka Streams, ksqlDB or Flink. Patterns that were fine at low volume can become expensive or fragile at scale.
4. Security
TLS, authentication, ACLs and certificate management tend to be set up once and then left alone. Certificates expire. Service accounts outlive the projects they were created for. Access granted for a quick fix is never taken away. On a platform that now carries sensitive data, these deserve regular attention. Patching belongs here too. Older versions eventually stop receiving fixes, and an out-of-date platform leaves known flaws open. As the recent PHP alert showed, the gap between a flaw being made public and attackers trying to exploit it can be very short.
5. People
This is the one that rarely appears on a dashboard. Many Kafka estates depend on one or two people who really understand them. If they left next month, who would patch the platform, tune it, or take the call when something fails out of hours? With open-source Kafka, the answer has to come from inside your organisation.
For a CIO, the question is simpler still. This is now a production system. Who stands behind it?
Open-source Kafka or Confluent: what actually changes?
Plenty of organisations run Apache Kafka well in-house and should carry on doing so. It gives you the core: brokers, topics, producers and consumers, plus Kafka Connect and Kafka Streams. There are no licence fees. What it leaves to you is everything around that core.
Schemas. Apache Kafka doesn’t include a schema registry, so teams add and run one themselves. Confluent includes Schema Registry as part of its governance layer, so producers and consumers work to agreed data contracts.
Stream processing. With open source, this usually means writing and running your own Kafka Streams applications. Confluent brings Apache Flink into the platform, so teams can filter, join and transform streams using SQL.
Connectors. Self-managed Kafka Connect means finding, building, testing and maintaining connectors yourself. Confluent offers a library of pre-built, managed connectors to common databases, applications and cloud services.
Security and governance. Kafka provides ACLs. Confluent adds role-based access control, audit logs and data lineage, which matter more as more teams share the platform.
Operations. Patching, upgrades and scaling sit with your team, including emergency fixes when a critical vulnerability lands. On Confluent Cloud they’re managed for you. Confluent Platform brings the same capabilities on-premises or in private cloud, and WarpStream runs inside your own cloud where data sovereignty demands it.
None of this makes open-source Kafka the wrong choice. It means the trade-off changes as streaming moves from useful to business-critical.
AI raises the stakes
Kafka is very good at one job: getting events from where they happen to where they’re needed. The value comes from what happens next.
AI makes that more pressing. Models and agents are only as current as the data they receive. A platform that delivers events reliably, with governance and lineage built in, means they act on what is happening now rather than on the last batch run. With watsonx.data integration, Confluent’s streaming layer connects into a wider pipeline and governance framework built for AI.
We work with organisations extending existing Kafka investments as well as those coming to streaming for the first time. We always start with the use case, because the right architecture depends on it.
Find out how we work with Confluent →
Know what you’re running
None of this means your Kafka platform is at risk tomorrow. Most self-managed estates carry on working, often for years. That is exactly why they slip down the list.
The PHP alert is a reminder of how quickly that can change. A platform that has been quiet for a long time can still be carrying old versions, old configuration and old assumptions. The teams in the strongest position are the ones that already know what they’re running, who looks after it, and what they would do if something changed overnight.
That doesn’t have to mean a migration or a big programme. It starts with an honest look at where you are.
KAFKA OPTIMISATION ASSESSMENT
Already running Kafka?
If Kafka has quietly become production-critical in your organisation, it’s worth knowing what’s building up underneath it.
Our Kafka Optimisation Assessment is a structured analysis for organisations running open-source Apache Kafka, Confluent Platform or Confluent Cloud. You get:
- A full review of cluster health, connectors, schemas, stream processing and security
- Prioritised recommendations, with critical issues kept separate from advisory findings
- A read-only approach. Nothing in your environment is changed
Whether you want a more reliable platform or you’re weighing up a move to Confluent, it gives you a clear picture to decide from.