Apache Kafka
The durable, replayable spine every real-time system we build sits on.
Kafka is the component that makes real-time architecture possible rather than merely fashionable. Its value is not speed — plenty of things are fast — but durability and replayability. Events are retained, so a consumer that fails can resume, a new consumer can read history, and a logic error can be corrected by reprocessing rather than by accepting a permanent gap in the record.
We use it as the ingestion spine for plant telemetry, transaction events, clickstream and change data capture. Producers write once; any number of independent consumers read at their own pace without coordinating with each other or with the producer.
It also carries real operational weight, and we are direct about that. A Kafka cluster needs monitoring, capacity planning and someone who understands consumer lag. For clients whose requirement is genuinely batch, we do not install it.
Where it fits
What Apache Kafka genuinely gives us
Durable and replayable
Retention means a consumer failure is a delay rather than a data loss, and history can be reprocessed after a logic fix.
Decouples producers from consumers
A new consumer can be added without touching the producing system — which matters when the producer is a plant control system nobody wants to modify.
Ordering where it matters
Per-partition ordering guarantees let event-sourced and state-machine consumers be correct.
Scales past what most estates need
The clusters we run handle peaks well beyond design load without architectural change.
Operating Kafka without a dedicated platform team
Most businesses in this market that adopt Kafka do not have a platform team to run it, and this is the practical consideration that should drive the hosting decision. Self-hosting means owning broker upgrades, partition rebalancing, disk capacity planning, and the operational knowledge to diagnose a consumer lag incident at short notice. That is a reasonable commitment for an organisation with a dedicated infrastructure function and an unreasonable one for a team of six application developers.
For most of our clients, managed Kafka on MSK or Confluent Cloud is the correct answer even at a visible price premium, because the alternative price is paid in engineering attention at unpredictable moments. We size that comparison honestly against real throughput and retention rather than against a hypothetical peak.
Whichever route is chosen, the monitoring that matters is small and specific: consumer lag per group, under-replicated partitions, broker disk headroom, and the age of the oldest unconsumed message. A team that watches those four numbers and has a documented procedure for each will handle almost every incident that occurs, without needing deep internal expertise.
Kafka is a commitment, not a component
Kafka solves a real and specific problem: many producers and many consumers needing a durable, replayable, ordered log of events, where consumers move at different speeds and new consumers may need to read history. If that is your problem, nothing else is quite as good. If it is not, Kafka is a substantial operational burden bolted onto an architecture that would have been fine with a database table and a queue.
We ask a short set of questions before recommending it. Do you have more than one consumer of the same events, and will you have more? Do you need to replay history to rebuild a consumer’s state? Does ordering within a key matter? Are you dealing with volumes where a database-backed queue would genuinely struggle? A single yes is usually not enough. Several is a strong signal.
Where Kafka is warranted, the decisions that matter are made early and are painful to reverse: topic and partition design, what the key is and therefore what ordering guarantee you actually have, schema management and how you will evolve it without breaking consumers, retention, and consumer group semantics. We have inherited more than one deployment where every event went to a single topic with one partition, which delivers all of Kafka’s complexity and none of its throughput.
Topic and partition design decides everything downstream
A topic should represent one kind of event with one schema and one retention policy. Partitions should be keyed by whatever the consumers need ordering on — a machine identifier for plant data, a customer for transactions — because ordering is guaranteed within a partition and nowhere else.
Getting the key wrong is expensive to fix later, because repartitioning a live topic means reprocessing. We spend time on this at design and write it down with the reasoning.
Edge buffering for Indian plant networks
The link between a plant and a central cluster will drop. A system that loses readings during those windows never earns trust, so we place a store-and-forward buffer at the edge: readings are written locally, forwarded when the link is available, and back-filled automatically with their original timestamps.
The report or dashboard then shows explicitly which period was back-filled, so nobody is misled about when data actually arrived.
How Apache Kafka projects usually go wrong
These are the failure modes we look for first when we are called in to rescue somebody else's implementation.
Apache Kafka questions we get asked
Including where we would recommend something other than Apache Kafka. Call +91 70033 91355 and you will get the same answer from an engineer.
Ask two questions. Do multiple independent consumers need the same events? Do you need to replay history after fixing a consumer or changing logic? If both answers are no, a managed queue such as SQS or RabbitMQ is simpler, cheaper and entirely adequate. If either is yes, Kafka earns its operational cost. We have talked clients out of Kafka more than once.
Managed — MSK, Confluent Cloud or Azure Event Hubs — unless data residency or cost at very large scale forces otherwise. Kafka operations are genuinely specialised, and the managed premium is usually less than the cost of building that capability in-house. For on-premise plant deployments we run self-managed, with runbooks, because there is no alternative.
Services built on Apache Kafka
Technologies we pair it with
Tell us what is slowing your business down.
A 30-minute call with a senior engineer — not a salesperson. You leave with an architecture sketch and an honest cost range, whether or not you hire us.
Direct line
+91 70033 91355Mon–Sat · 9:30 AM – 7:30 PM IST · Sealdah, Kolkata