Kafka.
Apache Kafka is a distributed event log. Producers append events, consumers read them at their own pace, and events are retained for a configured period, which means a consumer can be replayed from a point in the past rather than depending on a message being delivered once.
Where Kafka fits on a long engagement.
Retention is what distinguishes Kafka from a queue. Because the log persists, a new consumer can read the entire history, a broken one can be fixed and replayed, and a downstream system can be rebuilt from source events rather than restored from a backup. For data platforms that is a structural advantage.
It is also genuinely heavy. Partitions, consumer groups, offset management and rebalancing are real operational surface, and a team running Kafka without someone who understands it is carrying risk they cannot see. For many systems a managed queue is the correct, unglamorous answer.
What an assigned team does with Kafka.
Kafka is a platform commitment rather than a library choice. Somebody has to own partitions, consumer lag and the rebalancing behaviour that only misbehaves under load, and an organisation that adopts it without that person is carrying operational risk it cannot see until an incident.
On an assigned team that ownership usually sits with platform capacity rather than with the engineers producing events, which is why streaming work is commonly assigned next to DevOps managed services on one agreement.
What we use Kafka for.
- Decoupling systems that grew together Producers and consumers that no longer call each other directly, so one being down stops meaning the other is too.
- Rebuilding a downstream store Replaying history to repopulate a warehouse or search index after a change, without asking the source system for an export.
- Feeding several consumers from one stream Analytics, search and notification all reading the same events without the producer knowing any of them exist.
How Kafka capacity is assigned.
Streaming work is assigned under managed data services, and we will say plainly when a managed queue would serve the same purpose for a fraction of the operational cost.
Tell us what your roadmap needs Kafka for.
A service delivery manager replies with the disciplines we would assign, the monthly capacity and what the first month looks like.
Loading the contact form… You can also email hello@azendo.co.
We reply within one working day. No obligation, and no newsletter.