More articles from this series
Data in a Microservice World
From start-up to corporation – more and more companies are adopting microservice architectures. In this post, you'll learn how companies use Apache Kafka to simplify communication between their services.
Companies – of every size – are increasingly adopting microservice architectures. This means: A previously large and complex application is divided into several units. These microservices should communicate with each other exclusively via interfaces (APIs for short).
The supposed advantages are obvious.
Companies henceforth organise their teams into smaller units. These develop their solutions independently of each other. The benefit? They can test results more quickly and thereby identify errors earlier.
Equally important: Because teams are smaller, communication usually improves. As is well known, the number of potential misunderstandings increases with the number of people involved. I say: The pursuit of microservices is logical. At least in theory. Because there are problems precisely when it comes to communication.
What REST Often Really Means
In fact, in a development project, not only do people talk to each other, but IT systems also communicate with one another. We’re talking about: REST. Spelled out, this means: Representational State Transfer. The individual microservices exchange information with each other via the HTTP protocol. Questions are asked or answered.
REST, however, also means to be at rest. And that’s unfortunately fitting. In classic microservice architectures, users must do one thing above all: wait.
Figure 1. Typical in REST architectures: Service A calls B, which waits for C, and C needs results from D and E to respond. And meanwhile A waits
An example:
Service A wants to get an answer to a question from Service B. So far, uncomplicated. The problem?
B can’t answer it without first asking C. C in turn needs information from D and E for the answer. And A? Waits and waits and waits.
It gets worse:
Perhaps the team operating Service E has just released a new version that no longer matches C’s assumptions. E can’t answer the request as a result. The microservice architecture is thus doomed to failure before the project has even begun.
How Kafka Rescues Microservice Architectures
The obvious solution is well-known. Teams coordinate their activities and henceforth agree all changes amongst themselves. The problem: The units thereby build up exactly what they actually wanted to avoid with microservices: dependencies between teams and thus also between services.
What was supposed to make work easier now makes it more difficult. This costs time and money. Fortunately, there’s Apache Kafka, which can significantly reduce these dependencies even within a microservice architecture.
Figure 2. The alternative: Services notify others as soon as something happens. This way, every service knows the current state at all times.
Kafka dispenses with the request-response game. Instead, different (micro-)services permanently publish their events. The services that are interested in the information listen in real-time to these updates and react correspondingly quickly.
Moreover: Kafka can capture these events in the different services and distribute them company-wide. One can imagine the software as an effective and efficient hub for data streams.
What Kafka Means in Practice
To explain it more tangibly: If you use a service that’s responsible for addresses, it publishes the changes at the moment these adjustments are made.
Even more practical:
Figure 3. The address service notifies other services as soon as an address is changed. The greeting card service listens to these changes and updates its local address table.
If a greeting card service handles shipping, it can access a local copy of the relevant data thanks to Kafka. Querying the address service is no longer necessary. You can save yourselves delays due to downtimes or maintenance with the help of Kafka – or at least minimise them.
Another advantage: The chaos created by the mass connections of the communicating services is also reduced by the hub.
The Crucial Advantages of Kafka: Affordable, Secure, and Efficient
Of course, messaging systems existed before Kafka. But Apache Kafka brings two major architectural innovations:
-
Kafka’s performance and simultaneous reliability is hard to beat. Even with a small and relatively cost-effective Kafka cluster consisting of 3 standard (cloud) servers, we can achieve data throughputs of several hundred MBs and millions of messages per second. The risk of data loss is extremely low, as Kafka replicates and distributes data across multiple servers.
-
Kafka doesn’t just send messages between different systems, but can also store this data. When introducing a new service, it doesn’t need to somehow "synchronise" the data. Instead, it can simply read all necessary data from Kafka. Afterwards, it’s at the same level as all other services.
And if someone does deploy erroneously on Friday at 4:00 PM, Kafka allows you to "time travel" and restore the original data – before the bug was introduced. Because Kafka only forgets when we want it to.
About Anatoly Zelenin
Hi, I’m Anatoly! I love to spark that twinkle in people’s eyes. As an Apache Kafka expert and book author, I’ve been bringing IT to life for over a decade—with passion instead of boredom, with real experiences instead of endless slides.
Continue reading
Data as a Product: The 6 Principles of Effective Data Management
Data is available in real-time. The infrastructure for microservices is in place. The goal is clear: We now want the Data Mesh. But how do we reach our business departments with it? This post explains why data management is a cultural and product question.
Read more
Outlook on the Data Mesh: 4 Steps for the Paradigm Shift
The "Data Mesh" is a real hype in the IT. Why companies benefit from a decentralised data architecture and how Apache Kafka helps establish this new structure is discussed in this article.
Read more
We believe in a world where companies make better decisions with real-time data, act sustainably, and actively shape their digital future.
Copyright 2025, All Right Reserved
Contact
Address
Commit to Flow GmbH
Wiener Platz 11
01069 Dresden