Outlook on the Data Mesh: 4 Steps for the Paradigm Shift

But why is the Data Mesh a game-changer? First, the definition:

A Data Mesh is a concept for organising data architectures. This is based on the idea that responsibility for data is distributed to individual teams or domains. What does this mean more specifically?

When set up correctly, the scalability and flexibility of data architectures will improve.

A tangible example helps with understanding.

Imagine you post on LinkedIn. You would receive the likes and comments that your network leaves, but not live, but with a delay of several hours. Unthinkable and annoying, isn’t it?

We’re now accustomed to receiving an immediate response. But that’s by no means a given. The IT infrastructure must first be able to deliver this. Let’s stick with the LinkedIn post example. Because it gets worse.

Imagine you’ve posted something and receive countless comments and messages. So far in companies, it’s arranged that someone else collects and clusters this data for you. The problem: Unlike you, the person doesn’t know which messages are important to you and which aren’t. Under certain circumstances, the data essential to you is filtered out — or placed in false contexts. You don’t just get the data late — but also distorted.

Much better: All specialist departments may draw the most important information for themselves in real-time from the data pool. And, this is NEW: self-determined.

But we’re still far from this offering. In the vast majority of organisations, it still works like this instead: A team dedicates itself to data preparation. These are collected and curated according to best knowledge. The problem: These data teams often cannot evaluate the quality of information nearly as precisely as the departments themselves.

Through central data teams, all other departments depend on the work of one specialist area. This isn’t just a bottleneck, but an economic risk. Because what is supposed to happen if people without domain expertise make a pre-selection of information for the corresponding departments? Right: chaos.

With a new approach. We must reorganise data management. In such a way that specialist teams are allowed to look after their own data products themselves. And they later make these verified and sorted data available to everyone else — refined.

This doesn’t mean that central data teams will soon have no job. On the contrary: They provide the infrastructure via Apache Kafka and create an environment in which departments can efficiently juggle their relevant data. This is and remains a complex and important job on which those affected can focus even more strongly. Because: Only if the data platform can be operated easily will the departments also use it bilaterally. That is, not just pull data, but also return the clustered information.

Because in future, the departments should analyse the information, check the quality, and publish the extracted data in a better quality. These, in turn, become visible in real-time for all participants in the Data Mesh.

This new approach would be associated with a paradigm shift.

We must move away from a central team that evaluates data. A self-service portal is needed in which departments are responsible for processing relevant information.

With a Data Mesh, we create exactly this data infrastructure. We procure data without thousands of processing programs and without a data swamp. Instead, different teams can operate these systems with little effort and process the corresponding data for themselves and everyone else. This is the idea behind the Data Mesh — and thus the basis for us being able to understand data as internal products.