The data mesh and the enduring myth of a golden data source

Data accuracy is like communication within a relationship. We all like to believe we have the perfect solution for ensuring that all data is always up-to-date, but the harsh realities of life get in the way.

When we discuss data synchronisation solutions with prospective clients, we sometimes stumble upon the idea that a given system within a company is the company’s golden source for data. Often this is the invoicing platform, which makes some sense given that invoicing drives revenue. However, it is naïve to think that the finance team are likely to have the latest information. The finance team will typically engage with a limited number of people and they’re only likely to find out about information changes after the fact. Possibly only a few weeks later when invoices are returned.

Rather than a golden data source, we believe in the notion of a data mesh. A data mesh is a world where data is updated at the point that it becomes evident that it is out of date. This could be by any user within the organisation (subject to appropriate security considerations). Additionally, it is likely that different users have interest in pieces of information. For example, sales team may want to understand prospective buyers, the finance team are likely to engage with their finance counterparts.


A data mesh world

In a “data mesh world”

  • all users have access to the most up-to-date information available and
  • all users have the ability to add to and/or update the information on hand … which then becomes available to all other users (see above)

A data mesh world is created by allowing data to flow seamlessly across systems. That’s not to say that all data should be in all systems at all times. That would be a crazy world which undermines usability across systems. Different systems support different purposes and the relevant data within them should be up-to-date.

  • Customer data is likely to appear across most systems, particularly externally facing systems (e.g. CRM system, line of business system, customer support system and finance/accounting platform).
  • Transactional data may only appear in the line of business and finance systems.
  • Sales data (e.g. prospective opportunities) may only appear in the CRM system and not be necessary elsewhere.

It may also be that some data is summarised or aggregated when presented in other systems. Where appropriate, good user experience will allow relevant users to easily move between systems as their work requires them to do so. Naturally, this needs to be supported by relevant role based security policies across the various systems.

  • A summary of support activity (e.g. number of tickets raised, open, closed) may show within the CRM but to access individual tickets, users need to click through to the support system
  • Current account balances may show in the CRM system but individual transaction data is accessible only by clicking through to the finance system

In reality, this results in a complex web of systems where the same data is actually stored in multiple places. This is unlikely to meet a theoretically ideal software architecture, but it does reflect the far more likely reality of most organisations. Different software solutions used by different parts of the business relying on overlapping data sets.


Creating a data mesh world

Creating the data mesh requires enabling technology which seamlessly synchronises data across the various systems. The technology needs to deployed with underlying rules addressing relevant scenarios (e.g. what happens when data is updated in two places simultaneously). The technology also needs to be flexible enough to easily accommodate ongoing change, either of the business process or of the underlying software. It needs to be easy to extend the data being synchronised, to introduce new data fields, to change the data processing rules, etc.

The data mesh has to deliver meaningful information when needed while always being ready to handle changes both externally and internally. Sure, a data mesh needs updating and refreshing as the world changes, but that just reflects the messy reality of the world. Just like communication in a relationship.

Recommended Posts