As much as it pains me to admit it, I fear that Gwyneth Paltrow and Chris Martin were onto something when they coined the term “conscious decoupling”. In the data integration space, the ability to break down a service into components is a significant enabler of flexibility. That’s good because technical flexibility translates to business flexibility. If the only constant is change, as we’re so often told, then flexibility in technical architectures is the way to prepare for change.
Let’s a have a look. At a generic level, we call it the Seamless “service” because that is how we deliver it, but it is actually composed of a set of components which can be individually controlled. Imagine, if you will, a one-way integration from a source system to a target system.
In practical terms, the work of an integration is to:
- Get information from the source system. This is handled by a source system plugin, typically integrating against an API for the source system and looking for information (i.e. new/updated data), by certain criteria. Data is broken into messages and posted onto Azure Service Bus.
- The management of messages on the Service Bus. Queueing, monitoring for message expiration, etc.
- Put data into the target system. This is a bit more complex as typically there is some data manipulation that is required. This could be as simple as confirming that data meets the rules of the target system, it often includes mapping certain fields to new values (e.g. if a record is of type “Prospect” in the source system, in the target system it is called a “Lead”). It will often include logic about assessing which data is the most recent and determining whether the target system should be updated at all.
It’s an ever-changing world
Traditionally these integrations were built as a “point-to-point” integration (i.e. directly connecting source and target) and, assuming everything went as imagined when the integration was first built, everything was A-okay. But things don’t ever stay as imagined.
- The business acquires another company and now needs to consolidate information from two source systems.
- The business wants to assess a new software package and needs to simultaneously move data from the source system into another target during the trial period.
- Or, it is as simple as, the target system becomes unavailable for a short while.
Imagine that Seamless is like the Lego City of data integrations. Each of the steps detailed above is a single Lego model and each can be operated and managed separately. But as a Lego City, the whole thing works when all the models are put together on one play table.
- Two sources into one target? No problem – we deploy an additional source plugin to pull data from the new source as well. If the data that it produces looks different, we may need to update the target system plugin to apply the target system data rules.
- Two targets? No problem – an additional target plugin and some logic applied to the service bus about which target systems are interested in which data messages (i.e. do all pieces of data go to both systems? maybe only some go to both and some go to only one? that’s fine).
- What if the target system is unavailable for a while? No problem – we keep the messages queueing on the bus until it becomes available.
Lego is popular because it’s modular nature allows you to plug-and-play different parts of models. You can react to the world around you without having to rebuild from scratch every time. Unless my 5 year-old is involved, in which case there’s a lot of rebuilding, but thankfully we’ve not given him the admin passwords for Seamless.
Conscious decoupling – it’s about flexibility. Flexibility to quickly and affordably respond to the inevitable changing nature of your business needs.
[icon name=”exchange” class=”” unprefixed_class=””] More information on Seamless can be found here.