When we’re discussing integrations with prospective clients, one question that seems to come up every so often is “We’ve got a dev team as part of the project, why don’t they just build it?” It’s not an unreasonable question, if there’s no compelling argument for using an integration service then why bother with one?
Getting to grips with integration development
Having some insight into how an integration works will provide some useful context. In simple terms, a straightforward integration follows roughly these steps:
- Get specific data from the data source (i.e. construct the relevant queries)
- Assess whether that data has changed since the last time we got it
- Decide whether the data needs to be posted to the target system
- Encrypt the data
- Compress the data
- Move the data from source to target
- Decompress the data
- Decrypt the data
- Transform the data
- Write the data to target
I should add that those steps are of differing levels of complexity in different scenarios. At a simple level, some systems can provide data which has changed since we last saw it, some systems cannot and the integration needs to perform the compare. At a more complex level, some systems have sophisticated APIs which allow us to construct queries “on the fly” using configurable options, some systems require us to hard code the queries before we can execute them.
In the cold light of day, most of these steps are generic:
- Step 1 is specific to the source system but not necessarily to your business need, that is to say – “Dear system, please tell me about new records created in the past 30 seconds” is a generic request of whatever system you’re using. It is not specific to your business need. If the source is bespoke, then naturally it is a wholly bespoke step.
- Step 2 is typically generic, but will be driven by business need.
- Step 3 is a business decision largely informed by the outcome of Step 2.
- Steps 4 through 8 are generic for any (well designed) integration.
- Step 9 is specific to your setup in the target system. The data transformation will need to reflect the data structures in your target system, including both the overall data structure (an account in the source may be called a company in the target) and potential dropdown list values (Mister maps to Mr, status set to “Active” maps to status set to “Live”, etc.).
- Step 10 is specific to the target system in the same way that step 1 is specific to the source system.
Steps 1 and 10 are specific to the “technical scenario” (i.e. the systems in play) and steps 3 and 7 are where your business requirements are going to drive how the integration should work. The rest of it is pretty much the same, whether you’re connecting your CRM to your helpdesk or your accounting package to a front-end e-commerce system.
Paying to reinvent the wheel
With that in mind, let’s return to the question: “We’ve got a dev team as part of the project, why don’t they just build it?” The short answer is “Because the vast bulk of their development effort would be spent on generic integration features which are already accessible to you at a much lesser cost than building them yourself.”
Naturally, there is more complexity to the answer. In my experience, point-to-point integrations are usually developed to a “minimum functioning feature set”, i.e. they can get the right data from A to B on day one. That’s not necessarily a terrible place to be, but it is not as good as it could be.
A service built first and foremost as an integration service will offer sophisticated features which your project budget is unlikely to underwrite for a single point-to-point integration. Encryption, compression, elegant management of downtime at the source or target, ability to be easily extended, etc. All of these features need to be designed and built.
Additionally, developers are not “just developers”. That’s like saying builders are just builders. The technical skills required to marshal data, overcome infrastructure and connectivity complexity, manage reporting, etc. are quite different to the skills required to configure / build your new application. Sure, there’s some overlap. Most bricklayers can do some paintwork, but for painting decorative ceiling cornices, I’d rather get a painter.
The big picture
In a 2015 paper on digital transformation projects, Gartner noted that “many organizations already favor a new kind of “build” that does not include out-of-the-box solutions, but instead is a combination of application components that are differentiated, innovative and not standard software or software with professional services (for customization and integration requirements), or solutions that are increasingly sourced from startups, disrupters or specialized local providers.”
We’re happy to be one of those startups, even if it means using American spellings on our blog post.