Making integration simple: just don’t delete it

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

Record deletes

What happens if we delete records in one of the systems in the sync?

Well, you might not like the answer – but the best answer is that in day-to-day operations you probably shouldn’t be deleting records at all. (Cough cough GDPR … I know, we’ll get there).

In this post we’ll look at what it means to delete a record and then we’ll consider what that means for an integration.

Deleting records

Let’s talk about deletion in principle before we talk about integration.

In most systems, a record, let’s say a Contact record, will have connections to a handful (or many more) other records. A Contact might belong to a Company, they may be referred to on Orders, on Help Desk Tickets, on Sales Opportunties, etc. In database terms, each of these are relations between records. In the olden days as far back as the 1990’s, we talked about “relational databases” for this very reason.

Deleting just one Contact record could have a significant knock-on effect. What happens to all those other records referring to that Contact record? The answer is very system specific, we were looking at this recently for a sync between Freshdesk and Dynamics 365.

  • Freshdesk will delete the Contact and all associated Tickets. That has a significant implication on reporting and, in the case of a helpdesk system, trying to identify systemic problems which are causing multiple tickets to be raised.
  • Dynamics would leave the Tickets in situ but delete the reference to the Contact.

Ultimately, each system will have a different approach to this. In some cases it will be elegant (Dynamics), in some cases efficient (Freshdesk) and in some cases, the system just won’t have the ability to delete.

One term sometimes bandied about is a soft delete. This is more like an archiving status: the record still exists, it is just not visible to all users. This can solve the relational issues, but might have UI/expectations challenges to manage.

Okay, so deleting is not great, let’s talk about integrations

Let’s talk about deletions in the context of a Seamless integration. Seamless is architected as a message queue, it polls a source system for new/changed data and transmits it to a target system. Seamless doesn’t actually know about the data itself or maintain a list of all records on each system. It was designed this way for a range of security and flexibility reasons.

The challenge of this approach is that Seamless cannot know about records that no longer exist. When polling the source system, a record that no longer exists doesn’t appear as changed because it doesn’t appear. It is a classic “Donald Rumsfield scenario” of unknown unknowns.

Simply put, if you delete a record from a source system, Seamless will never know.

You’re smart guys, you can fix this

Of course, it it is possible to engineer a solution. As always, there are options.

  • Option one: A register of records. In this option Seamless would maintain a register of records in each system. We can do this by record id so there’s no content stored inside Seamless and this would address the pressing security consideration.
  • Option two: Flag for deletion. In this option, we would add a flag to the record in either/both systems titled “Flag for deletion” or similar. As this flag is effectively a change to an existing record, Seamless will recognise the flag and act accordingly. This could be deleting the record in both systems and cleaning up references as required.

Flagging for deletion can be achieved in config and gives Seamless flexibility to clean-up records as required by either system in the sync. To date, this is the option that has been preferred when the question is raised.

Sometimes you just gotta delete

There are can be legitimate drivers for deleting records. A request from an individual to delete all records on hand is one such example. Under the GDPR, this is something all businesses operating in Europe should be considering. SaaS app vendors will increasingly find this is a requirement but, given Seamless’ ability to connect to all sorts of endpoints (not just modern SaaS apps), we need to ensure we have a few tricks up our sleeve.

In this context, there are a few options (always with the options!). We could implement a deleting mechanism along the lines of those described above. Or we could consider data anonymisation.

If a system doesn’t have an elegant way to clean up relationships between records, it might be better to anonymise data rather than delete it. This could be done by changing names (and other personal info) to random strings. This is certainly something that could be done with Seamless’ data transform features.

So, what happens when someone deletes a record?

That very much depends what you mean by “delete”. Is it a hard delete where the record disappears? Is it a soft delete where the record is archived? Does the system have a way of telling us the record has been deleted or do we need to construct a way of doing this?

As always, there are options.

Making integration simple: syncing all over the world

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

Azure regions

One of the most powerful lines in all of the Seamless sales materials reads “Seamless can be deployed within an Azure region of your choice.” It seems so straightforward, but it provides really meaningful options for Recursyv and for Seamless users.

Let’s start with the basics: What is an Azure region?

Azure is the cloud computing environment provided by Microsoft. In practical terms, this means that when Recursyv operate Seamless, we’re actually running our code inside a Microsoft provided computing environment which, in physical terms, is hosted inside a Microsoft data centre.

An Azure region describes a set of data centres which are all connected with high-speed data lines. Typically an Azure region is aligned to a geographic region that you may recognise, within a single country or across a group of closely located countries. Over time, Microsoft continue to open more regions – most recently South Africa and the Middle East.

Using multiple data centres within a single region provides redundancy. If Data Centre A becomes unavailable, all services being run within Data Centre A can be switched over to Data Centre B. Ideally this will happen without users noticing a service interruption.

For security reasons, Microsoft typically won’t detail the specific location of any given data centre, except to confirm that if you subscribe to Azure services within a specific Azure region, the data centres will be located within that region.

Azure region size

At the time of writing this post (August 2019), Azure was available in 54 regions servicing 140 countries.


Okay, it’s about geography – but why does it matter?

There are two main reasons why it matters.

(1) Connectivity speed

If we’re syncing between two applications hosted on the West Coast of the USA but using Seamless within the UK, we’d be sending data across the continental USA and across the Atlantic to a UK data centre, performing Seamless core operations and then returning the data back across the Atlantic and back across the continental USA. This will have a significant time delay and, if you’re counting, unnecessary energy consumption. For two applications hosted on the West coast of the USA, we’d be wise to use an Azure region on the West coast of the USA.

Of course, we won’t always be syncing two applications hosted in the same place. In these scenarios, we’ll use data location as one of the considerations when advising customers where Seamless should be hosted.

(2) Relevant legislation

Arguably more important than data transfer speeds is the legislation/regulation that will apply to the Seamless user’s processing of data. There is a growing recognition of the importance of data housing commitments and associated legislation to support these. Within the EU, the GDPR (General Data Protection Regulation) is the most widely known such example.

Seamless users will typically have significant legislative commitments to look after the data that is being synced by Seamless. Choosing the appropriate data region will enable Seamless users to meet their commitments to their own customers/suppliers/partners.

Is it all as complicated as it sounds?

No. In most cases Recursyv will be able to provide advice on how to select the most appropriate Azure region. This will largely be driven by factors detailed here as well as some operational considerations which will be specific to the given integration.

At the time of writing, Recursyv are operating Seamless in the Azure regions in the UK, EU, USA and Australia with an implementation in Azure Germany being planned. In each case, the Seamless services are deployed within the Azure region in support of providing a mix of the appropriate connectivity speed and the most suitable legislative environment.

Making integration simple: all about API calls

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

API Calls

What is an API call?

Let’s start with an API and then describe an API call.

An API

An API, application programming interface, is the “public face” of one software application to other applications. One app, Seamless, can reach out to another app, let’s say Autotask, and ask the other app for something. Seamless could be asking Autotask for data, Seamless could be trying to update a record in Autotask, etc.

Typically SaaS vendors publish API guidelines which describe the various things that an external system can ask the application to do. As long as Seamless reaches out to Autotask and asks according to the structures laid out in the Autotask API Reference, Autotask should do what Seamless asks.

An API call

An API call is one instance of Seamless asking Autotask to do something. If Seamless asks Autotask for a Contact record where the Contact ID = 123454, that query is a single API call.

Continue reading

Does it have to take so long?

ClockOne thing that always strikes us about rolling out integrations is the difference between the time taken to prepare the integration and the time taken to deploy the integration. Our engineering team may estimate a few days to build the integration but the overall effort required often stretches beyond that estimate.

For many of our rollouts, we start with a template. For a purely templated integration, we can prepare the underlying sync setup in an hour or two. Underpinning the template is the fact that we’ve done many integrations with both systems, we have feature-rich system connectors in our library and a great working knowledge of how the systems work.

So… why does the project need consulting time to get from our Project Engineers to syncing data?

 

Prerequisites

There are some things that we need before we even get started. This usually means arranging connectivity details, user/api accounts, possibly creating some cross-reference fields, etc. It should also not be overlooked that Dan really works better with a good cup of coffee. If we’re deploying into a client’s Azure environment, we need it to be setup according the specification provided.

We provide the specs for all of these as part of our project initiation, but it often takes a few project calls to confirm that prereqs have been delivered so that we can begin work in earnest.

Taking advantage of the templates

If you’ve read some of the explanatory content on the website, you’ll know we love banging on about our templates. For sure, they help us make a massive leap on Day 1 of setting up an integration. They’re still only a starting point. Most systems have had some changes made since they were deployed. At the very least, option sets are likely to have been changed to meet business requirements. More commonly a few extra fields have been added to one system and need to be reflected in the other.

Often there are differences between the systems that require some flexible thinking to solve data problems. Some applications provide users with complete flexibility to make any sort of change, setup any sort of data entity, etc. On the other hand, other apps have limited customisation options, perhaps just a putting new fields on a form. As part of the integration design we need to come up with an approach to keep these different data structures in sync between the two systems.

Testing

Seamless makes changes to data and we want to be completely confident that the changes we’re making are as expected. When we transition from our workbench to a target environment, invariably we find things that are different. A round of testing will pick these up and allow us to make the necessary changes to ensure data integrity.

Additionally, a test cycle often results in evolution of the sync requirements. Users see data in situ and that sparks ideas which require changes to the sync spec.

Environments

The gold standard is an organisation who is running a complete set of test environments that are a mirror of their production environments. All we’d need is a different set of credentials for each environment and everything else would be like-for-like. It would also be great if everyone’s favourite sports team could win every match they play and that the ending of Game of Thrones was universally acclaimed.

Test environments might be behind production because of that hot fix that was quickly applied. Or we’re in the midst of a broader change project and test environments are ahead of production. Perhaps the environments are functionally similar but the test environment data doesn’t reflect a real-world data set. If the systems are not SaaS apps, it may be that connectivity for the systems is totally different.

Ultimately there needs to be confidence that testing results in the test environment are a meaningful representation of what will occur in the production environment.

Why is integration so often an afterthought?

HeartOn their own a heart, lungs and a liver would be pretty useless. But when you join them up using the vascular system (arteries, veins, capillaries), all of sudden the sum becomes greater than the whole. Each of our organs has a “specialist role” and when they perform to role, the work they do is invaluable. The heart pumps blood, the lungs oxygenate blood, the liver helps you recover from a big night out.

In this admittedly sketchy analogy, IT systems are no different. Each IT system in your business has a specialist role which it (hopefully) does very well. Most businesses will have a financial system (invoices, accounting, etc.). Some will have point-of-sales systems (POS), some have a sales or contact management system (CRM), most will have an email system. Many, possibly even most, will have a core line-of-business system which helps manage or control the primary business function of the company.

All too often there is no vascular system connecting all of these disparate applications together. Each system operates using its own data set “knowing” things which it is not “sharing” with other systems. Has Customer A paid us in full? Does Customer B have any open queries? These are pieces of information which users across the business could, and should, use to make more informed decisions.

Why are businesses not building their vascular systems?
Why do so many businesses not focus on linking up their IT applications?

 

Is there a fear that integration is too expensive?

Possibly. Traditionally the software that connects up applications – known as middleware – was expensive. It required specialist infrastructure and a lot of work just to get the application talking to the middleware.

Two market forces have arisen to challenge this. Firstly, in the era of cloud applications, the only infrastructure required to run modern infrastructure is often a data connection. Secondly, the advent of common APIs means that connecting to many applications has become much simpler.

Is there a fear that integration is too complicated?

Possibly. Connecting system A to system B requires some degree of planning. It cannot be denied that there will be some work to line up data fields in either system and to ensure that business processes on either side will still operate with data being added/updated.

Using templated connectors and a proven, rigorous, analysis means that the complications can be minimised. Most businesses are not in the business of running complicated IT projects. Using integration templates, supported by repeatable analysis for non-standard data, means the complications can be quickly resolved.

Is there a fear that integration will take too long?

Possibly. For a myriad of reasons, IT projects have a reputation for taking longer than expected. Most companies are not in the business of running long IT projects, they take energy and focus away from day-to-day operations.

Using a well designed toolset which has a demonstrable ability to rapidly deploy integrations removes a lot of the risk. Templates for many common applications, coupled with underlying foundation connectors for common integration scenarios, mean that any development is fast-tracked before it even begins.

Does it have to be expensive, complicated or time consuming?

Of course not. Although admittedly, if the answer were yes that would be a pretty surprising take for this post!

For all too long, sophisticated integration has been the preserve of large enterprises enterprises who could justify the cost, and associated complications, with managing traditional options. Cloud options and the uniquity of APIs has drastically reduced the cost profile of setting up and operating integrations. Complications can be controlled by using a reliable toolset. And the time to deliver can be accelerated by using re-usable components.

With the introduction of reliable, cloud based infrastructure-as-a-service options, coupled with well designed tooling, sophisticated integration is accessible to companies of all sizes.

Walking the walk

Walk the walkHow do you tell the difference between “walking the walk” and “talking the talk”? We’re delighted whenever we can meet a less typical client requirement that also enables us to demonstrate different aspects of the Seamless offering.

In this post, we’ve chosen to share the stories of 3 recent implementations that demonstrate different aspects of how Seamless offers flexibility when delivering application integration.

We’re not looking to integrate Microsoft apps

You’ll not be surprised to learn we’re big advocates of Microsoft, but they’re not the only app vendor out there. In a recent implementation, client is using Seamless to keep a pair of helpdesks in sync, one of which runs on BMC Remedy and the other on ManageEngine.

This implementation demonstrates how we can use Microsoft Azure to enable digital transformation outside of the Microsoft realm. Remedy and ManageEngine are both commercial applications from different vendors. We’re using Seamless, running in Azure, to keep them in sync.

We’re not just connecting two applications together

We have a client rolling out Dynamics 365 into an existing architecture which contains an Azure service bus for transporting data across the enterprise. Seamless has been setup to sync data to and from the service bus.

This demonstrates how the componentised nature of Seamless allows us to deploy into more complex contexts than simply syncing between System A & System B. In order to sync to/from a data bus, we only use half of a typical implementation. In turn, this means that we needed to setup some Seamless’ internal data transfer objects differently. The configurability of Seamless meant we were able to achieve this within a few days of joining the project team.

In this instance, we’ve deployed into the client’s Azure estate rather than our own. This enables the client to take advantage of their own existing security infrastructure within Azure, as well as their pre-existing preferential commercial terms from Microsoft.

We’d like to use some of our technical skills

Syncing contacts and accounts between Dynamics and Autotask is bread-and-butter stuff for Seamless, so why is it being mentioned here? In this example, our client, Verdant Services, a consulting and cloud services provider, is configuring their own integration.

Verdant are using the Seamless Integration Workbench to specify data mappings, update transformations and generally take ownership of their integration. As the project progresses, they’re also able to quickly change their integration to move data between various staging and live environments.

This demonstrates the power of the Seamless Integration Workbench. With a few hours training, our client has taken ownership of the integration inside their software delivery project. This enables them to be flexible and respond to the needs of the project as it progresses.

Debunking common myths about cloud computing

Cloud computingIt wouldn’t be remotely accurate to suggest that cloud computing is the “next” big thing. I had my first email address in 1993 and Salesforce, the golden child of cloud apps, was founded in 1999! However, there are still a number of misconceptions that we encounter when we mention that Seamless has been designed and built as a cloud-first integration solution.

When we say Seamless was built “cloud first”, we mean that it was built using capabilities that have only become available in the cloud-era (i.e. microservices delivered within a cloud computing platform – more about that later). We do not mean that it will only connect to other cloud based applications – Seamless can absolutely connect on-premises software to cloud applications.

To make sense of it all, we’ve debunked some common myths about cloud computing.Continue reading

Taking a moment to celebrate the engineering

Engineering

Since the beginning of the year we’ve been working on a particularly thorny deployment. An integration with lots of technical challenges which brought together many features of Seamless for the first time. Jon has been tucked away in “deep dev” mode and, as he prepared to put the integration into test, he wrote a wrap-up note for the rest of our team. The implementation validates a lot of the underlying engineering for Seamless. On reflection, I realised that this integration creates a lot of milestones.
Continue reading