Making integration simple: syncing all over the world

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

Azure regions

One of the most powerful lines in all of the Seamless sales materials reads “Seamless can be deployed within an Azure region of your choice.” It seems so straightforward, but it provides really meaningful options for Recursyv and for Seamless users.

Let’s start with the basics: What is an Azure region?

Azure is the cloud computing environment provided by Microsoft. In practical terms, this means that when Recursyv operate Seamless, we’re actually running our code inside a Microsoft provided computing environment which, in physical terms, is hosted inside a Microsoft data centre.

An Azure region describes a set of data centres which are all connected with high-speed data lines. Typically an Azure region is aligned to a geographic region that you may recognise, within a single country or across a group of closely located countries. Over time, Microsoft continue to open more regions – most recently South Africa and the Middle East.

Using multiple data centres within a single region provides redundancy. If Data Centre A becomes unavailable, all services being run within Data Centre A can be switched over to Data Centre B. Ideally this will happen without users noticing a service interruption.

For security reasons, Microsoft typically won’t detail the specific location of any given data centre, except to confirm that if you subscribe to Azure services within a specific Azure region, the data centres will be located within that region.

Azure region size

At the time of writing this post (August 2019), Azure was available in 54 regions servicing 140 countries.


Okay, it’s about geography – but why does it matter?

There are two main reasons why it matters.

(1) Connectivity speed

If we’re syncing between two applications hosted on the West Coast of the USA but using Seamless within the UK, we’d be sending data across the continental USA and across the Atlantic to a UK data centre, performing Seamless core operations and then returning the data back across the Atlantic and back across the continental USA. This will have a significant time delay and, if you’re counting, unnecessary energy consumption. For two applications hosted on the West coast of the USA, we’d be wise to use an Azure region on the West coast of the USA.

Of course, we won’t always be syncing two applications hosted in the same place. In these scenarios, we’ll use data location as one of the considerations when advising customers where Seamless should be hosted.

(2) Relevant legislation

Arguably more important than data transfer speeds is the legislation/regulation that will apply to the Seamless user’s processing of data. There is a growing recognition of the importance of data housing commitments and associated legislation to support these. Within the EU, the GDPR (General Data Protection Regulation) is the most widely known such example.

Seamless users will typically have significant legislative commitments to look after the data that is being synced by Seamless. Choosing the appropriate data region will enable Seamless users to meet their commitments to their own customers/suppliers/partners.

Is it all as complicated as it sounds?

No. In most cases Recursyv will be able to provide advice on how to select the most appropriate Azure region. This will largely be driven by factors detailed here as well as some operational considerations which will be specific to the given integration.

At the time of writing, Recursyv are operating Seamless in the Azure regions in the UK, EU, USA and Australia with an implementation in Azure Germany being planned. In each case, the Seamless services are deployed within the Azure region in support of providing a mix of the appropriate connectivity speed and the most suitable legislative environment.

Making integration simple: all about API calls

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

API Calls

What is an API call?

Let’s start with an API and then describe an API call.

An API

An API, application programming interface, is the “public face” of one software application to other applications. One app, Seamless, can reach out to another app, let’s say Autotask, and ask the other app for something. Seamless could be asking Autotask for data, Seamless could be trying to update a record in Autotask, etc.

Typically SaaS vendors publish API guidelines which describe the various things that an external system can ask the application to do. As long as Seamless reaches out to Autotask and asks according to the structures laid out in the Autotask API Reference, Autotask should do what Seamless asks.

An API call

An API call is one instance of Seamless asking Autotask to do something. If Seamless asks Autotask for a Contact record where the Contact ID = 123454, that query is a single API call.

Continue reading

Does it have to take so long?

ClockOne thing that always strikes us about rolling out integrations is the difference between the time taken to prepare the integration and the time taken to deploy the integration. Our engineering team may estimate a few days to build the integration but the overall effort required often stretches beyond that estimate.

For many of our rollouts, we start with a template. For a purely templated integration, we can prepare the underlying sync setup in an hour or two. Underpinning the template is the fact that we’ve done many integrations with both systems, we have feature-rich system connectors in our library and a great working knowledge of how the systems work.

So… why does the project need consulting time to get from our Project Engineers to syncing data?

 

Prerequisites

There are some things that we need before we even get started. This usually means arranging connectivity details, user/api accounts, possibly creating some cross-reference fields, etc. It should also not be overlooked that Dan really works better with a good cup of coffee. If we’re deploying into a client’s Azure environment, we need it to be setup according the specification provided.

We provide the specs for all of these as part of our project initiation, but it often takes a few project calls to confirm that prereqs have been delivered so that we can begin work in earnest.

Taking advantage of the templates

If you’ve read some of the explanatory content on the website, you’ll know we love banging on about our templates. For sure, they help us make a massive leap on Day 1 of setting up an integration. They’re still only a starting point. Most systems have had some changes made since they were deployed. At the very least, option sets are likely to have been changed to meet business requirements. More commonly a few extra fields have been added to one system and need to be reflected in the other.

Often there are differences between the systems that require some flexible thinking to solve data problems. Some applications provide users with complete flexibility to make any sort of change, setup any sort of data entity, etc. On the other hand, other apps have limited customisation options, perhaps just a putting new fields on a form. As part of the integration design we need to come up with an approach to keep these different data structures in sync between the two systems.

Testing

Seamless makes changes to data and we want to be completely confident that the changes we’re making are as expected. When we transition from our workbench to a target environment, invariably we find things that are different. A round of testing will pick these up and allow us to make the necessary changes to ensure data integrity.

Additionally, a test cycle often results in evolution of the sync requirements. Users see data in situ and that sparks ideas which require changes to the sync spec.

Environments

The gold standard is an organisation who is running a complete set of test environments that are a mirror of their production environments. All we’d need is a different set of credentials for each environment and everything else would be like-for-like. It would also be great if everyone’s favourite sports team could win every match they play and that the ending of Game of Thrones was universally acclaimed.

Test environments might be behind production because of that hot fix that was quickly applied. Or we’re in the midst of a broader change project and test environments are ahead of production. Perhaps the environments are functionally similar but the test environment data doesn’t reflect a real-world data set. If the systems are not SaaS apps, it may be that connectivity for the systems is totally different.

Ultimately there needs to be confidence that testing results in the test environment are a meaningful representation of what will occur in the production environment.

Do you really need to use an integration service to sync your SaaS data?

 

Many popular SaaS applications include a handful of vendor built integration options. Vendors want to make it easy for new users to become embedded in their applications. They also want to embed their apps into your enterprise, making it less likely that you will consider moving onto other options.

 

If you can get an integration as part of your monthly fee, why would you look to Recursyv Seamless?

 

Why should you consider vendor integrations?

  • Vendor integrations are often built into the existing monthly charge. They’re a value-add option for the vendor to make signing up more attractive.
  • They’re often ready to use “out of the box”. Capture a few details and you could be ready to go.
  • They will be supported by the vendor over time. If the vendor changes their platform, they’re likely to evolve the integration in-line with platform changes.

 

Why should you consider an integration specialist?

Vendor integrations are likely to be limited in their functional scope

They are likely to support standard fields and sync with pre-determined fields in the other system. This may be great for getting started, but it doesn’t allow much flexibility. Most of our integrations have required some additional configuration before they meet the overall business requirement. These changes run along an entire spectrum of complexity. They could be straightforward as adding a single extra field. They could be as complex as syncing custom tables, syncing subsets of data based on custom filtering, etc.

Vendor integrations are likely to limit flexibility over time

As your business evolves, you’ll want to ensure that the systems supporting your business evolve with it. This might mean additional data fields, additional data sources (or endpoints) or even simply new options for option sets. Vendor integrations will not be setup to support your constantly evolving business requirement.

Vendor integrations will often be limited to point-to-point integrations with other popular applications

Vendors will provide tools to sync their system with another system, drawn from a small selection of popular systems. It is unlikely that you will be able to connect more than one other system. You may be limited to only the most popular SaaS apps. Single point-to-point integrations may help get over the initial hurdle of keeping two systems broadly in sync, but there is no room for growth as you need to sync with more systems (billing, support, back-end platforms, etc.)

Vendors may not be quick to support changing endpoints

Imagine you’re using Vendor A’s app and their integration offering to Vendor B’s app. When Vendor B makes changes to their API, it is unlikely to be Vendor A’s immediate priority will be to update their integration. You may find yourself waiting an unpredictable amount of time before Vendor A gets a chance to update their integration. If not enough customers are using the integration, Vendor A may decide it is not worth the investment to update the integration. An integration vendor will be focused on keeping their endpoint libraries up-to-date and are likely to have the changes prepared in advance of the API change.

Vendor integrations typically include limited monitoring and recovery features

Unless you notice missing data, you may never know that your integration is not working properly. By the time you do, it’s often too late. A specialist solution will typically include various monitoring options. These should be able to confirm that the integration is running and to report on integration errors. Additionally, an integration provider will have options for recovering and redeploying integrations quickly.

 

Specialists live and breathe their area of focus. As an integration specialist, we’re regularly encountering unusual integration requirements and using those to constantly evolve our Seamless feature set. Like typical SaaS apps, as the Seamless service improves, those improvements are offered to all users.

Within Recursyv, our primary focus is about getting systems syncing quickly, easily and flexibly. Once integrations move from our engineering and project teams to our operations team, we’re solely focused on ensuring that those integrations continue to run, day-in and day-out.

We proactively monitor integrations for technical and data issues and address them as soon as they’re raised. Often our alerting will flag up SaaS app availability problems before they’re noticed by users.

 

 

How to choose?

In short, it is all about what you need an integration to do for you. There is, of course, a place for using vendor provided integration tools. They will often meet basic requirements where there is limited need for flexibility and control. Specialist integration providers will introduce flexibility, additional data endpoints and control features that vendor solutions are unlikely to offer.

When considering your own approach, there are some questions you should consider

  • What are the different apps you need to sync with? Does the vendor you’re considering offer integration endpoints for those apps?
  • What level of flexibility / customisation do you need when matching data items across different systems? Do you have custom fields, tables, option set values, etc.? Are these likely to change over time?
  • How often do you need data to refresh?
  • Are you likely to want to introduce additional data endpoints over time?

 

Ending the enduring legacy of data silos

Silos

“I understand what data syncing is, but why is it useful?” is something we hear all too often. Here’s how we try and explain it.

It should be that every time your business interacts with a customer, a piece of information is generated. This could be a phone call note, an invoice or a request for help from your customer. All common scenarios and, in most cases, all handled by different systems.

As our user community has grown, we’ve found that one way to consider the usefulness of syncing data is to think about all the places where a business interacts with customers.

  • The sales team are often using a CRM system (Dynamics, Salesforce, HubSpot, etc.)
  • The marketing team may be using email marketing (MailChimp, etc.)
  • The customer support team are typically using a helpdesk system (Zendesk, ServiceNow, etc.)
  • The finance team are likely to be using an accounting package to send and monitor invoices, debtors, etc. (Xero, etc.)
  • There may be collaboration software in use across the business, either to manage projects or just to maintain open communication channels (Jira, Slack, etc.)

That list just describes generic functions that most businesses have. It is quite likely that there will be some specialised software to support the primary function of your business. This could be point-of-sale software, manufacturing/control software, project management software, e-commerce, etc.

That’s a lot of places where there may be references to customers. When your sales team ring up a customer, they’ll want to know whether the customer has any outstanding complaints. When your accounts team need to follow-up with a query, they’ll want to know who is best to ring and ask. When your support team are working on open helpdesk tickets, it’s useful to know that the sales team have a big proposal submitted awaiting a decision.

This is where data syncing comes to the rescue. When I’m explaining the business value of Seamless, I often ask people to tell me all the different engagements their business may have with their own customers. I then ask what underlying system that interaction is recorded on. Typically, as we talk through these engagements, it becomes very evident how standalone silos of information are being built across the business.

Seamless is about making disparate silos of data disappear into the background. Staff across an organisation should all see the same information about customers (or orders or helpdesk incidents or … ) no matter which IT system they’re currently using.