Making integration simple: just don’t delete it

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

Record deletes

What happens if we delete records in one of the systems in the sync?

Well, you might not like the answer – but the best answer is that in day-to-day operations you probably shouldn’t be deleting records at all. (Cough cough GDPR … I know, we’ll get there).

In this post we’ll look at what it means to delete a record and then we’ll consider what that means for an integration.

Deleting records

Let’s talk about deletion in principle before we talk about integration.

In most systems, a record, let’s say a Contact record, will have connections to a handful (or many more) other records. A Contact might belong to a Company, they may be referred to on Orders, on Help Desk Tickets, on Sales Opportunties, etc. In database terms, each of these are relations between records. In the olden days as far back as the 1990’s, we talked about “relational databases” for this very reason.

Deleting just one Contact record could have a significant knock-on effect. What happens to all those other records referring to that Contact record? The answer is very system specific, we were looking at this recently for a sync between Freshdesk and Dynamics 365.

  • Freshdesk will delete the Contact and all associated Tickets. That has a significant implication on reporting and, in the case of a helpdesk system, trying to identify systemic problems which are causing multiple tickets to be raised.
  • Dynamics would leave the Tickets in situ but delete the reference to the Contact.

Ultimately, each system will have a different approach to this. In some cases it will be elegant (Dynamics), in some cases efficient (Freshdesk) and in some cases, the system just won’t have the ability to delete.

One term sometimes bandied about is a soft delete. This is more like an archiving status: the record still exists, it is just not visible to all users. This can solve the relational issues, but might have UI/expectations challenges to manage.

Okay, so deleting is not great, let’s talk about integrations

Let’s talk about deletions in the context of a Seamless integration. Seamless is architected as a message queue, it polls a source system for new/changed data and transmits it to a target system. Seamless doesn’t actually know about the data itself or maintain a list of all records on each system. It was designed this way for a range of security and flexibility reasons.

The challenge of this approach is that Seamless cannot know about records that no longer exist. When polling the source system, a record that no longer exists doesn’t appear as changed because it doesn’t appear. It is a classic “Donald Rumsfield scenario” of unknown unknowns.

Simply put, if you delete a record from a source system, Seamless will never know.

You’re smart guys, you can fix this

Of course, it it is possible to engineer a solution. As always, there are options.

  • Option one: A register of records. In this option Seamless would maintain a register of records in each system. We can do this by record id so there’s no content stored inside Seamless and this would address the pressing security consideration.
  • Option two: Flag for deletion. In this option, we would add a flag to the record in either/both systems titled “Flag for deletion” or similar. As this flag is effectively a change to an existing record, Seamless will recognise the flag and act accordingly. This could be deleting the record in both systems and cleaning up references as required.

Flagging for deletion can be achieved in config and gives Seamless flexibility to clean-up records as required by either system in the sync. To date, this is the option that has been preferred when the question is raised.

Sometimes you just gotta delete

There are can be legitimate drivers for deleting records. A request from an individual to delete all records on hand is one such example. Under the GDPR, this is something all businesses operating in Europe should be considering. SaaS app vendors will increasingly find this is a requirement but, given Seamless’ ability to connect to all sorts of endpoints (not just modern SaaS apps), we need to ensure we have a few tricks up our sleeve.

In this context, there are a few options (always with the options!). We could implement a deleting mechanism along the lines of those described above. Or we could consider data anonymisation.

If a system doesn’t have an elegant way to clean up relationships between records, it might be better to anonymise data rather than delete it. This could be done by changing names (and other personal info) to random strings. This is certainly something that could be done with Seamless’ data transform features.

So, what happens when someone deletes a record?

That very much depends what you mean by “delete”. Is it a hard delete where the record disappears? Is it a soft delete where the record is archived? Does the system have a way of telling us the record has been deleted or do we need to construct a way of doing this?

As always, there are options.

Don’t use Word for calculations – using the right tools to solve the right problems

The challenges of scale

Tools

In a recent conversation, someone told me about frustrations they had keeping their invoicing system in sync with their line of business application. The business in question is a rental agency of sorts and their core line of business system, a property management system, manages rentals. Every month, the accounting team was spending up to 5 days just to make sure the systems were in sync and the invoices were accurate.

As the business grew, so did the size of the problem. Over time, manually keeping the systems in sync became unworkable. The business leadership decided to address the issue. It was recognised that working from a single data set that everyone used would make processing much easier and more reliable.

In pursuit of this aim, the team decided to migrate all rental processing from the property management system to the accounting package. Commercially speaking, invoicing and revenue recognition were the driver of the business and so using this system at the heart of operations seemed to be a good compromise.

At first glance, it makes sense. One way to keep all the data consistent is to use a single system. In practice, well, there’s a reason you don’t do complex calculations on a table in Microsoft Word. It’s just the wrong tool for the job.

Using the right tools for the job

All the gains made by having a single data set were lost by the fact that the accounting package didn’t have the features to operate the business. Whenever customers looked for something even slightly different to the standard, the finance package just wasn’t flexible enough to support it. Ultimately, any time gained from using a single data set was lost in massaging the finance system to meet operational business requirements.

Typically the best systems for specialised jobs are specialised systems. It just makes sense to run a rental agency on a dedicated property management system. An alternate approach to addressing the underlying problem would, of course, be to sync the data between the systems.

Syncing data can, and often should, be invisible to the users. Users should continue to use the systems they know and trust – the systems that are best for the job at hand. Behind the scenes, a service such as Seamless data sync, can ensure that data within those systems are kept in sync.

Any decent handyman will tell you that you shouldn’t use a screwdriver to bash in a nail.

Seamless

Seamless is a data syncing tool that makes it easy to keep different systems in sync. The line of business system and the finance systems could be “loosely connected” using Seamless such that the core client list would be replicated in both systems and updated in either system when the other updates.

From an operations perspective, the rentals system could be used to manage lettings and generate invoices as appropriate. These invoices could then be passed by Seamless to the finance system which can, and should, manage sending invoices to customers and controlling the subsequent payments cycle.

Simply put, Seamless would enable different functions within the business to use best tools for their respective work. With an underlying data set kept in sync by Seamless, there is no danger of creating “information silos” or introducing rework and manual data entry. Data syncing becomes the glue that holds it all together.

Making integration simple: syncing all over the world

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

Azure regions

One of the most powerful lines in all of the Seamless sales materials reads “Seamless can be deployed within an Azure region of your choice.” It seems so straightforward, but it provides really meaningful options for Recursyv and for Seamless users.

Let’s start with the basics: What is an Azure region?

Azure is the cloud computing environment provided by Microsoft. In practical terms, this means that when Recursyv operate Seamless, we’re actually running our code inside a Microsoft provided computing environment which, in physical terms, is hosted inside a Microsoft data centre.

An Azure region describes a set of data centres which are all connected with high-speed data lines. Typically an Azure region is aligned to a geographic region that you may recognise, within a single country or across a group of closely located countries. Over time, Microsoft continue to open more regions – most recently South Africa and the Middle East.

Using multiple data centres within a single region provides redundancy. If Data Centre A becomes unavailable, all services being run within Data Centre A can be switched over to Data Centre B. Ideally this will happen without users noticing a service interruption.

For security reasons, Microsoft typically won’t detail the specific location of any given data centre, except to confirm that if you subscribe to Azure services within a specific Azure region, the data centres will be located within that region.

Azure region size

At the time of writing this post (August 2019), Azure was available in 54 regions servicing 140 countries.


Okay, it’s about geography – but why does it matter?

There are two main reasons why it matters.

(1) Connectivity speed

If we’re syncing between two applications hosted on the West Coast of the USA but using Seamless within the UK, we’d be sending data across the continental USA and across the Atlantic to a UK data centre, performing Seamless core operations and then returning the data back across the Atlantic and back across the continental USA. This will have a significant time delay and, if you’re counting, unnecessary energy consumption. For two applications hosted on the West coast of the USA, we’d be wise to use an Azure region on the West coast of the USA.

Of course, we won’t always be syncing two applications hosted in the same place. In these scenarios, we’ll use data location as one of the considerations when advising customers where Seamless should be hosted.

(2) Relevant legislation

Arguably more important than data transfer speeds is the legislation/regulation that will apply to the Seamless user’s processing of data. There is a growing recognition of the importance of data housing commitments and associated legislation to support these. Within the EU, the GDPR (General Data Protection Regulation) is the most widely known such example.

Seamless users will typically have significant legislative commitments to look after the data that is being synced by Seamless. Choosing the appropriate data region will enable Seamless users to meet their commitments to their own customers/suppliers/partners.

Is it all as complicated as it sounds?

No. In most cases Recursyv will be able to provide advice on how to select the most appropriate Azure region. This will largely be driven by factors detailed here as well as some operational considerations which will be specific to the given integration.

At the time of writing, Recursyv are operating Seamless in the Azure regions in the UK, EU, USA and Australia with an implementation in Azure Germany being planned. In each case, the Seamless services are deployed within the Azure region in support of providing a mix of the appropriate connectivity speed and the most suitable legislative environment.

Making integration simple: all about API calls

In this series of short blog posts, we’ll try to break down some of the engineering principles behind integrations. We’ll use examples from Seamless, but the principles will be universal.

API Calls

What is an API call?

Let’s start with an API and then describe an API call.

An API

An API, application programming interface, is the “public face” of one software application to other applications. One app, Seamless, can reach out to another app, let’s say Autotask, and ask the other app for something. Seamless could be asking Autotask for data, Seamless could be trying to update a record in Autotask, etc.

Typically SaaS vendors publish API guidelines which describe the various things that an external system can ask the application to do. As long as Seamless reaches out to Autotask and asks according to the structures laid out in the Autotask API Reference, Autotask should do what Seamless asks.

An API call

An API call is one instance of Seamless asking Autotask to do something. If Seamless asks Autotask for a Contact record where the Contact ID = 123454, that query is a single API call.

Continue reading

Do you really need to use an integration service to sync your SaaS data?

Laptop

Many popular SaaS applications include a handful of vendor built integration options. Vendors want to make it easy for new users to become embedded in their applications. They also want to embed their apps into your enterprise, making it less likely that you will consider moving onto other options.

 

If you can get an integration as part of your monthly fee, why would you look to Recursyv Seamless?

 

Why should you consider vendor integrations?

  • Vendor integrations are often built into the existing monthly charge. They’re a value-add option for the vendor to make signing up more attractive.
  • They’re often ready to use “out of the box”. Capture a few details and you could be ready to go.
  • They will be supported by the vendor over time. If the vendor changes their platform, they’re likely to evolve the integration in-line with platform changes.

 

Why should you consider an integration specialist?

Vendor integrations are likely to be limited in their functional scope

They are likely to support standard fields and sync with pre-determined fields in the other system. This may be great for getting started, but it doesn’t allow much flexibility. Most of our integrations have required some additional configuration before they meet the overall business requirement. These changes run along an entire spectrum of complexity. They could be straightforward as adding a single extra field. They could be as complex as syncing custom tables, syncing subsets of data based on custom filtering, etc.

Vendor integrations are likely to limit flexibility over time

As your business evolves, you’ll want to ensure that the systems supporting your business evolve with it. This might mean additional data fields, additional data sources (or endpoints) or even simply new options for option sets. Vendor integrations will not be setup to support your constantly evolving business requirement.

Vendor integrations will often be limited to point-to-point integrations with other popular applications

Vendors will provide tools to sync their system with another system, drawn from a small selection of popular systems. It is unlikely that you will be able to connect more than one other system. You may be limited to only the most popular SaaS apps. Single point-to-point integrations may help get over the initial hurdle of keeping two systems broadly in sync, but there is no room for growth as you need to sync with more systems (billing, support, back-end platforms, etc.)

Vendors may not be quick to support changing endpoints

Imagine you’re using Vendor A’s app and their integration offering to Vendor B’s app. When Vendor B makes changes to their API, it is unlikely to be Vendor A’s immediate priority will be to update their integration. You may find yourself waiting an unpredictable amount of time before Vendor A gets a chance to update their integration. If not enough customers are using the integration, Vendor A may decide it is not worth the investment to update the integration. An integration vendor will be focused on keeping their endpoint libraries up-to-date and are likely to have the changes prepared in advance of the API change.

Vendor integrations typically include limited monitoring and recovery features

Unless you notice missing data, you may never know that your integration is not working properly. By the time you do, it’s often too late. A specialist solution will typically include various monitoring options. These should be able to confirm that the integration is running and to report on integration errors. Additionally, an integration provider will have options for recovering and redeploying integrations quickly.

 

Specialists live and breathe their area of focus. As an integration specialist, we’re regularly encountering unusual integration requirements and using those to constantly evolve our Seamless feature set. Like typical SaaS apps, as the Seamless service improves, those improvements are offered to all users.

Within Recursyv, our primary focus is about getting systems syncing quickly, easily and flexibly. Once integrations move from our engineering and project teams to our operations team, we’re solely focused on ensuring that those integrations continue to run, day-in and day-out.

We proactively monitor integrations for technical and data issues and address them as soon as they’re raised. Often our alerting will flag up SaaS app availability problems before they’re noticed by users.

 

 

How to choose?

In short, it is all about what you need an integration to do for you. There is, of course, a place for using vendor provided integration tools. They will often meet basic requirements where there is limited need for flexibility and control. Specialist integration providers will introduce flexibility, additional data endpoints and control features that vendor solutions are unlikely to offer.

When considering your own approach, there are some questions you should consider

  • What are the different apps you need to sync with? Does the vendor you’re considering offer integration endpoints for those apps?
  • What level of flexibility / customisation do you need when matching data items across different systems? Do you have custom fields, tables, option set values, etc.? Are these likely to change over time?
  • How often do you need data to refresh?
  • Are you likely to want to introduce additional data endpoints over time?

 

Ending the enduring legacy of data silos

Silos

“I understand what data syncing is, but why is it useful?” is something we hear all too often. Here’s how we try and explain it.

It should be that every time your business interacts with a customer, a piece of information is generated. This could be a phone call note, an invoice or a request for help from your customer. All common scenarios and, in most cases, all handled by different systems.

As our user community has grown, we’ve found that one way to consider the usefulness of syncing data is to think about all the places where a business interacts with customers.

  • The sales team are often using a CRM system (Dynamics, Salesforce, HubSpot, etc.)
  • The marketing team may be using email marketing (MailChimp, etc.)
  • The customer support team are typically using a helpdesk system (Zendesk, ServiceNow, etc.)
  • The finance team are likely to be using an accounting package to send and monitor invoices, debtors, etc. (Xero, etc.)
  • There may be collaboration software in use across the business, either to manage projects or just to maintain open communication channels (Jira, Slack, etc.)

That list just describes generic functions that most businesses have. It is quite likely that there will be some specialised software to support the primary function of your business. This could be point-of-sale software, manufacturing/control software, project management software, e-commerce, etc.

That’s a lot of places where there may be references to customers. When your sales team ring up a customer, they’ll want to know whether the customer has any outstanding complaints. When your accounts team need to follow-up with a query, they’ll want to know who is best to ring and ask. When your support team are working on open helpdesk tickets, it’s useful to know that the sales team have a big proposal submitted awaiting a decision.

This is where data syncing comes to the rescue. When I’m explaining the business value of Seamless, I often ask people to tell me all the different engagements their business may have with their own customers. I then ask what underlying system that interaction is recorded on. Typically, as we talk through these engagements, it becomes very evident how standalone silos of information are being built across the business.

Seamless is about making disparate silos of data disappear into the background. Staff across an organisation should all see the same information about customers (or orders or helpdesk incidents or … ) no matter which IT system they’re currently using.

 

When is integration not integration?

Ice CreamIntegration, like ice cream, comes in many flavours. And like ice cream, some are better suited to your needs than others. On a hot summer’s day, a refreshing sorbet might be your preference. If you’re throwing the ice cream on top of a warm apple pie however, Madagascan vanilla (obviously!). Sometimes you’ll see a kiosk advertising ice cream, but really they’re flogging some vile frozen chemical cream-like product which is neither good for man nor beast.

So … (need to stop thinking about ice cream) when is integration not integration?

There are many ways you can expose data from one system when working in another. Different requirements will drive different approaches. In this post, we share some of the common ways to share data between systems.
Continue reading

Why is integration so often an afterthought?

HeartOn their own a heart, lungs and a liver would be pretty useless. But when you join them up using the vascular system (arteries, veins, capillaries), all of sudden the sum becomes greater than the whole. Each of our organs has a “specialist role” and when they perform to role, the work they do is invaluable. The heart pumps blood, the lungs oxygenate blood, the liver helps you recover from a big night out.

In this admittedly sketchy analogy, IT systems are no different. Each IT system in your business has a specialist role which it (hopefully) does very well. Most businesses will have a financial system (invoices, accounting, etc.). Some will have point-of-sales systems (POS), some have a sales or contact management system (CRM), most will have an email system. Many, possibly even most, will have a core line-of-business system which helps manage or control the primary business function of the company.

All too often there is no vascular system connecting all of these disparate applications together. Each system operates using its own data set “knowing” things which it is not “sharing” with other systems. Has Customer A paid us in full? Does Customer B have any open queries? These are pieces of information which users across the business could, and should, use to make more informed decisions.

Why are businesses not building their vascular systems?
Why do so many businesses not focus on linking up their IT applications?

 

Is there a fear that integration is too expensive?

Possibly. Traditionally the software that connects up applications – known as middleware – was expensive. It required specialist infrastructure and a lot of work just to get the application talking to the middleware.

Two market forces have arisen to challenge this. Firstly, in the era of cloud applications, the only infrastructure required to run modern infrastructure is often a data connection. Secondly, the advent of common APIs means that connecting to many applications has become much simpler.

Is there a fear that integration is too complicated?

Possibly. Connecting system A to system B requires some degree of planning. It cannot be denied that there will be some work to line up data fields in either system and to ensure that business processes on either side will still operate with data being added/updated.

Using templated connectors and a proven, rigorous, analysis means that the complications can be minimised. Most businesses are not in the business of running complicated IT projects. Using integration templates, supported by repeatable analysis for non-standard data, means the complications can be quickly resolved.

Is there a fear that integration will take too long?

Possibly. For a myriad of reasons, IT projects have a reputation for taking longer than expected. Most companies are not in the business of running long IT projects, they take energy and focus away from day-to-day operations.

Using a well designed toolset which has a demonstrable ability to rapidly deploy integrations removes a lot of the risk. Templates for many common applications, coupled with underlying foundation connectors for common integration scenarios, mean that any development is fast-tracked before it even begins.

Does it have to be expensive, complicated or time consuming?

Of course not. Although admittedly, if the answer were yes that would be a pretty surprising take for this post!

For all too long, sophisticated integration has been the preserve of large enterprises enterprises who could justify the cost, and associated complications, with managing traditional options. Cloud options and the uniquity of APIs has drastically reduced the cost profile of setting up and operating integrations. Complications can be controlled by using a reliable toolset. And the time to deliver can be accelerated by using re-usable components.

With the introduction of reliable, cloud based infrastructure-as-a-service options, coupled with well designed tooling, sophisticated integration is accessible to companies of all sizes.

The Seamless integration workbench – a primer

Over the past few months, we’ve dropped a few hints about the Seamless integration workbench. In this post we wanted to share some more details about the workbench.

Integration Workbench

What’s it good for?

The workbench is all about setting up an integration configuration. An integration configuration describes everything the core Seamless service needs to sync data between different systems. Within the workbench, a user can configure all the variables that make up an integration.

  • Specify the connector plugins that need to be used for this integration
  • Specify any properties the plugins require in order to process data (e.g. connection details for the systems being synced)
  • Specify where data in transit will be hosted (a service bus, a database, a local directory, etc.) and when it will be expired
  • Specify the data queries to retrieve data from the source system
  • Specify the data transforms required before posting data into the target system
  • Specify the timing of data sync from source to target and vice-versa

 

How does it work?

All aspects of an integration are ultimately saved into an XML file. These files can be loaded into the workbench (to continue working on a previous integration) and saved from the workbench (for subsequent deployment into Seamless).

The workbench can also run the integration locally (i.e. on the local machine). This is useful for troubleshooting and debugging integrations before they’re deployed into the cloud. Integrations are run by clicking play and all logging data is provided to the user for analysis.

What can it not do?

The workbench can not do the deployment of the config file to a live Seamless service. This was kept out of the workbench to enable separation of design and deployment.

The workbench can not run the integration on an ongoing basis. Options for running locally are limited to a handful of iteration cycles. In practice, you couldn’t use the workbench to run Seamless as a locally hosted integration service.

Who gets to use the workbench?

The workbench is provided to all Seamless delivery partners so that they can work on developing integrations. The workbench is also provided to clients who want to specify and build their own integrations – often as part of a broader software project.

Does every Seamless user need to use the workbench?

No. Only delivery partners and users who are specifying their own integrations need to use the workbench. Where you’re working with a delivery partner or directly with Recursyv, you won’t have any use for it.

How did it come to be called the workbench?

You’d think the term “workbench” was a pretty straightforward moniker for this piece of kit. In the early days, our engineering team creatively labelled this the “Seamless Integration Builder” – descriptive but not particularly exciting. After many, many minutes of discussing prospective names for it, David and Jon moved onto a more pressing conversation about why cycling is a better sport than running. In conversation some months later, one of our clients referred to the tooling as “the workbench” and the name has stuck. Also, the client prefers cycling.