Taking a moment to celebrate the engineering

Engineering

Since the beginning of the year we’ve been working on a particularly thorny deployment. An integration with lots of technical challenges which brought together many features of Seamless for the first time. Jon has been tucked away in “deep dev” mode and, as he prepared to put the integration into test, he wrote a wrap-up note for the rest of our team. The implementation validates a lot of the underlying engineering for Seamless. On reflection, I realised that this integration creates a lot of milestones.
Continue reading

Tomorrow’s IT operations are as important as today’s software delivery

When we set out to build Seamless, one of the things we focused on, quite obsessively, was that we should reduce risk in a software delivery project. Admittedly, this is a reflection of the backgrounds of our founders: delivery of software. It also vastly undermines the value of delivering Seamless as a managed service. Perhaps we’ve done ourselves a disservice?

Erm – let’s not get ahead of ourselves, have we actually addressed delivery risk?

In the first instance, I’d like trumpet our engineering a little bit. If we need to connect to a relatively modern API (think RESTful or SOAP), we typically have data syncing in under a week. Once we’ve “proved the pipe” (demonstrated that we can move data between source and target), we can shape that as necessary to meet the business requirement. As this is readily demonstrable, the delivery risk is greatly reduced. There is no longer a long period of writing code and hoping for the required outcome.

Getting there is only half the battle

As old hands at “change the business”, we may have fallen into the trap of forgetting about “run the business”. If you’ll forgive the cliche, delivery of a new software solution is the beginning of a journey rather than the end.

A data integration is a living piece of infrastructure that needs to be operated. Even if there are no changes to the business requirement, integrations need to be looked after and cared for.

  • Is the integration able to connect to both source and target?
  • Is the volume of data being moved roughly in line with expectations?
  • Does the integration have enough oomph to ensure that data is sync’ed within the required timeframes?

The Seamless monitoring features, including meaningful alert content, means that keeping tabs on these questions becomes part of the regular operations monitoring regime. Additionally, alerting on exception means that you can assume all is well until told otherwise.

Integration issues are often a symptom of broader problems

One thing we’ve observed is that often issues with the integration are just symptoms of broader problems. Our monitoring is proactive (we go looking for issues) and our reporting is sufficiently detailed that we’re able to help Seamless users spot and trouble-shoot underlying issues, often before they’ve noticed them themselves.

  • Are both the source system and target system accessible? If Seamless is unable to reach systems, is this because of an underlying systems issue that needs to be addressed?
  • Are there data mismatch issues between source and target? Often this is a case of changes to reference data in one system or another.
  • Are data volumes completely outside of the expected data volumes? If so, is this driven by a something in the business (e.g. a sales promotion) or is one of the systems experiencing an underlying issue?

In 2017 we had only one service availability issue. Sure – we’d like it to have been none but, if there’s going to be some downtime, let’s at least handle it elegantly.

In July 2017 there was a short-lived Azure UK connectivity issue. Our alerting meant that we knew it was happening before it was confirmed by Microsoft and we were able to manage expectations across our client base. In turn, this allowed clients to manage expectations within their user community. We also provided the option of redeploying to another Azure geography. Thankfully, the outage was contained to a few non-business hours (for our UK customers) and there was no need to exercise any of these options.

The underlying Seamless service remained live and, as connectivity returned, any data had been “caught out” between source and target was quickly sync’ed by the service.

The only rule of business requirements is that they change

One of our first prospects for Seamless was a company who, as a result of an acquisition, found themselves with two software platforms for managing helpdesk tickets across their considerable IT estate. As part of the merger process, a vendor had written a point-to-point integration which sync’ed relevant tickets back and forth.

Six months down the line, there was a need to change the rules for which tickets would sync between the apps. After months of avoiding the question and 3 days of consulting work (on fees), the vendor quoted tens of thousands of pounds to change the integration. These were well known, well documented apps with mature APIs, updating the integration should’ve been a piece of cake. It became easier to change business rules inside the company than to change the integration and that’s just not how IT should work.

The use of config and straight-forward mapping features within Seamless means that meeting the needs of changing business requirements doesn’t require re-engineering entire integrations every time a field, a data map or an option set changes.

Not every IT organisation is the same

This blog post was, in part, inspired by a conversation with a prospective client. In our initial conversation, the primary concern was about delivery timescales of the integration requirements. As the conversation progressed, it was also discussed that the broader procurement strategy was to use contractors to deliver software with a view to streamlining the size of the in-house IT team. That’s not uncommon, by their very nature, project teams are short-term creatures designed to achieve an outcome (rollout of new software).

In discussing why Seamless may be a better option than using contract developers to build a point-to-point solution, I stressed not only the delivery risk and timing considerations, but that the managed service remains in place after the project team is stood down. This enables the in-house IT team to focus their effort where they can deliver more “bang for buck” whilst the Seamless monitoring ensures that when issues occur, the IT team are quickly informed with meaningful information to trouble-shoot any underlying systems issues.

Why don’t we just get the dev team to build it?

When we’re discussing integrations with prospective clients, one question that seems to come up every so often is “We’ve got a dev team as part of the project, why don’t they just build it?” It’s not an unreasonable question, if there’s no compelling argument for using an integration service then why bother with one?

 

Getting to grips with integration development

Having some insight into how an integration works will provide some useful context. In simple terms, a straightforward integration follows roughly these steps:

  1. Get specific data from the data source (i.e. construct the relevant queries)
  2. Assess whether that data has changed since the last time we got it
  3. Decide whether the data needs to be posted to the target system
  4. Encrypt the data
  5. Compress the data
  6. Move the data from source to target
  7. Decompress the data
  8. Decrypt the data
  9. Transform the data
  10. Write the data to target

I should add that those steps are of differing levels of complexity in different scenarios. At a simple level, some systems can provide data which has changed since we last saw it, some systems cannot and the integration needs to perform the compare. At a more complex level, some systems have sophisticated APIs which allow us to construct queries “on the fly” using configurable options, some systems require us to hard code the queries before we can execute them.

In the cold light of day, most of these steps are generic:

  • Step 1 is specific to the source system but not necessarily to your business need, that is to say – “Dear system, please tell me about new records created in the past 30 seconds” is a generic request of whatever system you’re using. It is not specific to your business need. If the source is bespoke, then naturally it is a wholly bespoke step.
  • Step 2 is typically generic, but will be driven by business need.
  • Step 3 is a business decision largely informed by the outcome of Step 2.
  • Steps 4 through 8 are generic for any (well designed) integration.
  • Step 9 is specific to your setup in the target system. The data transformation will need to reflect the data structures in your target system, including both the overall data structure (an account in the source may be called a company in the target) and potential dropdown list values (Mister maps to Mr, status set to “Active” maps to status set to “Live”, etc.).
  • Step 10 is specific to the target system in the same way that step 1 is specific to the source system.

Steps 1 and 10 are specific to the “technical scenario” (i.e. the systems in play) and steps 3 and 7 are where your business requirements are going to drive how the integration should work. The rest of it is pretty much the same, whether you’re connecting your CRM to your helpdesk or your accounting package to a front-end e-commerce system.

 

Paying to reinvent the wheel

With that in mind, let’s return to the question: “We’ve got a dev team as part of the project, why don’t they just build it?” The short answer is “Because the vast bulk of their development effort would be spent on generic integration features which are already accessible to you at a much lesser cost than building them yourself.”

Naturally, there is more complexity to the answer. In my experience, point-to-point integrations are usually developed to a “minimum functioning feature set”, i.e. they can get the right data from A to B on day one. That’s not necessarily a terrible place to be, but it is not as good as it could be.

A service built first and foremost as an integration service will offer sophisticated features which your project budget is unlikely to underwrite for a single point-to-point integration. Encryption, compression, elegant management of downtime at the source or target, ability to be easily extended, etc. All of these features need to be designed and built.

Additionally, developers are not “just developers”. That’s like saying builders are just builders. The technical skills required to marshal data, overcome infrastructure and connectivity complexity, manage reporting, etc. are quite different to the skills required to configure / build your new application. Sure, there’s some overlap. Most bricklayers can do some paintwork, but for painting decorative ceiling cornices, I’d rather get a painter.

 

The big picture

In a 2015 paper on digital transformation projects, Gartner noted that “many organizations already favor a new kind of “build” that does not include out-of-the-box solutions, but instead is a combination of application components that are differentiated, innovative and not standard software or software with professional services (for customization and integration requirements), or solutions that are increasingly sourced from startups, disrupters or specialized local providers.”

We’re happy to be one of those startups, even if it means using American spellings on our blog post.

 

No-one likes downtime. But if it has to happen, at least make it happen elegantly

It’s hard to write a post which talks about any virtues associated with downtime but, in the wake of last night’s Azure UK outage, we had some small sense of satisfaction that Seamless behaved both as we designed and would’ve hoped. Connectivity to the UK South Azure data centre was lost between about 22:30 and 01:40.

 

 

Accepting that we weren’t going to fix Azure’s connectivity issues, where do we see the positives for Seamless?

 

1. Our alerting let us know it was happening (before Microsoft confirmed any issues)

The Seamless monitoring alerted us to the first connectivity alert at approx. 19:43. This seemed to be an isolated alert, however we received a second connectivity alert at 22:30 and they began to be raised regularly at that time. We then began our own investigation and the Azure service page confirmed a connectivity issue at approx. 23:30. Shortly after that, we issued our alert to the Seamless user community.

 

2. When Azure connectivity returned, Seamless shrugged its shoulders and carried on working

Bearing in mind that connectivity was the primary issue, the service itself actually continued running. As soon as connectivity was restored, data continued sync’ing with no intervention from us (or, of course, our customers). Any data that had not been collected at source would’ve then been collected and posted to target systems. Any data “in transit” when the connectivity issues started was processed and sync’ed to relevant target(s).

 

3. We had mitigation actions ready in case we needed to support customers

Our mitigation options are typically to redeploy the service and the config to other Azure locations (subject to relevant instructions from our customers bearing in mind potential data considerations). In the event that the UK South region was not restored by the start of the business day, we were in a position to relocate the service and client configurations to Western Europe (Dublin). This would’ve taken us less than an hour and no data would’ve been lost in this process.

As Microsoft restored the Azure service around 01:40, this was not necessary.

 

 

No-one likes downtime and we recognise that we’re lucky that these 3 hours were outside of core business hours for our current customer base. In practice, we continue to believe that a Microsoft data centre is more secure and robust than were we to build our own.  In this light, we architected the Seamless service to “fail elegantly” and although disappointed this had to be tested in anger, we’re delighted that it worked as we would’ve expected it to.

The counter-intuitive benefit of shorter projects

As we’re working to build the partner network around Seamless, I am often talking through (what we believe to be) the benefits of using Seamless for delivery of integrations. We have a background in delivering IT change projects and one of our favourite Seamless benefits is that developing integrations becomes much much quicker. If we have a connector plugin already built, it is simply a matter of specifying the right fields and associated rules. If we don’t, we can often build one in under a week (assuming being reasonable API access).

For companies who make a lot of their revenue selling time (i.e. time taken by software engineers to build applications), it seems counter-intuitive that we’re touting shorter projects as a benefit. Er, what’s up with that?

 

Our response is simple: I’d rather you sold a 60 day project than lost a 120 day project to a competitor.

Our partners will be competing with any number of other organisations who will have different ways to deliver software. Today’s average user has experience of sophisticated web services and expects that software delivery companies can quickly connect up different services. Agile approaches add an additional dimension of expectation because clients expect to see phased delivery of packages of requirements.

Sure, reducing delivery time reduces upfront professional services revenue, but it makes our partners’ projects more commercially attractive. If this can be done in a way that also reduces risk (using technology that is proven to work) and increases flexibility (extendable plugins, a myriad of configuration options, etc.), then the offer becomes more attractive on day one (commercially) and over time (ability to adapt to inevitable business change).

Finally, let’s not lose sight of that fact that, over time, “lost” revenue may be recovered (plus more) through sharing of the subscription fee.

 

 

[icon name=”exchange” class=”” unprefixed_class=””]  Learn more about partnering with Seamless. We could talk about it all day.

Rolling out the big guns – SQL Server, MySQL and Oracle

The initial development of Seamless focused on ensuring we could support clients and opportunities that we were, at the time, working on. That quickly resulted in connector plugins for Microsoft Dynamics, Freshdesk, Autotask, Hoopla and SharePoint lists. As these have been stable and in use for a while now, this has given us the chance to focus on building connector plugins for more popular enterprise-grade tools.

That left us with an interesting choice … what next? We decided we’d focus on a handful of widely used applications which are deployed in thousands of different ways at organisations across the globe: Microsoft SQL Server, MySQL and Oracle. We’re delighted to announce these are now built, tested and hungry for data.

 

Seamless wall of fame May 2017

 

These plugins are all configurable, offering the best mix of rapid deployment and ability to control and extend the plugin’s work to meet the requirement at hand.

  • Configure mappings (i.e. fields to integrate) using XML to identify relevant tables and fields
  • Extend the feature set of these plugins using .Net to code extensions which are deployed on top of the existing plugins (i.e. you are not constrained by the limitations of any given plugin)

 

[icon name=”phone” class=”” unprefixed_class=””]  If you have an integration need for any of these technologies, please reach out and discuss with us how we can address this, quickly and affordably.

[icon name=”exchange” class=”” unprefixed_class=””]  More information on Seamless can be found here.

Business flexibility, celeb news, Lego modelling and IT architectures – you heard it here first

As much as it pains me to admit it, I fear that Gwyneth Paltrow and Chris Martin were onto something when they coined the term “conscious decoupling”. In the data integration space, the ability to break down a service into components is a significant enabler of flexibility. That’s good because technical flexibility translates to business flexibility. If the only constant is change, as we’re so often told, then flexibility in technical architectures is the way to prepare for change.Continue reading