There is undoubtedly great value to be gained by linking the company's systems together and integrating them. The benefits are endless:
- Less manual work to keep all systems up to date
- Process automation (e.g. automatic invoicing)
- Better reporting and analysis (see data across systems)
One of the things to be aware of when planning integrations is the level of robustness required. By robustness, I mean the risk of failure, whether it goes down and how vulnerable this is. There's a difference between a monthly report, where everything must fit within two decimal points, and a live dashboard, where a small outage doesn't matter, as it is updated 2 minutes later.
Two types of integrations
There are generally two types of integrations, both of which have their pros and cons:
- Point-to-point integration (typically with REST services)
- Message-based (typically with a service bus)
The difference between the two is their robustness. Of course, there are also some financial decisions involved in this.
Point-to-point integration
With point-to-point integrations (the most common integration for e.g. webshops to accounting systems) you have an online system where you have access to its data (e.g. a webhook or an API). This system is one of the points in the "point-to-point" - e.g. a webshop. So, the other point is the system you want to integrate with; this could be an accounting system.
You can connect these two points together. Typically, you develop a small piece of code in the middle to translate from one system to another, and you can synchronize this at regular intervals (such as every hour). In basic terms, the software supplier arrives with a glue gun and glues the systems together. It’s that simple. A piece of software is required to extract data from point 1, transform it, and insert it in point 2 again.
The benefits of this type of integration are countless, and for many companies, this is clearly the only type of integration required.
Point-to-point integration, the grey circles and lines indicate several systems to be integrated. Each line is software integration.
Pros
This is a simple type of integration, which has its own benefits:
- Quick and easy to develop and therefore also relatively cheap (I will return to that shortly).
- Relatively robust, all three points (point 1, point 2 and the glue gun in the middle) are often available at the same time.
- Little or no business logic to consider.
- High performance.
Cons
Of course, there are some disadvantages as well. These typically only arise when you have other requirements such as uptime if several more systems are introduced, or the data starts to grow.
- All systems must be online at the same time (all three points).
- All systems must know the address of each other.
- It quickly becomes unmanageable when integrating multiple systems.
- The complexity increases when all the systems have to talk to each other (in a company with 10 systems, each system must know and be able to communicate with 9 others).
Point-to-point integrations are a very fast and simple integration between two systems, and we do a lot of them at Median Aps. It is quick to develop and cheap to operate. This type of integration fits really well into smaller companies, e.g. webshops, where you need one or two integrations that connect the systems together.
But it doesn’t take much before you have to consider another solution.
Message-based integration
This type of integration has some different pros and cons compared with point-to-point integration. Generally, we can say that this type of integration is the "big brother" of point-to-point integration.
We recommend that if you start integrating your systems, a point-to-point solution may be a sensible first step in order to get a better overview of your requirements and a better idea of the benefits you’ll get from integrating. In short - you have to see if the business case holds. If this turns out to be a benefit for the company, you can easily move from a point-to-point integration to a message-based integration.
The latter works by two or more systems sending messages to each other, rather than directly integrating. The messages run through an advanced queuing system (don't worry, this is something the software developer takes care of) and at the other end of the queue (or multiple queues), there is a recipient who handles the message.
Picture it as a "straw", where there is a small piece of software at one end which keeps track of e.g. orders in the webshop, and every time a new order arrives, it picks up the order and adds a message to the queue. It is simple and has only one job to do. It increases the robustness of this part of the system. This means that the individual parts in contact with e.g. the webshop are extremely simple and therefore also more robust and easier to maintain.
At the other end of the queue, another small piece of software removes a message away from the queue and inserts it into the other system. That is basically all that happens.
This "queue" in the middle is where all the benefits come from. It can contain a lot of messages, so it also acts as a kind of buffer so that the piece of software that inserts orders into the accounting system is not overloaded. This solution is somewhat more advanced, but it can solve virtually all the company's integration needs. The benefit is that you can have thousands of these queues, making it easier to have multiple integrations without increasing the complexity. This integration method is very flexible and easy to correct, adapt and introduce new integrations into.
Pros
Many of the pros of this type of integration have something to do with scaling your integrations, both in relation to the number of systems you connect and the amount of e.g. orders you have in the system.
- The source system and the destination do not have to be online at the same time (the queue handles this)
- The systems do not have to know each other (everyone only needs to know the central queue - that is, in a company with 10 systems, each system only needs to know 1 central queue) - this doesn’t increase too drastically as new systems come along
- It is easy to scale (you can start several pieces of software that read from the queue and insert them into the system - they work together to empty the queue of messages)
- Great flexibility, it is easy to connect new systems (e.g Business Intelligence systems)
Cons
Many of these cons need to be seen in relation to the simple integration we talked about in point-to-point integration.
- The queue is the bottleneck (most queues are in the cloud and as such are very robust) - this is typically never a problem
- More development and maintenance required (the value is also greater) - it is not "off the shelf"
- Slightly less performance (especially in situations with a lot of business logic)
- Eventually consistent – accept that all data is not immediately updated, but they will be over time (eventually consistency).
Adapting to message-based integration
Not all systems know about messaging, some systems may use other types of integration methods, and others may not be designed for integration. In the (albeit simplified) diagram above, the ERP System or whatever other systems we base our day-to-day business, have knowledge about the message queue. This is not correct. It's designed to simplify it a bit. We as a company can't possibly know which systems are best for our company (now? In the future?), they may change or evolve over time, and it is not certain that the next version/system fits the integration we already made. In these cases, you insert a small "adapter" between the source system and the message queue. This means that this small adapter is the one that can connect to source systems and has knowledge about the bus.
This has several benefits, including that (as a company) we start to build an enterprise architecture, which in principle can be independent of source systems (to prevent vendor-lock-in). Long-term, the goal might be to be completely independent and choose based on what is best for the company (that in itself is a long discussion about how to select systems and to a certain extend being agnostic in your architecture).
The job of an "Adapter" is to create the actual integration. When you can't guarantee that the source systems have some sensible "handles" to use to get data from (e.g. API, or services), you sometimes have no other options. e.g. to download from the system database or "screen-scraping". The latter options are far from the most optimal, but this may be the only solution if, for example, you have legacy systems that need integrating, or you retrieve data from the web that is not available in other ways.
The benefits of having this "adapter" are that you get a looser connection between the source systems and the other systems in your company, as they don't have to know each other. It also means that we start to build a strong enterprise architecture in the company to integrate all our systems. This enterprise architecture can give us a competitive advantage, as our competitors cannot copy this. It is unique to our company (all competitors can easily buy the same source systems, but the integration and knowledge in the integration - which is now increasingly reminiscent of a Service Oriented Architecture (SOA) - cannot be copied). In this case, it will also be easier to grow and replace a system in the long run. You' simply' replace the adapter and then all the systems run again.
More details on the message-oriented integration. Now with adapters.
Typical Integration Challenge in EAM industry
Recommendations
If you are in the process of integrating systems or are investigating whether there is any value in it, start by making simple point-to-point integrations. Many companies never need to do anything other than this, as it covers all needs and generates the necessary value in the company. As you grow, you can then consider whether to go in a different direction and look at more advanced types of integration.
One of the biggest challenges we see companies experiencing is making ad-hoc integrations without really having any control over them. This means that no one knows what integrations have been made (the one who made them stopped working for the company 3 years ago). And no one knows if there are several similar integrations, there is also no one who dares touch the systems - least of all replace the old accounting system (e.g. C5/AX/NAV). Our advice - be proactive:
- Buy/choose systems that can actually integrate with others – favor strong APIs
- Decide on integrations before others do it for you
- Think out of the box - did you know e.g. that you can actually use RPA to make integrations?
- Think more out of the box - now that you’re extracting data anyway, why not put it in a database so you can generate reports and analyses (Business Intelligence) e.g. with Microsoft Power BI?
This article is written by Brian Hvarregaard, owner of Median - software development and consulting company based in Aalborg. You can find the original article here.
