For over a decade, the omnipresent SOA architecture and ESBs were considered state-of-the-art for integration architecture. Still, there are lots of organizations where ESBs are in use. If you still have an ESB as the main hub of your integration stack, it is probably time to consider newer options. The world moved on, and “agile” has also reached the integration architecture.
But before we look at what agile integration is, we need to take a broader look at the integration architecture. An example of that is the reference architecture model shown below (Based on the IBM Think 2018 presentation: http://ibm.biz/HybridIntRefArch).
Integration architecture patterns are often divided into three main categories: synchronous, asynchronous, and batch. Synchronous integrations are often implemented as HTTP/HTTPS or REST interfaces, asynchronous integrations are mostly different kinds of pub-sub or streaming integrations, and finally, batch integrations are often referred to as ETL (Extract, Transform, Load) or more recent ELT (Extract, Load, Transform) and are very commonly used in connection with data warehouses, various data platforms, and data lakes.
With the march of cloud technologies, the integration architecture has also more and more adopted cloud as the execution environment, and there seems to be two main streams on how the integrations are being implemented in the cloud: either as the native PaaS or the “best of suite” iPaaS /iSaaS type of platform.
The native PaaS uses the basic components of one or more major PaaS platforms (AWS, Azure, Google Cloud). Here we talk about components like f.eks. AWS API GW, AWS Kinesis, AWS SNS/SQS, AWS Step Functions, Azure API Manager, Azure ESB, Azure Logic App, and so on. The “best of suite” iPaaS/iSaaS is basically a complete integration suite implemented as a SaaS service, e.g., like Dell Boomi, Informatica, or MuleSoft, which often provide a set of adapters for different protocols.
The integration architecture has also evolved over the last decade from the infamous centralized SOA architecture and ESB to a more distributed architecture. This evolution has affected three axes: people, architecture, and technology.
In the architecture and technology axis, as development becomes increasingly autonomous, with cloud services, big data, and microservice-oriented architecture, as well as new ways of running software natively in the cloud or in containers, the integration architecture has also evolved into a more distributed variant. The centralized ESB-like platforms disappear, and integration becomes either point-to-point for synchronous integrations or pub-sub and high-performance streaming for asynchronous integrations. The integration software itself became more distributed and, in some cases, also runs either in containers or natively in the cloud.
Finally, as integration is more distributed and often developed by separate autonomous teams, it is natural that different integrations are implemented using different technologies and programming languages, or become what we call polyglot integrations.
Another consequence of this evolution in the integration architecture is changes affecting the people axis. With autonomous teams and distributed integrations, there is no longer a need for centralized integration teams, and the integration resources are now spread over different teams. This also means that the integration architecture becomes more of an abstract aspect that the organization has to manage, often without explicitly allocated resources and without clear ownership. This trend basically follows the same pattern as for the other dimensions of the enterprise architecture, including security and information architecture.
The integration architecture also follows another important trend, called domain-driven architecture (DDD). DDD is another force that pushes integration architecture away from centralized, layer-oriented architectures toward a more distributed architecture, with tighter integrations within each domain and more loosely coupled integrations with other domains and external services. This makes it possible to reduce the complexity of long technical value chains with unnecessary transformations, increase ownership of integration artifacts, and reduce the amount of overlapping data that pops up everywhere. Here is an example of Domain Centric Integration Architecture at DNB (presented at IBM Think Summit Oslo 2019)
Process orientation is another important aspect, particularly in the context of digitalization, as process improvements and optimization are among the most important areas for driving a business toward greater digitalization. Also, integrations need to become more process-driven rather than solely technology-driven. However, traditional, centralized integration platforms leave little room for adjustments and adaptations to better facilitate process changes, making it difficult to tailor integrations to the improvements in processes. Because platform choice is often purely technology-driven, once a platform is selected and implemented, it is usually hard to adapt to the actual process. If you’re lucky, you have a wide enough range of adapters and tools to meet your needs, but there is no guarantee.
A cloud-based “À la carte” integration platform, where one can pick the most suitable integration components and pay only for the components in use and for the time they are in use, is therefore better suited to a process-driven integration approach.
The critics would, however, point out that with the rise of the modern, distributed, autonomous, and polyglot integration platforms, we lost some of the important capabilities that, e.g., SOA and ESB provided. The integrations are becoming more point-to-point, adding more complexity and increasing the “spaghetti factor”. There is no longer one place, one system, that hides the complexity and lets you see how your portfolio is integrated and all its dependencies. In practice, this is not such a big issue and can be addressed through documentation, reverse engineering, or self-discovery mechanisms; several tools make this task easier. The point-to-point challenge can also be alleviated, e.g., by using data lakes and data streaming mechanisms that reduce the need for direct point-to-point integrations, just to mention Sesam (https://sesam.io/) or Kafka (https://www.confluent.io/)
On the other hand, one could point out that the new platforms no longer support several aspects of the traditional ESB VETRO pattern, which stands for Validate, Enrich, Transform, Route, and Operate (https://www.oreilly.com/library/view/enterprise-service-bus/0596006756/ch11.html)
This is somewhat correct; however, with distributed, containerized, and polyglot integrations, it is relatively easy to implement all necessary validations, enrichments, and transformations. When it comes to routing, there are several components that can provide similar functionality in Azure (APIM) or AWS (API GW), and the Operate aspect is more of a task of the autonomous DevOps team that operates the service and its integrations.
In summary, the integration architecture has undergone significant changes across several dimensions and has evolved from a centralized SOA/ESB platform to a more distributed, autonomous, and polyglot architecture. This development has been catalyzed by underlying trends in IT development and architecture, in particular, DevOps and autonomous teams; digitalization and process orientation; and cloud, microservices, and containerization. The result is an integration architecture that is more flexible and adaptable to both business needs and the needs of the development organization itself, and, finally, the rise of what we call Agile Integration architecture.
Views expressed are my own.
This work, excluding photos, is licensed under a Creative Commons Attribution 4.0 International License.