• SOA Source Book
• Service-Oriented Architecture
• The Open Group Service Integration Maturity Model (OSIMM) Version 2
• SOA Reference Architecture
• Introduction
• Motivation
• Key Principles
• Basic Concepts
• Overview of the SOA RA Layers
• Capabilities and the SOA RA
• Description of Layers
• Operational Systems Layer
• Service Component Layer
• Services Layer
• Business Process Layer
• Consumer Layer
• Integration Layer
• Quality of Service Layer
• Information Layer
• Governance Layer
• Related Work and Usages of the SOA RA
• Relationship to Other SOA Standards
• Relationship to Open Group Guide: Using TOGAF to Define and Govern SOA
• Referenced Documents
• Acknowledgements
• Legal
• Service-Oriented Cloud Computing Infrastructure (SOCCI) Framework
• Using TOGAF to Define and Govern Service-Oriented Architectures
• SOA Governance Framework
• Service-Oriented Architecture Ontology Version 2.0
• Legacy Evolution to SOA
• Microservices Architecture
• SOA for Business Technology
• Navigating the SOA Open Standards Landscape Around Architecture
• SOA in the Digital Age

SOA Work Group

SOA Standards Discussion Forum

SOA Work Group Members Page

SOA Reference Architecture – Integration Layer

Overview

Context and Typical Flow

The Integration Layer is a key enabler for an SOA as it provides the capability to mediate which includes transformation, routing, and protocol conversion to transport service requests from the service requester to the correct service provider. Thus, it supports the capabilities required for enabling SOA such as routing, protocol support and conversion, messaging/interaction style, support for heterogeneous environment, adapters, service interaction, service enablement, service virtualization, service messaging, message processing, and transformation.

It is the layer in the SOA RA that supports the integration with solution platforms by the other layers in the SOA RA using “adapters”, the access of services by other layers, and the capabilities associated with service transport. It can be thought of as the plumbing which interconnects the SOA.

This layer enables the service consumer/requestor to connect to the correct service provider through the introduction of a reliable set of capabilities. The integration can start with modest point-to-point capabilities for tightly-coupled end-points and cover the spectrum to a set of much more intelligent routing, protocol conversion, and other transformation mechanisms often described as, but not limited to, an Enterprise Service Bus (ESB). WSDL specifies a binding, which implies location where a service is provided, and is one of the mechanisms to define a service contract. An ESB, on the other hand, provides a location-independent mechanism for integration, and service substitution or virtualization.

The integration that occurs here is primarily the integration of Layers 2 through 4 (the “functional” aka horizontal layers of the SOA RA). For example, this is where binding (late or otherwise) of services occurs for process execution. This allows a service to be exposed consistently across multiple customer-facing channels such as web, IVR, XML client, etc. The transformation of response to HTML (for web), VoiceXML (for IVR), XML String, can be done via XSLT functionality supported through a message transformation capability in the Integration Layer.

As shown in Usage of the Integration Layer, the Integration Layer:

• Provides a level of indirection between the consumer of functionality and its provider. A service consumer interacts with the service provider via the Integration Layer. Hence, each service interface is only exposed via the Integration Layer (e.g., ESB), never directly and point-to-point integration is done at the Integration Layer instead of consumers/requestors doing it themselves.
• Consumers and providers are decoupled; this decoupling allows integration of disparate systems into new solutions.

Usage of the Integration Layer

Capabilities

There are multiple set of categories of capabilities that the Integration Layer needs to support in the SOA RA. These categories are:

• Communication, Service Interaction, and Integration: This category of capabilities provides the ability to route requests to correct the provider after necessary message transformation and protocol conversion and to connect the service requestor to the service provider and its underlying solutions platforms realizing the requested service. It also provides the ability to discover services and, at runtime, to support the virtualization of services so that changes to the end-points (or locations from where the services are called and where the services are provided) can occur without impact to service consumers and service providers.
• Message Processing: This category of capabilities provides the ability to perform the necessary message transformation to connect the service requestor to the service provider and to publish and subscribe messages and events asynchronously.
• Quality of Service: This category of capabilities supports handling of transactions and exceptions and other NFRs.
• Security: This category of capabilities helps in enforcement of access privileges and other security policies.
• Management: This category of capabilities provides the ability to maintain service invocation history and monitor and track the service invocations.

This layer features the following capabilities:

Communication, Service Interaction, and Integration

1. Ability to take a service call and messages to the end-point; i.e., to enable a service consumer to connect/interact with service providers
2. Ability to handle service request and service response
3. Ability to support communication through a variety of protocols
4. Ability to support variety of messaging styles such as one-way, pub-sub, request-response
5. Ability to route messages to the correct service provider
6. Ability to transform protocol formats; e.g., from SOAP/HTTP to SOAP/Message Queue or SOAP/JMS
7. Ability to link a variety of systems that do not directly support service-style interactions so that a variety of services can be offered in a heterogeneous environment
8. Ability to store and forward messages using message queuing

Message Processing

1. Ability to transform data formats; e.g., from proprietary to standard format or industry standards and vice versa
2. Ability to transform semantic mapping (data positional mapping)
3. Ability to aggregate (including messages and data) from different services and service providers
4. Ability to propagate the events from producers to consumers

Quality of Service

1. Ability to handle transactions from the other layers, especially when a statically composed service invokes a service chain
2. Ability to handle exceptions raised in the process of service invocation and message passing

Security

1. Ability to authenticate/authorize for service invocation and message routing

Management

1. Ability to capture and record message routing and service invocation history
2. Ability to track and monitor the message routing and service invocation activities
3. Ability to configure the Integration Layer

Architecture Building Blocks (ABBs)

The ABBs responsible for providing these sets of capabilities in the Integration Layer are:

ABB to Capability Mapping for the Integration Layer

Details of ABBs and Supported Capabilities

Details of ABBs

This section describes each of the ABBs in the Integration Layer in terms of their responsibilities.

Integration Controller/Integration Gateway

This ABB serves as an entry point to this layer. Other layers interact with this ABB to leverage other ABBs in this layer to fulfill their respective responsibilities. In turn, this ABB is thus responsible for interfacing with other ABBs in this layer and managing the interaction flow among the ABBs in this layer. For example, it delegates to the Router ABB to service requests and it delegates to the Message Transformer ABB for the data, format, and message transformation needs of the other layers.

Adapter

This ABB is responsible for the interfacing/connectivity of SOA RA layers of a solution to external systems and components and taking a call (message) to the end-point. In particular, it addresses any necessary mediation (router, message transformer, protocol converter) of the elements outside the Integration Layer and the interaction with external systems and components. This ABB should typically be used by ABBs in all SOA RA layers to access external components of solution platforms, providing a consistent integration capability.

Mediator

This ABB is responsible for handling the service request/response interaction. It also supports the transformation between message formats, conversion of protocols, and routing of service call/messages to the service provider. It uses the Data Transformer ABB and optionally the Semantic Transformer ABB for the transformations. Finally, it supports static service composition to orchestrate and chain services or system calls or messages together in order to compose services statically. It uses ABBs such as Router, Message Transformer, and the Data Aggregator to do this.

Router

This ABB is responsible to route messages between service consumer/requestor and service provider including those based on both content-based routing, straight through message passing. It may change the route of a message, selecting among service providers that support the intent of the requester. Selection criteria for the provider can include content and context of the message as well as knowledge about capabilities of the target candidates and even versioning of service implementation. If a message is routed to one provider, which responds with a failure, the message can be re-routed to another provider. In certain circumstances it may be used to route messages without the involvement of the Mediator ABB to realize straight message pass-through. This ABB may use other ABBs from the Integration Layer such as the Message Transformer ABB, Auditor ABB, Logger ABB, and Exception Handler ABB. It may leverage the Access Controller ABB from the Quality of Service Layer.

Message Transformer

This ABB is responsible for transforming messages from one format to the other, including data format transforms into a single “canonical” form or its subset, and transformation across protocols to whatever protocols that are routed. It also supports data enrichment.

Semantic Transformer

This ABB is responsible for semantic/positional mapping of data, so that it conforms to standards. What it does is dependent on the associated standard; e.g., UDEF in general scenarios or in the case of specific lines of business, ICD 10 or LOINC. For example, this ABB transforms and maps a first name and last name in the in-bound data provided by the Mediator ABB to an Integration Layer standard format where the order is reversed. As the semantic interoperability of services becomes more prevalent with the adoption of cloud computing and SaaS, this becomes more important.

Data Transformer

This ABB is responsible for transforming data formats from source to target format; e.g., from proprietary to industry standards and vice versa. It integrates with the Information Layer to obtain metadata such as the canonical form, etc.

Protocol Converter

This ABB is responsible for transforming data across industry standard protocols. For example, a JSON to SOAP conversion or a SOAP/HTTP to a SOAP/JMS or SOAP/Message Queue conversion would be responsibilities of this ABB.

Asynchronous Messaging Manager

This ABB is responsible for storing and forwarding messages potentially using a message queue and it provides the underlying plumbing to transport messages between service invokers and providers as well as the ability to store and forward the messages. It supports both reliable transport and guaranteed delivery. Message queuing is a common example of a mechanism to support asynchronous messaging. A common feature of this ABB is to support message delivery guarantees and reliability.

Transaction Manager

This ABB manages transactions and encapsulates transaction handling from the remaining tiers, especially when a composite service invokes a service chain. Types of transaction handling include atomic ACID transactions, two-phase commit, long-running, journaled, and compensating transactions. It leverages standards such as WS* (WS-Coordination, with WS-Atomic Transaction for atomic, ACID transactions and WS-BusinessActivity for long-running transactions) standards to achieve this. Most of the transaction standards specify a set of transaction interaction patterns to address different kinds of transactions.

Data Aggregator

This ABB supports the ability to compose (aggregate) data from different services/service providers which are interacting with the Integration Layer using the Mediator ABB into one format, after they have been transformed into a consistent “enterprise” format (canonical form) by the Message Transformer ABB. It is used by the Mediator ABB.

Quality of Service Layer: Access Controller

See Access Controller ABB in the Quality of Service Layer.

Logger

This ABB is responsible for capturing and recording message routing and service invocation activities. It logs data to monitor system exceptions and system stability (data such as resource availability, etc.). This should be interoperable and integrate with the Quality of Service Layer’s monitoring features. It is important from a monitoring and support for compliance perspective.

Auditor

This ABB is responsible for tracking and monitoring the message routing and service invocation activities. It supports the capture of audit data, the conversion to standard formats such as XDAS and CBE, the encryption of that data during transport, and the obfuscation of sensitive data.

Exception Handler

This ABB is responsible for handling system exceptions raised during service invocation and message passing. System exceptions are caused due to software or hardware errors and not due to application logic errors. Application exceptions are treated as business events and handled through the Event Manager.

Event Broker

This ABB is responsible for facilitating event consumers (subscribers, sensors) to subscribe to events and event producers (publishers, emitters) to publish the event and to propagate the event from producers to consumers. It leverages the Asynchronous Messaging Manager ABB, Messaging Transformer ABB, and Router ABB.

Event Producer

This ABB is responsible for generating, publishing, emitting, or producing events.

Event Listener

This ABB is responsible for registering an interest in a particular type of event; i.e., for subscribing to a particular type of event.

Quality of Service Layer: Configuration Manager

See Configuration Manager ABB in the Quality of Service Layer.

Structural Overview of the Layer

The Integration Layer is a critical component of the SOA RA. Its ABBs can be thought of as being logically partitioned into categories which support:

• Ability to provide the integration of services with underlying solutions platforms, to discover services, and, at runtime, to support the virtualization of services so that changes to the end-points (or locations from where the services are called and where the services are provided) can occur without impact to service consumers and service providers
• Ability to support mediation; mediation can be thought of as the routing of the request to correct providers after necessary message transformation and protocol conversion and aggregation of the content from different service providers, from different formats and protocols into a common canonical form (data format) – while not normative, it is customary to support service messages in an XML format after mediation
• Ability to support different service standards such as WS-Security, etc.
• Ability to mediate reliability through the imposition of WS* and other standards-based protocols
• Ability to support routing and orchestration, including routing based on content (content-based routing), static service composition, and orchestration (calling services in a defined sequence) to deliver data
• Ability to perform message transformation
• Ability to support Quality of Service (QoS) requirements such as transaction management, performance criteria, exception handling, etc.
• Ability to support security requirements
• Ability to track, monitor, and manage service invocations

ABBs in the Integration Layer illustrates the ABBs partitioned into key categories:

ABBs in the Integration Layer

Inter-Relationships between the ABBs

The figures below show the inter-relationships and a non-normative sequence of interaction. The final, prescriptive sequence of interaction will be defined by the underlying solution architecture and the standards that the invoking other SOA RA layers support.

Simple Interactions between Consumer and Provider through the Integration Layer

Relationships among ABBs in the Integration Layer

Significant Intersection Points with other Layers

Interaction with Cross-Cutting Layers

The Integration Layer relies on other cross-cutting layers of the architecture to fulfill its responsibilities.

It relies on the Governance Layer for the following capabilities:

• Ability to store metadata for policies
• Ability to support management (storage, retrieval, etc.) of rules to support rules associated with decision points in service mediation, orchestration, and composition; the Integration Layer will leverage a common business rules capability that can be used by the ESB (the component in the Integration Layer which mediates – routes and transforms data)
• Ability to determine service end-points for service virtualization

It relies on the Quality of Service Layer for the following capabilities:

• Ability to authenticate/authorize for service invocation and messages

Key Interactions of the Integration Layer with Cross-Cutting Layers

Therefore, the Integration Layer interfaces with the following ABBs of cross-cutting layers of the architecture to provide its capabilities:

• It leverages the Service Registry ABB and Service Repository ABB from the Governance Layer for storing metadata such as policy, schema, etc. and for providing access to the metadata. The Service Registry ABB contains service definitions at runtime and supports service virtualization and service discovery.
• It leverages the Business Rule Manager ABB in the Governance Layer to support rule implementation for the Integration Layer.
• It leverages the Access Controller ABB in the Quality of Service Layer for authenticating/authorizing facility for service invocation and message routing. It also leverages the Policy Enforcer ABB in the Quality of Service Layer to enforce policies local to the Integration Layer.
• The Data Transformer ABB in the Integration Layer uses metadata from the Information Layer and leverages the Information Metadata Manager ABB from the Information Layer for data transformation.
• The Event Broker ABB, Event Listener ABB, and Event Producer ABB in the Integration Layer use the Event Manager ABB in the Governance Layer for event definition and related information.

Interaction with Horizontal Layers

Horizontal layers of the SOA RA leverage ABBs from this layer to provide their respective capabilities. Horizontal layers of the SOA RA use the Integration Controller ABB in the Integration Layer to access the ABBs in the Integration Layer and capabilities they provide, such as Mediator ABB, Router ABB, Message Transformer ABB, Data Transformer ABB, etc.

Key Interactions of the Integration Layer with Horizontal Layers

Detail Interactions of the Horizontal Layers with the Integration Layer provides a typical interaction with horizontal layers of the SOA RA with the Integration Layer:

Detail Interactions of the Horizontal Layers with the Integration Layer

In the example above:

• Through the service discovery and invocation process, the web client looks up the View Profile Service.
• A connection is made to the Integration Layer (ESB).
• The ESB performs protocol conversion, if necessary.
• It subsequently routes the call to the appropriate destination.
• It then receives the results of the call.
• It then aggregates the results of the call.
• The aggregated result is transformed and returned to the client in a format that can be consumed by it (for example, in the case of the web client, the aggregated result is returned in an HTML format).

Usage Implications and Guidance

The Integration Layer also incorporates the support for service virtualization using static routing rules or dynamically at runtime using a service repository and service registry. Service virtualization decouples the location of services for consumers of services. Services are exposed to consumers through a registry, but the exact location decoupled, to support versioning, change of service locality, and administration, without impacting the consumer.

To summarize, the Integration Layer supports capabilities required for enabling SOA such as routing, protocol support and conversion, messaging/interaction style, support for heterogeneous environment, adapters, service interaction, service enablement, service virtualization, service messaging, message processing, and transformation.