Legacy Evolution to SOA – Best Practices

 

Overview

Service-Oriented Architecture (SOA) is an architectural style for creating business solutions to enable business agility. The goal is to continuously improve business results by SOA services that generally enable flexibility, maintainability, and cost-effectiveness more than legacy and non-SOA solutions. This section emphasizes the key foundational aspects and considerations that would effectively transform legacy solutions to SOA solutions.

Key SOA Principles

SOA defines a set of (generally agreed) key principles for services. In this section a few of them are described, including key aspects regarding legacy systems which need to be taken into account in an Legacy to SOA (L2SOA) evolution. Though TOGAF itself is not intended to be prescriptive, these key SOA principles are essential for the service enablement of legacy.

Well-Defined Service Contract

Service interactions must be well-defined with a widely supported standard (e.g., WSDL) describing details to assist the service requestor to invoke the service(s) required.

This principle helps to prevent the automatic generation of contracts out of not-so-well documented existing legacy APIs (more or less “code first”), which might be tempting at first.

Another aspect of this is to avoid performing direct legacy database-to-schema conversion, which introduces tight-coupling between message and database.

Define Services with Appropriate Granularity

Services must be designed for appropriate granularities that offer greater flexibility to service requestors without impacting the performance and security. Services granularity should make it easy for service requestors to assemble services to execute business scenarios. This is not always possible, especially for (multi-step) transaction-oriented (legacy) systems. The granularity level should be clearly described per service (e.g., which steps of functionality and invocations of other services or modules take place). For certain legacy systems this might imply first a (re)documentation project.

Loosely-Coupled Services

Service requesters must not have any knowledge about the technical details associated with a service implementation. Seamless integration and exposure of legacy systems into a flexible IT environment with the use of SOA abstracts the technology implementation from the use of it. As long as the implementation meets the specified Service-Level Agreement (SLA), the service requester does not need to have any knowledge about the technical part of the service implementation. Therefore, legacy systems can be migrated or replaced by other technology, without affecting the service and therefore the service requester.

This also relates to the principle of well-defined service contract, described earlier.

Design Services for Stateless

Services invocation must be independent of the state of other services and each service invocation has all the required information from one request to another. This is a challenge in the functionality of legacy systems, as functions can be intertwined with other functions (e.g., not well structured COBOL modules). This is acceptable as long as it occurs in the background and the principle can still be upheld for the specified service and its contract.

Ensure Services have Appropriate Security Enforcement Standards

Services must be designed with appropriate security policy enforcement mechanisms to ensure that only authorized requesters can successfully invoke them. When centralizing policies and identity management in SOA, at least a mapping needs to be done from individual systems security information to the centralized information to be able to implement this. It might not be possible to integrate legacy systems with a central security mechanism, so redundancy mechanisms need to be developed and maintained (including pre-call mapping of credentials).

Adopt SOA Ontology/Vocabulary Standard

To effectively facilitate SOA adoption and have required alignment between the business and IT communities, a common vocabulary standard must be utilized in services lifecycle management. Ontology defines the SOA concepts and semantics commonly understood by all stakeholders and enables effective communications. The Open Group SOA Ontology [16] facilitates SOA adoption.

Operational Considerations

Define Metrics (including Key Business Performance Indicators)

Metrics are essential for the continuous communication and justification of an L2SOA engagement. What starts with a baseline measurement (which might be put in a business case), will be measured at appropriate intervals and reported. Here a few metrics are suggested.

  1. Application Usage: What is the actual usage of an application, before and after L2SOA? SOA can lead to a service which can be used by many more consumers than in the current state. This change can be measured if there is a baseline.
  2. Cost Reduction: Define various cost factors; e.g., maintenance, license, energy, etc. The ratio between relevant cost factors also need to be taken in to consideration; e.g., the ratio in IT budget between the projects supporting business growth initiatives and the projects for (legacy) maintenance.
  3. Functional Re-use: Define and measure functional re-use before and after the evolution of service-enabling business functionality. The metrics need to be measured and reported across business silos.
  4. Quality of Service: Quality of Service (e.g., service up/downtime, message throughput) is part of Service-Level Agreement (SLA) monitoring and should include appropriate performance-related KPIs.
  5. Revenue-generated: Creation of services can lead to additional revenue by exposing internal processes/functionality externally.
  6. Time-to-Market: Using SOA to enable re-usable and standardized services to access legacy systems should reduce time-to-market of certain functionality.
  7. Security KPIs(data protection-related KPIs): Especially concerning the key SOA principle: “Ensure services have appropriate security enforcement standards”, this metric helps to monitor frequency and severity of security incidents after modernization. The Open Group Security Forum, a Forum of The Open Group, covers a range of security aspects including SOA [10].

Define Architecture using the TOGAF ADM

There are several distinct approaches that can be leveraged to approach the architecture definition of the transformation (evolution) of any legacy enterprise to service-oriented. The approaches in this Guide are based on the foundation of the TOGAF 9 [4] architecture framework and Architecture Development Method (ADM), providing the step-by-step approach, iterative processes, and types of building blocks needed for the evolution. Approach to Enable L2SOA discusses this in detail.

To define the Architecture Building Blocks (ABBs) for the L2SOA evolution, modernization strategies are helpful. They define common strategies to use for a certain situation, and provide common building blocks for use in that context. The next section provides an overview of strategies for enabling L2SOA modernization.

Modernization Strategies

There are several strategies to enable the process of modernizing legacy applications using SOA. Enterprises need to select one or more strategies and evaluate against these strategies during the process. The key strategies are service enablement, language conversion, re-architect, and re-hosting of applications. The following sections detail each of them, describing the approach, problem solved/value-add, risks and mitigations, and architecture building blocks involved.

Service Enablement

Approach

This strategy enables the access of legacy applications through services hosted by integration platforms and/or service containers (comparable with NCOIC Integration Level 3 – Using Existing Interfaces [8]).

Two implementation models can be distinguished [11]:

  1. Take the Data to the Service: Collate slowly changing information into a holding data store (a warehouse or data grid, for example), and act upon the data there. Extract, Transform, and Load (ETL) and object caching technologies can be used to make the data more readily available to the service and less dependent upon the legacy application.
  2. Take the Service to the Data: For rapidly changing transactional data, the action must go to the physical source to manipulate it. The integration layer will use connector and adapter technology to make those accesses (for example, read and update).

These services can be categorized into presentation services (e.g., screen scraping), and task services (e.g., web service querying data).

Capabilities to consolidate multiple legacy assets are possible by creating services that invoke multiple legacy systems or data.

Problem Solved/Value-Add

  • Re-use and leverage existing assets while protecting huge investments made in legacy applications.
  • Improve value of core applications by using SOA methods and technologies to unlock value.
  • Leverage legacy functionality in automated business processes spanning multiple systems and departments.

Risks and Mitigations

  • Difficult to identify useful legacy functionality to be leveraged; mitigated by detailed business process analysis and mapping, and conducting (both static and run-time) code analysis may help to answer the question: “Which applications contain the designed functionality?”.
  • Poor documentation can lead to unknown functionality versus clear service interface.
  • System transformations may be disruptive. Proper analysis is necessary to understand organizational, process, and technology impact.
  • It is important to communicate the vision of service enablement to collaborate in enterprise-wide SOA program governance, which is not always easy.

Architecture Building Blocks Involved

  • Information System Architecture: New information system services, providing legacy system functionality.
  • Technology Architecture: Integration platform, connected to legacy systems.
  • Technology Architecture: Service containers hosting the services.

Language Conversion

Approach

This approach aims at converting applications written on legacy languages into more supportable languages. The main goal of this conversion is to be able to convert to a language that provides SOA enablement by default (e.g., Java platform), so the legacy system functionality can be directly provided through services by the system itself. This only works if the conversion can be done automatically. Otherwise, a re-architecture (see next section) might be necessary in case the software structure is very complex and interwoven, or even a re-build might be necessary in case the previous strategies are not sufficient.

Problem Solved/Value-Add

  • Reduce Total Cost of Ownership (TCO) of the applications – mainframe, hardware, software license fees, and learning costs.
  • Reduce time-to-market with equal or better performance, availability, and scalability.
  • Applications can be made adaptable to SOA implementation.

Risks and Mitigations

  • Legacy functionality is not structured in modules, but has intertwined business logic which makes it impossible to provide structured autonomous services; mitigation is to re-architect (see section below).
  • No supported functions in the latest version of the languages; mitigation could be to analyze transformation and convert to compatible modules providing equivalent functionality.
  • Skill/training used for analyzing old platform/technology/languages might be scarce; mitigation could be to ensure access to key resources with legacy business and program skills.
  • No automated conversion possible; only mitigation is to select another strategy.

Architecture Building Blocks Involved

  • The logical components for the legacy system stay the same, but this might imply a new technology component for platform changes (language changes, platform might change).
  • Additional information system services can be defined, which can be implemented using the SOA enablement options provided by the new language. This is comparable with the service enablement strategy as described in Service Enablement; the difference is that the system is able to provide the services by itself as a provider (which still can be mediated by an integration platform).

An overall important aspect is what to do with the bugs that are found: fix them or keep them. Fixing brings additional impact analysis, testing, and budget and planning discussions. Keeping them makes the unknown known and needs to be communicated.

Re-Architect

This approach aims at re-architecting the legacy system. It restructures the business logic towards a modularized design. This design enables the creating of autonomous services that can access the functional modules.

Two sub-strategies are identified below.

Re-Architect, but Use the Same Environment

Approach

In this strategy the mainframe source is first restructured into specific functional modules. Then these modules are opened up with adapters and integrated with a service mediation component, as described in Service Enablement. But, this means a change in the legacy system.

Problem Solved/Value-Add (in addition to Service Enablement (Section 4.5.1))

  • Re-use and leverage existing business functionality and intellectual property.

Risks and Mitigations (in addition to Service Enablement (Section 4.5.1) and Language Conversion (Section 4.5.2))

  • Poor and old documentation especially on the system structure is a huge risk. Part of re-architecting will be spent on documenting the system as-is.
  • Re-architecting might lead to programming changes (e.g., split functionalities and code). Needs additional testing, especially regression testing.

Architecture Building Blocks Involved (in addition to Service Enablement (Section 4.5.1))

  • Depending on the granularity of the ABBs in the specific architecture (e.g., one level deeper than the legacy component itself), re-architecting can lead to less or more ABBs.

Re-Architect, but Use a New Environment (e.g., Language, Platform)

Approach

This is a combination of restructuring in specific functional modules, combined with a language conversion. This is also called re-build, but in the case of L2SOA evolution re-architecture must be done first before re-building in a new language. Re-building is not the same as language conversion described in Language Conversion, because it is not an automated conversion due to the necessary re-architecture work.

Problem Solved/Value-Add (in addition to Language Conversion (Section 4.5.2))

  • Possible to fix old problems and introduce new functionalities.

Risks and Mitigations

  • Budget discussion; need good arguments why one of the other approaches was not chosen.
  • See previous sub-strategy.

Architecture Building Blocks Involved

  • See previous sub-strategy.

Re-Hosting of Applications

Approach

This is the need to reduce the cost of the mainframe platform in cases where mainframes support the majority of business-critical applications to distributed and cloud computing models. Note that this strategy on its own does not transform the legacy system to SOA. It needs to be combined with one or more of the strategies described earlier.

Problem Solved/Value-Add

  • Preserve business logic and protect existing investments in legacy applications.
  • Improve their competitiveness by integrating legacy applications with new applications.
  • Maintain the end user's experience on the front end of the application and yield cost savings.
  • Reduce time-to-market with equal or better performance, availability, and scalability.

Risks and Mitigations

  • Integration of multiple product vendors and tools; need a program to manage required integration changes as separate projects.
  • Estimating capacity for re-hosting from one platform to another requires very detailed analysis.

Architecture Building Blocks Involved

  • The information system building blocks will stay the same.
  • The technology ABBs, especially the physical, will change to the new host type.

Pattern-Based Approach

The L2SOA enablement key architecture styles are elaborated further by leveraging Enterprise Integration Patterns (EIPs) which provide the capabilities to service enable legacy applications.

However, large enterprises having complex enterprise systems with various types of legacy applications often leverage a combination of approaches to provide the capabilities to modernize their enterprise solutions depending upon the architecture strategy that is employed.

The following patterns are mostly used (one or a combination):

Enterprise Integration Patterns

Pattern

Description

Enterprise Integration (Service Bus)

An Enterprise Service Bus (ESB) represents an environment designed to foster sophisticated interconnectivity between services. It establishes an intermediate layer of processing that can help to overcome common problems associated with reliability, scalability, and communications disparity [9].

Queue-based Mediation

Queues enable guaranteed delivery through persistence of the data (messages) being sent. They operate in an asynchronous manner, which enables the complete decoupling between sender and receiver.

Service Enablement

Solution logic can be encapsulated by a service so that it is positioned as an enterprise resource capable of functioning beyond the boundary for which it is initially delivered. It is subject to additional design and governance considerations [9]. Important is the use of a standardized service contract, to reduce implementation coupling. This includes both providing legacy functionality as services, and adding mechanisms to legacy systems to consume services.

Screen Scraping

Screen-scraping software electronically “reads” and “puts” the information from a mainframe terminal screen. But, the service specification is totally dependent on the screen transaction definition.

Re-design

Instead of keeping the application as-is and using one of the patterns above, this pattern actually re-designs the application which provides opportunities to service enable the application by design (using re-architecture and/or language conversion strategies, as described in Modernization Strategies).

Based on these patterns and the statistical information gathered by research, the approaches can be categorized into matrices which provide indicators of the benefits and pitfalls of each approach. This provides a starting point to add more detailed multi-dimensional models that provide a more holistic view of the various approaches.

Based upon a set of L2SOA projects, it is possible to qualitatively analyze cost and complexity ranges per pattern.

Integration Patterns Cost and Complexity

One thing that is immediately visible is the relation to return-on-investment. The directly SOA-related patterns have a high cost and medium-to-high complexity. But these patterns result in a solution/enterprise-level infrastructure and transformation that is beneficial to many more (legacy) systems than if we talk about screen scraping, typically delivered as a web service. A strong conclusion can be made that SOA enablement of legacy systems needs to be seen in the context of the enterprise as a whole and not as individual systems. In this manner, the high (infrastructure) costs can be spread across many service-enabled applications – both legacy and new. This can only be accomplished if the process is viewed from an Enterprise Architecture perspective.

Relationship with OSIMM

The OSIMM model [5] provides a holistic approach to the maturity of applications in an enterprise. Leveraging the model and the patterns that are described above can provide a view of the silo'ed legacy applications' maturity level, providing the organization with the information to progress by making an evolution from a legacy system to a service-oriented approach. For example, providing a queue-based mediation between an SOA application and a mainframe that currently resides in a silo that is targeted to be integrated. This could provide the organization with the capability to strategize enablement of a portfolio of legacy applications leveraging the OSIMM. Chosen architectural styles and patterns can be mapped on the matrix fields between desired maturity level and dimensions (application, architecture, and information).

Organization and Process

Legacy evolution has impact on current organization and process:

  • L2SOA is a transformation program and has to be approached and managed as such.
  • Current IT operations and management teams undergo impact. A change program is needed to guide them.
  • New technologies will enter the landscape and need to be learned.
  • New roles will be added to create/support the modernization. This can be an opportunity for employees to gain new knowledge, but also a threat that they might not keep up or are not given the opportunity to re-learn (e.g., other people are hired).
  • Current change request procedures need to be extended.
  • SOA brings the concept of services implemented across business boundaries, both from a functional and technical perspective. While the legacy systems might currently support clearly defined dedicated business processes and even departments, this will certainly change. Service owners will need to be able to operate in that context.
  • It is advisable to create a competence center where the (at least technical) knowledge of modernizing L2SOA is consolidated.
  • How can organizations see the value and how do they “sell” it to other departments in the organization? Definition and measurement of return-on-investment of an L2SOA engagement is crucial. See Define Metrics for metrics and Key Performance Indicators (KPIs) which can be used for this.

Governance

Introduction

In general, governance means establishing and enforcing how people and solutions work together to achieve organizational objectives. This focus on putting controls in place distinguishes governance from day-to-day management activities [6].

SOA governance should extend the organization’s existing IT and Enterprise Architecture governance models to take into account the new SOA assets and SOA policies [6]. SOA, when done correctly, leads to services which connect and also transcend traditional business boundaries. Functionality and data in legacy systems might therefore be used by new consumers under new Service-Level Agreements (SLAs). SOA governance is essential for organizations that make pervasive use of SOA in their business and IT systems regardless of legacy system modernization efforts.

The Open Group SOA Governance Framework and Relation to Legacy Systems

The Open Group SOA Governance Framework [6] is based on the definitions of governance and SOA governance as stated in the previous section. Its goal is to enable organizations to define and deploy their own focused and customized SOA governance model. As a framework, it consists of an SOA Governance Reference Model (SGRM) which is utilized as a starting point, and an SOA Governance Vitality Method (SGVM) which aims at continuous improvement of the governance process.

The SGRM defines a set of principles, governing processes, artifacts, roles and responsibilities, and governance technologies. Legacy systems are explicitly taken into account in the SGRM, where there is a need to distinguish legacy from any other type of system.

SOA Governance Reference Model Touch Points shows an overview of the defined touch points of the SOA Governance Framework with legacy systems.

SOA Governance Reference Model Touch Points

Reference Model Part

Where

What

Process

Service Lifecycle Governance/
Service Realization Planning

Explicitly analyze whether the service should use existing legacy functionality.

Process

Service Lifecycle Governance/
Service Modeling

When creating the service implementation model, identify appropriate legacy integration patterns, if necessary. (*)

Checkpoint

Service Lifecycle Governance/
Approve Service Model

Ensure conformance to legacy modernization guidelines. (**)

Checkpoint

Service Lifecycle Governance/
Approve Service Implementation

Ensure conformance to legacy modernization guidelines. (**)

Artifact

Service Description

Service implementation design model needs to include legacy re-use patterns used and which legacy functionality is re-used.

Also, in the general description of the implemented service, the used legacy functionality needs to be described.

(*) See also the patterns described in Pattern-Based Approach.

(**) This needs to be explicitly taken into account as an architectural deliverable.

Additional Considerations and Challenges, with Recommendations

Additional Governance Considerations

Consideration/Challenge

Recommendation

Legacy systems are explicitly sized and not ready for an unknown number of users, as a result of opening up as a service.

Strict SLAs with maximum named and concurrent users defined. Explicitly test whether these numbers can be supported, because several “older” technologies have strictly defined boundaries in this.

Organization’s sourcing policies to detail the architecture for the evolution from legacy results in changes to governance structures at portfolio, program, and third-party contractor levels.

Review current governance structure and extend it to cover the people (stakeholders, employees, etc.), technologies (legacy, SOA, etc.), financial implications, and processes impacted by the evolution [6][7].

The stability of the security in the legacy system can be impacted by exposing it as a service.

Security design and solid implementation covering the legacy system and the services architecture together is essential to make sure there are no “rogue” uses of services causing unexpected service usage.

Skills/Education/Professional Development

Employees who are used to job stability and somewhat resistant to adapt new technology often feel stress about their ability to transform to a new skill set required for evolved legacy application technology. In general, some people are uncomfortable with change and tend to resist change. Management should consider the following in planning for legacy applications transformations to SOA:

  • Carefully assess the SOA implementation styles that best suit the organization while keeping the current expertise of employees. Plan to proceed gradually to adapt simple implementation models rather than those that are complex and require advanced skills.
  • Identify the ideal path of least resistance that provides maximum return with minimum risk to SOA program adaptation.
  • Organizations should account for the complexity and learning curve required in adopting SOA technologies and standards in their legacy applications transformation.
  • Provide necessary training to progressively evolve employee skills required to effectively perform new roles.
  • An effective governance process should be put in place to ensure that training and education is an ongoing process of skills development.
  • Identify, involve, and cultivate agents of change early in the organization of a competency center to lead additional educational efforts. Would-be detractors may also be embraced at this stage to turn negativity of change into positive behavior.

Security

SOA and Security

SOA by definition is a loosely-coupled architecture, where applications are composed by connecting exposed service endpoints through a set of SOA enabling components. However, this open architecture along with its benefits also presents a complex set of requirements to secure information. Organizations that started with SOA originally used web services for simple point-to-point integration with basic security requirements, but are moving towards a more matured SOA model. To do this right they will need a solid and robust security model. Any security model should build on an organization’s existing security infrastructure and hence it is not possible to come up with a one-size-fits-all model. This implies that the existing security infrastructure should be well documented, including the exact way in which security in legacy systems is implemented and integrated with its surroundings (if any).

Within SOA the general way of approaching security is to move security and policy checks out of the actual business logic into a layer above it, making it possible to integrate with a centralized policy and identity management solution including Single Sign On (SSO). In most cases this is not done or not possible on the data layer with the data layer platforms, as data is considered the core asset in the organization and therefore will have its own security layer. Ideally these are aligned with the centralized solutions mentioned earlier.

The Open Group Security for the Cloud and SOA project published a White Paper: Security Principles for Cloud and SOA [12]. It describes two sets of principles:

  1. A set of 20 generic security principles common to all designs aiming to assure a secure IT architecture, ranging from “security by design” to “data protection lifecycle”.
  2. A specific set of principles regarding assuring security in architecting the cloud and the SOA environment, mainly focused on policies, data protection and privacy, and cloud-specific principles.

Those principles are also relevant for L2SOA engagements.

Legacy and Security

One of the first concerns with legacy systems is the risk of outdated technology and software, making it extremely vulnerable when the system is exposed to more parties and technologies than currently required. Assessing the state of the software is important.

Because of the direct link from system to terminal, encryption of data was not directly important. Encryption of transport and/or information itself needs to be considered when opening up the system with services.

Some legacy systems have already reached a maximum amount of named users in the years that they are used. This means that the people that use the system might have an ingenious manual administration of user names and passwords. This is a serious risk for integrating the systems with services using centralized security components.

Another concern is the way a session is defined in the legacy system. Introducing SOA can also introduce SSO. Depending on the implementation and platform chosen, a generic session is defined with session durations for an account before it needs to authenticate again. If it is not possible to send an “invalidate session” to a legacy system, other measures need to be taken to be sure that the session cannot be taken over by an account that is de-activated centrally.

US Homeland Security: Build Security In documents 623 and 624 [13] and [14] describe detailed guidance in assessing the security state of legacy systems, which is also useful in L2SOA engagements.

 

⇒ Case Study: Automotive Company