SOA Ecosystem

In the previous post, I’ve discussed the role of the ESB in the SOA ecosystem. However, I failed to adequately describe what an SOA ecosystem was and what were all the inter-relationships that must exist in order to make it truly effective.


If you refer to the diagram above, you will notice several major components that make up the SOA Ecosystem.

1. ESB
2. Registry/Repository (RegRep)
3. Security
4. Service Management
5. Shared Service Environments
6. Service Consumers
   

The SOA Ecosystem also encompasses all the related elements such as the application and service developers and testers as well as all the tools being used to accomplish these activities.

To truly comprehend how the SOA ecosystem operates, a clear understanding must be developed of what each component does and what its role is. Let’s start from the service consumer side.

• Service Consumers
  
  • Application Developers build applications that consume services. They use IDEs and other development tools to construct service requests and parse responses. Developers interact with the Registry/Repository to find the right services, obtain service metadata, and understand usage patterns.
  • Application Testers perform quality assurance tasks on the final product.
  • Application Servers that execute the application code interact directly with the SOA platform hosting the services.
• SOA Infrastructure
• Service Management Platform acts as an entry point into the SOA infrastructure. It retrieves policy information about the service being executed and applies it appropriately to the request. The policy is used to understand service security and authority, associated SLAs, constraints, contracts, etc. The Service Management Platform is often utilized to keep track of the service consumption and run-time metrics, which are then fed into the Registry/Repository.
• The role of the Enterprise Service Bus has already been discussed.
• Registry/Repository acts as a central repository for services and their metadata. Its uses and integrations are discussed at each related point.
• Security / Authentication Platform is a part of the larger IT infrastructure and is typically represented by either LDAP or Active Directory technology.
• Shared Service Environments are used to host reusable services. While different organizations choose to approach service hosting differently, if a common service hosting platform can be established, many issues related to service scalability, performance, reuse, security, implementation, standardization, etc. can be easily resolved. A centrally managed platform can be easily upgraded to accommodate additional – foreseen or unforeseen – volume. Standard capabilities can be provided to perform security, authentication, logging, monitoring, instrumentation, deployment, and many other tasks.
• Service Creation
• Service Architects and Developers create reusable services using the appropriate design and development tools. They also interact with the Registry/Repository to discover existing services and register new services and related metadata. The created services should ideally be deployed into a Shared Service Environment.
• Service Testers perform quality assurance tasks on the new or modified services. They use special SOA testing tools to create test cases and automate their execution. These tools interface with the Registry/Repository to retrieve metadata about the services and update related information once testing is complete.
  

Many vendors have slightly different view of how their products integrate and interact with the SOA ecosystem, what components exist, and where they are located. One can also include other elements into the SOA ecosystem such as the EA Repository, CMDB, IT Governance Tools, Monitoring Tools, etc. However, the general dynamics remain the same. The key point is that the SOA infrastructure interacts with the rest of the IT platforms and people in a certain way. In order for SOA to be truly successful, these interactions must become natural, automated, and symbiotic.

SOA Testing

Many articles, books, and conferences dealing with the SOA tend to ignore one of the most important topics in software development – testing. Often, testing is just an afterthought in many software development efforts. A lot has been said and written about this problem. However, it is important to note that testing in an SOA program plays a much more important and prominent role than in any other software development effort. Without a proper testing foundation, the whole SOA initiative will either fail or become too unwieldy or expensive to maintain.

One of the cornerstones of SOA is service reuse. Success of the SOA program is often measured through the amount of services created and reused. The biggest problem with testing in an SOA environment manifests itself when a service has several consumers and changes are made to it. How do you validate that this change does not impact service consumers? How do you determine the best way to deal with this change? Do you ask all of the service consumers to perform their own regression testing to make sure internal service changes do not impact them? Obviously, this is not an effective solution. With more and more services getting more and more reuse, you need a solution that minimizes the amount of manual testing you need to do but, at the same time, provides a clear understanding of how the service changes impact its consumers.

The services are composed of three primary elements – interface, contract, and implementation. Interface represents the protocol and communication mechanism between service and its consumers. Contract defines all of the interaction details such as message formats, SLAs, policies, etc. Service implementation is self-explanatory. A service can expose multiple interfaces and may potentially support multiple contracts. The key to understanding the impact on service consumers is to verify whether or not changes to any of the service elements invalidate how it behaves today. Changes that have no impact are called non-breaking; changes that modify the behavior are called breaking.

Each shared service needs to have an automated test created as part of its normal implementation. It will address two issues – provide an initial test bed for the service and automate all future testing needs. The test should inspect what changes are made to each service element. When service is modified in any way, the automated test suite should be executed to understand the impact of the changes. If all tests pass, the changes should be considered non-breaking and consumers should be unaffected. If any of the tests fail, this would indicate a breaking change and a new version of the service would need to be created. Alternatively, the impacted consumers can change but, ideally, breaking changes should trigger a new service version.

The biggest problem with SOA many companies face is lack of a consistent, comprehensive testing approach. Without automated regression testing for shared services, organizations are exposed to risk of high manual testing costs every time a service is changed or new consumer is added. Additionally, it can drive service versioning and serve as a formal validation mechanism that service consumers can trust. Automation can save millions of dollars in manual labor and ensure stability of the whole SOA platform.

SOA Façade Pattern

In the last post, I discussed the value of the canonical modeling and described how to minimize the impact of canonical model changes on the service consumers. The solution was to use the façade pattern. I would like to elaborate on this topic since a more in-depth discussion is needed to define the pattern and understand its uses.

A good definition of the façade pattern can be found on Wikipedia: http://en.wikipedia.org/wiki/Facade_pattern. In general terms, it is described as “a simplified interface to a larger body of code”. This is exactly how it should be applied to SOA. A façade should be built in front of any service whose interface is based on the canonical model. Consumers would not access the service directly but rather through its exposed façade interface. In fact, the canonical interface should only be exposed for internal consumption. Each façade should be designed to be specific for each consumer or a group of consumers and not directly tied to the canonical model. The diagram below depicts the pattern details and its usage.

There are several distinct benefits of using the façade pattern.

1. Façade shields service consumers from the changes in the canonical model.
   If every consumer was dependent on the canonical model, even the smallest change could have disastrous effects. All of the services as well as potentially all of the service consumers would need to be re-tested. Lacking automated regression and functional tests already developed, this would be a major undertaking. Using the façade pattern would minimize the impacts of any canonical model changes. Since the facades are specific to each consumer and are not directly tied to the canonical model, the only thing that would need to change is internal mapping between the façade interface and the canonical model.
   
2. Façade hides the complexity of the canonical model.
   Modeling the whole business domain is not a simple task. Therefore, canonical models are usually large and complex. Service consumers do not typically want to know the entire canonical model and understand all of its intricacies. They want to get the data they need and continue performing their business functions. Exposing a consumer-specific interface via the façade prevents service consumers from having to know any canonical model details. Additionally, since canonical models are fairly generic, most of the data elements in the returned entity may not be relevant to the consumer. A façade simplifies the request and response data structures and ensures that only relevant information is returned.
   
3. Façade returns data representation understood by the consumer.
   A canonical model is generic. It is designed to describe the whole organization. However, service consumers typically operate in their own specific domains. Service façade that is designed to return data in a format that consumers understand simplifies the overall consumption experience and reduces the overall efforts. The consumer does not need to perform any translations and can start working with the data right away. Additionally, a façade can help representing the same entity differently for different consumers if so required. There may be instances, for example, when one Line of Business (LoB) thinks of a customer one way while another LoB views a customer completely differently. These views may even be largely incompatible but as long as they are represented in the canonical model, a façade can be created to address specific LoB needs.

The façade pattern introduces a small translation layer between the service consumer and the canonical service interface. It should not contain any generic business logic but can perform some consumer-specific operations. While it may cause reduction in performance and increased development costs, the negative impacts should be minimal and will be offset by the benefits described above.

SOA & Canonical Modeling

There has been a number of discussions in various articles and blogs on the topic of application and value of canonical modeling for SOA. Majority of industry experts support the use of canonical models as one of the key SOA patterns. However, there are some that consider canonical modeling as detrimental to SOA (http://service-architecture.blogspot.com/2006/08/single-canonical-form-not-for-soa.html). I would like to add my voice to those that consider canonical modeling a critical aspect in the success of the SOA program.

Primary goal of any SOA program is to introduce a variety of reusable services. Reusability typically implies that a service has a number of different consumers and data providers. The first and most obvious value of a canonical model is that it acts as an abstraction layer between all those consumers and providers. It is an old and well-known design pattern – when you have a number of data sources and their consumers that need to be integrated together, you introduce an abstraction layer, so that neither is aware of the internal details of the other. This way, any changes made to a consumer or a provider will have minimal impact on all of its integration points. This is the second benefit of using a canonical model. It minimizes the impact of internal service changes, modifications of data sources, or switch to a new backend data source on the service consumers. The canonical model should remain unchanged regardless of what happens inside the service, which, in turn, ensures that the contract between the service consumer and the service itself remains unaffected. The maximum possible impact on the service may be the need to change the mappings between internal service data structures and the canonical model.

Since the canonical model minimizes the impact of internal service changes on its consumers, it also reduces the need for regression testing. (This is the third benefit of canonical modeling.) If services did not provide a layer of abstraction between its internal implementation, backend data sources, and its consumers, any change inside the service or data provider would be reflected in its interface and thus would require a full regression test. Usage of a canonical model eliminates the need to perform rigorous regression testing since, as we discussed above, any such changes would not impact the service consumer. The only thing that would need to be done is perform a test validating that the service contract did not change.

Representing standard structures in a canonical model maximizes service reuse (fourth benefit). Consider all the entities that your business deals with every day. It could be customer, account, price, payment, etc. However, different parts of the organization may view these entities differently. One division, for example, may care about the customer household information while other about his/her geo-positioning. Without a canonical model, you would end up with multiple slightly different representations of the same entity. This results in services being built based on disparate models targeted for only specific audiences. Other groups trying to reuse these services would require changes to address their needs or would simply not be able to consume them. Representing all entities in a standard way eliminates this incongruence and allows different parts of the organization to speak the same language. This, in turn, maximizes the potential and real reuse of services built across the company.

Of course, some would argue that any changes to the canonical model would impact all of the service consumers and they would be right. In order to minimize this risk, a façade pattern needs to be used. Rather than exposing the canonical interface directly to the consumers, a façade would need to be build based on each consumer’s needs. It would expose data contracts specific to each consumer or a group of consumers. The service would never be called directly but only through one of its façade interfaces. This way, any changes made to the canonical model would not impact the service consumers directly. The façade would remain intact. The only change that would potentially need to be made is modification of mappings between the façade and canonical data structures.

Used together, the canonical modeling technique and the façade pattern will maximize the service reuse and minimize the impact of internal changes on service consumers. The approach will save costs and time on regression testing efforts. Regardless of what the opponents say, use of these techniques is critical to the overall SOA program success.

Links to some good SOA & canonical modeling articles:
http://www.it-eye.nl/weblog/2007/06/13/soa-best-practice-9-use-a-canonical-data-model/
http://www.ibm.com/developerworks/db2/library/techarticle/dm-0803sauter/
http://www.soapatterns.org/canonical_interface_expression.asp
http://reallifeserviceorientedarchitecture.blogspot.com/2007/11/how-to-create-canonical-form.html