SoftwareTesting: The Old and the New

Software testing has evolved considerably over the last decade from single dimensional test approaches to multi-dimensional ones. Testing now encompasses broader aspects such as test management through push button test environment creation on the Cloud to dedicated tool evaluations for each kind of testing. There is general consensus worldwide though, on the need for flexibility and creativity in testing and the need to treat testing as more than just another phase performed towards the end of the project.

Traditional software testing that focused on code-level testing has evolved with Distributed and Web Service architectures. Security takes on more importance as there are now multiple integrated systems and multiple interfaces, internal and external. In this context, I would like to delve more into the details of testing Service-oriented architecture (SOA) and SOA based solutions, a subject close to my heart!

The V-Model:

For Web Service architectures and SOA, the V-Model is typically a suitable test methodology that enforces testing discipline throughout the project lifecycle. The project starts with defining the Business User Requirements. The V-Model would recommend that the Business Acceptance Test Criteria for these defined requirements are defined and agreed before moving to the start of the technical design phase. Before moving to the next phase/level of technical design, the model recommends test criteria defined for that level of technical requirements, and so on. The V-Model is simply encouraging the project team to continually determine how they would successfully test the project deliverables.

The V-Model is a suitable test methodology to deliver SOA projects for the following reasons:

1. It encourages a top-down approach with respect to defining the business process requirements, high-level functional technical design, security etc.
2. It then encourages a bottom-up test approach - test individual functions within a service, test an individual service then test a composite set of services through to testing an integrated process and finally testing a complete business system. SOA is loosely coupled services and that is why a bottom up test approach is recommended.
3. The levels reflect different viewpoints of testing on different levels of detail.
4. The V-Model encourages testing throughout the entire Software Development Life Cycle.

Challenges:

Some of the new challenges in testing SOA solutions as opposed to the traditional applications, typically are:

1. Services that do not have a user interface
2. Data driven business logic within services
3. External services to the organization
4. The quality of service software will be vital to promote reuse and facilitate business agility. Services that have known bugs and quality issues will not be reused by the development teams. A significant increase in testing activities and test assets (functional, performance and security regression suites that include sophisticated harnesses and stubs) will be required at a service (program) level
5. Predicting the future usage of services to assist with performance, load, stress, scalability
6. As your SOA evolves, security testing will have a higher priority and profile within your organizations test strategyIn SOA, Services are based on heterogeneous technologies.

No longer can we expect to test an application that was developed by a unified group, as a single project, sitting on a single application server and delivering through a standardized browser interface. The ability to string together multiple types of components to form a business process requires unconstrained thinking from an architects perspective, and test planning and scheduling complexities from a testers perspective.

In SOA, application logic is in the middle-tier, operating within numerous technologies, residing outside the department, or even outside the company. We know that to test SOA, you need to go far beyond merely testing a user interface or browser screen. Web Services (WSDL/SOAP) will be an important component for many SOAs, but if youre only testing Web Services, you are not likely to test the entire technology stack that makes up the application. What are the transactions happening at the messaging layer? Is the right entry being reflected in the database? In fact, many perfectly valid SOA applications will house business logic entirely outside of the Web Services.

To address the above challenges, organizations need to review and enhance their current test methodology. Many Test Tool vendors have now recognized the new SOA test challenges and have developed a new breed of tools to help organizations to plan, manage and automate SOA functional, performance and security testing.

Testing SOA solutions:

How do you test SOA architecture? You dont. Instead, you learn how to break down the architecture to its component parts, working from the most primitive to the most sophisticated, testing each component, then the integration of the holistic architecture. In other words, you have to divide the architecture into domains, such as services, security, and governance and test each domain separately using the recommended approach and tools. SOA is loosely coupled with complex interdependencies and a SOA testing approach must follow the same pattern.

You can broadly categorize SOA testing into the following phases:

1. Governance Testing
2. Service-component-level testing
3. Service-level testing
4. Integration-level testing
5. Process/Orchestration-level testing
6. System-level testing
7. Security Testing

Test Tools:

As discussed earlier, tools are key in every phase of testing. SOA requires a comprehensive test tool strategy. Some of the commercial and open source tools available in the market that could be explored for SOA testing initiatives would be:

Commercial:

1. Green Hat GH Tester
2. Mercury suite of products
3. Parasoft SOAtest
4. AdventNet QEngine
5. Borland SilkPerformer SOA edition
6. iTKOs LISA product suite

Open Source/ Freeware:

1. SOAP UI
2. PushToTest TESTMAKER
3. AutoStub (an internal tool developed by Torrry Harris, belonging to the SOAizer™ stack)

Conclusion:

As conclusion, a few key recommendations for testing SOA initiatives would be:

1. Design your SOA project test approach alongside the definition of business requirements and highlevel technical design. Ensure the test teams are involved right from the start.
2. Static test techniques such as formal inspections are a must to ensure the business requirements and technical designs are defined to standards and are complete. Strong governance and discipline are essential to successfully delivering SOA.
3. SOA is driven by business and not technology. The test team will have to ensure the key business stakeholders and users are actively involved throughout the project life cycle and not just at the start and the end.
4. The Test team must be aligned to business domains not technologies. This will enable a successful implementation of a Risk Based approach to testing. The test team must also be technical and be able to white box test the entire SOA platform.
5. Security assessment and testing must take place throughout the entire project life cycle.
6. Many organizations will need to perform a leap of faith. The majority of testing activities now need to be performed during the business requirements analysis, design and service build phases. To achieve delivery agility and true service reuse, each individual service must be delivered to the Integration and UAT test phases with well-defined functional test coverage and guarantees on performance, scalability and security. Simply put, services must come with a Quality statement!
7. Test Tools are now a must. Organizations must invest appropriately in a test tool strategy.

You should not be surprised that many of the recommendations outlined in this paper are not new. Many of them are already defined as best practices. SOA demands strong governance, well-defined standards, processes and disciplines. If Quality is at the heart of your SOA, then the promises of SOA will be delivered.