Performance Testing with Open Source

Client: A company providing IT solutions for leading financial services companies

Application: An online banking application, developed in ASP.NET with a multi-tier client/server architecture

Issues: The client was developing a new version of an online banking application to be implemented by a number of different financial services companies. There had been complaints about the performance of the previous site, especially the amount of time it took to log in and retrieve account information. The client wanted to make sure that performance of the new site was acceptable and that it could handle the expected user load.

The project had a number of challenging requirements:

• The client did not have a lot of money to spend on testing tools
• The client did not have people with expertise in performance testing
• On-line banking required secure HTTPS transactions
• At least four (4) servers were involved in different parts of every user session

ProtoTest’s Solution: After analyzing the project requirements ProtoTest set about finding the right tool to generate the needed load on the servers. With our experience in open source software we found a product that could meet the project goals without the cost of expensive virtual user licenses.

Once the tool was selected a comprehensive Performance Test Plan was created according to the ProtoTest Software Quality Process. The test plan covered the approach, strategy, user scenarios, and metrics. A proof of concept test was also run to demonstrate the viability of the selected test tool. ProtoTest worked closely with client employees to mentor them on the testing process and make sure that knowledge was retained after ProtoTest was finished.

Test scripts and load scenarios were created to simulate user activity, and procedures were established for monitoring server performance. Multiple load scenarios were run and results were then analyzed and reported. When problems were uncovered we gave the client time to resolve the issues, and then we returned to retest the system. When the project was finished ProtoTest delivered a detailed report of the tests run and the performance testing project results.

Results:

• Numerous performance-related issues were uncovered and resolved during this project, and the improvements in server response times were very dramatic.
• Response times for users logging in were reduced by more than 90% from the beginning to the end of the project, from an average of over 40 seconds per user under heavy load to less than 4 seconds. In addition, login times were reduced by more than 50% from the previous production site.
• A severe hardware deficiency on one of the database servers was discovered in time to upgrade the server before launch of the product.
• Middleware processes were identified as system bottlenecks and were able to be optimized to provide substantial improvements.
• Web page sizes were reduced by as much as 80%, decreasing the amount of time to download them.
• Final production systems were able to easily handle anticipated user loads.
• Significant cost savings resulted from using an open source testing tool, which could be applied directly to improving systems rather than buying software.
• The client was extremely satisfied with the process and the outcome of this performance testing project. Not only did the client achieve considerable improvements in the speed of their systems, but also saved a lot of money in the process.