Overview
With agile practices getting more popular Test Automation is important factor in achieving continuous delivery. There are so many tools available in market for test automation. It includes both Open Source and Proprietary Tools/Framework. It’s very challenging as well as interesting to pick up best tool for your test automation. Many times I have seen people choosing Open Source tools just because they are free. On the other hand it also true that most of the proprietary products are very costly. It was also challenging for me to come up with best tool for Test Automation for my project which included applications in ASP.NET and WPF.
In today’s post, I would more talk about Web Test Automation and try to compare well known open source framework “Selenium” (http://seleniumhq.org/) and proprietary tool “Test Studio” from Telerik (http://www.telerik.com/automated-testing-tools.aspx). The prior one is framework using which we need to create whole sort of test script by writing code while later one is product with nice UI. Somebody might say why I am doing comparison of open source framework with a proprietary product? The answer is – ultimate we have to come up with solution that satisfies my “requirement”. So before going ahead I have to clarify my or project’s requirement. Solution should provide: Automation of Web Application (as of now just considering ASP.NET application for ease of analysis).Test support on Internet Explorer, Firefox, Chrome and Safari
Less learning curve as my resources would be from Testing team and may not coding expert
Better productivity for creating new scripts so that it would not increase much cost for new enhancements.Better Reporting feature for easy analysis of failure scenarios
Ease of Maintenance.Overall Test Script creation and execution and maintenance cost (initial + resource) should be less.
Analysis.The first two requirements are straight forward. Both tools in context (Selenium and Test Studio) satisfy this.
The later ones are more subjective and many vary from project to project. Let’s discuss these in details one by one.Learning Curve
Selenium – To work with frameworks that mainly require script coding, developer should have knowledge of both script coding as well as test case development. This increases learning curve for script developer. Also without powerful UI support, developer need to rely on trial and error to find out bugs in script itself. I must mention here that Selenium has very good support from community and that is very useful.
Test Studio – It provides nice UI that helps person self-learning the tool. It also provides multiple views by which you can carry out your work. One can learn usage of Test Studio by view various videos available on Telerik site. Also you can ask for any help from support team.
Resource Productivity
Here I would like analyse ease of development of a complex test script by average skilled user.
Selenium – It has very good apis and skilled resource can develop script with better productivity. The usage of IDE in Firefox is limited. So for any test case one has to do sufficient coding exercise. The coding itself is bound to follow its coding life cycle including debugging and sanity testing. Reading just steps as a story board is difficult in selenium. Handling of timeouts, AJAX calls are the areas where developer can make easy mistakes. So quality of script will more dependent on experience of developer in selenium.
Test Studio – It provides complete UI based script capturing as well as modifications to script test. You can easily see story board that gives you better idea about your test case and steps followed in the same. Tool provides various UI based options for capturing element, comparing values against required properties. You can easily add data driven testing without much effort. Even logical and looping conditions can be easily incorporated using UI. Capturing of mouse and java script event is UI based. This gives lot of relief to developer and person can concentrate on Test Case logic or Business validation planning rather than coding or development. UI provides high speed in capturing element, comparing the required values and verifying the output. This is biggest advantage of Test Studio.
Ease of Maintenance
It’s an important factor where you have large number of test cases.
Selenium – People find it difficult to manage test scripts than creation. For small change also it might need to recapture whole script. It requires lot of planning for large scale project to use selenium effectively in there project. Tester needs to do debugging and find out where it’s an application failure or it’s a script issue. Many times dedicated effort has to be given to keep pace with changes incurring in agile sprints.
Test Studio – Here Test Studio has upper hand. One can create modular test cases and reuse them in various test scenarios as and when required. It provides detailed failure log that includes images of failed step and expected step. So just by viewing image one can easily do initial analysis and decide whether script to be modified or bug to be raised for application issue. Steps list and Storyboard help even new developer to understand test case better way and one can do modification easily to the same.
Overall Cost
Let’s try to put together cost statics for say 500 test cases. The productivity of 3-4 test cases per day can be achieved for Selenium while we can consider around 6-7 test cases for Test Studio. I personally believe we can achieve much more than 7 with Test Studio. Let’s assume 20 test cases need to be modified or created for code changes per month that includes modification and enhancements to system.
Item | Selenium | Test Studio |
License Cost | $0 | $2500 |
New Script Development Effort with low productivity | 167 PD (3 cases per day) | 83 PD (6 cases per day) |
New Script Development Effort with better productivity | 125 PD (4 cases per day) | 72 PD (7 cases per day) |
Script change effort per month 70% change & 30% new | 4-5 PD | 2-3 PD |
One Year Maintenance effort | 50-60 PD | 25-35 PD |
So from long term perspective and with increase in no. Of test cases Test Studio not only pays you off initial cost but also saves the overall cost.
Other benefit of using Test Studio
WPF & Silverlight
Test Studio support WPF and Silverlight application test automation.
Manual Script customization
The Test Studio generates code behind the scene which is based on Telerik Testing Framework i.e. Similar to codedui framework of Microsoft. This gives additional flexibility to developer it he/she at all need to write done manual scripts.
Performance Testing
The upcoming version (2011 R2) of Test Studio includes Performance testing with In-Depth analysis for .NET applications. I am eagerly waiting for this release.
Comparison Table
Below is the detailed comparison between Selenium and Test Studio:
Item | Selenium | Telerik Test Studio |
IDE / Script Capture | Firefox 3+ | Firefox 2+, IE 7+, Safari 5+, Chrome 7+ |
Run Test supported Browsers | Firefox 3+, IE 7+, Safari 2+, Opera 8+
Chrome doesn’t have support as of now | Firefox 2+, IE 7+, Safari 5+, Chrome 7+ |
Script Development | Firefox IDE + Manual coding Good API collection. But resource must be skilled in API
Manual coding is more time consuming and error prone Testing of script itself is required Complex cases mostly done using manual coding | Well-designed UI that can help capture all types of scripts Codebase gets generated at background for customization if required Complex cases can also be captured with ease using UI |
Supported OS | Windows, OS X, Linux, Solaris | Windows |
Browser Objects Recognition | Good object recognition. Objects can be identified by name, ID or located by using xpath.
Complex elements are not easy to capture Time consuming | Very Rich UI based object identification Easy identification of html element, DOM member Provides easy way to identify element from hierarchical view Supports mouse action, drag and drop events Support java script events of control |
Recording and Playback | Supported in Firefox | Supported in Firefox, IE, Safari, Chrome |
Capture once & Run for all browsers | Supported | Supported |
AJAX & jquery | Supported | Supported |
Java Script Alerts / Dialog | Supported | Supported |
Code Generation Language | Java, C#, Ruby, Perl, PHP | C#. Language knowledge is not mandatory. UI based application |
Data Driven Testing | Supported Need to plug in custom code for reading data from store | Supports multiple data stores like flat file, in build table, database, excel etc. Easy to add / modify data in data store |
Test Results Reports | Reporting is supported through the test runner and various logs, screen shots can also be captured.
Default Reporting is not rich | Very Rich image based reporting. Step story board or failure step comparison with image is available |
Performance Testing | Not supported | Supported in 2011 R2 getting release by end of Sep 2011. Support In-Depth details with .NET profiler for .NET application |
Price | Free | $2500 |
Conclusion
We analysed Selenium and Test Studio for our requirement. The Test Studio provides better value if we do cost benefit analysis.
The projects that are being executed on Windows environment and if you are thinking of using open source framework like Selenium or watir for cost saving, then you should try Test Studio before final decision. Even for web sites that are developed in Java or PHP can take benefit of Test Studio as Web Testing is based on HTML and browser.
