.NET Web Test Automation: Selenium vs. Telerik’s Test Studio

Overview

With agile practices getting more popular Test Automation is important factor in achieving continuous delivery. There are so many tools available in market for test automation. It includes both Open Source and Proprietary Tools/Framework. It’s very challenging as well as interesting to pick up best tool for your test automation. Many times I have seen people choosing Open Source tools just because they are free. On the other hand it also true that most of the proprietary products are very costly. It was also challenging for me to come up with best tool for Test Automation for my project which included applications in ASP.NET and WPF.

In today’s post, I would more talk about Web Test Automation and try to compare well known open source framework “Selenium” (http://seleniumhq.org/) and proprietary tool “Test Studio” from Telerik (http://www.telerik.com/automated-testing-tools.aspx). The prior one is framework using which we need to create whole sort of test script by writing code while later one is product with nice UI. Somebody might say why I am doing comparison of open source framework with a proprietary product? The answer is – ultimate we have to come up with solution that satisfies my “requirement”. So before going ahead I have to clarify my or project’s requirement. Solution should provide:

Automation of Web Application (as of now just considering ASP.NET application for ease of analysis).Test support on Internet Explorer, Firefox, Chrome and Safari

Less learning curve as my resources would be from Testing team and may not coding expert

Better productivity for creating new scripts so that it would not increase much cost for new enhancements.Better Reporting feature for easy analysis of failure scenarios

Ease of Maintenance.Overall Test Script creation and execution and maintenance cost (initial + resource) should be less.

Analysis.The first two requirements are straight forward. Both tools in context (Selenium and Test Studio) satisfy this.

The later ones are more subjective and many vary from project to project. Let’s discuss these in details one by one.Learning Curve

Selenium – To work with frameworks that mainly require script coding, developer should have knowledge of both script coding as well as test case development. This increases learning curve for script developer. Also without powerful UI support, developer need to rely on trial and error to find out bugs in script itself. I must mention here that Selenium has very good support from community and that is very useful.

Test Studio – It provides nice UI that helps person self-learning the tool. It also provides multiple views by which you can carry out your work. One can learn usage of Test Studio by view various videos available on Telerik site. Also you can ask for any help from support team.

Resource Productivity

Here I would like analyse ease of development of a complex test script by average skilled user.

Selenium – It has very good apis and skilled resource can develop script with better productivity. The usage of IDE in Firefox is limited. So for any test case one has to do sufficient coding exercise.  The coding itself is bound to follow its coding life cycle including debugging and sanity testing. Reading just steps as a story board is difficult in selenium.  Handling of timeouts, AJAX calls are the areas where developer can make easy mistakes. So quality of script will more dependent on experience of developer in selenium.

Test Studio – It provides complete UI based script capturing as well as modifications to script test. You can easily see story board that gives you better idea about your test case and steps followed in the same. Tool provides various UI based options for capturing element, comparing values against required properties. You can easily add data driven testing without much effort. Even logical and looping conditions can be easily incorporated using UI. Capturing of mouse and java script event is UI based. This gives lot of relief to developer and person can concentrate on Test Case logic or Business validation planning rather than coding or development. UI provides high speed in capturing element, comparing the required values and verifying the output. This is biggest advantage of Test Studio.

Ease of Maintenance

It’s an important factor where you have large number of test cases.

Selenium – People find it difficult to manage test scripts than creation. For small change also it might need to recapture whole script. It requires lot of planning for large scale project to use selenium effectively in there project. Tester needs to do debugging and find out where it’s an application failure or it’s a script issue. Many times dedicated effort has to be given to keep pace with changes incurring in agile sprints. 

Test Studio – Here Test Studio has upper hand. One can create modular test cases and reuse them in various test scenarios as and when required. It provides detailed failure log that includes images of failed step and expected step. So just by viewing image one can easily do initial analysis and decide whether script to be modified or bug to be raised for application issue. Steps list and Storyboard help even new developer to understand test case better way and one can do modification easily to the same.

Overall Cost

Let’s try to put together cost statics for say 500 test cases. The productivity of 3-4 test cases per day can be achieved for Selenium while we can consider around 6-7 test cases for Test Studio. I personally believe we can achieve much more than 7 with Test Studio. Let’s assume 20 test cases need to be modified or created for code changes per month that includes modification and enhancements to system.

Item	Selenium	Test Studio
License Cost	$0	$2500
New Script Development Effort with low productivity	167 PD (3 cases per day)	83 PD (6 cases per day)
New Script Development Effort with better productivity	125 PD (4 cases per day)	72 PD (7 cases per day)
Script change effort per month 70% change & 30% new	4-5 PD	2-3 PD
One Year Maintenance effort	50-60 PD	25-35 PD

So from long term perspective and with increase in no. Of test cases Test Studio not only pays you off initial cost but also saves the overall cost. 

Other benefit of using Test Studio

WPF & Silverlight

Test Studio support WPF and Silverlight application test automation.

Manual Script customization

The Test Studio generates code behind the scene which is based on Telerik Testing Framework i.e. Similar to codedui framework of Microsoft. This gives additional flexibility to developer it he/she at all need to write done manual scripts.

Performance Testing

The upcoming version (2011 R2) of Test Studio includes Performance testing with In-Depth analysis for .NET applications. I am eagerly waiting for this release.

Comparison Table

Below is the detailed comparison between Selenium and Test Studio:

Item	Selenium	Telerik Test Studio
IDE / Script Capture	Firefox 3+	Firefox 2+, IE 7+, Safari 5+, Chrome 7+
Run Test supported Browsers	Firefox 3+, IE 7+, Safari 2+, Opera 8+ Chrome doesn’t have support as of now	Firefox 2+, IE 7+, Safari 5+, Chrome 7+
Script Development	Firefox IDE + Manual coding Good API collection. But resource must be skilled in API Manual coding is more time consuming and error prone Testing of script itself is required Complex cases mostly done using manual coding	Well-designed UI that can help capture all types of scripts Codebase gets generated at background for customization if required Complex cases can also be captured with ease using UI
Supported OS	Windows, OS X, Linux, Solaris	Windows
Browser Objects Recognition	Good object recognition. Objects can be identified by name, ID or located by using xpath. Complex elements are not easy to capture Time consuming	Very Rich UI based object identification Easy identification  of html element, DOM member Provides easy way to identify element from hierarchical view Supports mouse action, drag and drop events Support java script events of control
Recording and Playback	Supported in Firefox	Supported in Firefox, IE, Safari, Chrome
Capture once & Run for all browsers	Supported	Supported
AJAX & jquery	Supported	Supported
Java Script Alerts / Dialog	Supported	Supported
Code Generation Language	Java, C#, Ruby, Perl, PHP	C#. Language knowledge is not mandatory. UI based application
Data Driven Testing	Supported Need to plug in custom code for reading data from store	Supports multiple data stores like flat file, in build table, database, excel etc. Easy to add / modify data in data store
Test Results Reports	Reporting is supported through the test runner and various logs, screen shots can also be captured. Default Reporting is not rich	Very Rich image based reporting. Step story board or failure step comparison with image is available
Performance Testing	Not supported	Supported in 2011 R2 getting release by end of Sep 2011. Support In-Depth details with .NET profiler for .NET application
Price	Free	$2500

Conclusion

We analysed Selenium and Test Studio for our requirement. The Test Studio provides better value if we do cost benefit analysis.

The projects that are being executed on Windows environment and if you are thinking of using open source framework like Selenium or watir for cost saving, then you should try Test Studio before final decision. Even for web sites that are developed in Java or PHP can take benefit of Test Studio as Web Testing is based on HTML and browser. 

Posted in: