Coded UI Test: Technologies Evaluation

As of this Christmas day, this is my assessment of the main technologies & products out there supporting Coded UI Testing. For those who don't know, Coded UI Testing is about programmers writing tests in code that automate functional testing by ways of replaying user actions on an application.

Product Comparison:

Category	MS Coded UI Tests	Selenium	Awesomium	PhantomJs	Watir	TestComplete	Unified Functional Testing (UFT)
Cross Browser Support	Not by default. Requires Selenium components to be installed.	Yes Requires WebDrivers.	No (Uses Chrome core)	No	Yes	Yes	No
Ease of programming	A bit clunky unless additional wrapper library installed: CodedUI CodeFirst	Fairly simple	Clunky and too low level. Missing documentation.	Cool if you like JS programming	Cool if you like Ruby. A bit hard to setup.	Very hard to use. Not a full programming language	Clunky, using VBScript
Testing Framework Support	TestTools.UITesting	NUnit fits well	Any .Net testing framework	Many	test-unit fits well	N\A	N\A
Cost	Requires VS Premium / Ultimate. We currently just have Professional.	Free (Apache 2.0)	Professional license ($$)	Free (BSD)	Free (BSD)	Expensive ($$)	Proprietary ($$)
Headless Support	No	Yes (HTMLUnit)	Offscreen view	Yes	Yes	No	No
Record and Playback	Yes (can record via IE only but can do cross browser playback)	Can be recorded with Selenium IDE (Firefox plugin)	No	Not built-in. Can be done in Resurrectio	Minimal	Yes but with issues when playback cross browser.	Yes

The order of the category is in decreasing importance. For my use case, cross browser support is the most important because the applications under test are websites requiring to be run on multiple browser types. In the mean time, record and playback is least important as this style of test generation create brittle tests that can be run on a browser not another, and changing the application UI results in broken tests that are hard to maintain.

In short, Selenium is the clear winner if you want the most mature Coded UI testing support, plus it is free. Microsoft's Coded UI test is a runner up for its close integration with Visual Studio. But price is a factor since this feature is only available in the premium or ultimate edition. Watir is third to me as it is somewhat hard to setup on Windows and the documentation is a bit sketchy. But if you are in the Ruby camp, you definitely should give this a try.