Published on April 27, 2014 by Toran Billups
The older I get, the closer I want my tests to mirror the interactions my customers have day-to-day in the browser. This doesn't mean I quit unit testing, but generally when I work through a bug or feature I try to simulate it end-to-end with the browser as much as possible.
The biggest problem with writing Selenium tests is that they take a long time to run. And if your team builds up a suite of 300+ Selenium tests you are in for a world of hurt (ie- slow feedback).
I've found that when your project's total build time gets over 60 seconds you simply don't want to run it very often. And even when you have a progressive team that works hard to build small web apps you might find yourself with a few hunderd tests to exercise all the features of that seemingly "tiny" app or component.
You get a ton of other benefits when you build your application using pure client technologies, but as a web developer I've been waiting for integration style tests that give me all the benefits of selenium at a fraction of the cost (ie- how long it takes to run a single test or suite of tests). So I decided to setup a fairly low tech experiment/benchmark to see just how costly running these integration tests are with Selenium.
The basic idea is that I wrote 3 integration tests with Selenium, then reproduced each of them using pure JS with QUnit / Testem / Ember-Testing. But because running just 3 tests wouldn't show much of a gap, I copy/pasted these until I had a total of 303.
If you want to see all 3 of the tests give the benchmark example a look. For brevity I wanted to focus in on just the test that removes an item from the list.
I didn't want to inline the animated gifs I cut from each benchmark video because of the size, but if you want to download them it's worth a quick watch. Also keep in mind you can pull down the app and run the entire suite of tests yourself (just set aside 27+ minutes of your time for the Selenium run if you do so)
Here is the animated gif for the full Selenium run showing it took 27+ minutes to run 303 tests Selenium
And finally, the animated gif for the full testem run showing it took 30 seconds to run 306 tests Testem