The older I get, the closer I want my tests to mirror the interactions my customers have day-to-day in the browser. This doesn't mean I quit unit testing, but generally when I work through a bug or feature I try to simulate it end-to-end with the browser as much as possible.
Since web applications today are so complex this often requires something like Selenium. I've found these tests are the best way to reproduce a production bug in my web apps because they exercise both the server generated html and the dynamic javascript on the page.
The biggest problem with writing Selenium tests is that they take a long time to run. And if your team builds up a suite of 300+ Selenium tests you are in for a world of hurt (ie- slow feedback).
I've found that when your project's total build time gets over 60 seconds you simply don't want to run it very often. And even when you have a progressive team that works hard to build small web apps you might find yourself with a few hunderd tests to exercise all the features of that seemingly "tiny" app or component.
This has been the challenge for web developers since we started sprinkling javascript code on top of server generated html. The core problem has been that we don't have a clean separation of client and server so we are forced to test both when we just want to verify the client part of the application works.
But imagine a world where you could just verify the client (html/css/javascript) using a simple api that lets you find dom nodes/click buttons/toggle checkboxes/navigate around the app. This is the number one benefit of writing a pure javascript app using something like ember.js.
You get a ton of other benefits when you build your application using pure client technologies, but as a web developer I've been waiting for integration style tests that give me all the benefits of selenium at a fraction of the cost (ie- how long it takes to run a single test or suite of tests). So I decided to setup a fairly low tech experiment/benchmark to see just how costly running these integration tests are with Selenium.
The basic idea is that I wrote 3 integration tests with Selenium, then reproduced each of them using pure JS with QUnit / Testem / Ember-Testing. But because running just 3 tests wouldn't show much of a gap, I copy/pasted these until I had a total of 303.
If you want to see all 3 of the tests give the benchmark example a look. For brevity I wanted to focus in on just the test that removes an item from the list.
The test is really simple, just find the table and verify it has 2 people/delete one of them/verify that person is no longer shown in the client. I then created a nearly identical test in pure javascript.
The non scientific results showed that running 303 of these integration tests w/ Selenium took 1629 seconds. Compare this to the javascript version where running 306 tests only took 30 seconds. So although Selenium works, it's not the best economic model for teams that strive for a high level of test coverage.
I didn't want to inline the animated gifs I cut from each benchmark video because of the size, but if you want to download them it's worth a quick watch. Also keep in mind you can pull down the app and run the entire suite of tests yourself (just set aside 27+ minutes of your time for the Selenium run if you do so)
Here is the animated gif for the full Selenium run showing it took 27+ minutes to run 303 tests Selenium
And finally, the animated gif for the full testem run showing it took 30 seconds to run 306 tests Testem
Keep in mind this was not the most scientific test possible, but I think it does show the cost of feedback today is high for developers building traditional web applications where both server generated html and javascript are mixed together.
