Exact Magic Software, LLC: Precision tools for serious developers


Punishing Test Coverage

Looking around for something complicated to test that folks actually use, I decided to pick on Lucene.NET. My idea was to make sure the code coverage works well on what is a complex piece of software with some really complex tests – and that it isn’t so painfully slow that nobody would ever actually use it.

I ran the tests in Lucene 1.9rc1. After a little bit of setup, and changing the NUnit references to 2.4 using the Fix NUnit References Option in the Test Explorer I ran the entire solution’s Lucene Test solution’s test using the TestExplorer. 301 tests with coverage ran in 3 minutes 9 seconds. There were a few failures in there – looks like the Lucene test run at least a few web connections that I didn’t set up my local IIS to respond:

As a comparison, I ran again with coverage disabled, I just hit the toggle button in the Test Explorer. 2 minutes 8 seconds.

Coverage does have some overhead since it computes activity on each individual line, but not an overwhelming overhead. I’ve had profiler/coverage experiences in the past that have 400% to 800% overhead – this one is just 50%.

  Permalink |  Comments[0]