Archive for December, 2007
‘Experimental’ Update
Fresh experimental update today, build 2221. This largely revolves around the metrics feature, which we’ve been tuning down the overall memory usage and adding in data to do impact analysis.
A new # of Methods metric is included.
| Permalink | Comments[0] |
Find in Solution Explorer
New feature as part of StudioTools today, this is available on the experimental updates version 2180 and higher.
It does what I’d hope you’d expect, finding the current file in Solution Explorer for you, focusing Solution Explorer and highlighting the file. Very handy in Solutions with a large number of files or when you are ‘new’ to a codebase. Enjoy.
| Permalink | Comments[1] |
TestMatrix Updates
Uploaded build 2179 this morning, it includes exporting the call graph to XML as a new feature, along with an error check to prevent updates when more than one Visual Studio is running. This scenario was causing the occasional file lock and a ‘partial update’ would result, so now make sure that you are running only one Visual Studio before you check for updates — the restart to have them take effect.
| Permalink | Comments[0] |
Auto-Install Restore
When files are pulled from the auto-install, they land into
C:\Users\Setup\AppData\Local\ExactMagic\install
and when they are finally installed (on your next Visual Studio startup) they are archived into
C:\Users\Setup\AppData\Local\ExactMagic\installed
Now, it is possible if you have a TestMatrixDebugShim.exe running (orphaned really) by a Visual Studio crash, that you’ll need to… ‘redo’ an auto update. Simply copy the contents on ‘installed’ to ‘install’ and restart Visual Studio.
| Permalink | Comments[0] |
TestMatrix Updated
Uploaded the 2008.2.0.2177 release this morning, and tidied up the version numbers a bit. Now the last number is tied into the Subversion build repository of the release branch.
Get this from the Check for Updates…
Main new features are saving the call trace to XML. The coverage window will now automatically open when coverage is collected, and you can turn this off in Tools | Options (thanks for David Veeneman for the screenshot!). Main bug fix is for coverage, where if the last instruction of a method was an explicit throw, it was previously possible to get an InvalidProgramException.
In the call trace, there are two new icons ‘blocked’ and ‘exception’.
- Blocked: the method time is taking 1000ms more than the CPU time, indicating some kind of lock or I/O is blocking the call
- Exception: the method exited with an exception, this is useful to quickly see how far up the stack an exception carries
| Permalink | Comments[0] |
Fancy new website!
We’ve just completed a major update to the look and feel of the website. Please let us know if we’ve broken something. We’ll be adding additional content
about the new releases after the holidays.
| Permalink | Comments[0] |
External Method Calls
To make the profiler even more useful, the latest experimental builds now show external calls. External in the case of the profile really means anything not in your solution.
This approach lets you focus on your code, and see at least one more level into referenced libraries as well as the .NET BCL. It keeps performance high by not deeply tracing into assemblies that are part of the framework that you can’t really modify. This is a performance boosting compromise with the goal of letting you pinpoint parts of your code to tune, without cluttering up your view with a lot of .NET framework internal call stacks.
External methods have a little arrow over the icon, and are grayed out indicating they aren’t in your solution …
Remember to get tracing, you’ll need to enable coverage (TestMatrix | Collect Coverage Data)
You can get this feature from the (Experimental) updates starting… now!
| Permalink | Comments[0] |
Profiling and Tracing
We’ve been working on our profiling and tracing system rather a lot in this release. The main innovation being a system of instrumentation that doesn’t require and C++, or COM, or even a profiler DLL. One of the benefits of this new approach is sheer speed at runtime and the ability to collect some really clever data during the test runs.
The most exciting thing is the Call Trace. When you are running with coverage enabled (TestMatrix | Collect Coverage Data), this call trace is populated along with the line level code coverage.
Think of this as showing you the Call Stack that you see while debugging, but after your tests are finished. This lets you review what was called, which can often be surprising to see just how many methods are involved in a test case.
This also lest you see the time and percentages of time in a method, separating out the actual clock time (Time in Method) from the CPU elapsed time (CPU in Method). Gaps in those two numbers indicate some kind of block or wait, such as I/O, Sleep, or a database call.
One thing to keep in mind, in order to be very fast, and to focus on code you can really do something about, the trace only collects data from your classes and methods, not from the base class library. Very specifically, the times for calls nested ‘under’ another call don’t sum up when you call out to the framework as those methods just aren’t in the call trace shown.
Coming soon on the online updates, we’re planning on enhancing this feature with some additional data:
- Calls out the framework from your methods, one level deep so it stays fast but so that the times ‘add up’
- Allocations of your solution’s objects, count and size - but not cluttering up the data with huge numbers of allocations of string, which is the usual problem with memory profiling.
- File, Network, and Registry I/O similar to procmon/filemon/regmon, but inline in the call trace
- Exceptional Exits - marking on the call trace when a method exited via a a thrown exception rather than returning normally
| Permalink | Comments[0] |
Tracking Exceptions while Testing
Tracking exceptions up the Call Trace lets you see where a test failure starts and to visually see how far an exception ‘leaks’ up your call stack before being handled.
The little indicates which calls exited via an exception, as opposed to exiting normally out the bottom of the function, or through a return.
You can double click on any call in the Trace and navigate there in the editor. And — you’ll see the error fingers in the gutter if this exception exited all the way out of your test case.
You can right click on the fingers and navigate up and down the exception stack trace. This feature is available even without profiling being on, as it intercepts exceptions coming out of the test engines and saves that metadata along with the test. Very handy for tracking down why a test failed.
| Permalink | Comments[0] |
Command Line Support
This has been one of the more requested features, and we’ve got it in the latest (Experimental) update downloads. The basic idea is that you can run your Visual Studio solution’s tests with any supported test engine (NUnit, CsUnit, and MbUnit currently — more on the way), as well as collect coverage.
This command line program runs the same sequence as the Visual Studio integrated test engine, but with one exception:
Build Your Solution First
the command line runner looks at the solution and project files in order to determine the location of the output assemblies in order to run tests. In terms of a NAnt script, just build your solution, then execute TestMatrix.exe as a command line task. Here is a sample run, using our self-test testing solution.
C:ExactMagictrunk>TestMatrix.exe /engine:nunit /solution:Test.sln /coverage:coverage.xml 100.00 % Pass: 24 Fail: 2 Not Run: 1 Testing complete in 00:07.032 |
The actual program is located under your user profile — where the hot updates extract. On Vista, this is generally:
C:\Users\[your username]\AppData\Local\ExactMagic\addins\TestMatrix
Copy the entire contents of this folder to wherever you’d like on your build server, or reference it directly, it has all the files you’d require. The guid-named subdirectories are the actual different test engine plugins. Take a peek at the framework.xml and you’ll get a sense as to how the system works.
This command line version doesn’t currently check for a license, so you don’t need to worry about installing Visual Studio on your build server.
| Permalink | Comments[0] |