Exact Magic Blog
Coming Soon: “Vegas”
We are happy to announce the first details about the next release of TestMatrix for Visual Studio, code named “Vegas”. We’ve gotten lots of great feedback from our customers, and think you will all find this release very exciting. We’ve got a new platform with a lot of great features, here are some of the highlights, stay tuned to the blog for more updates and specifics.
New Platform with Plug-in Support
We have rewritten our platform to improve speed and reliability, and have introduced a new plug-in based architecture. With plug-ins, you’ll be able to install just the features you wish, and leave out those you don’t need. This will give us much better compatibility with other Visual Studio add ons, and allow us to add new features more quickly. A built in auto-update feature will keep everything up to date.
Support for Visual Studio 2008 (Orcas)
Microsoft should be releasing its next version of Visual Studio around the end of the year, and we plan to be right there with them. Many of you brave souls are already running the betas, and have asked for Orcas support. Contact us if you want to try out an early build.
Multi-Framework Unit Testing
The next release will add support for a number of additional unit testing frameworks, including MbUnit, MSTest, csUnit, and NUnit. Support will be fully transparent, and you can use several frameworks within the same project without difficulty. Great for those of you transitioning from one framework for another.
Super Fast Unit Tests
The fastest by far! We’ve added a number of features specifically designed to help speed unit test execution time. We’ve added real time code analysis so that unit tests are identified even before you have compiled and advanced dependency tracking to reduce test startup time in solutions with lots of projects. Fast, pain-free unit testing is what it’s all about.
And Much More
Lots of other enhancements and new features will be included. Improved unit coverage and reports, more flexible spell checking, and enhanced navigation tools.
Availability and Pricing
We are really excited about the new platform, and want all our users to have the best possible experience. Therefore, we’ve decided that this will be a free upgrade to all registered users. TestRunner users who upgrade to TestMatrix 1.0 before Vegas ships will also receive a free upgrade. So, buy your copy today and you’ll get the next release for free!
The final release won’t be available until the end of the year, but we will soon begin a private beta period to be followed by a public beta. We’ll begin posting more details to the blog as well, so stay tuned for more information.
If you want to get an early look at things, and are not afraid of bugs, contact us and tell us you’d like to participate in the private beta. Tell us the platform, language, and version of Visual Studio you typically use. We’ll contact you within the next couple of weeks and begin making builds available.
| Permalink | Comments[1] |
Disabling the Spell Check Feature
This question has come up a lot recently. Some people love the integrated spell checking, others hate it. The number one comment we get regarding this feature is the flagging of variable names as spelling errors.
The reasoning here was that variable names should be meaningful words, rather than abbreviations or random letters. This produces more meaningful code that is easier to read. The spell check engine is smart enough to understand camel case, so standard programmer idioms can still be applied. However, realizing that this isn’t for everyone, you do have a couple of options. To configure spell checking, bring up the options panel for Studio Tools from the Visual Studio menu:
Tools | Options
First, you can set Comments Only to true, disabling spell checking inside the code itself.
Alternatively, you can set As-You-Type Spell Checking to false to disable it all together.
You can always execute a manual spell check from the Studio Tools | Spelling menu.
Hopefully this clears up some confusion. We’d love your comments on this feature. For example, should the default state be to only spell check inside of comments? Are there other rules we should teach the engine about that would make the code level spell checking more useful?
| Permalink | Comments[1] |
TestRunner upgrade pricing
As many of you may already know, the TestMatrix product is a descendant of the popular TestRunner plugin from Mailframe, and also includes technology originally distributed as the product CodeSpell.
Good news for registered user of the earlier Mailframe products TestRunner or CodeSpell, you can upgrade your licenses to TestMatrix for a special reduced rate of $40 per user from the online store.
You will be asked to enter the registration information from your original order at checkout. You must have a valid license key to complete your upgrade order. If you have any questions or problems, please contact us.
| Permalink | Comments[0] |
Orcas Test Support
Looking forward to future versions, we’re looking at bridging the gap to provide NUnit, MBUnit, Selenium, and maybe CSUnit testing support inside of the Team System Test framework. I’m assuming folks will start moving to the official MS test solution as Orcas rolls out, which leaves you in a pickle. What do you do with all your NUnit tests? Convert them? It seems like a lot of work, and possibly error prone, and what about third party components that you use that have integrated tests, or multiple project teams on multiple schedules?
So, that leaves me with the following theory – it’s better to integrate into what will be the omnipresent MS Test solution by making plug-ins that will run all your existing tests as-is, without modification, so that you can mix and match the best test tool for the job, yet have all the results merge together into one collected test output provided by the Orcas test system.
- Mark the failure line in the editor
- Bridge Integration
- NUnit
- MBUnit
- CSUnit
- Selenium
- Allow ‘Debug Tests’ from the context menu in the editor
- Provide a Test Statistics panel with Pass/Fail/Inconclusive ratios and charts
-
Custom Test Types
- Repeating / Timed
- ‘With Coverage’
- ‘With Performance’
What do you think?
| Permalink | Comments[3] |
NUnit Integrated
TestMatrix is all about making NUnit easier and faster to use while you are programming. It’s most useful if you are already using NUnit for testing, and are looking for a faster way to run your tests, gather coverage, and debug you tests without shuffling through a lot of windows or external tools by giving you direct feedback in the editor about your test cases. It’s a free trial to download, give it a shot and it’ll save you time testing.
We’ve got
- NUnit 2.4
- Code Coverage, Memory, and Performance Profiling
- Fully 64 bit support (better than NCover!)
From your test cases, just right click and run. The menu even shows you which test, whether you are in a [TestFixture] or a [TestCase].
Right there, in the editor the test result is marked with a colored indicator (which you can pick the colors)
And most convenient, when you have a failed test the failure and root cause is show to you right there in the editor with the ‘fingers’ test failure marker and a tooltip.
And, we’ve got NUnit GUI style test ‘tree’ as a docking window.
| Permalink | Comments[0] |
More than One Way to Test it
Sometimes when I’m testing, I just want to smoke test the entire solution, particularly when I just synched up multiple changes from the rest of the team. This is when I use the TestMatrix | Test Explorer, then pick the solution in the drop down.
From here I can just hit the play arrows and run the entire solution. See the icon for solution, and the
nested assembly icon? I can click and explore the tests visually like I would with NUnit GUI but one notch more convenient by avoiding the window shuffle toggling back and forth.
I also tend to work in one assembly/project at a time with modular changes. This is where selecting a single project makes sense. The drop down is filtered to be only those projects that reference NUnit to keep it nice and short. So in exploring and running the tests I can work on the solution or a single project.
If I’m just working in a single class, doing pure TDD unit testing as I code, I tend to run the tests right from the editor with the context menu or Ctrl-R,T as a hotkey (remember it as Control Run Test).
And having run the test, the test results show up right there in the editor.
I think about it in terms of big test runs and small test runs. Big runs I tend to do with the explorer, mainly having been hooked on NUnit GUI early on. Small runs I tend to do right in the editor, similar to TestDriven.NET – but with graphical feedback right on the test case. Pass/Fail is a bit more ‘in your face’. Particularly on failed tests, which is what it’s all about. Just hover over the ‘fingers’ – the 5 horizontal markers. This shows you exactly where a test failed, and prints out the message and stack trace. You can see why your test failed without a lot of hopping around.
| Permalink | Comments[2] |
Punishing Test Coverage
Looking around for something complicated to test that folks actually use, I decided to pick on Lucene.NET. My idea was to make sure the code coverage works well on what is a complex piece of software with some really complex tests – and that it isn’t so painfully slow that nobody would ever actually use it.
I ran the tests in Lucene 1.9rc1. After a little bit of setup, and changing the NUnit references to 2.4 using the Fix NUnit References Option in the Test Explorer I ran the entire solution’s Lucene Test solution’s test using the TestExplorer. 301 tests with coverage ran in 3 minutes 9 seconds. There were a few failures in there – looks like the Lucene test run at least a few web connections that I didn’t set up my local IIS to respond:
As a comparison, I ran again with coverage disabled, I just hit the toggle button in the Test Explorer. 2 minutes 8 seconds.
Coverage does have some overhead since it computes activity on each individual line, but not an overwhelming overhead. I’ve had profiler/coverage experiences in the past that have 400% to 800% overhead – this one is just 50%.
| Permalink | Comments[0] |
It Was Mailframe
After having Mailframe and TestRunner in the market for 3 years now, we’ve decided to take things a bit more ‘company’. First off – Mailframe doesn’t say anything about what we’re doing, Exact Magic does. We’re looking to provide programmers with tools that work for you. Taking the drudgery out. Collecting data in the background. Automating common tasks. We want it to seem like magic.
StudioTools is the combination of TestRunner and CodeSpell, along with some new and rather cool navigation tools that let you get right to your types (Alt-G) or open a file by quick name match (Alt-O). We’ve incorporated hundreds of user’s feedback based on TestRunner 2005. The biggest thing, we made it a lot faster. Coverage and profiling is really fast. In our solutions, we’ve seen overhead as low as 25% in running unit tests to collect coverage metrics. The idea here is coverage that is so fast, you can leave it on all the time. Folks that have licenses for TestRunner 2005 or CodeSpell 2005 should contact us about upgrade pricing.
| Permalink | Comments[2] |