Greg Young [MVP]

While it may not be a "failure", it's not a pass either. None of your tests have passed in this case, so you haven't verified anything. That's no better than failure.

As the place I work has just started using TDD, and we're using the Visual Studio built-in tests, please take these comments with a grain of salt.

Right now I've found that with our legacy code we'll have failures more often (though not exclusively) in the "execute" part of the test and these failures will identify further "implied" requirements of the legacy framework (all written in C#, but very very complex and coupled in our case).  We try to add further tests to point out these "requirements," but we aren't always able to do so.  I would suppose this has to do with the legacy domain we're using, but it does make things difficult.

So to sum up; for us a failed setup when dealing with legacy code means that we're missing a previous test that should have been written to flesh out an implied requirement.

Yes, a test shouldn't show up as "failed" if it was never run because SetUp failed.

I think it depends on how setup fails. If it fails because of something uncontrollable (windows crash issues, out of memory, network suddenly fails, etc) then I wouldn't worry about it because either it's a much bigger issue than just my setup failing or it's a rare, probably-one-time thing that probably won't happen again (this is really rare though). If it fails because it needs something that it doesn't have, however (like a dll, write permissions on a folder, the existence of a config file, etc.) then this is something that you should have environment tests to detect. These tests verify the existence of various environment properties such as permissions, file existence, IIS settings, etc. that your app needs to run and will tell you explicitly what is missing if something goes wrong. That way, if a setup task fails, there should be a failing environment test also (or a test that can be run to determine why setup failed) or your code is just messed up as Greg says and needs to be fixed/refactored/updated/removed.

It's the difference between "Error" and "Failure". Errors are when things that are needed to exercise and teardown your fixture throw exceptions, while failures are when the test itself throws exceptions. i.e. "testing failed" vs. "the test failed".

(ok, I wrote this once and the submission failed.. gonna try again and if you get it twice, keep the original :) Sorry, my friend... you are assuming the (unit) testing framework you use should support the SetUp method. My very own testing framework (totally vaporware, of course) doesn't, so I don't have to deal with that scenario. I hear what you're saying, but if you concede that a SetUp method is simply a short-hand for things that all tests in the given unit should do before they move to the 'meat' of their test, then you should see that reporting a failure in a SetUp method as a failed test(s) is reasonable. After all, if I have 2 or more tests that need to do the same preparatory steps, without a SetUp method I'd simply call an ad-hoc function (implementing those steps) at the beginning of each of my tests. Therefore a failure in this ad-hoc method (which in your case happens to be named 'SetUp') is just a failure of my test(s) initial step. No ?

I think only the FIRST test that depends on a given SetUp should be shown as failed. Test that fail outside of the SetUp should be listed before tests that fail in the SetUp, therefore working down the list will likely get to the root problem quickly. It would be great if there was someway to automatically order tests, so that the tests for the most low level code are run first, or even the tests for classes that have just been changed are run first. A profiler looking at path lengths to the methods under test, should be able to find for each method, the tests with the shortest path length, and therefore run those tests first when the method is changed.