There is something called flow psychology ( http://en.wikipedia.org/wiki/Flow_(psychology) ) which describes the conditions that need to be fulfilled if you are to experience “flow” (Which at least for me is a precondition for not being caught my my mail reader). I learnt a lot from that as a project manager.

On to a question I have:

What would happen if all your tests took less than a second to run? Would you structure your tests differently?

There is something called flow psychology ( http://en.wikipedia.org/wiki/Flow_(psychology) ) which describes the conditions that need to be fulfilled if you are to experience “flow” (Which at least for me is a precondition for not being caught my my mail reader). I learnt a lot from that as a project manager.

On to a question I have:

What would happen if all your tests took less than a second to run? Would you structure your tests differently?

“If all my tests took less than a second”? Do you mean more than a second?

Most of the tests I write first are independent of infrastructure, which means that they run really fast. I try to separate out my slow-running tests (e.g. test that a search generates SQL that retrieves the correct result from the database). I haven't doing any soft of formal categorization here, though.

I find that either I run a single test in the IDE. Then it should run fast. Or I run all my tests for the command line. Then everything should be run. So I don't need to categorize stuff now.

“If all my tests took less than a second”? Do you mean more than a second?

Most of the tests I write first are independent of infrastructure, which means that they run really fast. I try to separate out my slow-running tests (e.g. test that a search generates SQL that retrieves the correct result from the database). I haven’t doing any soft of formal categorization here, though.

I find that either I run a single test in the IDE. Then it should run fast. Or I run all my tests for the command line. Then everything should be run. So I don’t need to categorize stuff now.

If all of your tests - system tests, unit tests, integration tests, the works, all completed in less than a second - would you structure your test suite differently? Would you still keep your unit tests around? Would you hang on to your CI server?

Thanks for a very interesting question. I think I might want to expand this into a full entry eventually.

I think the risk of my tests running this quickly are fairly slim. But if it did happen, I would still want to separate out the functional tests that we create in cooperation with domain experts and the system level tests that simulate realistic operating conditions. I'm happy with the system level tests we have now, which stress the system. There's no way these could run in under 1 second.

But neither of these types tests are JUnit tests. Our functional tests are implemented in FitNesse and our system tests are scripts that push data on the system (for a web based system, I would consider JMeter).

For tests implemented with JUnit, I would not see any usefulness in the distinction if things ran sufficiently fast. I might consider deleting tests that were covered sufficiently by the functional tests, but since these are run by different tools, it might be hard to find out exactly what tests could be deleted.

I would keep my CI server, though, so I was sure that changes by every member of the team (myself included!) actually passed the test suite. I might even add a failure condition if the tests ran for too long, to make sure that we didn't start down the slippery slope of slow tests.

My idea when it comes to functional testing doesn't really lend itself to fitnesse. I want to also veryify that all interfaces to the system do what they should. I can't really see that happening with fitnesse the way I want it to.

The good news is that we are almost there. I have created functional test suites which ran a full regression test for a module in less than a minute or so. The drawback was the setup time, which also was on the order of a minute. This minute was caused by spring/hibernate.

Good point about the CI server. What if eclipse (or whatever) continually ran your tests, using the second processor everyone seems to have but not use these days, just as your code gets compiled “continually” behind the scenes?

If all of your tests - system tests, unit tests, integration tests, the works, all completed in less than a second - would you structure your test suite differently? Would you still keep your unit tests around? Would you hang on to your CI server?

Thanks for a very interesting question. I think I might want to expand this into a full entry eventually.

I think the risk of my tests running this quickly are fairly slim. But if it did happen, I would still want to separate out the functional tests that we create in cooperation with domain experts and the system level tests that simulate realistic operating conditions. I’m happy with the system level tests we have now, which stress the system. There’s no way these could run in under 1 second.

But neither of these types tests are JUnit tests. Our functional tests are implemented in FitNesse and our system tests are scripts that push data on the system (for a web based system, I would consider JMeter).

For tests implemented with JUnit, I would not see any usefulness in the distinction if things ran sufficiently fast. I might consider deleting tests that were covered sufficiently by the functional tests, but since these are run by different tools, it might be hard to find out exactly what tests could be deleted.

I would keep my CI server, though, so I was sure that changes by every member of the team (myself included!) actually passed the test suite. I might even add a failure condition if the tests ran for too long, to make sure that we didn’t start down the slippery slope of slow tests.

My idea when it comes to functional testing doesn’t really lend itself to fitnesse. I want to also veryify that all interfaces to the system do what they should. I can’t really see that happening with fitnesse the way I want it to.

The good news is that we are almost there. I have created functional test suites which ran a full regression test for a module in less than a minute or so. The drawback was the setup time, which also was on the order of a minute. This minute was caused by spring/hibernate.

Good point about the CI server. What if eclipse (or whatever) continually ran your tests, using the second processor everyone seems to have but not use these days, just as your code gets compiled “continually” behind the scenes?

I think functional tests will tend to vary a lot from project to project. It really depends on what your interface to the outside world is.

A minute to start up Hibernate (Spring is seldom to blame) is unacceptable for a functional test. In my functional tests, I have replaced the whole persistence layer with a stupid in-memory replacement.

I think functional tests will tend to vary a lot from project to project. It really depends on what your interface to the outside world is.

A minute to start up Hibernate (Spring is seldom to blame) is unacceptable for a functional test. In my functional tests, I have replaced the whole persistence layer with a stupid in-memory replacement.