On Tue, 2011-04-26 at 18:44 +0000, Gabe Westmaas wrote:
> First, the name of the type of tests I think is a little off,
> especially given the set of tests termie pointed out.  [...]  It seems
> what you have here is in the Functional/Smoke tests range rather than
> integration.  What do you think?  Obviously we just have to pick a
> name and go with it, but I think "integration" is typically above the
> asterisk line that I drew above.

When the project was presented to me, it was described as an
"integration test suite."  That was the name I chose for the directory,
and it wasn't until later, when we involved Rackspace QA guys, that we
started discussing the differences between what constitutes an
integration test vs. a functional test.  We came to the conclusion that
these were actually functional tests, but I decided to hold off on
renaming things until the code was written.  I'm certainly happy to
rename the suite, now that we have it pretty much complete (at least to
this point; there's talk of adding further tests to perform stressing,
like trying to create 1000 instances simultaneously and that sort of
thing).  The only reason I haven't gone ahead and renamed it is just
because I wanted to see some discussion on the merge prop to discover a
consensus.

> Second, I think there is a question about whether we need to use
> another testing framework to support our tests, or if we use nose
> tests to write tests and something else to manage parallelization and
> dependencies.

The problem is that adding parallelization after the fact is as
difficult as adding security after the fact.  You often have to think
about a problem differently if you're trying to solve that problem in a
parallel fashion rather than a sequential fashion.  This came up in the
course of developing this test suite, in fact: we started with a simple
adaptation of unittest that incorporated threading at the suite level,
and which involved only a single extra file (orbitals.py).  However,
when we then needed to add support for class-level test fixtures
(setUpClass/tearDownClass), Trey and I spent 2 or 3 hours discussing how
to do it, and the solution we came up with still wasn't very optimal.
Then there was the question of what to do if we need module-level or,
heaven forbid, package-level test fixtures.  Another problem was that
the question of which tests could be run in parallel had to be managed
directly by the user in a more-or-less sideways fashion; orbitals
required every test to be explicitly listed as part of a suite() method
for all the test classes.

As it happens, though, I had already solved this problem for a C unit
test framework I had written for some of the C libraries I've worked on.
I had designed that framework (I called it "test-harness") to run test
programs, but I'd incorporated the idea of dependencies ("If program A
fails, it makes no sense to run tests that require the interfaces A
tests to work"); I had then added some support to run several test
programs in parallel.  That dependency-based paradigm translated easily
into the Python DTest framework that I then wrote.  I should point out
that I did try to keep several of the unittest paradigms as well, so
that DTest would be at least a little familiar to most Python
programmers.  I was also inspired by some (OK, many!) of the nose
features, so I included many of those in DTest as well...and I intend to
add others once I have time to sit down and think out how to do them
(things like support for generators).

The major feature of DTest, for our purposes in this "integration" test
suite, is that it, for the most part, handles scheduling which tests
need run when, while still allowing test writers to manipulate the
ordering if needed.  This makes it easier for anyone to write tests,
whether they understand threading or not, and yet we still get the time
win of having several tests run in parallel (one of the reasons I think
it would be great to adapt the existing ~600 unit test suite over to
DTest).

Is DTest the only way to solve the problem?  Of course not; but it is a
good way, it's a whole lot easier than trying to get nose to do it for
us (we'd have to figure out how to write a plugin that replaces a
fundamental test scheduling algorithm!), and DTest being separate from
nova allows anyone to use it (or extend it, or add features to it) if
they wish.

I should, perhaps, point out that the dependencies--which give DTest its
name--also happen to solve the test fixtures problem I described above:
DTest sets up the dependencies automatically to cause set up fixtures to
run before all affected tests, and tear down fixtures don't run until
all the affected tests have finished.  This ensures that, for instance,
test Nova instances are created before they are referred to, or that
test Glance images have been uploaded before they are accessed.  That
this happens for free is evidence that DTest is a good solution to the
parallel testing problem.

I hope this clarifies why I've taken the approach I have to this test
suite, but please feel free to ask other questions.  Gabe and I actually
talked this question out in person at the conference, but I'm happy to
discuss the test suite either here or in person with anyone; just hunt
me down and ask away!
-- 
Kevin L. Mitchell <email address hidden>