Merge lp://qastaging/~klmitch/nova/os_int_tests into lp://qastaging/~hudson-openstack/nova/trunk

Wow, bunch of nice stuff. I'm a little worried it may be duplication of effort / code however, have you looked in nova/tests/integrated? It is doing much of the same stuff you are doing. It doesn't use DTest but it may be interesting for you to look into porting them to it, or adding your tests alongside them if the overlap in testing is small.

On Wed, 2011-04-20 at 21:48 +0000, termie wrote:
> Wow, bunch of nice stuff.

Thank you.

> I'm a little worried it may be duplication of effort / code however,
> have you looked in nova/tests/integrated?

No, I hadn't been aware of it.

> It is doing much of the same stuff you are doing.

The thing that worries me about it is this, from
nova/tests/integrated/__init__.py:

        """
        :mod:`integrated` -- Tests whole systems, using mock services where needed
        =================================
        """

The purpose of my integrated test suite is to test the OpenStack API of
an existing nova cluster (or huddle, or whatever term you want to use),
which is why mine uses gflags in the first place to provide connection
and login information. This is also only a base; we eventually intend
to add stress tests, to really stress out clusters and make sure they
can handle the load and do things properly.

(Incidentally, nova isn't yet quite to that state, unfortunately :/ )

> It doesn't use DTest but it may be interesting for you to look into
> porting them to it, or adding your tests alongside them if the overlap
> in testing is small.

I think it doesn't make sense to merge these two test suites, as the
purpose of the existing one seems to be testing nova during the test
run, whereas mine is more intended to exercise a nova cluster. That
said, I have certainly thought it might be a good idea to port nova's
existing unit test suite over to DTest--the parallelization would have
to be a big win. (And I'm rather proud of DTest ;)
--
Kevin L. Mitchell <email address hidden>

Alright, if you are planning on doing tests against live clusters, perhaps then look at the smoketests directory. I don't like the name of it but it is doing exactly that. I think we'd probably love to get your tests in alongside those then, as the existing ones are only really testing the ec2 api.

Regarding DTest, it looks nice and we certainly want parallelization of tests, but we also are pretty happy with most of what nose gets us, what do you think the options are for integration with nose are?

On Thu, 2011-04-21 at 17:47 +0000, termie wrote:
> Alright, if you are planning on doing tests against live clusters,
> perhaps then look at the smoketests directory. I don't like the name
> of it but it is doing exactly that. I think we'd probably love to get
> your tests in alongside those then, as the existing ones are only
> really testing the ec2 api.

Yeah, that was the one I was worried about when I started this branch.
Because I use a different test infrastructure--for parallelization--I
decided to make it a different directory, though.

> Regarding DTest, it looks nice and we certainly want parallelization
> of tests, but we also are pretty happy with most of what nose gets us,
> what do you think the options are for integration with nose are?

Well, I used nose as one of my sources for ideas and features for DTest.
I know there are some features that don't have equivalents in DTest,
like nose's plug-ins. As for integrating with nose, I haven't looked,
but I suspect it's not going to be possible; even though I've tried to
have similar concepts as unittest and nose, I've deliberately gone with
a completely separate implementation. This allowed me to omit concepts
that didn't make sense (e.g., unittest.TestSuite has no true equivalent
in DTest, and the suite() functions are never called; trying to support
TestSuite or a variant would have complicated DTest too much with too
little benefit, imo). That said, I think most things you may want to be
able to do in nose you can do in DTest, with perhaps a few exceptions (I
don't yet have support for generators yet, for instance).

(BTW, if you see bugs, or think of things that should be in DTest, log
an issue at https://github.com/klmitch/dtest/issues and I'll be happy to
address it :)
--
Kevin L. Mitchell <email address hidden>

Good to see us making progress on this level of testing on the OS API, great stuff! I have a couple comments, and I'm interested in what you and others think.

First, the name of the type of tests I think is a little off, especially given the set of tests termie pointed out. I see (at least) 5 different types of tests:

Unit Tests
Integration Tests
*****
Functional Tests
Smoke Tests
Performance Tests

Those tests above the asterisks can use mocks/fakes, etc and those below should not. It seems what you have here is in the Functional/Smoke tests range rather than integration. What do you think? Obviously we just have to pick a name and go with it, but I think "integration" is typically above the asterisk line that I drew above.

Second, I think there is a question about whether we need to use another testing framework to support our tests, or if we use nose tests to write tests and something else to manage parallelization and dependencies.

On Tue, 2011-04-26 at 18:44 +0000, Gabe Westmaas wrote:
> First, the name of the type of tests I think is a little off,
> especially given the set of tests termie pointed out. [...] It seems
> what you have here is in the Functional/Smoke tests range rather than
> integration. What do you think? Obviously we just have to pick a
> name and go with it, but I think "integration" is typically above the
> asterisk line that I drew above.

When the project was presented to me, it was described as an
"integration test suite." That was the name I chose for the directory,
and it wasn't until later, when we involved Rackspace QA guys, that we
started discussing the differences between what constitutes an
integration test vs. a functional test. We came to the conclusion that
these were actually functional tests, but I decided to hold off on
renaming things until the code was written. I'm certainly happy to
rename the suite, now that we have it pretty much complete (at least to
this point; there's talk of adding further tests to perform stressing,
like trying to create 1000 instances simultaneously and that sort of
thing). The only reason I haven't gone ahead and renamed it is just
because I wanted to see some discussion on the merge prop to discover a
consensus.

> Second, I think there is a question about whether we need to use
> another testing framework to support our tests, or if we use nose
> tests to write tests and something else to manage parallelization and
> dependencies.

The problem is that adding parallelization after the fact is as
difficult as adding security after the fact. You often have to think
about a problem differently if you're trying to solve that problem in a
parallel fashion rather than a sequential fashion. This came up in the
course of developing this test suite, in fact: we started with a simple
adaptation of unittest that incorporated threading at the suite level,
and which involved only a single extra file (orbitals.py). However,
when we then needed to add support for class-level test fixtures
(setUpClass/tearDownClass), Trey and I spent 2 or 3 hours discussing how
to do it, and the solution we came up with still wasn't very optimal.
Then there was the question of what to do if we need module-level or,
heaven forbid, package-level test fixtures. Another problem was that
the question of which tests could be run in parallel had to be managed
directly by the user in a more-or-less sideways fashion; orbitals
required every test to be explicitly listed as part of a suite() method
for all the test classes.

As it happens, though, I had already solved this problem for a C unit
test framework I had written for some of the C libraries I've worked on.
I had designed that framework (I called it "test-harness") to run test
programs, but I'd incorporated the idea of dependencies ("If program A
fails, it makes no sense to run tests that require the interfaces A
tests to work"); I had then added some support to run several test
programs in parallel. That dependency-based paradigm translated easily
into the Python DTest framework that I then wrote. I should point out
that I did try to keep several of the unittest paradigms a...

Unrelated to gabe's response... posting this with kevin sitting behind me but....

I'm hesitant to bring in a new and, no offense, largely untested test framework into the mix, especially one that has a rather different style from our existing tests, doing so would make dealing with any issues in it largely gated on a single person (you) and that's not usually a very good practice.

I would be much happier if DTest was additive to the existing system rather than completely parallel. I've done a decent amount of hacking on nose and unittest and have even ported unittest to other languages once or twice, so I am aware of how tangled a bunch of the code is in there, however I do think that the goals of DTest (threaded execution and dependencies) can be built as a new set of tools layered on top of those systems rather than as a completely separate entity, and I think in general the project would get a lot more traction that way.

I have some example POC code i can show you as well, will update this bug again when I upload it.

Here you go... sort of messy but I can walk you through it: https://github.com/termie/nova/tree/parallel_tests

On Wed, 2011-04-27 at 17:34 +0000, termie wrote:
> Here you go... sort of messy but I can walk you through it:
> https://github.com/termie/nova/tree/parallel_tests

I'd appreciate it if you did; this looks like a copy of the nova tree,
and I don't immediately see the nose plugin you were talking about.
--
Kevin L. Mitchell <email address hidden>

On Wed, 2011-04-27 at 17:32 +0000, termie wrote:
> Unrelated to gabe's response... posting this with kevin sitting behind
> me but....

Sorry for the delay in response, but my ADD and the distraction of the
summit conspired together...

> I'm hesitant to bring in a new and, no offense, largely untested test
> framework into the mix, especially one that has a rather different
> style from our existing tests, doing so would make dealing with any
> issues in it largely gated on a single person (you) and that's not
> usually a very good practice.

I understand your objections. However, I should point out that the
copyright on DTest is assigned to OpenStack LLC and that the license is
the Apache 2 license. I'm happy to give access to DTest away to other
Nova developers, and I've tried to document the code such that it's
clear what it's trying to do. That should address the "gated on a
single person" objection (tell me what else I can offer/do if it isn't
enough). As for "largely untested"--that's going to be a factor
whatever we do, I think, be it a nose plugin or DTest, and I have made
an effort to test DTest to the best of my (admittedly somewhat limited)
test-writing abilities. I'm fairly confident at this point that DTest
covers the vast majority of the subtleties inherent to a threaded test
framework, and every run I've made on the two existing DTest-based
suites (this integration merge prop and DTest's own test suite) has
worked perfectly.

> I would be much happier if DTest was additive to the existing system
> rather than completely parallel. I've done a decent amount of hacking
> on nose and unittest and have even ported unittest to other languages
> once or twice, so I am aware of how tangled a bunch of the code is in
> there, however I do think that the goals of DTest (threaded execution
> and dependencies) can be built as a new set of tools layered on top of
> those systems rather than as a completely separate entity, and I think
> in general the project would get a lot more traction that way.

Perhaps, and I'd be happy to work with you toward that end, but I know
that DTest, as it stands now, works perfectly well, and that there are
subtleties, like output capturing, that could easily be missed if we
have to do it as a plugin. That's not to say it can't be done; just to
say that I have working code now :)

I'll also point out that layering threading on top of unittest, at least
using the approach of orbitals (code for which should be available in an
earlier version of my branch) turned out to be somewhat of a headache,
and we weren't even trying to do the output capturing that I manage to
incorporate in DTest.
--
Kevin L. Mitchell <email address hidden>

I'm currently working on refactoring this branch a little bit so it can be used with either nose or DTest. I'm not going to try to do the threading in nose, and I may or may not be able to get dependencies to work; I'll leave the former exclusively to DTest, and the latter may involve some deep surgery to nose.

The integration suite can now be run under either DTest (multi-threaded) or nose (single-threaded). By default, it runs under nose. Support has been added to make nose honor dependencies. To run integration tests under DTest, pass the --use_dtest option to run_tests.py.