Code review comment for lp://qastaging/~klmitch/nova/os_int_tests

On Wed, 2011-04-27 at 17:32 +0000, termie wrote:
> Unrelated to gabe's response... posting this with kevin sitting behind
> me but....

Sorry for the delay in response, but my ADD and the distraction of the
summit conspired together...

> I'm hesitant to bring in a new and, no offense, largely untested test
> framework into the mix, especially one that has a rather different
> style from our existing tests, doing so would make dealing with any
> issues in it largely gated on a single person (you) and that's not
> usually a very good practice.

I understand your objections. However, I should point out that the
copyright on DTest is assigned to OpenStack LLC and that the license is
the Apache 2 license. I'm happy to give access to DTest away to other
Nova developers, and I've tried to document the code such that it's
clear what it's trying to do. That should address the "gated on a
single person" objection (tell me what else I can offer/do if it isn't
enough). As for "largely untested"--that's going to be a factor
whatever we do, I think, be it a nose plugin or DTest, and I have made
an effort to test DTest to the best of my (admittedly somewhat limited)
test-writing abilities. I'm fairly confident at this point that DTest
covers the vast majority of the subtleties inherent to a threaded test
framework, and every run I've made on the two existing DTest-based
suites (this integration merge prop and DTest's own test suite) has
worked perfectly.

> I would be much happier if DTest was additive to the existing system
> rather than completely parallel. I've done a decent amount of hacking
> on nose and unittest and have even ported unittest to other languages
> once or twice, so I am aware of how tangled a bunch of the code is in
> there, however I do think that the goals of DTest (threaded execution
> and dependencies) can be built as a new set of tools layered on top of
> those systems rather than as a completely separate entity, and I think
> in general the project would get a lot more traction that way.

Perhaps, and I'd be happy to work with you toward that end, but I know
that DTest, as it stands now, works perfectly well, and that there are
subtleties, like output capturing, that could easily be missed if we
have to do it as a plugin. That's not to say it can't be done; just to
say that I have working code now :)

I'll also point out that layering threading on top of unittest, at least
using the approach of orbitals (code for which should be available in an
earlier version of my branch) turned out to be somewhat of a headache,
and we weren't even trying to do the output capturing that I manage to
incorporate in DTest.
--
Kevin L. Mitchell <email address hidden>