Merge lp://qastaging/~doanac/uci-engine/webui-amqp into lp://qastaging/uci-engine

PASSED: Continuous integration, rev:554
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/812/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/812/rebuild

Can we have an integration test to go with this? A simple HTTP GET on /status would've caught this:

http://imgur.com/pXsNmNh

Note that the TS availability is already covered by many tests (tests/test_ticket_system.py and tests/test_integration.py).

I think we should keep our integration tests as slim as possible for now, and only verify/document user-visible-features, at least until we have better performance. It's not that we don't want granularity and precise verification, but 12 minutes for each test makes you think better about what benefits are you bringing to the project.

In fact, in the integration test scope, we could simply (and efficiently) test CLI and WEBUI interactions, because currently those are the only ways users interact with the system (i.e. if the TS is not available, creating a new ticket will not pass).

> Can we have an integration test to go with this? A simple HTTP GET on /status
> would've caught this:
>
> http://imgur.com/pXsNmNh

sure - i thought we had a test for this already.

> Can we have an integration test to go with this? A simple HTTP GET on /status
> would've caught this:
>
> http://imgur.com/pXsNmNh

Just to note: in my new "pre-baked" setup world an integration test wouldn't have caught this because our base image comes with this package pre-installed. This is a good example of "the only thing you can really trust is testing a real deployment".

Agreed and yes, I ran into this as well with bakery. This is why we shouldn't use pre-baked images for testing. We should never assume dependencies just exist in the image as we're not building the images for Prodstack.

> Note that the TS availability is already covered by many tests
> (tests/test_ticket_system.py and tests/test_integration.py).

The test I was asking for wasn't looking at the ticket system. It was merely exercising that viewing the /status page on the webui did not produce an exception.

> In fact, in the integration test scope, we could simply (and efficiently)
> test CLI and WEBUI interactions, because currently those are the only
> ways users interact with the system (i.e. if the TS is not available,
> creating a new ticket will not pass).

That's a fair point, though we probably want to test specific circumstances which are hard or too time consuming to simulate without going straight to the underlying components. I think we should also test the contracts between components.

> > Note that the TS availability is already covered by many tests
> > (tests/test_ticket_system.py and tests/test_integration.py).
>
> The test I was asking for wasn't looking at the ticket system. It was merely
> exercising that viewing the /status page on the webui did not produce an
> exception.

Right, you have a point here, we do need a "User can view system status information clicking on <Status>." story verified/documented.

> > In fact, in the integration test scope, we could simply (and efficiently)
> > test CLI and WEBUI interactions, because currently those are the only
> > ways users interact with the system (i.e. if the TS is not available,
> > creating a new ticket will not pass).
>
> That's a fair point, though we probably want to test specific circumstances
> which are hard or too time consuming to simulate without going straight to the
> underlying components. I think we should also test the contracts between
> components.

Regarding testing components contracts/APIs, once we draw this like about only being allowed to interact with the system the way users do (CLI & Browser), we will find it complex to check for internal components specifics, since they are not entirely exposed to users.

Instead of checking contracts, and have to instrument the system for it, we could check behaviour, following the idea mentioned some time ago for creating a ticket and walk it through the the whole process (create ticket, custom workflow, override/revision, status, artifacts, etc).

On 10 June 2014 16:59, Celso Providelo <email address hidden> wrote:
> Regarding testing components contracts/APIs, once we draw this like about only being allowed to interact with the system the way users do (CLI & Browser), we will find it complex to check for internal components specifics, since they are not entirely exposed to users.
>
> Instead of checking contracts, and have to instrument the system for it, we could check behaviour, following the idea mentioned some time ago for creating a ticket and walk it through the the whole process (create ticket, custom workflow, override/revision, status, artifacts, etc).

Yes, I don't disagree that this should be the primary focus, but I
don't think it can be entirely in absence of tests against the
services themselves.

There will be workflows through the system that won't touch the ticket
system at all. Britney talking to the components of the Airline won't
need to create tickets - it will just talk to the Test Runner. So we
should have tests that verify the test runner API calls that Britney
uses.

> On 10 June 2014 16:59, Celso Providelo <email address hidden> wrote:
[snip]
> Yes, I don't disagree that this should be the primary focus, but I
> don't think it can be entirely in absence of tests against the
> services themselves.
>
> There will be workflows through the system that won't touch the ticket
> system at all. Britney talking to the components of the Airline won't
> need to create tickets - it will just talk to the Test Runner. So we
> should have tests that verify the test runner API calls that Britney
> uses.

Agreed, I see what you mean now. In that case there will be already an *external* API to be called (as opposed to one that have to be created only for checking if the tst_runner is there) and tested and Britney will be treated as a client expecting <something> to work in this story test.

revno 555 should fix things. I've also got some other ideas we can start building off this with.

Big +1. Thanks Andy

PASSED: Continuous integration, rev:555
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/817/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/817/rebuild