Merge lp://qastaging/~doanac/uci-engine/restish-logging into lp://qastaging/uci-engine

[r=Chris Johnston, PS Jenkins bot] Add missing testtools import. from Vincent Ladeuil

[r=PS Jenkins bot, Celso Providelo] restish-charm: complete transition to charmhelpers

This moves the restish charm completely to charm helpers and
adds a little more unit testing from Andy Doan

[r=Chris Johnston, PS Jenkins bot] ppa-cleaner: fix regression caused by recent django charm change

The lates django charm changes screwed up the python path
used by the upstart charm. I fixed the upstart charm, and this
is all we need fixed in trunk.

We actually have code that catches this regression, it just got
ignored. from Andy Doan

[r=Francis Ginther, PS Jenkins bot] ticket-system: add concept of "reviews"

This allows us to attach reviews to a ticket. The reviews can be
mapped to workflow steps so that the lander can be paused at the
point a review is needed. Once all reviews are completed, the
ticket-system will resume the ticket by notifying the lander. from Andy Doan

[r=Andy Doan, PS Jenkins bot] Provides the 'series' parameter to the test runner 1328193 from Vincent Ladeuil

[r=PS Jenkins bot, Evan Dandrea] Skip charm and integration tests when check_environment complains about missing env vars 1319703 from Vincent Ladeuil

[r=Vincent Ladeuil, PS Jenkins bot] feature probing: support juju-lxc

This allows our feature probing to account for juju-lxc deployments from Andy Doan

[r=Francis Ginther] Update apache charm revno to include apt-get update fix from Chris Johnston

[r=PS Jenkins bot, Celso Providelo] Update rabbitmq-server revision number to fix apt-get install problem 1328864 from Chris Johnston

[r=Andy Doan, PS Jenkins bot] Refactoring 'get_image' CLI action to download images using tempurls provided by Gatekeeper. from Celso Providelo

[r=Andy Doan, PS Jenkins bot, Vincent Ladeuil] Silencing the virtualenv installation in the integration tests. from Celso Providelo

[r=PS Jenkins bot, Celso Providelo] Switching lander to call publisher worker and run ppa_sync.copy directly, instead of running the ppa_sync script. from Ursula Junque

[r=PS Jenkins bot, Celso Providelo] Switching lander to call publisher worker and run ppa_sync.copy directly, instead of running the ppa_sync script. from Ursula Junque

[r=Ursula Junque] Import lp:cupstream2distro rev 605. from Francis Ginther

[r=Celso Providelo, PS Jenkins bot] restish charm: support pgsql relations

This allows the restish charm to pair up with a postgres server. It
stores the DB settings as a json file like:

 {
   "database": "ci-airline-gatekeeper-restish",
   "schema_user": "db_2_ci_airline_gatekeeper_restish_schema",
   "schema_password": "mGHMMdLp5hWwb9nqW7whwCttHhCWfLP3Cm9Y62b3",
   "host": "10.0.3.169",
   "user": "db_2_ci_airline_gatekeeper_restish",
   "password": "mLL8s7dsZ82sPdzyV5dyYwSrwCRcrcPffdbfNSmfXg",
   "port": "5432"
 }
  from Andy Doan

[r=PS Jenkins bot, Celso Providelo] restish charm: allow deploying the python-pyramid framework from Andy Doan

[r=PS Jenkins bot, Francis Ginther] Fixes cupstream2distro unit tests. from Ursula Junque

[r=PS Jenkins bot, Vincent Ladeuil] Installing python-apt for testing charms. from Celso Providelo

[r=Evan Dandrea, PS Jenkins bot] build-essential is required to install python-apt. 1329688 from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] Bootstrap run-tests tests. from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] Change test ids to better reflect their location on disk. 1285647 from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] Determine deployment env in order to determine if sshuttle is needed from Chris Johnston

[r=Francis Ginther] Fix minor oversite from sshuttle workaround from Chris Johnston

[r=Vincent Ladeuil, PS Jenkins bot] Don't rebuild charms/precise/*/hooks/charmhelpers if there is no need to. Shaves nearly 24 seconds off subsequent deployment runs. from Evan Dandrea

[r=Evan Dandrea] Handle the test selection regexps in load_testing_tests. from Vincent Ladeuil

[r=Evan Dandrea] Better isolate TestFloatingIP so it passes even if dev defines a JUJU_ENV before running the tests 1330420 from Vincent Ladeuil

[r=Evan Dandrea] Move webui cm.py one level up to align with other charms from Vincent Ladeuil

[r=Francis Ginther, PS Jenkins bot] Add a tool to bake an image with our dependencies, upload it, and point the juju image metadata at it. from Evan Dandrea

[r=Vincent Ladeuil, PS Jenkins bot] Fixing integration tests failures (test_integration -> replace venv by the corresponding sys.paths for speed and test_ticket_system -> setup new webui configuration) from Celso Providelo

[r=Celso Providelo, PS Jenkins bot] restish charm: support db migrations

This is a super simplistic way, but should work. You simply add something
like:

 db_migration_cmd: ./nf-data-collector/migrate.sh

to your deployer yaml and the script will get run when the pgsql relation
changes or the charm is upgraded
  from Andy Doan

[r=Evan Dandrea, PS Jenkins bot] Support configurable tempurl "public_host" (different than OS_AUTH_URL hostname) to cope with the prodstack scenario (see https://rt.admin.canonical.com/Ticket/Display.html?id=72096). from Celso Providelo

[r=Paul Larson, PS Jenkins bot] Extend --concurrency support to all tests except for the integration ones from Vincent Ladeuil

[r=Vincent Ladeuil, PS Jenkins bot] Including ci_utils in the test sys.path so CLI runs correctly. from Celso Providelo

[r=Joe Talbott] webui: fix live-logging view

This was broke during the move to django. from Andy Doan

[r=Paul Larson, PS Jenkins bot] Source package re-signing script and a couple of unit tests. from Parameswaran Sivatharman

[r=PS Jenkins bot, Francis Ginther] Using datastore for fetching files from Swift in BSB, so it works with private files uploaded via Gatekeeper. from Celso Providelo

[r=PS Jenkins bot, Joe Talbott] webui: add a read-only "reviews" page

The lander now has the ability to "pause" a ticket. A paused ticket
can mean one of two things:

1) the ticket needs one or more reviews performed
2) the ticket needs to retry a step

This handles displaying a ticket "needs attention" and provides a link
to see what actions need to happen. For now it just shows the missing
actions. The next step will be to add buttons to allow the ticket to continue. from Andy Doan

[r=Francis Ginther, PS Jenkins bot] restish charm: fix db-migrate bug

the pgsql relation join/change doesn't always include all the data
required to make a db connection. This ensures we only call the
migration method once all data is ready.
  from Andy Doan

[r=Evan Dandrea, PS Jenkins bot] Remove mount-callback-umount script from imagebuilder from Paul Larson

[r=PS Jenkins bot, Celso Providelo, Andy Doan] Assume a valid deployment, rather than asking amulet to check if it has one and built it out otherwise with every, single, test. This runs the 30 integration tests in 196.538s under LXC. Further improvements can be made. from Evan Dandrea

[r=PS Jenkins bot, Celso Providelo] Fix the last know integration test failure. 1332018 from Vincent Ladeuil

[r=PS Jenkins bot, Andy Doan, Francis Ginther] Extending deploy.py for pulling external charms branches into the working-dir even when it's in "build-only" mode. This is part of the IS requirements for staging deployment. from Celso Providelo

[r=Andy Doan, PS Jenkins bot] Series of assorted small fixes related to the task-force for staging deployment and acceptance tests. from Celso Providelo

[r=Andy Doan, PS Jenkins bot] ticket-system - Add openid and oauth2 support.

  * This is essentially a NOP at this point and shouldn't affect deployments as the
    API does not yet require authentication.
  from Joe Talbott

[r=Evan Dandrea, PS Jenkins bot] rabbit-worker charm: update logs directory

This makes the upstart logging location for our rabbit workers consistent
to what i/s requires. The log directories will become:

 /srv/ci-airline-imagebuild-worker/logs/upstart.log
 /srv/ci-airline-bsb-worker/logs/upstart.log

 etc.

This had a side-effect of changing the upstart service names, which
I fixed the integration test cases for (as well as added one for
bsbuilder since one didn't exist)
  from Andy Doan

[r=PS Jenkins bot, Evan Dandrea] Ensures setup.sh fails when any setup.py component fails. 1333590 from Vincent Ladeuil

[r=Francis Ginther, PS Jenkins bot] Fallback to current dir (/srv/<service>) if HOME is not set for the test runner worker. 1331989 from Vincent Ladeuil

[r=PS Jenkins bot, Evan Dandrea] Add pep8/flake tests for juju-deployer and tests test files and fix found issues. from Vincent Ladeuil

[r=PS Jenkins bot, Evan Dandrea] lander charm: fix logging

i/s confirmed logging for this componet should go to:

 /srv/ci-airline-lander/logs

This required a few changes:

1) change the "service_dir" to use dashes and not underscores
This made a new upstart service name. This broke that json_status cgi
script *again*. So I tried to change this in a way that it can be
updated in the future without breakage.

2) update the "lander logs" location
We were logging results from processing tickets under "logs". I chaged
this to "lander-logs" so it won't collide with our upstart logs.

3) make the charm / upstart script smarter
This gets us logging in the correct place, but it also lets the upstart
job ensure the proper directories exist before the job is launched. This
makes things safer in the event someone deletes the logs directory and
restarts the service
  from Andy Doan

[r=PS Jenkins bot, Celso Providelo] Fixes glitches on publisher from Ursula Junque

[r=Andy Doan, PS Jenkins bot] Trivial fixes for ci-airline staging. from Celso Providelo

gunicorn: set logging level to DEBUG for all

restish charm: fix to work with i/s requirement

As discussed with i/s this will get us going with:

 /srv/ci-airline-gatekeeper-restish/logs/access.log
 /srv/ci-airline-gatekeeper-restish/logs/gunicorn.log