Merge lp://qastaging/~doanac/uci-engine/lander-logs-multi into lp://qastaging/uci-engine

[r=Paul Larson, PS Jenkins bot] django charm: add relationship support for rabbitmq

This lets you add a relation between rabbitmq and django like we
do in our rabbitmq-worker charm. This allows the ticket system to use rabbit from Andy Doan

[r=Evan Dandrea, PS Jenkins bot] Fix the tests that didn't catch the broken call nor the broken assumptions about rabbit behavior. Also fix the broken basic_publish() call. 1320000 from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] lander/ticket-system: convert to rabbit

This changes the lander/ticket relationship so that the ticket
system simply sends amqp messages that the lander can handle. This
is the first step in allowing us to scale to multiple landers. from Andy Doan

[r=Chris Johnston, PS Jenkins bot] remove incorrect comment about ticket-states

this comment is incorrect and just leads to confusion from Andy Doan

[r=Chris Johnston, PS Jenkins bot] ticket-system: make test_style automatically ignore migrations

Its a pain to remember adding your migration to this file. This
discovers them for you.
  from Andy Doan

[r=Vincent Ladeuil, PS Jenkins bot] webui: allow view ticket progress live

This is a super simplistic start, but allows you to follow the progress of a ticket landing in real time. from Andy Doan

[r=PS Jenkins bot, Andy Doan] Uprgade to ucitests 0.1.3 and remove the corresponding upstreamed code. from Vincent Ladeuil

[r=Andy Doan, PS Jenkins bot, Celso Providelo] This branch adds the publisher skeleton to our tree, also adds PUBLISHER_QUEUE type to amqp_utils.py from Ursula Junque

[r=PS Jenkins bot, Ursula Junque, Chris Johnston] ticket-system: fix series choices for tickets

Things have been working by dumb luck because we were creating tickets
via the REST api with a series in all lowercase (ie "saucy").

I was playing with editing tickets from the django admin panel and
discovered we saved the series as SAUCY. This breaks the bsbuilder's
interaction with the LP api which expectes a series to be all lowercase.

I fixed that, removed the obsolete quantal series and included trusty. from Andy Doan

[r=Francis Ginther, PS Jenkins bot] ticket-system: allow updating of just status or workflow-step

This fixes a bug and annoyance in the ticket system.

bug - you can't just update the status or workflow-step. They have
to both be provided

annoyance - we return status and workflow step back as words like
"in progress", but our write-api requires the inputs to be numerical.

This fixes both issues so that you can now just update a ticket's status
to something like 'failed'.
  from Andy Doan

[r=PS Jenkins bot, Paul Larson] Put the right data into the lp_creds.txt file. from Evan Dandrea

[r=PS Jenkins bot] Updating restish charm according to IS guidelines. from Celso Providelo

[r=PS Jenkins bot, Evan Dandrea] Updating our charms to cope with prodstack guidelines from Celso Providelo

[r=PS Jenkins bot, Andy Doan] Add restish charm to local charms in deploy.py from Chris Johnston

[r=Andy Doan, PS Jenkins bot] This branch adds Merge Proposals to the Ticket System. from Ursula Junque

[r=Vincent Ladeuil, PS Jenkins bot] Add more deployment padding for the ticket_system amulet tests from Chris Johnston

[r=PS Jenkins bot, Andy Doan, Vincent Ladeuil, Evan Dandrea] Introduces the 'intercom' charm interface as a proposed solution for internal components communication. from Celso Providelo

[r=PS Jenkins bot, Andy Doan] Contextualising Gatekeeper sandboxes as part of a ticket from Celso Providelo

[r=Andy Doan, PS Jenkins bot] Fix test failures introduced while renaming the services in an IS compatible way. 1323498 from Vincent Ladeuil

[r=PS Jenkins bot, Andy Doan] More changes/updates for staging deployment. from Celso Providelo

[r=PS Jenkins bot, Andy Doan] Move logger creation earlier so that we can get sensible logging when problems arise during message handling. Also log the actual exception rather than just a generic error message. from Paul Larson

[r=Ursula Junque, PS Jenkins bot, Andy Doan] Creating the corresponding ticket container on ticket creation. from Celso Providelo

tarmac: allow use of TMPFS

[r=PS Jenkins bot, Andy Doan] Use a fake data store to test the test runner worker. from Vincent Ladeuil

[r=PS Jenkins bot, Andy Doan] Switch the CLI to use python-requests from Chris Johnston

Fix ticket system integration tests failures related to the new swift requirement.

ubuntu-ci-autoconfig change to adapt the webui name change to ci-airline-webui

[r=Andy Doan, PS Jenkins bot] Adds a subordinate hook to create chroots for building source packages. from Francis Ginther

[r=PS Jenkins bot, Celso Providelo] More charms fixing for staging deployment. from David Ames

[r=Parameswaran Sivatharman, PS Jenkins bot] lander: handle rename to ci-airlines-<unit> for staging from Andy Doan

[r=Celso Providelo, PS Jenkins bot] engine health: support private ips

in staging our engine health providers are behind private IPs.
This creates a proxy so that we can connect to them. from Andy Doan

[r=PS Jenkins bot, Andy Doan] Integrates CLI and Gatekeeper for file uploading without Swift credentials. from Celso Providelo

[r=Andy Doan, Tom Haddon, PS Jenkins bot] Add hostname config option to python-django for specifying a domain from Chris Johnston

[r=Chris Johnston, PS Jenkins bot] Upgrading UCI-E charms to use a more recent charmhelpers revision from Celso Providelo

[r=Andy Doan, PS Jenkins bot] Renaming deployment configuration to 'ci-airline' (removing the '-staging' suffix). 1324941 from Celso Providelo

[r=PS Jenkins bot, Andy Doan] Retry package installation (fatal=True) when dpkg is locked (master/subordinate racing). from Celso Providelo

[r=Andy Doan, PS Jenkins bot] Fixing charmhelpers.payload.execd loading/importing. from Celso Providelo

[r=Celso Providelo, PS Jenkins bot] remove juju-gui

At the sprint we all concluded nobody used this. In the event its needed,
a user can simply run:

  juju deploy juju-gui from Andy Doan

[r=Chris Johnston] remove juju-gui test

I removed this service from the deployment but forgot to remove the test from Andy Doan

[r=Andy Doan, PS Jenkins bot] Upload a '.uploaded_version_info' file in a python format and enough data to clearly identify the payload. from Vincent Ladeuil

[r=Paul Larson, PS Jenkins bot] webui charm: fix broken upgrade logic

The backup logic had a typo causing it to not work. Additionally,
the json-status backup logic isn't necessary in install() and the
queues file doesn't need to be backed up because its built from
config data.
  from Andy Doan

[r=PS Jenkins bot, Francis Ginther, Andy Doan] Basic publisher node deployment configuration. from Celso Providelo

[r=Evan Dandrea, Vincent Ladeuil, PS Jenkins bot] engine-health: include revision level for components

Include branch/revno information in our engine health page for the
services we have that expose json-status. This doesn't account for
our rabbit-workers, but should be a nice, helpful start. from Andy Doan

[r=Andy Doan, PS Jenkins bot] flake8 fixes to python-django/hooks/hooks.py from Chris Johnston

[r=Andy Doan] Increase amulet timeout from 10 to 20 minutes. from Vincent Ladeuil

[r=PS Jenkins bot, Chris Johnston] django charm: use charm helper for apt-get install

This ensures we won't hit the dpkg lock issue from Andy Doan

[r=Evan Dandrea, PS Jenkins bot] deploy.py: juju-deployer script path fix

The PYTHONPATH set in the juju-deployer script we create only
works when called from deploy.py (that has it set). This
exports the path so you can run this command by hand.
  from Andy Doan

[r=Francis Ginther, Andy Doan, PS Jenkins bot] Moving ppa_sync from lander to publisher worker and adding entry-points for validate_subticket() and copy(). from Celso Providelo

[r=Francis Ginther, Evan Dandrea, PS Jenkins bot] ticket-system: add concept of "workflow"

We have a "todo" in our lander to get the workflow for a ticket
from the ticket-system. This adds a new table that we can store
support workflow-steps in. We'll assign all tickets a workflow. We'll
keep a "default" one for convienice and have django create it
during our initial migration. But over time we could add/remove/change
these workflows based on CI train needs
  from Andy Doan

[r=Francis Ginther, PS Jenkins bot] lander/ticket-system: introduce concept of pause/continue

now that the lander can "resume", we'll add the notion of a ticket
that can be paused and then continued. By default this won't be
enabled. However, if you have a ticket created with a workflow that
has a step like like:

 {'name': 'bsbuilder', 'pause':{}}

Then the lander will see a ticket has a pause step that hasn't been
handled, update the ticket-system and exit. Later you can mark
the ticket to "CONTINUE" from something like the django admin panel.
This will send a message back to the lander to resume work
  from Andy Doan

[r=PS Jenkins bot, Vincent Ladeuil] Fixing integration tests to cover Gatekeeper and support multiple configuration deployments. 1326809 from Celso Providelo

[r=PS Jenkins bot, Andy Doan] Charm for system-image-server from Paul Larson

[r=Andy Doan, PS Jenkins bot] Integration tests typo fixing from Celso Providelo

[r=PS Jenkins bot, Andy Doan] Add ability to specify where gunicorn logging happens from the python-django charm. from Chris Johnston

[r=Francis Ginther, PS Jenkins bot] django charm: get upgrade path consistent with install

We noticed a bug where the upgrade path doesn't install packages
if the config changed to include new ones. This makes changes the
upgrade path to use the install path. from Andy Doan

[r=Celso Providelo, PS Jenkins bot] ppa-assigner health: improve reporting

show the ppa owner so users know where the ppa's are.

also don't show "private ppas only" as a pass/fail. its really just
useful debug info that doesn't reall reflect a "bad" state
  from Andy Doan

[r=Andy Doan, PS Jenkins bot] webui - Convert to Django

* splits webui and ticket-system juju-deployer yaml files
* not yet working
  - Completed tickets view
  - Failed tickets view
  - package listing
  - engine status from Joe Talbott

[r=PS Jenkins bot, Joe Talbott] webui: add back support for engine health

With the new django app, we needed to split things a little. The
webui charm is still subordinate to apache but its now essentially
just hooks for the proxy logic and json_status urls.

The monitoring of queues was simplified since we now have django.
Its much easier/cleaner to deal with them now. from Andy Doan

[r=Joe Talbott, PS Jenkins bot] deploy.py: fix bug with --build-only

The build-only logic fails to create our swift tarball. This
means that i/s is actually deploying us by branch instead of
our tarball. This fixes the issue by always uploading the
tarball.
  from Andy Doan

[r=Joe Talbott, PS Jenkins bot] add unit testing for the engine-health portion of the webui from Andy Doan

[r=PS Jenkins bot, Evan Dandrea] Remove duplication between the test runner tesbed and uci-vms. from Vincent Ladeuil

[r=Andy Doan, PS Jenkins bot] A few small changes to make our test suite not break all over the shop on OSX. from Evan Dandrea

[r=PS Jenkins bot, Celso Providelo, Evan Dandrea] webui: engine health requires amqp lib

my engine health changes cause this code to import amqp_utils. This
means the webui now requires amqplib. 1328518 from Andy Doan

[r=PS Jenkins bot, Chris Johnston] CLI end2end integration test fix. 1328518 from Celso Providelo

[r=Chris Johnston, PS Jenkins bot] Add missing testtools import. from Vincent Ladeuil

[r=PS Jenkins bot, Celso Providelo] restish-charm: complete transition to charmhelpers

This moves the restish charm completely to charm helpers and
adds a little more unit testing from Andy Doan

[r=Chris Johnston, PS Jenkins bot] ppa-cleaner: fix regression caused by recent django charm change

The lates django charm changes screwed up the python path
used by the upstart charm. I fixed the upstart charm, and this
is all we need fixed in trunk.

We actually have code that catches this regression, it just got
ignored. from Andy Doan

[r=Francis Ginther, PS Jenkins bot] ticket-system: add concept of "reviews"

This allows us to attach reviews to a ticket. The reviews can be
mapped to workflow steps so that the lander can be paused at the
point a review is needed. Once all reviews are completed, the
ticket-system will resume the ticket by notifying the lander. from Andy Doan

[r=Andy Doan, PS Jenkins bot] Provides the 'series' parameter to the test runner 1328193 from Vincent Ladeuil

[r=PS Jenkins bot, Evan Dandrea] Skip charm and integration tests when check_environment complains about missing env vars 1319703 from Vincent Ladeuil

[r=Vincent Ladeuil, PS Jenkins bot] feature probing: support juju-lxc

This allows our feature probing to account for juju-lxc deployments from Andy Doan

[r=Francis Ginther] Update apache charm revno to include apt-get update fix from Chris Johnston

[r=PS Jenkins bot, Celso Providelo] Update rabbitmq-server revision number to fix apt-get install problem 1328864 from Chris Johnston

[r=Andy Doan, PS Jenkins bot] Refactoring 'get_image' CLI action to download images using tempurls provided by Gatekeeper. from Celso Providelo

[r=Andy Doan, PS Jenkins bot, Vincent Ladeuil] Silencing the virtualenv installation in the integration tests. from Celso Providelo

[r=PS Jenkins bot, Celso Providelo] Switching lander to call publisher worker and run ppa_sync.copy directly, instead of running the ppa_sync script. from Ursula Junque

[r=PS Jenkins bot, Celso Providelo] Switching lander to call publisher worker and run ppa_sync.copy directly, instead of running the ppa_sync script. from Ursula Junque

[r=Ursula Junque] Import lp:cupstream2distro rev 605. from Francis Ginther

[r=Celso Providelo, PS Jenkins bot] restish charm: support pgsql relations

This allows the restish charm to pair up with a postgres server. It
stores the DB settings as a json file like:

 {
   "database": "ci-airline-gatekeeper-restish",
   "schema_user": "db_2_ci_airline_gatekeeper_restish_schema",
   "schema_password": "mGHMMdLp5hWwb9nqW7whwCttHhCWfLP3Cm9Y62b3",
   "host": "10.0.3.169",
   "user": "db_2_ci_airline_gatekeeper_restish",
   "password": "mLL8s7dsZ82sPdzyV5dyYwSrwCRcrcPffdbfNSmfXg",
   "port": "5432"
 }
  from Andy Doan

[r=PS Jenkins bot, Celso Providelo] restish charm: allow deploying the python-pyramid framework from Andy Doan

[r=PS Jenkins bot, Francis Ginther] Fixes cupstream2distro unit tests. from Ursula Junque

[r=PS Jenkins bot, Vincent Ladeuil] Installing python-apt for testing charms. from Celso Providelo

[r=Evan Dandrea, PS Jenkins bot] build-essential is required to install python-apt. 1329688 from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] Bootstrap run-tests tests. from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] Change test ids to better reflect their location on disk. 1285647 from Vincent Ladeuil

[r=Celso Providelo, PS Jenkins bot] Determine deployment env in order to determine if sshuttle is needed from Chris Johnston

[r=Francis Ginther] Fix minor oversite from sshuttle workaround from Chris Johnston

[r=Vincent Ladeuil, PS Jenkins bot] Don't rebuild charms/precise/*/hooks/charmhelpers if there is no need to. Shaves nearly 24 seconds off subsequent deployment runs. from Evan Dandrea

[r=Evan Dandrea] Handle the test selection regexps in load_testing_tests. from Vincent Ladeuil

[r=Evan Dandrea] Better isolate TestFloatingIP so it passes even if dev defines a JUJU_ENV before running the tests 1330420 from Vincent Ladeuil

[r=Evan Dandrea] Move webui cm.py one level up to align with other charms from Vincent Ladeuil

[r=Francis Ginther, PS Jenkins bot] Add a tool to bake an image with our dependencies, upload it, and point the juju image metadata at it. from Evan Dandrea

[r=Vincent Ladeuil, PS Jenkins bot] Fixing integration tests failures (test_integration -> replace venv by the corresponding sys.paths for speed and test_ticket_system -> setup new webui configuration) from Celso Providelo

[r=Celso Providelo, PS Jenkins bot] restish charm: support db migrations

This is a super simplistic way, but should work. You simply add something
like:

 db_migration_cmd: ./nf-data-collector/migrate.sh

to your deployer yaml and the script will get run when the pgsql relation
changes or the charm is upgraded
  from Andy Doan

[r=Evan Dandrea, PS Jenkins bot] Support configurable tempurl "public_host" (different than OS_AUTH_URL hostname) to cope with the prodstack scenario (see https://rt.admin.canonical.com/Ticket/Display.html?id=72096). from Celso Providelo

[r=Paul Larson, PS Jenkins bot] Extend --concurrency support to all tests except for the integration ones from Vincent Ladeuil

[r=Vincent Ladeuil, PS Jenkins bot] Including ci_utils in the test sys.path so CLI runs correctly. from Celso Providelo

[r=Joe Talbott] webui: fix live-logging view

This was broke during the move to django. from Andy Doan

[r=Paul Larson, PS Jenkins bot] Source package re-signing script and a couple of unit tests. from Parameswaran Sivatharman

[r=PS Jenkins bot, Francis Ginther] Using datastore for fetching files from Swift in BSB, so it works with private files uploaded via Gatekeeper. from Celso Providelo

[r=PS Jenkins bot, Joe Talbott] webui: add a read-only "reviews" page

The lander now has the ability to "pause" a ticket. A paused ticket
can mean one of two things:

1) the ticket needs one or more reviews performed
2) the ticket needs to retry a step

This handles displaying a ticket "needs attention" and provides a link
to see what actions need to happen. For now it just shows the missing
actions. The next step will be to add buttons to allow the ticket to continue. from Andy Doan

[r=Francis Ginther, PS Jenkins bot] restish charm: fix db-migrate bug

the pgsql relation join/change doesn't always include all the data
required to make a db connection. This ensures we only call the
migration method once all data is ready.
  from Andy Doan

[r=Evan Dandrea, PS Jenkins bot] Remove mount-callback-umount script from imagebuilder from Paul Larson

webui: include unit number for lander logs

We are going to need to support multiple landers. So we can't just
proxy with "logs-lander". This does it with logs-lander/<unit>/

ticket-system: add concept of lander_unit

This allows the ticket-system to know which lander is currently
working on a ticket.

lander: keep "lander_unit" in sync for the ticket-system

webui: handle multiple lander-units

oops

merge with trunk