Merge lp://qastaging/~brad-marshall/charms/trusty/hacluster/add-nrpe-checks-update-charmhelpers into lp://qastaging/~openstack-charmers/charms/trusty/hacluster/next

Brad, thanks for the submission! The hacluster charm is maintained by the openstack-charmers team and development operates off the /next branch. I've re-targeted this merge proposal towards the /next branch.

charm_lint_check #1956 hacluster-next for billy-olsen mp249570
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  hooks/hooks.py:22:1: F401 'relations_of_type' imported but unused
  make: *** [lint] Error 1

Full lint test output: http://paste.ubuntu.com/10193691/
Build: http://10.245.162.77:8080/job/charm_lint_check/1956/

charm_unit_test #1746 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/1746/

charm_amulet_test #1897 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10193709/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1897/

Brad, please fix the linting issue that's been identified by uosci-testing-bot.

When expanding a cluster by adding a node, the hacluster charm is somewhat inefficient as it will restart corosync whenever the corosync.conf file changes. If the nrpe check runs during this process I think it's going to trigger a false alarm. Can you confirm this? I'm thinking we may want to be a bit more intelligent about possibly disabling certain nagios checks while changes to the cluster are made. This is more of a question than anything.

As another comment, in the 15.01 release of the OpenStack charms, haproxy is now setup and configured on each service by default (it avoids port jumps, reconfig, the amount of work needed to be performed when clustering, etc). So I think that those charms need the haproxy check as well, but this will need to make sure it is okay to pair with a charm that has already enabled the haproxy nagios plugin. It may do so (I'm not overly familiar with nagios), but thought I'd drop it in and let you comment. Thanks.

Hi,

> Brad, please fix the linting issue that's been identified by uosci-testing-bot.

Done.

> When expanding a cluster by adding a node, the hacluster charm is somewhat
> inefficient as it will restart corosync whenever the corosync.conf file changes.
> If the nrpe check runs during this process I think it's going to trigger a false
> alarm. Can you confirm this? I'm thinking we may want to be a bit more intelligent
> about possibly disabling certain nagios checks while changes to the cluster are
> made. This is more of a question than anything.

Hmm, its an interesting question. Enabling and disabling checks in nagios isn't trivial,
so I'm not actually sure how it would work. Also, you need to be down for an extended
period before it'll alert, so I don't know if its worth the effort. I'd like to see
what we have here landed and we can make work out if there's a more efficient way
down the track?

Thanks,
Brad

> As another comment, in the 15.01 release of the OpenStack charms,
> haproxy is now setup and configured on each service by default
> (it avoids port jumps, reconfig, the amount of work needed to be
> performed when clustering, etc). So I think that those charms need
> the haproxy check as well, but this will need to make sure it is
> okay to pair with a charm that has already enabled the haproxy
> nagios plugin. It may do so (I'm not overly familiar with nagios),
> but thought I'd drop it in and let you comment. Thanks.

My understanding is that it should already be monitoring haproxy
with the latest charms, but it could be made better - it seems
that there's only a process check, which isn't really sufficient.

I'll try to look into what we could do with this, and how it would
interact with the subordinate charms.

Brad.

charm_lint_check #1958 hacluster-next for billy-olsen mp249570
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1958/

charm_unit_test #1748 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/1748/

charm_amulet_test #1899 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10196422/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1899/

charm_lint_check #2044 hacluster-next for billy-olsen mp249570
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/2044/

charm_unit_test #1834 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/1834/

charm_amulet_test #1982 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10268549/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1982/

charm_lint_check #2107 hacluster-next for billy-olsen mp249570
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/2107/

charm_unit_test #1897 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/1897/

charm_amulet_test #2013 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10302877/
Build: http://10.245.162.77:8080/job/charm_amulet_test/2013/

charm_lint_check #2205 hacluster-next for billy-olsen mp249570
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/2205/

charm_unit_test #1994 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/1994/

charm_amulet_test #2151 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10396224/
Build: http://10.245.162.77:8080/job/charm_amulet_test/2151/

This should now be ready for a re-review.

charm_lint_check #2490 hacluster-next for billy-olsen mp249570
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/2490/

charm_unit_test #2280 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/2280/

charm_amulet_test #2362 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10545084/
Build: http://10.245.162.77:8080/job/charm_amulet_test/2362/

Hey Brad,

I think this is almost there... almost there... I just have a question about the size of your charmhelpers sync and why you included more files on your sync. I couldn't figure it out.

Also, its somewhat easier to review if the c-h sync comes as a different merge proposal (rumors are this is to be automated), which this can then depend on so we can focus on the changes in here. No biggie, just makes the diff review much larger.

As always, thanks for the submission :)

It looks like I changed it so it would pull in the haproxy checks we have in charmhelpers.contrib.openstack, but later on they were moved to the individual charms. I'll drop it back to charmhelpers.contrib.openstack.utils and see how that goes.

charm_lint_check #3666 hacluster-next for billy-olsen mp249570
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3666/

charm_unit_test #3454 hacluster-next for billy-olsen mp249570
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3454/

charm_amulet_test #3456 hacluster-next for billy-olsen mp249570
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10854117/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3456/

Approve