Merge lp://qastaging/~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support into lp://qastaging/~chad.smith/charms/precise/block-storage-broker/trunk

Proposed by Chad Smith
Status: Rejected
Rejected by: Chad Smith
Proposed branch: lp://qastaging/~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support
Merge into: lp://qastaging/~chad.smith/charms/precise/block-storage-broker/trunk
Prerequisite: lp://qastaging/~chad.smith/charms/precise/block-storage-broker/bsb-ec2-support
Diff against target: 1107 lines (+602/-255)
2 files modified
hooks/test_util.py (+394/-143)
hooks/util.py (+208/-112)
To merge this branch: bzr merge lp://qastaging/~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support
Reviewer Review Type Date Requested Status
Chad Smith Disapprove
Alberto Donato (community) Needs Fixing
Review via email: mp+211963@code.qastaging.launchpad.net

Description of the change

This branch does a couple things:
 1. stops the block-storage-broker from importing euca2ools.commands.euca.describevolumes and euca2ools.commands.euca.describeinstances directly to avoid being affected by internal class/module changes on next major euca release. Instead block-storage-broker sets up a couple of simple parse_ec2* functions to help parsing the output of euca-describe-instances, euca-describe-volumes and euca-describe-tags commands. The euca response types supported by the parse_ec2* functions are INSTANCE, ATTACHMENT, VOLUME and TAG.

 2. To handle the fact that euca-2.X doesn't report volume TAG responses in the euca-describe-volumes command output, _ec2_describe_volumes also calls _ec2_describe_tags to augment any volume tag data (which we use to label volumes in EC2).

 3. In using euca-describe-tags to augment existing volume data I found that tags can be orphaned once a volume is removed by an admin via euca-delete-volume, if that is the case we will ignore tags for volumes that don't exist and log a message

 4. In unit tests drop all MockEuca* classes as they are no longer needed because we mock the subprocess.check_output results for the euca-* commands we run.

By parsing the output from the euca-* commands, we shield ourselves from the internal changes in class structure definition in future euca releases. This branch adds very basic validation of euca-describe-instance output. We can add more validation in separate branches as I didn't want this MP to grow too large.

To post a comment you must log in.
87. By Chad Smith

merge fixes from bsb-ec2-support and resolve conflicts + lint

88. By Chad Smith

revert Makefile change

89. By Chad Smith

Ignore any orphaned TAG lines from euca-describe-tags that don't map to current existing volume-ids

Revision history for this message
Chad Smith (chad.smith) wrote :

Here's a deployer bundle to deploy.
rm -rf ./trusty wherever you are about to juju-deployer from to ensure you don't use cached charm checkouts

cat block-storage-trusty.yaml

common:
    services:
        storage:
            branch: lp:~chad.smith/charms/precise/storage/storage-volume-label-availability-zone
            options:
                provider: block-storage-broker
                volume_size: 9
        block-storage-broker-ec2:
            branch: lp:~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support
            options:
                provider: ec2
                key: <YOUR_EC2_ACCESS_KEY>
                endpoint: <YOUR_EC2_URL>
                secret: <YOUR_EC2_SECRET_KEY>
        postgresql:
            branch: lp:~chad.smith/charms/precise/postgresql/postgresql-using-storage-subordinate
            constraints: mem=2048
            options:
                extra-packages: python-apt postgresql-contrib postgresql-9.3-debversion
                max_connections: 500

doit:
    inherits: common
    series: trusty
    relations:
        - [postgresql, storage]
        - [storage, block-storage-broker-ec2]

juju-deployer -c block-storage-trusty.yaml doit

Revision history for this message
Chad Smith (chad.smith) wrote :

The bundle above works from the w/ juju 1.17.6 be wary of 1.17. < 6 as it suffers from bug 1285901

Revision history for this message
Alberto Donato (ack) wrote :

As discussed on IRC, I think the code could be refactored to reduce duplication, for example as in https://pastebin.canonical.com/107195/.
Tests could then mock _run_command rather than subprocess.check_output.

It should be even possible to perform the line.split("\n") in the _run_command method, returning a list of lists (so the parse_* methods don't need to perform it), I'm not sure if it fits all the use cases, though.

#1:
+ data["device"] = ""
+ data["volume_label"] = ""
+ data["instance_id"] = ""
+ data["tags"] = {"volume_name": ""}

These can be set with a single data.update()

#2:
+ volume_data.update({"instance_id": data["instance_id"]})
+ volume_data.update({"device": data["device"]})

Same as #1

#3:
+ for tag_volume_id in volume_tag_data.keys():
+ additional_tags = volume_tag_data[tag_volume_id]["tags"]
+ if tag_volume_id not in result:
+ hookenv.log(
+ "Ignoring tags for volume-id %s that doesn't exist" %
+ tag_volume_id)
             else:

The first line inside the loop can be moved inside the else clause.
Also there's an extra indentation space on the hookenv.log() call

#4:
I think you can drop the checks on the response_type in the parse_* methods, since it's already checked by call sites (related tests can be dropped then).

review: Needs Fixing
90. By Chad Smith

pull in ack's _run_command patch for simplification

Revision history for this message
Chad Smith (chad.smith) wrote :

Rejecting this branch in favor of dropping euca2ools and using Amazon's python SDK: python-boto as boto is more stable across Ubuntu releases than trying to parse and unify the output of euca2ools commandline utilities.

review: Disapprove

Unmerged revisions

90. By Chad Smith

pull in ack's _run_command patch for simplification

89. By Chad Smith

Ignore any orphaned TAG lines from euca-describe-tags that don't map to current existing volume-ids

88. By Chad Smith

revert Makefile change

87. By Chad Smith

merge fixes from bsb-ec2-support and resolve conflicts + lint

86. By Chad Smith

now that _ec2_describe_volumes calls euca-describe-tags, we need to mock that in unit tests. Add unit tests for parse_ec2_*_reponse functions

85. By Chad Smith

docstring updates

84. By Chad Smith

perform validation of INSTANCE reponse_type before attmpting to parse the value in the line. lint fixes in unit tests

83. By Chad Smith

parse euca-describe-instances and euca-describe-volumes output instead of tying into euca2ools.commands.euca.describeinstances directly

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches

to all changes: