Merge lp://qastaging/~danilo/simplestreams/insert-item-factoring into lp://qastaging/simplestreams

Proposed by Данило Шеган
Status: Merged
Merged at revision: 435
Proposed branch: lp://qastaging/~danilo/simplestreams/insert-item-factoring
Merge into: lp://qastaging/simplestreams
Diff against target: 828 lines (+681/-79)
2 files modified
simplestreams/mirrors/glance.py (+202/-79)
tests/unittests/test_glancemirror.py (+479/-0)
To merge this branch: bzr merge lp://qastaging/~danilo/simplestreams/insert-item-factoring
Reviewer Review Type Date Requested Status
simplestreams-dev Pending
Review via email: mp+296828@code.qastaging.launchpad.net

Description of the change

This change refactors GlanceMirror.insert_item() to allow for easier and more contained testing. I needed to do this to understand everything that was going on inside insert_item and other bits of code. There are now four distinct and, imo, clear things happening in it:

 1. Download image to a local file from a ContentSource
 2. Construct extra properties to store in Glance along with image
 3. Prepare arguments for GlanceClient.images.create() call
 4. Adapt source simplestreams entry for an image for use in the target simplestreams index

It should be fully backwards compatible, and test coverage for all the individual steps should be much better (I admit to it not being perfect, but it's a step in the right direction, imho at least).

This branch is the base for follow-up branch (coming tomorrow) to introduce changes where we ensure a problem after the first image is fully uploaded to Glance won't result in all the metadata being lost due to it not being saved anywhere (bug 1584938).

The test for the insert_item() itself is far from perfect: it could be made simpler, but this one is based on the input and output data from the previous version of the code and actual live image data from cloud-images.ubuntu.com, so it tests a real-world scenario. And, insert_item still does too many things, but I was worried about breaking backwards compatibility and all the potential call-sites, so I didn't do anything about it.

Also, I've refrained from changing some of the awkward variable names in it, to simplify review ('flat' has remained 'flat', instead of perhaps 'flat_source_image_metadata', and 't_item' could switch to 'target_simplestreams_item'). I've also stopped making use of 'data' since it's now directly available in the flattened metadata, and was originally taken from 'src' in callsites anyway.

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

Danilo, thank you for cleanups. Looks mostly good.

The one thing i'm concerned about is changing the signature on GlanceMirror.

I'm not opposed to variable name improvements. but maybe something a bit shorrter.
  flattened_img_data
  target_sstream_item

clearly we're in bike shed territory here.

do you think its better or worse to have hypervisor_mapping at the class level ?
rather than getting it in insert_item and passing it around.

460. By Данило Шеган

Address review comments: rename a few internal variables in insert_item() to make them clearer.

Revision history for this message
Данило Шеган (danilo) wrote :

Thanks for the review, Scott.

What worries you about the changed signature of GlanceMirror? It's an optional parameter put at the very end, thus no existing call-site could be affected. Dependency injection of "OpenStack module" like this allows for simpler testing without having to re-factor the constructor itself (which does too much anyway). Alternative is to move any calls to openstack.* and whatever is using the returned data outside the constructor, and call that in every method which relies on actual connections (thus following the ideology of only setting up the data in the constructor, but not really doing anything), but that's a bit outside the scope of this branch.

As for hypervisor_mapping, it could very well be at the class level since we initialize GlanceMirror with the config object. However, for helper methods themselves, I prefer to have them rely as little as possible on their "context": makes it clear what their inputs and outputs are (and they are easier to test as well). Thus, I've changed nothing in this regard, but if you prefer it any other way, just shout. :)

I've done the variable renames and pushed that up.

461. By Данило Шеган

Document 'openstack' argument on GlanceMirror.

Revision history for this message
Scott Moser (smoser) wrote :

At first i didn't understand or missed the 'openstack=openstack' basically masking the outer scoped import. It seems odd dont you think ? I'm fine with it if you think this sort of thing is "pythonic", but would just as well accept:

    def __init__(self, config, objectstore=None, region=None,
                 name_prefix=None, progress_callback=None,
                 client=None):
     ...
     if client is None:
         client = openstack

I'm willing to defer to you on whatever you think is reasonable. I just find the inner varibable name overriding the module level import at the top easily confusing.

462. By Данило Шеган

Make the dependency injection of openstack module in GlanceMirror clearer as suggested by Scott.

Revision history for this message
Данило Шеган (danilo) wrote :

Sure thing: if one person finds it confusing, so might the next one. Used your suggestion instead!

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches