Merge lp://qastaging/~yamahata/glance/lp802893 into lp://qastaging/~hudson-openstack/glance/trunk

This is an interesting idea, but I'm a little hesitant to call this a bug. Keeping metadata simple was an explicit design design. I think this should probably be filed as a blueprint targeted at diablo-3, assuming this is something we want. Thoughts, Jay?

I would also like to see a lot more testing around this. I would want to guarantee this will work at every level, not just the utils function that does the mapping.

I agree that this is probably something we should file as a blueprint and discuss. I think that if we need more complex metadata in Glance that it's more appropriate to make a specific effort for Glance to support querying against OVF manifests/metadata than rolling our own.

On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
> This is an interesting idea, but I'm a little hesitant to call this a bug. Keeping metadata simple was an explicit design design. I think this should probably be filed as a blueprint targeted at diablo-3, assuming this is something we want. Thoughts, Jay?

Fair enough, I'll file a blueprint and let's start discussion.
I think the discussion involves both nova and glance. So I'll also start
a discussion thread on openstack devel ml in order to draw attention from
nova developer. So far I've thought it's Glance issue so that I created
this patch to Glance. Other option would be to change nova metadata handling.

Jay, any comments?

thanks,

> I would also like to see a lot more testing around this. I would want to guarantee this will work at every level, not just the utils function that does the mapping.
--
yamahata

> On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
> > This is an interesting idea, but I'm a little hesitant to call this a bug.
> Keeping metadata simple was an explicit design design. I think this should
> probably be filed as a blueprint targeted at diablo-3, assuming this is
> something we want. Thoughts, Jay?
>
> Fair enough, I'll file a blueprint and let's start discussion.

Cool. Agreed it's big enough to be a blueprint discussion.

> I think the discussion involves both nova and glance. So I'll also start
> a discussion thread on openstack devel ml in order to draw attention from
> nova developer. So far I've thought it's Glance issue so that I created
> this patch to Glance. Other option would be to change nova metadata handling.
>
> Jay, any comments?

I think it's a good feature to add, but I'd almost prefer to add it as new middleware on the registry, or even as an extension? That way you could do something like:

PUT /images/<ID>/container-format

with a body of something like:

'format': {
  'mappings': [
                   {'device_name': '/dev/sda1',
                    'snapshot_id': 0x12345678,
                    'delete_on_termination': False},
                   {'device_name': '/dev/sda2',
                    'no_device': True}
  ],
  'block_device_mapping': [
                   {'virtual': 'ami', 'device': 'sda1'},
                   {'virtual': 'root', 'device': '/dev/sda1'},
                   {'virtual': 'swap', 'device': 'sdb1'},
                   {'virtual': 'swap', 'device': 'sdb2'},
                   {'virtual': 'ephemeral0', 'device': 'sdc1'},
                   {'virtual': 'ephemeral1', 'device': 'sdc2'}
  ],
}

And then, the extension/middleware can store specific information about the container in separate tables in the database that can be queried using a more specific API than the very limited custom key/value properties we currently have.

Thoughts?
-jay

Is there any reason you can't just do a json.dumps and save it as a text blob, then use json.loads on the other end? I don't see why complex metadata needs to be split into a bunch of separate properties.

Vish

On Jun 29, 2011, at 2:08 PM, Jay Pipes wrote:

>> On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
>>> This is an interesting idea, but I'm a little hesitant to call this a bug.
>> Keeping metadata simple was an explicit design design. I think this should
>> probably be filed as a blueprint targeted at diablo-3, assuming this is
>> something we want. Thoughts, Jay?
>>
>> Fair enough, I'll file a blueprint and let's start discussion.
>
> Cool. Agreed it's big enough to be a blueprint discussion.
>
>> I think the discussion involves both nova and glance. So I'll also start
>> a discussion thread on openstack devel ml in order to draw attention from
>> nova developer. So far I've thought it's Glance issue so that I created
>> this patch to Glance. Other option would be to change nova metadata handling.
>>
>> Jay, any comments?
>
> I think it's a good feature to add, but I'd almost prefer to add it as new middleware on the registry, or even as an extension? That way you could do something like:
>
> PUT /images/<ID>/container-format
>
> with a body of something like:
>
> 'format': {
> 'mappings': [
> {'device_name': '/dev/sda1',
> 'snapshot_id': 0x12345678,
> 'delete_on_termination': False},
> {'device_name': '/dev/sda2',
> 'no_device': True}
> ],
> 'block_device_mapping': [
> {'virtual': 'ami', 'device': 'sda1'},
> {'virtual': 'root', 'device': '/dev/sda1'},
> {'virtual': 'swap', 'device': 'sdb1'},
> {'virtual': 'swap', 'device': 'sdb2'},
> {'virtual': 'ephemeral0', 'device': 'sdc1'},
> {'virtual': 'ephemeral1', 'device': 'sdc2'}
> ],
> }
>
> And then, the extension/middleware can store specific information about the container in separate tables in the database that can be queried using a more specific API than the very limited custom key/value properties we currently have.
>
> Thoughts?
> -jay
>
> --
> https://code.launchpad.net/~yamahata/glance/lp802893/+merge/66096
> Your team Glance Core is requested to review the proposed merge of lp:~yamahata/glance/lp802893 into lp:glance.

On Wed, Jun 29, 2011 at 4:43 PM, Vish Ishaya <email address hidden> wrote:
> Is there any reason you can't just do a json.dumps and save it as a text blob, then use json.loads on the other end?  I don't see why complex metadata needs to be split into a bunch of separate properties.

That's a perfectly reasonable solution.

Of course, it would preclude any ability to filter on such things. If
I wanted to find all images where delete_on_terminate was set, the
query would be a beast and issue a full table scan over
image_properties, since there would be no ability to do an equality
search.

So, long term, I think if it's information we want to query, we should
look to a more robust method of storing the pieces of information.

Cheers,
jay

On Wed, Jun 29, 2011 at 09:08:42PM -0000, Jay Pipes wrote:
> > On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
> > > This is an interesting idea, but I'm a little hesitant to call this a bug.
> > Keeping metadata simple was an explicit design design. I think this should
> > probably be filed as a blueprint targeted at diablo-3, assuming this is
> > something we want. Thoughts, Jay?
> >
> > Fair enough, I'll file a blueprint and let's start discussion.
>
> Cool. Agreed it's big enough to be a blueprint discussion.

I created the blueprint and the spec page.

https://blueprints.launchpad.net/glance/+spec/structured-data-in-metadata
http://wiki.openstack.org/StructuredMetadata

I've listed ideas for implementation candidate for the discussion base.
--
yamahata

On Wed, Jun 29, 2011 at 09:49:27PM -0000, Jay Pipes wrote:
> On Wed, Jun 29, 2011 at 4:43 PM, Vish Ishaya <email address hidden> wrote:
> > Is there any reason you can't just do a json.dumps and save it as a text blob, then use json.loads on the other end? ??I don't see why complex metadata needs to be split into a bunch of separate properties.
>
> That's a perfectly reasonable solution.
>
> Of course, it would preclude any ability to filter on such things. If
> I wanted to find all images where delete_on_terminate was set, the
> query would be a beast and issue a full table scan over
> image_properties, since there would be no ability to do an equality
> search.
>
> So, long term, I think if it's information we want to query, we should
> look to a more robust method of storing the pieces of information.

If there is no major objection, I'd like to go for the middleware
idea that Jay suggested.

--
yamahata

> If there is no major objection, I'd like to go for the middleware
> idea that Jay suggested.

Go for it! No need for permission :)

-jay

Yamahata,

Feel free to hop onto IRC freenode.net #openstack-dev if you want to discuss this or need some pointers on where to start :)

-jay