Posted on Wed 04 February 2015

Nova cells and block device mapping

There is a bug related to block device mapping for those using Nova cells which affects all versions of OpenStack.

One of the result is the inability to create an image of an instance booted from volume.

How to reproduce the problem

The bug can be reproduced by booting an instance from a volume with a block device mapping missing the device name. Although the instance boots fine, you won't be able to create an image from the instance.

The nova boot command can be used to create the instance (note the lack of device= in the --block-device value).

nova boot \
  --flavor m1.small \
  --block-device source=image,id=<image-id>,dest=volume,size=10,bootindex=0 \
  demo.example.org

And the nova image-create command to create an image of the instance:

nova image-create <instance-uuid> demo.example.org-snapshot01

It will then fail with the following error:

ERROR: Block Device Mapping is Invalid: failed to get volume XXX. (HTTP 400)

The problem explained

This is because 2 rows end up being created in the block_device_mapping table (BDM) of the API cell which confuses Nova when creating the image.

The first one is initially created by the API cell without most of the volume informations. The reason can be found in a comment found in nova/db/sqlalchemy/api.py:

# NOTE(xqueralt): Only update a BDM when device_name was provided. We
# allow empty device names so they will be set later by the manager.

Since the device_name value is missing from the initial request, a new BDM will be created as explained by an other comment in the same file:

# Either the device_name doesn't exist in the database yet, or no
# device_name was provided. Both cases mean creating a new BDM.

Here is an example of such BDM entry which ends up being created in the API cell:

           created_at: 2015-01-30 21:14:49
           updated_at: NULL
           deleted_at: NULL
                   id: 45
          device_name: NULL
delete_on_termination: 0
          snapshot_id: NULL
            volume_id: NULL
          volume_size: 10
            no_device: NULL
      connection_info: NULL
        instance_uuid: d5d4961a-42d2-40f5-abc7-02db9cc81fdb
              deleted: 0
          source_type: image
     destination_type: volume
         guest_format: NULL
          device_type: NULL
             disk_bus: NULL
           boot_index: 0
             image_id: df3aeaa2-cf9a-42e8-a2a4-6c854462a5f9

As briefly explained by one of the above comment, the device_name can be left empty to let the virt driver set the device_name itself.

Once the instance created in the compute cell, the BDM will bubble up through nova-cells with the device_name value populated. However no corresponding BDM entry will be found in the API cell. This is because the instance_uuid and device_name fields are part of a "soft" composite key when finding and updating a BDM entry. A new BDM entry is created with all the volume informations. This is the right BDM entry. The first BDM is kept.

Unfortunately, the side-effect is that Nova won't be able to find the associated volume (NULL) in Cinder for the first (wrong) BDM when creating an image from the instance.

Fixing the problem

There is currently a change pending review refactoring BDM to use objects.

This change hides a fix to nova/compute/cells_api.py for the above mentioned bug which consists of not creating the initial BDM entry in the API cell when cells are enabled. It will instead let the compute cell create the BDM in its own database and wait for nova-cells to synchronize it to the API cell. Thanks to Matt Van Winkle from Rackspace (VW_ on IRC) from pointing that one out.

A bug has been created and a change against master proposed and merged. A backport to Juno is currently pending review at the time of writing.

For those running the unsupported Icehouse version of OpenStack Nova, I backported the change.

© Mathieu Gagné. Built using Pelican. Theme by Giulio Fidente on github.