[vmware] Cache images as VM templates by leust · Pull Request #206 · sapcc/cinder

leust · 2024-07-11T15:37:07Z

Upon user request, the driver can cache the image as a VM Template and reuse that to create the volume(s). This feature is useful when creating many volumes in parallel from the same image.

Users can request the image cache feature when creating the volume, by passing the use_image_cache='true' as a property (metadata).

The feature must be enabled per backend, for example:

[vmware]
enable_image_cache = true

This will enable the image cache feature for the vmware backend.

The image templates will then be stored in a folder similar to the volumes folder: OpensStack/Project (vmware_image_cache)/Volumes, where {backend}_image_cache is used as a project name.

The driver will periodically delete the cached images that are expired. The expiry time can be controlled via the property image_cache_age_seconds set on the backend configuration.

Only images smaller than the configured image_cache_max_size_gb will be cached.

Change-Id: I6f5e481f6997a180a455b47abe525b93bcf9aa4e

leust · 2024-07-11T15:42:22Z

Needs sapcc/oslo.vmware#46

hemna · 2024-07-11T15:45:57Z

Why not use cinder's built in cache mechanism?

joker-at-work · 2024-07-12T07:08:00Z

Why not use cinder's built in cache mechanism?

I agree that the reasoning for this should be in the commit message.

leust · 2024-07-15T12:09:23Z

I added the reason to the commit message

We're not using the cinder built-in cache functionality because we
need a few extra features:

the built-in cache doesn't account for shards. The cache entry
will be placed on any backend/shard and could trigger a lot of
slower cross-vc migrations when creating volumes from it.
the built-in cache doesn't have a periodic task for deleting the
expired cache entries
we want to cache the images only when the customer requests it

hemna · 2024-07-15T12:49:31Z

How do we account for the consumed space of the cached volume in the specific datastore so that the scheduler knows how much we are using on that datastore?

leust · 2024-07-15T12:58:48Z

How do we account for the consumed space

The driver reports the free_capacity_gb directly from the datastore.summary.freeSpace so I believe we get this information straight from the VMware API response.

hemna · 2024-07-15T13:03:48Z

The scheduler still needs to account for the capacity allocated against the datastore though. These cached images will be hidden from what is actually allocated against the datastore.

leust · 2024-07-17T08:46:13Z

The scheduler still needs to account for the capacity allocated against the datastore though

OK, thanks for the hint.

Now the "image cache" capacity is being added to the pool's provisioned_capacity_gb which seems to be used while weighing as well as in the calculate_virtual_free_capacity.

The volume backend reports a new extra_provisioned_capacity_gb that's being recognised by the host_manager and added to the final provisioned_capacity_gb.

Could you please check the new code @hemna ?

joker-at-work

If we disable enable_image_cache we might still have images around. Would it make sense to run some cleanup or at least count the existing cached images as usage or we we have to make sure that we clean up manually?

joker-at-work · 2024-08-19T12:52:20Z

+        return list(itertools.chain(
+            *[self._get_cached_images_in_folder(folder_ref)
+              for folder_ref in folder_refs]))


This looks like you could use itertools.chain.from_iterable() instead of itertools.chain(*[…]).

joker-at-work · 2024-08-19T12:55:29Z

+          "created_at": date}]
+
+        Where
+            - name: the name of the template VM (set to the image_id)


Should we maybe add a prefix or postfix or something that makes it possible to distinguish image-cache template-vms from shadow-vms? Especially when things are orphaned and only left as directories on the datastore, it's helpful to distinguish them by name and know what can definitely be deleted.

joker-at-work · 2024-08-20T10:43:13Z

+        max_objects = self.configuration.vmware_max_objects_retrieval
+        options.maxObjects = max_objects
+        try:
+            result = self.session.vim.RetrievePropertiesEx(


Why can't we use WithRetrieval as in _get_image_cache_folder_ref()? This looks to be so much code and I would have expected this code to already exist.

Do you mean using oslo.vmware's get_objects() ?
We couldn't use it because that one looks in the rootFolder and we only want to look into the cache folder here.

with vutil.WithRetrieval(self._session.vim, retr_res) as retr_objects is what I mean. We only get result here and I would assume all the exception handing and such would already be handled in WithRetrieval. We later only use result.objects.

How do we do it in Nova? Did we also copy the contents of get_objects() there? (That's what I understood from you we basically have to do here)

Nova has it's own get_objects()-like called get_inner_objects()
Additionally, WithRetrieval doen't handle this NOT_AUTHENTICATED exception that's thrown when there are no objects in the folder (see nova's _get_image_template_vms).

Ah, ok. Thank you.

What's the use of WithRetrieval then? I thought we added it everywhere in Nova, because code missed to handle cases at times.

joker-at-work · 2024-08-20T10:48:40Z

+
+        cached_images = []
+        for obj in result.objects:
+            props = vim_util.propset_dict(obj.propSet)


Could it be that propSet doesn't exist? We sometimes have exceptions like that in Nova.

joker-at-work · 2024-08-20T11:01:09Z

+            img_volume = copy.deepcopy(volume)
+            img_volume['id'] = image_id
+            img_volume['project_id'] = self._cache_project_name()
+            img_volume['size'] = image_size_in_bytes / units.Gi


Why do we keep the other data and what happens with it? Do we really need whatever else is in the volume dict? Would it maybe make sense to be explicit about what we expect to be use going forward? Can there be private information (metadata) that somehow gets copied to another project's cloned root-disk metadata?

joker-at-work · 2024-08-20T11:01:49Z


+    def _can_use_image_cache(self, volume, image_size):
+        requested = (volume['metadata']
+                     .get(CREATE_PARAM_USE_IMAGE_CACHE) == "true")


We we maybe want to add a .lower() or does that come in automatically?

joker-at-work · 2024-08-20T11:03:22Z

+            LOG.debug("The requested image cannot be cached because it's "
+                      "too big (%(image_size)s > %(max_size)s)",
+                      {'image_size': image_size,
+                       'max_size': max_size})


Should we error out here? The customer requested to use the image-cache and thus expect speedy creations, but with the image they can't get it.
Same question towards requested but not enabled, I guess.

From the user experience point of view, I'd expect such an error to be returned in the API response.
Otherwise they will ask why their volume is in error state.

Fair point. We're too late in the process, because scheduling needs to have happened already. On the other hand, nobody will know ever ...

Any conclusion for this?

afaik, there's a way to add error messages and retrieve them with cinder message list. I think we can go with "ignore what the customer requested" for now, but should investigate this as a follow-up and maybe bring it up in a bigger round for a decision.

joker-at-work · 2024-08-20T11:05:42Z

+
+        img_backing = None
+        if self._can_use_image_cache(volume, metadata['size']):
+            img_backing = self._get_cached_image_backing(


I would rename _get_cached_image_backing() to _get_or_create_cached_image_backing(). Mainly because I was wondering where we would create the cached image backing if we only call a get.

joker-at-work · 2024-08-20T11:07:11Z

+                LOG.exception("Failed to delete the expired image %s",
+                              cached_image['name'])
+
+    def _get_cached_images(self):


We run this function every minute with the stats generation (I think). How long does it take to run in a real-world environment? Do we have to optimize somewhere?

Upon user request, the driver can cache the image as a VM Template and reuse that to create the volume(s). This feature is useful when creating many volumes in parallel from the same image. We're not using the cinder built-in cache functionality because we need a few extra features: - the built-in cache doesn't account for shards. The cache entry will be placed on any backend/shard and could trigger a lot of slower cross-vc migrations when creating volumes from it. - the built-in cache doesn't have a periodic task for deleting the expired cache entries - we want to cache the images only when the customer requests it Users can request the image cache feature when creating the volume, by passing the use_image_cache='true' as a property (metadata). The feature must be enabled per backend, for example: ``` [vmware] enable_image_cache = true ``` This will enable the image cache feature for the vmware backend. The image templates will then be stored in a folder similar to the volumes folder: OpensStack/Project (vmware_image_cache)/Volumes, where {backend}_image_cache is used as a project name. The driver will periodically delete the cached images that are expired. The expiry time can be controlled via the property `image_cache_age_seconds` set on the backend configuration. Only images smaller than the configured `image_cache_max_size_gb` will be cached. Change-Id: I6f5e481f6997a180a455b47abe525b93bcf9aa4e

hemna · 2025-11-14T17:52:15Z

Also we can enable cinder's image cache on a per backend basis. the backend is the shard effectively. Yes there is no background job to clear out old images, but the great thing is, the cached images are just volumes under a tenant. They can easily be managed (deleted via aging out), via the nanny. Just fetch all the tenant's images, look at the date when it was created, and delete it via the standard volume api. I think we should try at least test/try the cinder baked in image cache in qa-de-1 first before going to this length to hack up the driver.

joker-at-work · 2025-11-17T08:12:59Z

The first comments on the PR provide the reasoning why we think the Cinder-based approach is not helping. But it's your call as service-owner in the end.

Scsabiii · 2025-11-17T08:45:38Z

My oppinion is, that the driver level image_cache is a must with VMware, and sharding.
The problem is, we are not fast enough with the migrations so, that we can't survive another gardener update cycle.
If the sharding is removed, it sill a problem with VMware, that even cinder has the image as source volume, the VMware driver thinks the source not "read-only" so it will attempt to create implicit snapshots, that will create another locking issue/conflict during the clone op. The image-template is a good solution on the driver level, because VMware knows that template VM's are ready-only, and does not create this implicit objects, so there is no problem in parallel cloning ...
Also the cross-shard migration can be slower than the direct glance fetch ..... As VMware does also copy 0-s .....

leust requested review from hemna and joker-at-work July 11, 2024 15:42

leust force-pushed the imagecache branch from b38add9 to fc481de Compare July 15, 2024 12:08

leust force-pushed the imagecache branch 2 times, most recently from da700b1 to c763bce Compare July 17, 2024 08:20

hemna previously approved these changes Jul 25, 2024

View reviewed changes

joker-at-work reviewed Aug 20, 2024

View reviewed changes

leust force-pushed the imagecache branch 2 times, most recently from 3f6d754 to 8333237 Compare August 26, 2024 06:47

hemna force-pushed the stable/wallaby-m3 branch from 2b59588 to 17e2b86 Compare August 27, 2024 15:13

leust force-pushed the imagecache branch from 8333237 to c2475be Compare September 3, 2024 09:49

leust dismissed hemna’s stale review via c2475be May 26, 2025 13:26

joker-at-work closed this Nov 17, 2025

Conversation

leust commented Jul 11, 2024

Uh oh!

leust commented Jul 11, 2024

Uh oh!

hemna commented Jul 11, 2024

Uh oh!

joker-at-work commented Jul 12, 2024

Uh oh!

leust commented Jul 15, 2024

Uh oh!

hemna commented Jul 15, 2024

Uh oh!

leust commented Jul 15, 2024

Uh oh!

hemna commented Jul 15, 2024

Uh oh!

leust commented Jul 17, 2024

Uh oh!

joker-at-work left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hemna commented Nov 14, 2025

Uh oh!

joker-at-work commented Nov 17, 2025

Uh oh!

Scsabiii commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Scsabiii commented Nov 17, 2025 •

edited

Loading