KubeVirt instancetype.kubevirt.io Update #2

Comments

Welcome to part #2 of this series following the development of instancetypes and preferences within KubeVirt! Please note this is just a development journal of sorts, more formal documentation introducing and describing instancetypes will be forthcoming in the near future!

What’s new

s/VirtualMachineFlavor/VirtualMachineInstancetype/g

https://github.com/kubevirt/kubevirt/pull/8039

If you haven’t already guessed from the title of this post the long awaited rename has landed, goodbye flavors and hello instancetypes!

The PR is unfortunately huge but did include an interesting upgrade problem the fix for which will require additional cleanup in a future release.

Versioning with ControllerRevisions

https://github.com/kubevirt/kubevirt/pull/7875

Versioning through ControllerRevisions has been introduced. As previously discussed the underlying VirtualMachineInstancetypeSpec or VirtualMachinePreferenceSpec are stored by the VirtualMachine controller in a ControllerRevision unique to the VirtualMachine being started. A reference to the ControllerRevision is then added to the VirtualMachine for future look ups with the VirtualMachine also itself referenced as an owner of these ControllerRevisions ensuring their removal when the VirtualMachine is deleted.

Bug fixes

Upcoming

VirtualMachine subresource APIs to expand VirtualMachineInstancetype

https://github.com/kubevirt/kubevirt/pull/7549

This PR will introduce new VirtualMachine subresource APIs to expand a referenced instancetype or set of preferences for an existing VirtualMachine or one provided by the caller.

Hopefully these APIs will be useful to users and fellow KubeVirt/OpenShift devs who want to validate or just present a fully rendered version of their VirtualMachine in some way.

It’s worth noting that during the development of this feature we encountered some interesting OpenAPI behaviour that took a while to debug and fix.

AutoattachInputDevice and PreferredAutoattachInputDevice

https://github.com/kubevirt/kubevirt/pull/8006

While working on a possible future migration of the common-templates project to using VirtualMachineInstancetypes and VirtualMachinePreferences it was noted that we had no way of automatically attaching an input device to a VirtualMachine.

This change introduces both a AutoattachInputDevice attribute to control this in a vanilla VirtualMachines and a PreferredAutoattachInputDevice preference to control this behaviour from within a set of preferences.

The PR includes a simple rework of the application of DevicePreferences, applying them before any of the Autoattach logic fires within the VirtualMachine controller. This allows the PreferredAutoattach preferences to control the Autoattach logic with the original application of preferences after this logic has fired ensuring any remaining preferences are also applied to any new devices.

VirtualMachineInstancePreset deprecation in favor of VirtualMachineInstancetype

https://github.com/kubevirt/kubevirt/pull/8069

This proposal still has to be raised formally with the community but as set out in the PR I’d like to start the deprecation cycle of VirtualMachineInstancePreset now as VirtualMachineInstancetype starts to mature as a replacement.

Making DomainSpec optional

https://github.com/kubevirt/kubevirt/pull/7969

Previous work has gone into removing the need to define Disks for all referenced Volumes within a VirtualMachineInstanceSpec and also ensuring preferences are applied correctly to the automatically added Disks.

The end goal for this work has been to make the entire DomainSpec within VirtualMachineInstanceSpec optional, hopefully simplifying our VirtualMachine definitions further when used in conjunction with instancetypes and preferences.

Revisit the removal of SharedInformers

https://github.com/kubevirt/kubevirt/pull/7935

The use of SharedInformers within the webhooks and VirtualMachine controller had proven problematic and their use was previously removed.

While no discernable performance impact has been seen thus far this change will likely be revisited again in the near future as many controllers appear to have a pattern of retrying failed lookups using SharedInformers with vanilla client calls.

Referenced VirtualMachineInstancetypes can be deleted before a VirtualMachine starts

https://github.com/kubevirt/kubevirt/issues/8142

The bug above covers a race with the current versioning implementation. This race allows a user to delete a referenced instancetype before the VirtualMachine referencing it has started, stashing a copy of the instancetype in a ControllerRevision. For example:

# ./cluster-up/kubectl.sh apply -f examples/csmall.yaml 
virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
# ./cluster-up/kubectl.sh apply -f examples/vm-cirros-csmall.yaml 
virtualmachine.kubevirt.io/vm-cirros-csmall created
# ./cluster-up/kubectl.sh delete virtualmachineinstancetype/csmall
virtualmachineinstancetype.instancetype.kubevirt.io "csmall" deleted
# ./cluster-up/virtctl.sh start vm-cirros-csmall
Error starting VirtualMachine Internal error occurred: admission webhook "virtualmachine-validator.kubevirt.io" denied the request: Failure to find instancetype: virtualmachineinstancetypes.instancetype.kubevirt.io "csmall" not found

I believe we need one or more finalizers here ensuring that referenced instancetypes and preferences are not removed before they are stashed in a ControllerRevision.

An alternative to this would be to create ControllerRevisions within the VirtualMachine admission webhooks earlier in the lifecycle of a VirtualMachine. I had tried this originally but failed to successfully Patch the VirtualMachine with a reference back to the ControllerRevision, often seeing failures with the VirtualMachine controller attempting to reconcile the changes.

Moving the API to v1beta1

With the rename now complete and the future direction hopefully set out above I believe now is a good time to start looking into the graduation of the API itself from the experimental v1alpha1 stage to something more stable.

The Kubernetes API versioning documentation provides the following summary of the beta version:

  • The software is well tested. Enabling a feature is considered safe. Features are enabled by default.

  • The support for a feature will not be dropped, though the details may change.

  • The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, migration instructions are provided. Schema changes may require deleting, editing, and re-creating API objects. The editing process may not be straightforward. The migration may require downtime for applications that rely on the feature.

  • The software is not recommended for production uses. Subsequent releases may introduce incompatible changes. If you have multiple clusters which can be upgraded independently, you may be able to relax this restriction.

I believe the instancetype API can meet these criteria in the near future if it isn’t already and so I will be looking to start the process soon.

User-guide documentation

With the rename complete I have finally started drafting some upstream user-guide documentation that I hope to post in a PR soon.

Introductory kubevirt.io blog post(s)

Following on from the user-guide documentation I also plan on writing and publishing some material introducing instancetypes and preferences on the kubevirt.io blog.

Contents

comments powered by Disqus