Welcome to part #3 of this series following the development of instancetypes and preferences within KubeVirt!
What’s new
v1alpha2
https://github.com/kubevirt/kubevirt/pull/8282
A new v1alpha2 instancetype API version has now been introduced in the above PR from my colleague akrejcir. This switches the ControllerRevisions over to using complete objects instead of just the spec of the object. Amongst other things this means that kubectl can present the complete object to the user within the ControllerRevision as shown below:
$ kubectl.sh apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
virtualmachine.kubevirt.io/vm-cirros-csmall created
$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype
{
"kind": "VirtualMachineInstancetype",
"name": "csmall",
"revisionName": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1"
}
$ kubectl get controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1 -o json | jq .
{
"apiVersion": "apps/v1",
"data": {
"apiVersion": "instancetype.kubevirt.io/v1alpha2",
"kind": "VirtualMachineInstancetype",
"metadata": {
"creationTimestamp": "2022-09-30T12:20:19Z",
"generation": 1,
"name": "csmall",
"namespace": "default",
"resourceVersion": "10303",
"uid": "72c3a35b-6e18-487d-bebf-f73c7d4f4a40"
},
"spec": {
"cpu": {
"guest": 1
},
"memory": {
"guest": "128Mi"
}
}
},
"kind": "ControllerRevision",
"metadata": {
"creationTimestamp": "2022-09-30T12:20:19Z",
"name": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1",
"namespace": "default",
"ownerReferences": [
{
"apiVersion": "kubevirt.io/v1",
"blockOwnerDeletion": true,
"controller": true
"kind": "VirtualMachine",
"name": "vm-cirros-csmall",
"uid": "5216527a-1d31-4637-ad3a-b640cb9949a2"
}
],
"resourceVersion": "10307",
"uid": "a7bc784b-4cea-45d7-8432-15418e1dd7d3"
},
"revision": 0
}
Please note that while the API version has been incremented this new version is fully backwardly compatible with v1alpha1 and as a result requires no user modifications to existing v1alpha1 resources.
expand-spec APIs
https://github.com/kubevirt/kubevirt/pull/7549
akrejcir also landed two new subresource APIs that can take either a raw VirtualMachine definition or an existing VirtualMachine resource and expand the VirtualMachineInstanceSpec within using any referenced VirtualMachineInstancetype or VirtualMachinePreference resources.
expand-spec for existing VirtualMachines
The following expands the spec of a defined vm-cirros-csmall VirtualMachine resource that references the example csmall instancetype using diff to show the changes between the original and expanded definition returned by the API:
$ ./cluster-up/kubectl.sh apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
[..]
$ ./cluster-up/kubectl.sh proxy --port=8080 &
[..]
$ diff --color -u <(./cluster-up/kubectl.sh get vms/vm-cirros-csmall -o json | jq -S .spec.template.spec.domain) \
<(curl http://localhost:8080/apis/subresources.kubevirt.io/v1/namespaces/default/virtualmachines/vm-cirros-csmall/expand-spec | jq -S .spec.template.spec.domain)
[..]
--- /dev/fd/63 2022-10-05 15:51:23.599135528 +0100
+++ /dev/fd/62 2022-10-05 15:51:23.599135528 +0100
@@ -1,4 +1,9 @@
{
+ "cpu": {
+ "cores": 1,
+ "sockets": 1,
+ "threads": 1
+ },
"devices": {
"disks": [
{
@@ -16,5 +21,8 @@
"machine": {
"type": "q35"
},
+ "memory": {
+ "guest": "128Mi"
+ },
"resources": {}
}
expand-spec for a RAW VirtualMachine definition
The following expands the spec of a raw undefined VirtualMachine passed to the API that references the example csmall instancetype again using diff to show the changes between the original raw definition and the returned expanded definition:
$ ./cluster-up/kubectl.sh apply -f examples/csmall.yaml
[..]
$ ./cluster-up/kubectl.sh proxy --port=8080 &
[..]
$ diff --color -u <(jq -S .spec.template.spec.domain ./examples/vm-cirros-csmall.yaml) <(curl -X PUT -H "Content-Type: application/json" -d @./examples/vm-cirros-csmall.yaml http://localhost:8080/apis/subresources.kubevirt.io/v1/expand-spec
--- /dev/fd/63 2022-10-05 16:19:56.035111587 +0100
+++ /dev/fd/62 2022-10-05 16:19:56.035111587 +0100
@@ -1,4 +1,9 @@
{
+ "cpu": {
+ "cores": 1,
+ "sockets": 1,
+ "threads": 1
+ },
"devices": {
"disks": [
{
@@ -16,5 +21,8 @@
"machine": {
"type": "q35"
},
+ "memory": {
+ "guest": "128Mi"
+ },
"resources": {}
}
Please note that there remains some on-going work around the raw VirtualMachine definition API.
AutoattachInputDevice
https://github.com/kubevirt/kubevirt/pull/8006
A new AutoattachInputDevice toggle to control the attachment of a default input device has been introduced:
$ ./cluster-up/kubectl.sh apply -f examples/csmall.yaml
$ ./cluster-up/kubectl.sh apply -f - << EOF
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: demo-autoattachinputdevice
spec:
instancetype:
name: csmall
kind: virtualmachineinstancetype
running: true
template:
spec:
domain:
devices:
autoattachInputDevice: true
volumes:
- containerDisk:
image: registry:5000/kubevirt/cirros-container-disk-demo:devel
name: containerdisk
EOF
$ ./cluster-up/kubectl.sh get vmis/demo-autoattachinputdevice -o json | jq .spec.domain.devices.inputs
selecting docker as container runtime
[
{
"bus": "usb",
"name": "default-0",
"type": "tablet"
}
]
An associated PreferredAutoattachInputDevice preference has also been introduced to control AutoattachInputDevice along with the existing preferredInputType and preferredInputBus preferences:
$ ./cluster-up/kubectl.sh apply -f examples/csmall.yaml
[..]
$ ./cluster-up/kubectl.sh apply -f - << EOF
---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachinePreference
metadata:
name: preferredinputdevice
spec:
devices:
preferredAutoattachInputDevice: true
preferredInputType: tablet
preferredInputBus: virtio
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: demo-preferredinputdevice
spec:
instancetype:
name: csmall
kind: virtualmachineinstancetype
preference:
name: preferredinputdevice
kind: virtualmachinepreference
running: true
template:
spec:
domain:
devices: {}
volumes:
- containerDisk:
image: registry:5000/kubevirt/cirros-container-disk-demo:devel
name: containerdisk
EOF
[..]
./cluster-up/kubectl.sh get vms/demo-preferredinputdevice -o json | jq .spec.template.spec.domain.devices
{}
$ ./cluster-up/kubectl.sh get vmis/demo-preferredinputdevice -o json | jq .spec.domain.devices.autoattachInputDevice
true
$ ./cluster-up/kubectl.sh get vmis/demo-preferredinputdevice -o json | jq .spec.domain.devices.inputs
[
{
"bus": "virtio",
"name": "default-0",
"type": "tablet"
}
]
common-instancetypes
https://github.com/lyarwood/common-instancetypes
The KubeVirt project has for a while now provided a set of common-templates to help users to define VirtualMachines. These OpenShift/OKD templates cover a range of guest OS’s and workloads (server, desktop, highperformance etc).
I’ve created an instancetype based equivalent to this outside of KubeVirt for the time being. My common-instancetypes repo provides instancetypes and preferences covering all of the combinations covered by common-templates with some hopefully useful additions such as preferences for CirrOS and Alpine Linux.
The repo currently uses kustomize to generate everything so deployment into a cluster is extremely simple:
$ ./cluster-up/kubectl.sh kustomize https://github.com/lyarwood/common-instancetypes.git | ./cluster-up/kubectl.sh apply -f -
[..]
virtualmachineclusterinstancetype.instancetype.kubevirt.io/highperformance.large created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/highperformance.medium created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/highperformance.small created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/server.large created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/server.medium created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/server.small created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/server.tiny created
virtualmachineclusterpreference.instancetype.kubevirt.io/alpine created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7 created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7.desktop created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7.i440fx created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.8 created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.8.desktop created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.9 created
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.9.desktop created
virtualmachineclusterpreference.instancetype.kubevirt.io/cirros created
virtualmachineclusterpreference.instancetype.kubevirt.io/fedora.35 created
virtualmachineclusterpreference.instancetype.kubevirt.io/fedora.36 created
virtualmachineclusterpreference.instancetype.kubevirt.io/pc-i440fx created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7 created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7.desktop created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7.i440fx created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.8 created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.8.desktop created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.9 created
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.9.desktop created
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.18.04 created
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.20.04 created
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.22.04 created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.10 created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.10.virtio created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.11 created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.11.virtio created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k12 created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k12.virtio created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k16 created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k16.virtio created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k19 created
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k19.virtio created
virtualmachineinstancetype.instancetype.kubevirt.io/highperformance.large created
virtualmachineinstancetype.instancetype.kubevirt.io/highperformance.medium created
virtualmachineinstancetype.instancetype.kubevirt.io/highperformance.small created
virtualmachineinstancetype.instancetype.kubevirt.io/server.large created
virtualmachineinstancetype.instancetype.kubevirt.io/server.medium created
virtualmachineinstancetype.instancetype.kubevirt.io/server.small created
virtualmachineinstancetype.instancetype.kubevirt.io/server.tiny created
virtualmachinepreference.instancetype.kubevirt.io/alpine created
virtualmachinepreference.instancetype.kubevirt.io/centos.7 created
virtualmachinepreference.instancetype.kubevirt.io/centos.7.desktop created
virtualmachinepreference.instancetype.kubevirt.io/centos.7.i440fx created
virtualmachinepreference.instancetype.kubevirt.io/centos.8 created
virtualmachinepreference.instancetype.kubevirt.io/centos.8.desktop created
virtualmachinepreference.instancetype.kubevirt.io/centos.9 created
virtualmachinepreference.instancetype.kubevirt.io/centos.9.desktop created
virtualmachinepreference.instancetype.kubevirt.io/cirros created
virtualmachinepreference.instancetype.kubevirt.io/fedora.35 created
virtualmachinepreference.instancetype.kubevirt.io/fedora.36 created
virtualmachinepreference.instancetype.kubevirt.io/pc-i440fx created
virtualmachinepreference.instancetype.kubevirt.io/rhel.7 created
virtualmachinepreference.instancetype.kubevirt.io/rhel.7.desktop created
virtualmachinepreference.instancetype.kubevirt.io/rhel.7.i440fx created
virtualmachinepreference.instancetype.kubevirt.io/rhel.8 created
virtualmachinepreference.instancetype.kubevirt.io/rhel.8.desktop created
virtualmachinepreference.instancetype.kubevirt.io/rhel.9 created
virtualmachinepreference.instancetype.kubevirt.io/rhel.9.desktop created
virtualmachinepreference.instancetype.kubevirt.io/ubuntu.18.04 created
virtualmachinepreference.instancetype.kubevirt.io/ubuntu.20.04 created
virtualmachinepreference.instancetype.kubevirt.io/ubuntu.22.04 created
virtualmachinepreference.instancetype.kubevirt.io/windows.10 created
virtualmachinepreference.instancetype.kubevirt.io/windows.10.virtio created
virtualmachinepreference.instancetype.kubevirt.io/windows.11 created
virtualmachinepreference.instancetype.kubevirt.io/windows.11.virtio created
virtualmachinepreference.instancetype.kubevirt.io/windows.2k12 created
virtualmachinepreference.instancetype.kubevirt.io/windows.2k12.virtio created
virtualmachinepreference.instancetype.kubevirt.io/windows.2k16 created
virtualmachinepreference.instancetype.kubevirt.io/windows.2k16.virtio created
virtualmachinepreference.instancetype.kubevirt.io/windows.2k19 created
virtualmachinepreference.instancetype.kubevirt.io/windows.2k19.virtio created
Users can also deploy by generating specific resource Kinds such as VirtualMachineClusterPreferences below:
./cluster-up/kubectl.sh kustomize https://github.com/lyarwood/common-instancetypes.git/VirtualMachineClusterPreferences | ./cluster-up/kubectl.sh apply -f -
[..]
virtualmachineclusterpreference.instancetype.kubevirt.io/alpine unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7.i440fx unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.8 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.8.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.9 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.9.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/cirros unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/fedora.35 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/fedora.36 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/pc-i440fx unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7.i440fx unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.8 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.8.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.9 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.9.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.18.04 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.20.04 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.22.04 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.10 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.10.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.11 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.11.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k12 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k12.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k16 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k16.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k19 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k19.virtio unchanged
Finally users can also deploy using a set of generated bundles in the repo:
# ./cluster-up/kubectl.sh apply -f https://raw.githubusercontent.com/lyarwood/common-instancetypes/main/common-clusterpreferences-bundle.yaml
selecting docker as container runtime
virtualmachineclusterpreference.instancetype.kubevirt.io/alpine unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.7.i440fx unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.8 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.8.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.9 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/centos.9.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/cirros unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/fedora.35 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/fedora.36 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/pc-i440fx unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.7.i440fx unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.8 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.8.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.9 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/rhel.9.desktop unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.18.04 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.20.04 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/ubuntu.22.04 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.10 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.10.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.11 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.11.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k12 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k12.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k16 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k16.virtio unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k19 unchanged
virtualmachineclusterpreference.instancetype.kubevirt.io/windows.2k19.virtio unchanged
The next step with the repo is obviously to move it under the main KubeVirt namespace which will lots of housekeeping to get CI setup, generate releases etc.In the meantime issues and pull requests against the original repo would still be welcome and encouraged!
We also have plans to have the cluster wide instancetypes and preferences deployed by default by KubeVirt. This has yet to be raised formally with the community and as such a deployment mechanism hasn’t been agreed upon yet. Hopefully more on this in the near future.
Bug fixes
8142 - Referenced VirtualMachineInstancetypes can be deleted before a VirtualMachine starts
https://github.com/kubevirt/kubevirt/issues/8142
https://github.com/kubevirt/kubevirt/pull/8346
As described in the issue the VirtualMachine controller would previously wait until the first start of a VirtualMachine to create any ControllerRevisions for a referenced instancetype or preference. As such users could easily modify or even remove the referenced resources ahead of this first start causing failures when the request is eventually made:
# ./cluster-up/kubectl.sh apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
[..]
virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
virtualmachine.kubevirt.io/vm-cirros-csmall created
# ./cluster-up/kubectl.sh delete virtualmachineinstancetype/csmall
virtualmachineinstancetype.instancetype.kubevirt.io "csmall" deleted
# ./cluster-up/virtctl.sh start vm-cirros-csmall
Error starting VirtualMachine Internal error occurred: admission webhook "virtualmachine-validator.kubevirt.io" denied the request: Failure to find instancetype: virtualmachineinstancetypes.instancetype.kubevirt.io "csmall" not found
The fix here was to move the creation of these ControllerRevisions to earlier within the VirtualMachine controller reconcile loop, ensuring that they are created as soon as the VirtualMachine is seen for the first time.
$ ./cluster-up/kubectl.sh apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
[..]
virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
virtualmachine.kubevirt.io/vm-cirros-csmall created
$ ./cluster-up/kubectl.sh get vms/vm-cirros-csmall -o json | jq .spec.instancetype
selecting docker as container runtime
{
"kind": "VirtualMachineInstancetype",
"name": "csmall",
"revisionName": "vm-cirros-csmall-csmall-6486bc40-955a-480f-b38a-19372812e388-1"
}
8338 - PreferredMachineType never applied to VirtualMachineInstance
https://github.com/kubevirt/kubevirt/issues/8338
https://github.com/kubevirt/kubevirt/pull/8352
Preferences are only applied to a VirtualMachineInstance when a user has not already provided a corresponding value within their VirtualMachine [1]. This is an issue for PreferredMachineType however as the VirtualMachine mutation webhook always provides some kind of default [2], resulting in the PreferredMachineType never being applied to the VirtualMachineInstance.
[1] https://github.com/kubevirt/kubevirt/blob/bcfbd78d803e9868e0665b51878a2a093e9b74c2/pkg/instancetype/instancetype.go#L950-L952
[2] https://github.com/kubevirt/kubevirt/blob/bcfbd78d803e9868e0665b51878a2a093e9b74c2/pkg/virt-api/webhooks/mutating-webhook/mutators/vm-mutator.go#L98-L112
The fix here was to lookup and apply preferences during the VirtualMachine mutation webhook to ensure we applied PreferredMachineType when the user hasn’t already provided their own value.
$ ./cluster-up/kubectl.sh apply -f examples/csmall.yaml
[..]
$ ./cluster-up/kubectl.sh apply -f - << EOF
---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachinePreference
metadata:
name: preferredmachinetype
spec:
machine:
preferredMachineType: "pc-q35-6.2"
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: demo-preferredmachinetype
spec:
instancetype:
name: csmall
kind: virtualmachineinstancetype
preference:
name: preferredmachinetype
kind: virtualmachinepreference
running: true
template:
spec:
domain:
devices: {}
volumes:
- containerDisk:
image: registry:5000/kubevirt/cirros-container-disk-demo:devel
name: containerdisk
EOF
$ ./cluster-up/kubectl.sh get vms/demo-preferredmachinetype -o json | jq .spec.template.spec.domain.machine
selecting docker as container runtime
{
"type": "pc-q35-6.2"
}
8435 - VirtualMachineSnapshot of a VirtualMachine referencing a VirtualMachineInstancetype becomes stuck InProgress
https://github.com/kubevirt/kubevirt/issues/8435
https://github.com/kubevirt/kubevirt/pull/8448
As set out in the issue attempting to create a VirtualMachineSnapshot of a VirtualMachine referencing a VirtualMachineInstancetype would previously leave the VirtualMachineSnapshot InProgress as the VirtualMachine controller was unable to add the required snapshot finalizer.
The fix here was to ensure that any VirtualMachineInstancetype referenced was applied to a copy of the VirtualMachine when checking for conflicts in the VirtualMachine admission webhook, allowing the snapshot finalizer to later be added.
Documentation
kubevirt.io blog post
https://kubevirt.io/2022/KubeVirt-Introduction-of-instancetypes.html
After finally fixing a build issue with the blog we now have a post introducing the basic instancetype concepts with examples.
User-guide
https://kubevirt.io/user-guide/virtual_machines/instancetypes/
We now have basic user-guide documentation introduction instancetypes. Feedback welcome, please do /cc lyarwood on any issues or PRs related to this doc!
Misc
VirtualMachineInstancePreset deprecation in favor of VirtualMachineInstancetype
https://github.com/kubevirt/kubevirt/pull/8069
This deprecation has now landed. VirtualMachineInstancePresets are based on the PodPresets k8s resource and API that injected data into pods at creation time. However this API never graduated from alpha and was removed in 1.20 [1]. While useful there are some issues with the implementation that have resulted in alternative approaches such as VirtualMachineInstancetypes and VirtualMachinePreferences being made available within KubeVirt.
As per the CRD versioning docs this change updated the generated CRD definition of VirtualMachineInstancePreset marking the currently available versions of v1 and v1alpha3 as deprecated.
More context and discussion is also available on the mailing-list [3].
[1] https://github.com/kubernetes/kubernetes/pull/94090 [2] https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-deprecation [3] https://groups.google.com/g/kubevirt-dev/c/eM7JaDV_EU8
area/instancetype label on kubevirt/kubevirt
https://github.com/kubevirt/kubevirt/labels/area%2Finstancetype
We now have an area label for instancetypes on the kubevirt/kubevirt repo that I’ve been manually applying to PRs and issues. Please feel free to use this by commenting /area instancetype on anything you think it related to instancetypes! I do hope to automate this for specific files in the future.
Upcoming
Moving the API towards v1beta1
https://github.com/kubevirt/kubevirt/issues/8235
With the new v1alpha2 version landing I also wanted to again draw attention to the above issue tracking our progress to v1beta1. Obviously a long way to go but if you do have any suggested changes ahead of v1beta1 please feel free to comment there.
Support for default instancetype and preference PVC annotations
https://github.com/kubevirt/community/pull/190
https://github.com/kubevirt/kubevirt/pull/8480
This topic deserves its’ own blog post but for now I’d just like to highlight the design doc and WIP code series above looking at introducing support for default instancetype and preference annoations into KubeVirt. The following example demonstrates the current PVC support in the series but I’d also like to expand this to other volume types where possible. Again, feedback welcome on the design doc or code series itself!
$ wget https://github.com/cirros-dev/cirros/releases/download/0.5.2/cirros-0.5.2-x86_64-disk.img
[..]
$ ./cluster-up/virtctl.sh image-upload pvc cirros --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img
[..]
$ ./cluster-up/kubectl.sh kustomize https://github.com/lyarwood/common-instancetypes.git | ./cluster-up/kubectl.sh apply -f -
[..]
$ ./cluster-up/kubectl.sh annotate pvc/cirros instancetype.kubevirt.io/defaultInstancetype=server.medium instancetype.kubevirt.io/defaultPreference=rhel.7
[..]
$ cat <<EOF | ./cluster-up/kubectl.sh apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: cirros
spec:
running: true
template:
spec:
domain:
devices: {}
volumes:
- persistentVolumeClaim:
claimName: cirros
name: disk
inferInstancetype: true
inferPreference: true
EOF
[..]
$ ./cluster-up/kubectl.sh get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'
selecting docker as container runtime
{
"kind": "virtualmachineclusterinstancetype",
"name": "server.medium",
"revisionName": "cirros-server.medium-85cd3327-1825-45b1-9c8f-7ca18b2e4124-1"
}
{
"kind": "virtualmachineclusterpreference",
"name": "rhel.7",
"revisionName": "cirros-rhel.7-0b081d8a-f216-44e9-8248-9f0c373b2fdc-1"
}
Reintroduce use of instancetype informers within virt controller
https://github.com/kubevirt/kubevirt/pull/8537
The use of these informers was previously removed by ee4e266. After further discussions on the mailing-list however it has become clear that the removal of these informers from the virt-controller was not required and they can be reintroduced.
virtctl create vm based on instancetype and preferences
https://github.com/kubevirt/kubevirt/issues/7760
I’d like to make a start on this in the coming weeks. The basic idea being that the new command would generate a VirtualMachine definition we could then pipe to kubectl apply, something like the following:
$ virtctl create vm --instancetype csmall --preference cirros --pvc cirros-disk | kubectl apply -f -