auto

KubeVirt Summit 2023

Streamlining VirtualMachine creation

Lee Yarwood & Felix Matouschek

Introductions

Lee Yarwood
lyarwood@redhat.com
Software Engineer @ Red Hat
https://github.com/lyarwood

Felix Matouschek
fmatouschek@redhat.com
Software Engineer @ Red Hat
https://github.com/0xFelix

Agenda

  • Why?
  • Goals
  • tl;dr
  • API, CRDs and features
  • virtctl improvements
  • Next?

fit

https://kubevirt.io/user-guide/virtual_machines/instancetypes

Why?

  • The VirtualMachine API/CRD is rich but overwhelming for users
  • Most IaaS and public cloud providers have a simplified creation process
  • Two previous solutions to the problem within KubeVirt are no longer viable
    • VirtualMachinePresets are deprecated as of KubeVirt v0.57.0
    • Templates are a downstream concept from Red Hat

Goals

  • Simplify VirtualMachine creation for users
  • By reducing a users decision matrix during VirtualMachine creation
  • Do this in a KubeVirt native and upstream first way
  • Using separate resources to encapsulate workload resources and runtime preferences

tl;dr

  • We have introduced two new families of CRDs, each family containing a namespaced and cluster-wide CRD with a common spec
  • Instance types define resource related attributes of a VirtualMachine
  • Preferences define preferred values for the remaining runtime attributes of the VirtualMachine
  • Only a single instance type and preference can be referenced by a VirtualMachine
  • These are then applied to and expand within VirtualMachineInstance at runtime
https://kubevirt.io/user-guide/virtual_machines/instancetypes

tl;dr

fit

https://kubevirt.io/user-guide/virtual_machines/instancetypes

tl;dr

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: windows-ql8vf1
spec:
  dataVolumeTemplates:
    - apiVersion: cdi.kubevirt.io/v1beta1
      kind: DataVolume
      metadata:
        annotations:
          cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
        creationTimestamp: null
        name: windows-ql8vf1
      spec:
        source:
          blank: {}
        storage:
          resources:
            requests:
              storage: 60Gi
  running: false
  template:
    spec:
      domain:
        clock:
          timer:
            hpet:
              present: false
            hyperv: {}
            pit:
              tickPolicy: delay
            rtc:
              tickPolicy: catchup
          utc: {}
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
            - disk:
                bus: sata
              name: rootdisk
          inputs:
            - bus: usb
              name: tablet
              type: tablet
          interfaces:
            - macAddress: '02:3e:ee:00:00:00'
              masquerade: {}
              model: e1000e
              name: default
        features:
          acpi: {}
          apic: {}
          hyperv:
            frequencies: {}
            ipi: {}
            reenlightenment: {}
            relaxed: {}
            reset: {}
            runtime: {}
            spinlocks:
              spinlocks: 8191
            synic: {}
            synictimer:
              direct: {}
            tlbflush: {}
            vapic: {}
            vpindex: {}
        machine:
          type: pc-q35-rhel9.0.0
        resources:
          requests:
            memory: 4Gi
      networks:
        - name: default
          pod: {}
      volumes:
        - dataVolume:
            name: windows-ql8vf1
          name: rootdisk


apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: windows-ql8vf1
spec:
  instancetype:
    name: n1.medium
  preference:
    name: windows.10
  dataVolumeTemplates:
    - apiVersion: cdi.kubevirt.io/v1beta1
      kind: DataVolume
      metadata:
        annotations:
          cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
        creationTimestamp: null
        name: windows-ql8vf1
      spec:
        source:
          blank: {}
        storage:
          resources:
            requests:
              storage: 60Gi
  running: false
  template:
    spec:
      domain:
        devices: {}
      volumes:
        - dataVolume:
            name: windows-ql8vf1
          name: rootdisk

API, CRDs and features

  • instancetype.kubevirt.io
    • v1alpha1 as of the v0.56.0 release of KubeVirt
    • v1alpha2 as of the v0.58.0 release of KubeVirt
    • v1beta1 as of the upcoming v0.60.0 release of KubeVirt (planned)
  • CustomResourceDefinitions:
    • VirtualMachineInstancetype
    • VirtualMachineClusterInstancetype
    • VirtualMachinePreference
    • VirtualMachineClusterPreference
https://kubevirt.io/user-guide/virtual_machines/instancetypes

VirtualMachineInstancetype & VirtualMachineClusterInstancetype

  • Resource related attributes of VirtualMachineInstanceSpec
  • Required values, will conflict with user choices in the VirtualMachine
  • VirtualMachine can reference only one
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachineClusterInstancetype
metadata:
  annotations:
    instancetype.kubevirt.io/class: General Purpose
    instancetype.kubevirt.io/description: |-
      The N Series is quite neutral and provides resources for
      general purpose applications.
      *N* is the abbreviation for "Neutral", hinting at the neutral
      attitude towards workloads.
      VMs of instance types will share physical CPU cores on a
      time-slice basis with other VMs.
    instancetype.kubevirt.io/version: "1"
  labels:
    instancetype.kubevirt.io/vendor: kubevirt.io
  name: n1.medium
spec:
  cpu:
    guest: 1
  memory:
    guest: 4Gi
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#virtualmachineinstancetype

VirtualMachinePreference & VirtualMachineClusterPreference

  • All remaining attributes of VirtualMachineInstanceSpec
  • Preferred values, do not overwrite or conflict with user choices in VirtualMachine
  • VirtualMachine can reference only one
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachineClusterPreference
metadata:
  annotations:
    iconClass: icon-fedora
    openshift.io/display-name: Fedora
    openshift.io/documentation-url: https://github.com/kubevirt/common-instancetypes
    openshift.io/provider-display-name: KubeVirt
    openshift.io/support-url: https://github.com/kubevirt/common-instancetypes/issues
    tags: hidden,kubevirt,fedora
  labels:
    instancetype.kubevirt.io/os-type: linux
    instancetype.kubevirt.io/vendor: kubevirt.io
  name: fedora
spec:
  devices:
    preferredDiskBus: virtio
    preferredInterfaceModel: virtio
    preferredNetworkInterfaceMultiQueue: true
    preferredRng: {}
  features:
    preferredSmm: {}
  firmware:
    preferredUseEfi: true
    preferredUseSecureBoot: true
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#virtualmachinepreference

InstancetypeMatcher & PreferenceMatcher

  • Name
  • Kind (defaults to cluster-wide)
  • RevisionName
  • InferFromVolume
  • No support for cross namespace references
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  creationTimestamp: null
  name: vm-lt2gg
spec:
  instancetype:
    name: n1.medium
  preference:
    name: fedora
  runStrategy: Always
  template:
    metadata:
      creationTimestamp: null
    spec:
      domain:
        devices: {}
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - containerDisk:
          image: https://quay.io/repository/containerdisks/fedora:latest
        name: fedora
status: {}
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#virtualmachine

InstancetypeMatcher & PreferenceMatcher

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  creationTimestamp: null
  name: vm-lt2gg
spec:
  instancetype:
    name: n1.medium
  preference:
    name: fedora
  runStrategy: Always
  template:
    metadata:
      creationTimestamp: null
    spec:
      domain:
        devices: {}
      volumes:
      - containerDisk:
          image: https://quay.io/repository/containerdisks/fedora:latest
        name: fedora
status: {}

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  annotations:
    kubevirt.io/cluster-instancetype-name: n1.medium
    kubevirt.io/cluster-preference-name: fedora
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1alpha3
    kubevirt.io/vm-generation: "2"
  creationTimestamp: "2023-03-27T15:30:44Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  - foregroundDeleteVirtualMachine
  generation: 4
  name: vm-lt2gg
  namespace: default
[..]
spec:
  domain:
    cpu:
      cores: 1
      model: host-model
      sockets: 1
      threads: 1
    devices:
      disks:
      - disk:
          bus: virtio
        name: fedora
      interfaces:
      - bridge: {}
        model: virtio
        name: default
      networkInterfaceMultiqueue: true
      rng: {}
    features:
      acpi:
        enabled: true
      smm:
        enabled: true
    firmware:
      bootloader:
        efi:
          secureBoot: true
      uuid: d201a615-d331-5a55-8403-dbc738c919b8
    machine:
      type: q35
    memory:
      guest: 4Gi
    resources:
      requests:
        memory: 4Gi
  networks:
  - name: default
    pod: {}
  volumes:
  - containerDisk:
      image: https://quay.io/repository/containerdisks/fedora:latest
      imagePullPolicy: Always
    name: fedora
[..]
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#examples

RevisionName

  • Name of a ControllerRevision containing a point in time copy of the resource
  • Created when the VirtualMachine is first seen by KubeVirt (specifically the VirtualMachine controller)
  • Ensures the same VirtualMachineInstance is always created when starting the VirtualMachine
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example
spec:
  instancetype:
    kind: VirtualMachineInstancetype
    name: csmall
    revisionName: example-csmall-f73c7d4f4a40-1
  preference:
    kind: VirtualMachinePreference
    name: cirros
    revisionName: example-cirros-143571b5489c-1
[..]
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#versioning

InferFromVolume - New in v0.59.0

  • Takes the Name of a Volume listed in the VirtualMachine
  • Looks for Labels to determine default Name and Kind
  • Kind is optional and defaults to the cluster wide resource
  • Supports DataVolume, DataSource and PVC based volumes
  • Failure to find the resource and/or labels will cause the creation request to be rejected
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-sbdvs
spec:
  dataVolumeTemplates:
  - metadata:
      name: fedora
    spec:
      source:
        pvc:
          name: fedora-pvc
          namespace: default
      storage:
        resources: {}
  instancetype:
    inferFromVolume: fedora
  preference:
    inferFromVolume: fedora
[..]
  template:
    spec:
[..]
      volumes:
      - dataVolume:
          name: fedora
        name: fedora
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#inferfromvolume

InferFromVolume - New in v0.59.0

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-sbdvs
spec:
  dataVolumeTemplates:
  - metadata:
      name: fedora
    spec:
      source:
        pvc:
          name: fedora-pvc
          namespace: default
      storage:
        resources: {}
  instancetype:
    inferFromVolume: fedora
  preference:
    inferFromVolume: fedora
[..]
  template:
    spec:
[..]
      volumes:
      - dataVolume:
          name: fedora
        name: fedora

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
[..]
  labels:
    instancetype.kubevirt.io/default-instancetype: n1.medium
    instancetype.kubevirt.io/default-preference: fedora
  name: fedora-pvc
  namespace: default
[..]
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#inferfromvolume

InferFromVolume - New in v0.59.0

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-sbdvs
spec:
  dataVolumeTemplates:
  - metadata:
      name: fedora
    spec:
      source:
        pvc:
          name: fedora-pvc
          namespace: default
      storage:
        resources: {}
  instancetype:
    inferFromVolume: fedora
  preference:
    inferFromVolume: fedora
[..]
  template:
    spec:
[..]
      volumes:
      - dataVolume:
          name: fedora
        name: fedora

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
[..]
  name: vm-sbdvs
  namespace: default
spec:
  dataVolumeTemplates:
  - metadata:
      name: fedora
    spec:
      source:
        pvc:
          name: fedora-pvc
          namespace: default
      storage:
        resources: {}
  instancetype:
    kind: virtualmachineclusterinstancetype
    name: n1.medium
    revisionName: vm-sbdvs-n1.medium-3fc9183cab3a-1
  preference:
    kind: virtualmachineclusterpreference
    name: fedora
    revisionName: vm-sbdvs-fedora-f26c072cbea5-1
[..]
  template:
    spec:
[..]
      volumes:
      - dataVolume:
          name: fedora
        name: fedora
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#inferfromvolume

common-instancetypes v0.1.0

  • Set of Kustomize based instance type and preference resources for KubeVirt
  • Provides 29 instance types & 28 preferences
  • Both namespaced and cluster-wide resources provided
$ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git | kubectl apply -f -
virtualmachineclusterinstancetype.instancetype.kubevirt.io/cx1.2xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/cx1.4xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/cx1.8xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/cx1.large created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/cx1.medium created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/cx1.xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/gn1.2xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/gn1.4xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/gn1.8xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/gn1.xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/highperformance.large created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/highperformance.medium created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/highperformance.small created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/m1.2xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/m1.4xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/m1.8xlarge created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/m1.large created
virtualmachineclusterinstancetype.instancetype.kubevirt.io/m1.xlarge created
[..]
https://kubevirt.io/user-guide/virtual_machines/instancetypes/#common-instancetypes

virtctl improvements

  • Given the new API and CRDs can we also provide a better CLI UX?

What do others do?

GCP

gcloud compute instances create myvm --image-family=rhel-9

AWS

aws ec2 run-instances --image-id ami-0abcdef1234567890

Azure

az vm create -n myvm -g myrg --image RedHat

OpenStack

openstack server create --flavor small --image rhel-9

Introducing: virtctl create vm

Away with cloning the same template definition over and over again

  • Available starting with release v0.59.0
  • Fixed set of CLI flags to adjust VirtualMachine parameters
    • e.g. name, boot volume, instance type and preference
    • will be extended in the future
  • Outputs manifests which can be piped into kubectl or oc
    • Useable on Kubernetes and OKD / OpenShift (!)
https://kubevirt.io/user-guide/virtual_machines/creating_vms

Making it really easy

Inferring instance type and preference

virtctl create vm --infer-instancetype --infer-preference --volume-datasource kubevirt-os-images/fedora
  • Additional flags still required
  • Will make creating VirtualMachines really easy once enabled by default
https://kubevirt.io/user-guide/virtual_machines/creating_vms

Usage of virtctl create vm

--name string                        Specify the name of the VM. (default "vm-6z7w9")
--run-strategy string                Specify the RunStrategy of the VM. (default "Always")
--termination-grace-period int       Specify the termination grace period of the VM. (default 180)
--instancetype string                Specify the Instance Type of the VM.
--infer-instancetype                 Specify that the Instance Type of the VM is inferred from the booted volume.
--preference string                  Specify the Preference of the VM.
--infer-preference                   Specify that the Preference of the VM is inferred from the booted volume.
--volume-containerdisk stringArray   Specify a containerdisk to be used by the VM. Can be provided multiple times.
                                     Supported parameters: name:string,src:string
--volume-datasource stringArray      Specify a DataSource to be cloned by the VM. Can be provided multiple times.
                                     Supported parameters: name:string,src:string,size:resource.Quantity
--volume-clone-pvc stringArray       Specify a PVC to be cloned by the VM. Can be provided multiple times.
                                     Supported parameters: name:string,src:string,size:resource.Quantity
--volume-pvc stringArray             Specify a PVCs to be used by the VM. Can be provided multiple times.
                                     Supported parameters: name:string,src:string
--volume-blank stringArray           Specify a blank volume to be used by the VM. Can be provided multiple times.
                                     Supported parameters: name:string,size:resource.Quantity
--cloud-init-user-data string        Specify the base64 encoded cloud-init user data of the VM.
--cloud-init-network-data string     Specify the base64 encoded cloud-init network data of the VM.
https://kubevirt.io/user-guide/virtual_machines/creating_vms

Next?

  • v1beta1 with Kubevirt v0.60.0
    • Add more virtctl flags
    • Reduce number of created ControllerRevisions
    • Resource requirements provided by preferences
    • Required attributes provided by preferences
  • v1 with KubeVirt >= v0.61.0
    • Possible deployment of kubevirt/common-instancetypes by virt-operator

Questions?

fit

Thank you!

Caveat about time Q&A at the end etc.....