Provisioning Fedora CoreOS on VMware

This guide shows how to provision new Fedora CoreOS (FCOS) nodes on the VMware hypervisor.

Fedora CoreOS supports VMware ESXi ≥ 6.5, VMware Workstation ≥ 16, and VMware Fusion ≥ 12. It may be possible to modify the metadata of the OVF to run in older VMware products, but compatibility and supportability cannot be guaranteed.

Prerequisites

Before provisioning an FCOS machine, you must have an Ignition configuration file containing your customizations. If you do not have one, see Producing an Ignition File.

You also need to have access to a working VMware infrastructure, supporting VMs with at least hardware version 13. The examples below use the govc command-line tool for remote vSphere provisioning and the ovftool for local Workstation or Fusion provisoning.

Downloading the OVA

Fedora CoreOS is designed to be updated automatically, with different schedules per stream. Once you have picked the relevant stream, you can download the latest OVA:

STREAM="stable"
coreos-installer download -s "${STREAM}" -p vmware -f ova

Alternatively, OVA images can be manually downloaded from the download page.

Encoding Ignition configuration

For the vmware provider, Ignition requires two "guestinfo" fields to be present when the VM is first booted:

  • guestinfo.ignition.config.data.encoding: the encoding of the Ignition configuration.

  • guestinfo.ignition.config.data: the content of the Ignition configuration, encoded according to the format above.

For maximum compatibility, it is recommended to use base64 encoding and to prepare the Ignition configuration as such:

CONFIG_ENCODING='base64'
CONFIG_ENCODED=$(cat example.ign | base64 -w0 -)

An alternative to plain base64 encoding is gzip+base64 as described in the Ignition supported platforms. This is especially useful when submitting the Ignition config via govc as an inline parameter. In that case the encoded config is limited to slightly under 128 KiB on Linux, 256 KiB on macOS, and 32 KiB on Windows (8 KiB if using cmd.exe or PowerShell). If your config is larger than that limit, you may be able to submit it inline after compressing it with gzip.

CONFIG_ENCODING='gzip+base64'
CONFIG_ENCODED=$(cat example.ign | gzip -9 | base64 -w0 -)

Booting a new VM on Workstation or Fusion

This section shows how to use Workstation and Fusion facilities to configure and run VMs from the command-line. Some steps can potentially be performed via the graphical UI too.

Importing the OVA

The downloaded OVA has to be imported into the Workstation or Fusion library locally. At the same time the Ignition has to be provided for it to be applied to the VM.

VM_NAME='fcos-node01'
FCOS_OVA='./ova-templates/fedora-coreos-31.20200210.3.0-vmware.x86_64.ova'
LIBRARY="$HOME/Virtual Machines.localized"
ovftool \
  --powerOffTarget \
  --name="${VM_NAME}" \
  --allowExtraConfig \
  --extraConfig:guestinfo.ignition.config.data.encoding="${CONFIG_ENCODING}" \
  --extraConfig:guestinfo.ignition.config.data="${CONFIG_ENCODED}" \
  "${FCOS_OVA}" "${LIBRARY}"

Afterwards you can refresh the list of VMs in the Workstation or Fusion UI and the new fcos-node01 VM should appear ready for booting. Its hardware configuration can be further customized at this point, and then powered-up.

Booting a new VM on vSphere

This section shows how to use vSphere facilities to configure and run VMs from the command-line. Similar steps can be performed via the graphical UI too.

Importing the OVA

The downloaded OVA has to be first imported into vSphere library:

FCOS_OVA='./ova-templates/fedora-coreos-31.20200210.3.0-vmware.x86_64.ova'
LIBRARY='fcos-images'
TEMPLATE_NAME='fcos-31.20200210.3.0'
govc session.login -u 'user:password@host'
govc library.create "${LIBRARY}"
govc library.import -n "${TEMPLATE_NAME}" "${LIBRARY}" "${FCOS_OVA}"

Setting up a new VM

You can now deploy a new VM, starting from the template in the library and the encoded Ignition configuration:

VM_NAME='fcos-node01'
govc library.deploy "${LIBRARY}/${TEMPLATE_NAME}" "${VM_NAME}"
govc vm.change -vm "${VM_NAME}" -e "guestinfo.ignition.config.data.encoding=${CONFIG_ENCODING}"
govc vm.change -vm "${VM_NAME}" -e "guestinfo.ignition.config.data=${CONFIG_ENCODED}"

A new fcos-node01 VM is now available for booting. Its hardware configuration can be further customized at this point, and then powered-up:

govc vm.info -e "${VM_NAME}"
govc vm.power -on "${VM_NAME}"

First-boot networking and Ignition

Ignition supports referencing remote content in configuration and fetching it at provisioning time. For this reason, on first-boot FCOS instances try to perform network autoconfiguration via DHCP.

If your VMware setup employs static network configuration instead, you can override this automatic DHCP setup with your own custom configuration. Custom networking command-line ip= parameter can be configured via guestinfo properties as shown below, before booting a VM for the first time.

The provisioning flow follows the usual steps, plus an additional guestinfo.afterburn.initrd.network-kargs entry.

if you are using a provisioning method other than govc, make sure that the guestinfo attribute is provisioned in the VM’s Advanced Configuration Parameters (also known as ExtraConfig). Some management tools may default to a vApp Property instead, which does not work in this scenario. 
VM_NAME='fcos-node02'
IFACE='ens192'
IPCFG="ip=192.0.2.42::192.0.2.1:255.255.255.0:${VM_NAME}:${IFACE}:off"

govc library.deploy "${LIBRARY}/${TEMPLATE_NAME}" "${VM_NAME}"
govc vm.change -vm "${VM_NAME}" -e "guestinfo.ignition.config.data.encoding=${CONFIG_ENCODING}"
govc vm.change -vm "${VM_NAME}" -e "guestinfo.ignition.config.data=${CONFIG_ENCODED}"
govc vm.change -vm "${VM_NAME}" -e "guestinfo.afterburn.initrd.network-kargs=${IPCFG}"
govc vm.info -e "${VM_NAME}"
govc vm.power -on "${VM_NAME}"

The full syntax of the ip= parameter is documented in Dracut manpages.

For further information on first-boot networking, see Afterburn documentation.

Troubleshooting First-boot Problems

You may encounter problems with your Ignition configuration that require access to the system log which appears during first-boot. To make a copy of the system log you can attach a serial device to the VM before booting. vSphere as well as Workstation and Fusion allow this and will save the output to a file of your choice.

To attach a serial device simply modify the hardware settings of the powered off VM and add a Serial Port. Select the destination and name of the file to be created. Afterwards power on the VM. When encountering an error, check the file you initially specified - it should contain a copy of the system log.

The serial device can also be added to the VM via govc as described in the official usage documentation:

VM_NAME='fcos-node01'

govc device.serial.add -vm "${VM_NAME}"
govc device.serial.connect -vm "${VM_NAME}" "[datastore] ${VM_NAME}/console.log"

Modifying OVF metadata

While we provide these instructions for modifying the OVF metadata, we cannot guarantee that any modifications to the OVF metadata will result in a usable guest VM.

Fedora CoreOS is intended to run on generally supported releases of VMware ESXi, VMware Workstation, and VMware Fusion. Accordingly, the Fedora CoreOS VMware OVA image specifies a virtual hardware version that may not be compatible with older, unsupported VMware products. However, you can modify the image’s OVF metadata to specify an older virtual hardware version.

The VMware OVA is simply a tarball that contains the files disk.vmdk and coreos.ovf. In order to edit the metadata used by FCOS as a guest VM, you should untar the OVA artifact, edit the OVF file, then create a new OVA file.

The example commands below change the OVF hardware version from the preconfigured value to hardware version 13. (Note: the defaults in the OVF are subject to change.)

tar -xvf fedora-coreos-36.20220505.3.2-vmware.x86_64.ova
sed -iE 's/vmx-[0-9]*/vmx-13/' coreos.ovf
tar -H posix -cvf fedora-coreos-36.20220505.3.2-vmware-vmx-13.x86_64.ova coreos.ovf disk.vmdk