Virtual Machines based on Cloud Images

Author: Peter Boy (pboy) | Creation Date: N/A | Last update: N/A | Related Fedora Version(s): 33,34

As an alternative to using the Fedora Server Edition installation image and installation program, Anaconda, you can use the Fedora Cloud Base image. A base image incorporates a generic preconfiguration, as opposed to a cloud platform specific preconfigurations, e.g. Amazon or Google. Fedora Cloud Base Image is a pre-built Fedora server, ready to run in a virtual machine. Most distributions offer such an image. So you can use this guideline with adaptations also for a simple installation of a virtual machine with another distribution.

How it works

The Cloud image is used directly as ready to use (virtual) system disk in a virtual machine to be created. The first barrier is to perform the necessary initial basic configuration (network connection, root account, etc) so that the VM can start at all and you can log in. In a standard installation this is part of the distribution-specific installer, Anaconda in case of Fedora. In case of a Cloud image there is nothing to install and therefore no installer. Instead, a more or less standardized cloud-specific initialization procedure is used, cloud-init. In the absence of cloud, the system administrator must provide a replacement. The developers of cloud-init fortunately had some foresight and provided a 'nocloud' procedure of that said standardized cloud-specific initialization procedure. As you may guess, in a cloud centric development a nocloud option has a tough time. It remains somewhat of a challenge.

For an autonomous Fedora server, i.e. not integrated into a cloud or into a development environment such as Vagrant, we use virt-install, the libvirt standard installer. With version 3, as included in Fedora since 33, it provides an easy to use emulation of the cloud-init 'noCloud' procedure. It allows both a very simple installation in a default configuration using a single parameter and a very elaborate configuration using two quite simple configuration files. Before version 3 this was relatively laborious, since a special iso image had to be crafted to inject configuration data.

What you get

In particular, you get time. The workload is significantly lower and correspondingly faster. But by Cloud Base Image you (currently) don’t get an alternatively built but otherwise identical build of Fedora Server Edition. There are some subtle differences. For example, Fedora Server Edition uses xfs as its file system, Cloud Base Image still uses the older ext4. Fedora Server Edition now persists the network configuration completely and stringently in NetworkManager, Cloud Base Image still uses the old ifcfg plugin. Other differences are conceptual. For example, Cloud Image does not install a firewall. This function is usually managed by the cloud system. The use concept for the persistent storage is also different due to technical differences. But overall, the functionality is so far identical and the advantages noticeable that it is worthwhile and makes sense to use it.

How to proceed

First of all you need a working Fedora Server Edition including virtualization support added and libvirtd daemon active. We assume an internal network 'default' with virbr0, DHCP, and DNS set up as well (see section 'Add Virtualization Support'). External network connectivity will be provided by macvlan (ethernet interface) rsp. macvtap (libvirt naming).

Just a reminder, due to an inconvenience in systemd-resolved implementation in Fedora 33 and 34, you need a separate name service on the host if the host is to be able to address the virtual machines (e.g. dnsmasq). Details are described in the aforementioned guidelines.

Preparations

  1. Fetch a Fedora Cloud Base Image file, here F33, and store it into the directory /var/lib/libvirt/boot, which is the libvirt default location for images to install from by convention. Check the integrity of the download.

    […]# wget https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2  -O /var/lib/libvirt/boot/Fedora-Cloud-Base-33-1.2.x86_64.qcow2
    […]# wget https://getfedora.org/static/checksums/Fedora-Cloud-33-1.2-x86_64-CHECKSUM -O /var/lib/libvirt/boot/Fedora-Cloud-33-1.2-x86_64-CHECKSUM
    […]# sha256sum --ignore-missing -c *-CHECKSUM

    Because the *CHECKSUM file contains the values for all cloud images, we ignore the missing. So the check should result in one OK.

  2. Check the forwarding configuration:

    […]# cat /proc/sys/net/ipv4/ip_forward
    […]# cat /proc/sys/net/ipv6/conf/default/forwarding

    In both cases, an output of value of 1 is required. If necessary, activate forwarding temporarily until next reboot:

    […]# echo 1 > /proc/sys/net/ipv4/ip_forward
    […]# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

    For permanent setup create the following file:

    […]# vim /etc/sysctl.d/50-enable-forwarding.conf
    # local customizations
    #
    # enable forwarding for dual stack
    net.ipv4.ip_forwarding=1
    net.ipv6.conf.all.forwarding=1

    These preparations all done, the external access to the VMs should work flawlessly.

The virt-install program knows a large number of parameters. Only a few are needed here, which will be explained briefly.

Table 1. Virt-install Parameters

--name VM_NAME

Unique name of the VM to install as shown e.g.in VM list

--memory 3074

Amount of memory to allocate

--cpu host --vcpus 3

same cpu type as host, adjust numbers as appropriate

--os-type linux

Fix here, used by virt-install to determine defaults

--os-variant fedora33

Adjust distribution and version as needed

--import

Fixed, skips installation procedure and boots from the first (virtual) disk as specified by the first ‐-disk parameter.

--graphics none

Fixed, enforces a redirect of the VM login prompt to the host terminal window for immediate access.

--disk /var/lib/libvirt/images/VM_NAME.qcow2, format=qcow2,bus=virtio

disk image file, adjust VM_NAME

--network direct,source=enpXsY,source_mode=bridge, model=virtio

specify external netwok (macvlan) first, it will get the name eth0 as usual. Adjust interface name as appropriate.

--network bridge=virbr0,model=virtio

specify the internal network (libvirt generated bridge) second. It will get the name eth1 as usual.

--cloud-init

new with version 3 to handle nocloud configuration

Installation alternative 1: minimal configuration effort

This type of installation uses the --cloud-init parameter without any value or subparameters. This approach causes the generation and display of a root password shortly after the start of installation, enabling a one-time login. You have to note or copy it, of course. To get the full benefit, DHCP should be available for all Ethernet interfaces. Otherwise, the interface will be set up, but no connection will be established. Beyond that cloud-init is executed with sensible default settings. Finally, it is deactivated and not executed during subsequent boot processes.

The installation begins by creating a copy of the download image as a (fully installed) virtual disk in the directory /var/lib/libvirt/images, by convention the virtual disk pool.

Optionally, the size of the virtual disk can be increased. The default is about 5 GiB. You can resize the virtual disk later, too. Therefore, there is no reason to plan too generously in terms of size now. Due to the qcow2 format resizing does not affect the current image file size. It is dynamically adjusted as needed up to the maximum specified.

The example below adds 10 GiB to a total size of about 15 GiB.

[…]# cp  /var/lib/libvirt/boot/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 \
         /var/lib/libvirt/images/VM_NAME.qcow2
[…]# qemu-img resize  /var/lib/libvirt/images/VM_NAME.qcow2  +10G
[…]# qemu-img  info   /var/lib/libvirt/images/VM_NAME.qcow2

Thereafter, use the virt-install program to immediately start installation. The parameters pass all the required information. No further intervention, no further preparation is required.

[…]# virt-install  --name VM_NAME\
     --memory 3074  --cpu host --vcpus 3 --graphics none\
     --os-type linux --os-variant fedora33\
     --import  \
     --graphics none \
     --disk /var/lib/libvirt/images/VM_NAME.qcow2,format=qcow2,bus=virtio \
     --network type=direct,source=enp1s0,source_mode=bridge,model=virtio \
     --network bridge=virbr0,model=virtio  \
     --cloud-init

You see a lot of output:

WARNING  Defaulting to --cloud-init root-password-generate=yes,disable=yes

Starting install...
Password for first root login is: OtMQshytI0E8xZGD
Installation will continue in 10 seconds (press Enter to skip)...
Connected to Domain: VM_NAM
Escape character is ^] (Ctrl + ])
[    0.000000] Linux version … … …
…
…
…
[…] cloud-init[757]: Cloud-init v. 20.4 finished …  Datasource DataSourceNoCloudc …
[FAILED] Failed to start Execute cloud user/final scripts.
See 'systemctl status cloud-final.service' for details.
[  OK  ] Reached target Cloud-init target.

Fedora 34 (Cloud Edition)
Kernel 5.11.12-300.fc34.x86_64 on an x86_64 (ttyS0)

localhost login:

The VM terminal appears when installation is complete. Note that the first root login password is displayed early in the process and is used for the initial login. This password is single use and must be replace during the first login.

The error message is unsightly, but does not affect operation. It might be the reason for cloud-init service still enabled. You may disable it manually or remove it at all.

Still on the host you may check the network status:

[…]# less /var/lib/libvirt/dnsmasq/virbr0.status

[
  {
    "ip-address": "192.168.122.109",
    "mac-address": "52:54:00:57:35:3d",
    "client-id": "01:52:54:00:57:35:3d",
    "expiry-time": 1615665342
  }
]

As you see, the VM got an internal IP, but no hostname because no one has been set until now. That is one of the post-installation tasks to perform.

Post-Installation Tasks

The initially displayed password enables a login and forces the setting of a new one.

Of particular interest is the network configuration.

# ip a

If DHCP was available, a complete interface configuration is displayed.

Check for connectivity:

# ping host
# ping host.example.lan
# ping host.example.com
# ping guardian.co.uk

The VM can connect to internal and external destinations. The name resolution for the vm itself can’t work because the static hostname is not set yet.

The network configuration is stored in /etc/sysconfig/network-scripts. There is only a file for the first, external interface. The internal interface to libvirt virbr0 is generated with every boot process.

In case the virtual disk size has been changed, the partition sizes must be adjusted.

[…]# cfdisk  /dev/vda

The only partition should already have the adjusted size. Otherwise select resize and then write.

What remains is the resizing of the file system.

[…]# resize2fs -p /dev/vda1

Finally, let’s set the hostname

[…]# hostnamectl set-hostname  VM_NAME.example.lan

Exit and close the console by <ctrl>+].

You may reboot the VM and than check /var/lib/libvirt/dnsmasq/virbr0.status again. It’s now listing a hostname, internal name resolution is working now.

If your external DHCP server provides dynamic DNS as well, you should be able to connect to your VM from the public network:

[…]# ping VM_NAME.example.com

Last action is to enable autostart of the VM.

[…]# virsh  autostart  VM_NAME

Everything is working fine now, nearly out of the box. You would now start configuring the VM in detail according to its intended use. Just as it would be required after a standard installation.

In the end it takes only some 5 minutes to set up a fully functional system with minimal effort. It is ideal to quickly create a virtual machine for an ad-hoc solution or as an interim solution for a test.

And it is the ideal way to install a Cloud Base Image of another distribution, which may differ in detail from Fedora Cloud Base Image and its elaborate configuration as below. This sets the stage for tailoring an elaborate configuration of another guest distribution as described below for Fedora.

Installation alternative 2: elaborate configuration effort

Usually, a VM is more complex and there are several issues to deal with. There is a public facing interface that requires a firewall that is not included in the Fedora Cloud Base image. You have either to configure the host to protect the VM or install a firewall. Furthermore it is a peculiarity of the cloud-init process that the second, internal interface is not configured persistently. Instead, it is set up anew each time the system is booted. This makes it impossible to assign a firewall zone to this interface. The public interface also provides ssh access. So a root key file is needed to secure the login.

As soon as multiple VMs are to be installed, the above installation alternative 1 gets quite repetitive and requires a considerable amount of time. Instead, the cloud-init procedure can get detailed configuration information by 2 configuration files, meta-data and user-data. These files can be passed by reference via 2 subparameters of --cloud-init.

While you are free where to store these files, the boot directory is the preferred location, e.g. a subdirectory …/boot/cloud-init.

Preparing meta-data

The file referenced by meta-data contains information about the runtime environment, including a static network configuration, if required. Here only the mandatory parameter instance-id is filled in, which must be unique in a cloud environment, but can be chosen arbitrarily in a nocloud environment.

[…]# mkdir /var/lib/libvirt/boot/cloud-init
[…]# vim /var/lib/libvirt/boot/cloud-init/meta-data
instance-id: myapp

According to specifications a static network connection may get configured in meta-data. Basically it works. But some – obviously long standing – bugs require manual intervention in any case. So it is easier to configure a static network by adjusting the default initialization in the other, custom specific file user-data.

Preparing user-data

That file, referenced by user-data, holds the main configuration work. The tool is very powerful. Just about all configuration work that an administrator would perform to configure a server in detail can be recorded here. Configuration includes several steps.

  1. Setting the hostname

  2. Set up the user root, the public SSH key is copied into the file as well as set up the fallback account "hostmin" (or alike) which should also be able to log in by password. It will be assigned to the group wheel and his public key will be copied into the file

  3. Set up a first-time password for both users for initial login and to be changed immediately

  4. Install required additional packages, e.g. the firewall, fail2ban, postfix (needed by fail2ban) and custom specific software, maybe a webserver

  5. Some packages need additional configuration files

  6. The VM needs an update of all packages

  7. Several configuration commands are required

    1. Optional: convert interface eth0 to static

    2. Assign zone trusted to the interface eth1 (2nd position in the dbus path, so the order of the network parameters when calling libvirt is crucial!) and rename it according to naming convention. The modification also persists to a configuration file (still in /etc/sysconfig/network-scripts/ )

    3. Start the firewall and add required services

    4. If the size of the virtual disk has been changed, the file system must be updated.

    5. Finally disable cloud-init

The first line must necessarily contain some kind of shebang, which cloud-init uses to determine the format of the following data. The formatting itself is yaml.

[…]# vim /var/lib/libvirt/boot/cloud-init/user-data
#cloud-config

# (1) setting hostname
preserve_hostname: False
hostname: VM_NAME
fqdn: VM_NAME.EXAMPLE.COM

# (2) set up root and fallback account including rsa key copied into this file
users:
  - name: root
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQA...jSMt9rC4uKDPR8whgw==

  - name: hostmin
    groups: users,wheel
    ssh_pwauth: True
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAIAQDix...Mt9rC4uKDPR8whgw==

# (3) set up a first-time password for both accounts
chpasswd:
  list: |
    root:MY_PASSWORD
    hostmin:TOP_SECRET
  expire: True

# (4) install additional required packages
packages:
  - firewalld
  - postfix
  - fail2ban
  - vim
  - letsencrypt

# (5) some packages need additional configuration files
write_files:
  - path: /etc/fail2ban/jail.local
    content: |
      # /etc/fail2ban/jail.local
      # Jail configuration additions for local installation

      # Adjust the default configuration's default values
      [DEFAULT]
      ##ignoreip = <your save permanent network>/24  <your save permanent IP>/32
      bantime  = 6600
      backend = auto

      # The main configuration file defines all services but
      # deactivates them by default. We have to activate those neeeded
      [sshd]
      enabled = true

# (6) perform a package upgrade
package_upgrade: true

# (7) several configuration commands are executed on first boot
runcmd:
  # (a) If needed, convert interface eth0 as static
  #     IMPORTANT: external interface eth0 have to be specified
  #     BEFORE internal eth1 in virt-instal option list
  # comment in and modify as required
  ##- nmcli con mod path 1 ipv4.method static ipv4.addresses '<IPv4>/24'
  ##- nmcli con mod path 1 ipv4.gateway '<IPv4>' ipv4.dns '<IPv4>'
  ##- nmcli con mod path 1 ipv6.method static ipv6.addresses '<IPv6>/64'
  ##- nmcli con mod path 1 ipv6.gateway '<IPv6>' ipv6.dns '<IPv6>'
  ##- nmcli con up path 1
  #
  # (b) assign a zone to internal interface as well as some other adaptations.
  # results in the writing of a configuration file
  # IMPORTANT: internal interface have to be specified SECOND after external
  - nmcli con mod path 2 con-name eth1  connection.zone trusted
  - nmcli con mod path 2 ipv6.method disabled
  - nmcli con up  path 2

  # (c) activate and configure firewall and additional services
  - systemctl  enable  firewalld  --now
  - systemctl  enable  fail2ban   --now

  # (d) try to grow partition and filesystem just in case disk was enlarged
  #     Does no harm if not.
  - growpart  /dev/vda  1
  - resize2fs -p /dev/vda1

  # (e) finally disable cloud-init and reboot
  - systemctl  disable  cloud-init
  - reboot
# done

A concentrated overview of the user-data configuration options provides the examples section of the cloud-init project documentation.

Launch installation

After the configuration files are created, the virt-install process can be initiated. First copy the downloaded Cloud Image file and optionally adjust the virtual disk size.

[…]# cp  /var/lib/libvirt/boot/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 \
         /var/lib/libvirt/images/VM_NAME.qcow2
[…]# qemu-img  resize  /var/lib/libvirt/images/VM_NAME.qcow2  +10G
[…]# qemu-img  info    /var/lib/libvirt/images/VM_NAME.qcow2

Execute virt-install and adjust the values of CPU, memory, external network interface etc. to the requirements!

[…]# virt-install  --name VM_NAME \
     --memory 3074  --cpu host --vcpus 3 --graphics none \
     --os-type linux --os-variant fedora33 \
     --import  \
     --graphics none \
     --disk /var/lib/libvirt/images/VM_NAME.qcow2,format=qcow2,bus=virtio \
     --network type=direct,source=enpXsY,source_mode=bridge,model=virtio \
     --network bridge=virbr0,model=virtio  \
     --cloud-init meta-data=/var/lib/libvirt/boot/cloud-init/meta-data,user-data=/var/lib/libvirt/boot/cloud-init/user-data

It takes some time, be patient. After a while a login prompt is shown. Don’t try to login immediately. After some seconds the initialization process will continue. Finally, you see a message like

...
[  OK  ] Finished Network Manager Wait Online.
[  OK  ] Reached target Network is Online.

Fedora 34 (Cloud Edition)
Kernel 5.11.12-300.fc34.x86_64 on an x86_64 (ttyS0)

eth0: 192.168.yyy.zzz 2003:ca:xxxx:yyyy:zzzz:aa:bb:cc
eth1: 192.168.mmm.nnn
my-vm login:

If the network environment issues IP addresses based on MAC addresses via DHCP, add to the the first network configuration the MAC address:

--network type=direct,source=enpXsY,source_mode=bridge,mac=52:54:00:aa:bb:cc,model=virtio

Remember, that the first 3 pairs in the MAC address must be the sequence '52:54:00' for KVM virtual machines.

Check installation

Finally, login and check the installation. All interfaces should be up and running.

[…]#  ip a

Name resolution should work without further configuration

[…]# resolvectl domain
  Global:
  Link 2 (eth0): ~.
  Link 3 (eth1): <YOUR_DOMAIN>.lan
[…]# resolvectl dns
  Global:
  Link 2 (eth0): 213.133.98.98 2a01:4f8:0:1::add:1010
  Link 3 (eth1): 192.168.122.1

You should be able to connect to internal and external destinations

[…]# ping <YOUR_INTERNAL_HOST_NAME>.lan
[…]# ping <SOME_EXTERNAL_NAME>
[…]# ping6 <SOME_EXTERNAL_NAME>

Everything should work flawlessly.

Back on host finally enable autostart of the VM

[…]# virsh  autostart  VM_NAME

Everything done now.

Conclusion

Configuring the cloud-init process by virt-install version 3 is highly efficient and flexible. You may create a dedicated set of files for each VM or you may keep one set of generic files and adjust them by commenting in and out as required. A combination of both can be use. You can quickly and easily change configuration settings to test suitability for your purposes.

Thus, a configuration process is initiated quite comfortably that would otherwise be quite time-consuming. It is this performance that makes the use of cloud images so attractive.

In summary, the use of Cloud Base Images comes with some inconveniences and suffers from shortcomings in documentation, overall the combination of Cloud Base Images and virt-install version 3 is a great combination for creating virtual machines for Fedora Server Edition.

These nice people helped write this page:

Peter Boy, Jan Kuparinen

Want to help? Learn how to contribute to Fedora Docs.