Managing virtual machines with Cockpit

Peter Boy අනුවාදය F35-F36 Last review: 2022-05-10

A virtual machine is created under certain assumptions about required memory, data storage, processing capacity, and so on. All too often, these assumptions need to be adjusted based on actual use practices, and require additional configuration work by a system administrator. Cockpit greatly simplifies and facilitates this work, especially for the Fedora Server Edition, that do not include a graphical interface.

The article describes selected often recurring reconfiguration tasks.

Manage virtual machines configurations

In your local browser, open Cockpit on the host system using port 9090, e.g. https://example.com:9090. Log in either as root or with your administrative account. Select "Virtual Machines" in the left navigation column.

Cockpit Virtual Machines Overview

If there is no tab "Virtual Machines" the corresponding Cockpit module, cockpit-machines, is not installed yet. Consult the guide Adding Virtualization Support for information how to install and prepare the module.

The virtual machines overview page lists in the central area all virtual machines installed along with their current state. In the example above there are 3 virtual machines, 2 of them running. The most frequently executed action, shutdown rsp. launch, is directly accessible as a button. Additional commonly needed functions are offered in a drop-down menu to the right of it. It even offers a (simple) migration to another machine.

Manage a virtual machines basic host runtime environment

If you click onto a VMs name, a details page opens. In the top half it shows the basic runtime parameter along with a terminal window into the VM.

Cockpit basic virtual machines runtime parameter

Some of the parameters displayed are mostly informative, e.g. the hypervisor details, or you will hardly ever want to change them, e.g. the CPU type. These things can become useful when it comes to diagnosing issues.

More interesting is the option to quickly enable and disable auto-start on host reboot [1]. An example would be a (inactive) backup VM that can take over when problems with the main VM arise. A kind of 'cold-standby', maybe the only possibility in case of tight CPU and RAM resources on a machine. The action is taken immediately and is effective (on the next host reboot), no need for an explicit submit.

Very useful in everyday operations are the real-time information about the resource consumption for CPU and RAM [2] and the simultaneous ability to adjust these parameters [3]. Although you will need to reboot the VM to make these changes or have them take effect.

And of course, especially convenient is the terminal window into the virtual machine. An uncomplicated way to quickly perform a CLI operation in the graphical interface - the best of both worlds.

Manage a virtual machines storage

A VM usually uses a (virtual) disk provided as a disk image file in the Libvirt pool images (/var/lib/libvirt/images), either in qcow2 or raw format. And managing storage means manipulation of the respective file.

Increase a virtual disk

Cockpit is powerful but not (yet) perfect. The current version does not provide a graphical tool for that purpose. The system administrator still has to rely on CLI tools like qemu-img or the virt-* utilities of guestsfs-tools. Fortunately, we just have to use the CLI tool to increase the virtual disk image while the VM is stopped. Afterwards, we can start the VM again and use the VM’s Cockpit to reconfigure the storage.

As an example, use vm2 Cockpit to display the current configuration of storage. Open http://vm2:9090 (again, adjust the address as appropriate) in your browser, log in, and select Storage from the left navigation column.

Cockpit filesystems overview

In the Filesystems section of the central content area [1] there are 2 file systems: root of 11 GiB and boot of 1 GiB. Additional information is in the right column. The Devices section indicates a Volume group of 11 GiB [2]. This implies that the root file system completely fills this Volume group. The Drives section lists a virtual disk of 12 GiB [3]. This implies that this disk is completely filled with the two file systems.

After shutting down vm2, we can increase the virtual disk by 20 GiB to a total of 32 GiB in the hosts Cockpit terminal.

[…]# qemu-img  resize  /var/lib/libvirt/images/vm2.qcow2  +20G
Image resized.
[…]# qemu-img  info  /var/lib/libvirt/images/vm2.qcow2
file format: qcow2
virtual size: 32 GiB (34359738368 bytes)
disk size: 2.01 GiB
...

As the info shows, the virtual disk size is now 32 GiB. The file still occupies only 2.01 GiB in the directory, due to the qcow2 dynamic file format.

This is due to the qcow2 file format, which dynamically occupies the necessary space as needed. Therefore, the size of the extension should be chosen quite generously to potentially save the work of a later re-extension. In the case of multiple virtual machines, the sum of their size can also exceed the real available space. This is not a problem as long as not all virtual machines at the same time grow to a much greater extent than planned. However, flexibility is gained in the case of disproportionate growth of some single virtual machines.

Reconfigure an increased virtual disk

After starting vm2 again the Storage page now shows unsurprisingly a disk of 32 GiB, but nothing else changed. So the disk has now 20 GiB unallocated space.

Fedora Servers default storage setup is based on the concept of segmenting storage to facilitate system administration, prevent spillover of errors, and others. For details, see the Fedora Server Installation Guide. Therefore, if you need additional space for an application or service, e.g. web service, you would create a separate Logical volume in Fedora Server following that rationale.

You have three options now:

  1. Enlarge the existing partition and Volume group to add logical Volume as needed. That’s the way of Fedora Server’s default partitioning scheme. Unfortunately, you can’t enlarge partition and VG using Cockpit but have to use the terminal CLI in a one time action. Later, you can use to add or enlarge Logical volumes as needed using Cockpit.

  2. Avoid using the terminal and create an additional partition and Volume Group using just Cockpit. As with 1. you can use Cockpit again to add or enlarge Logical volumes as needed.

  3. Ignore the Fedora Server storage rationale and extend just everything to its maximal size.

We will continue with alternative 2 here. It is just as committed to the Fedora Server rationale as alternative 1, if not even stricter. It is with Cockpit the more comfortable variant and in the end implements the same actions, i.e. creating a Logical volume with file system and mounting it at a fitting position in the directory tree. And alternative 3 is conceptually the worst of all.

On the vm2 Storage page select the 32 Gib virtual disk and in the new page that opens click on the "Create partition" button right below the /dev/vda2 partition [1].

Cockpit create partition form

You only have to change the type to "No filesystem" [2] and create the partition. It takes some time before a third partition is displayed in the list.

Go back to the Storage page. At the top of the right column, activate the menu icon next to Devices and select "Create volume group".

Cockpit create volume group form

In the form that opens, enter a name for the Volume group, here in analogy to fedora_fedora of the system area fedora_user, select the newly created partition /dev/vda3, and create the Volume group. The form closes and on the Storage page you find a new device, the fedora_user Volume group of 20.0 GiB.

Setup separated disk space for user data

As already mentioned, in compliance with the Fedora Server storage concept, a data area separate from system data should contain the service and user data. One FSH-compatible way is to mount the /srv directory in a separate Logical volume and store therein the respective data for all services. Another way is to mount a well-known dedicated application directory into its own Logical volume. For example, in Fedora Server, PostgreSQL stores all its data into /var/lib/pgsql or Apache httpd in /var/www. These directories are each mounted in their own Logical volumes, preferably before the software is installed.

As an example we install httpd data into a separate logical volume. Just in case the software is already installed, stop httpd and move the data temporarily out of the way. We will restore them later. The directory /var/www should be empty.

The Storage page of the virtual machine (vm2) displays the Devices section in the top right column, and within the volume group fedora_user that we configured earlier. Clicking fedora_user opens the volume group configuration page. Below the box with details of the volume group is a list of the existing logical volumes of this volume group.It provides a link "Create new logical volume" to the right of the title, which opens a configuration form.

Cockpit create logical volume form

Replace the preset name by a descriptive term, here e.g. webservice, and choose a suitable size. Due to the dynamic space allocation of the qcow2 format you may choose the size definitely a bit generous. However, a later extension with Cockpit is very easy to perform.

With Purpose the form also offers the option to create a pool for "thinly provisioned volumes", which in turn accommodates logical volumes allocating space dynamically. In case of a VM this is usually not as helpful as on physical machines, because the qcow2 disk image files already provide for a kind of thin provisioning. But your mileage may vary.

After creating the Logical volume you find an entry in the list "Logical volumes". Expand the line using the triangle icon. You will find a Format Button. The associated form allows you to create and mount a file system in one pass.

Cockpit file system creation

You just have to provide a (descriptive) name and the mount point. After formatting you will find the volume group mounted and ready to use.

[…]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
...
/dev/mapper/fedora_fedora-root       11G  1.8G  9.3G  16% /
tmpfs                               1.5G  4.0K  1.5G   1% /tmp
/dev/vda1                          1014M  187M  828M  19% /boot
/dev/mapper/fedora_user-webservice  8.0G   90M  8.0G   2% /var/www

Cockpit handled all the various configuration steps involved on its own. This saves the system administrator quite a bit of effort. It even includes creating the mount directory if it does not already exist. At the end of the day, Cockpit makes it very easy for the system administrator to implement the elaborate storage concept of Fedora Server.

If you moved files out of the way at the beginning, you can restore them now. To be on the safe side, restore the SELinux labels.

[…]# /sbin/restorecon -R -vF /var/www

Now is the time to install the appropriate programs. Software provided by Fedora distribution as rpm installs into the appropriate system directories via dnf tool. External software should install in either /opt or /usr/local to keep it separated from distribution files to prevent any potential mutual interference. In accordance with the storage rationale, you should also create a separate logical volume and mount it under /opt. An example of this would be Wildfly application server, which stores program, configuration and possibly data in a common place.

Add a virtual disk

Expanding the virtual (system) disk is not the only way to provide additional storage, nor is it necessarily the best way. In the case of virtual disk images, creating a dedicated virtual disk instead of creating Logical volumes is a solid, often better alternative. It adds more flexibility. For example, in case of the "cold standby" VM mentioned above, the system administrator can assign the (virtual) disk with the entire current dataset to the other VM in less than a minute.

Cockpit supports the creation of additional virtual disks (but unfortunately not to modify an existing one). On the virtual machine details page in the section Disks you find an Add Disk button. It opens a form to fill in.

Cockpit add a virtual disk form

In the example above we create a 30gb disk for Web service. Add creates a disk image in /var/lib/libvirt/images and adds it to the Cockpit VM details page. You may check in the terminal window using lsblk. It shows a disk vdb without any partition so far.

The Host Cockpit simply creates a new (virtual) hard disk and "plugs" it into the virtual machine. Configuration and setup of the hard disk occurs within the VM. Their Cockpit can perform that very easily. Open a new tab in your browser, open Cockpit of the VM and select Storage over there.

Cockpit additional virtual disk

On the right side you see the newly created disk listed as part of the (virtual hardware) equipment of the VM. You can work on this disk like on any other. A click opens a corresponding dialog form.

The system administrator can create a partition table and one or more partitions, can create a Volume group and Logical Volumes the same way as described in the previous section. It is also possible to set up and mount a file system directly without a partition table. Cockpit enables and supports all these options comfortably and efficiently.

Manage Libvirt disk pool

It is not only the disk images of virtual machines that need to be adjusted at times. The space on the host for these files can also run out and require expansion.

A system administrator committed to the Fedora Server storage rationale has set up a dedicated separate file system for the image pool or overall for the libvirt directory. On the Storage page of the hosts Cockpit identify that file system and select it. Enlarge the row using the triangle icon.

Cockpit enlarge libvirt pool

In the example above, the system administrator choose to provide a separate storage for the dedicated libvirt directory tree in /dev/usrvg/libvirt. Selecting Grow you get a form where you have to enter just the desired new size. Cockpit does everything else, including the adjustment of the file system, completely on its own.

It couldn’t be simpler or faster.

Manage network connectivity

Manage libvirt virtual network

The current version of Cockpit (255) doesn’t offer such many configuration options for a libvirt managed virtual network so far. Cockpit lists the existing virtual networks under the "Network" link at the top line by line. Mostly this is only the default network. By expanding the line, the editing options become visible.

Cockpit libvirt virtual networks overview

For a production operation the most important one might be to ensure or to be able to check the start of the network on a reboot of the host. All other configuration options, in particular e.g. switching NAT on and off, require the use of " virsh net-edit default" on the command line.

Quite interesting is the option to create a new virtual network. It opens a simple form collecting the most important options

Cockpit create a libvirt virtual network

The system administrator can select a Forward mode, either NAT, Open, or none (isolated), which can be very convenient. The default Device is automatic, alternatively you can select an existing device. The automatic option creates a new device virbr<x>. More advanced configuration such as domain name and name resolution would require working with CLI tools over here as well.

All in all, a system administrator will be rather reluctant to modify anything on the network infrastructure frequently, if at all. So nothing important for daily operation is missing here.

Manage a virtual machines network connectivity

Cockpit offers a broader support for the network configuration of virtual machines. You can add and delete a network interface, but fully edit its properties as well.

The network options are located at the bottom of the screen of the respective virtual machine, in most cases outside the viewable pane, and are easy to overlook.

Add an interface

You may have configured a virtual machine with just a public interface in order to provide some public service. According to the general recommendations it uses mac-vlan as an efficient connection type. In the virtual machines Cockpit GUI it shows up as "type: direct" and "Source: enp1s0", the host’s physical interface [1].

Cockpit public libvirt virtual interface

After some time you may need to access a service provided by the host, e.g. a database service. Because the public interface uses mac-vlan, a direct communication between host and virtual machine is not possible (due to technical limitations of that type). Additionally, you may want to protect that communication and its data from the public. The solution is to add an interface to the virtual machine that connects to the libvirt internal virtual network, named default and using virbr0.

With Cockpit it is just a click on the "Add network interface" link right of the section title [2] and to fill in a simple form.

Cockpit add internal libvirt virtual interface

In case of an internal interface leave everything as suggested, but just activate "Always attach" to make it automatically available after reboot. Use the button "Add" to create the interface.

The interface shows up below the public interface. And inside the virtual machine a "ip a" shows and additional connection "enp7s0". Cockpit sets everything up on its own, no further involvement of the system administrator required.

Modify an interface

You may have configured a virtual machine with just an internal interface, which works perfectly as (the only) interface enp1s0. Now you need public access instead. The easiest way is to edit the interface definition in Cockpit.

In the network section of the VMs detail page there is just one interface with 3 buttons: Delete, Unplug, and Edit. The latter opens a interface properties form.

Cockpit edit libvirt virtual interface

The form shows the definition of the current internal network. Just modify the Interface type from "Virtual network" to "Direct attachement" and Source from "default" to "enp1s0", the host physical interface. Save your selection and reboot. Assuming DHCP works, the virtual machine has a perfectly working connection to the public network, including name resolution and default firewall configuration. Otherwise you have to define a static network configuration.

Even if you need a public interface in addition to the existing internal interface you may prefer to edit the existing interface and later add a new internal interface. That way you follow the common practice of always setting up the public interface as the "first" one, i.e. enp1s0 or eth0.

Delete an interface

As an example, the virtual maschine is equipped with two interfaces, a public (enp1s0) and an internal (enp7s0).

Cockpit delete a libvirt virtual interface

Probably you don’t want a public interface but that machine well protected from the public network. So you delete the public interface. A simple click on Delete will remove that interface from the virtual machine (after confirmation). If you check the network configuration inside the machine an nmcli device show lists only one device as attached to the still existing connection. And ip a lists just lo and enp7s0.

An nmcli conn show still lists the connection enp1s0, but without an attached device. The reason is that it was the first interface explicitly set up during installation. Therefore it was persisted in /etc/NetworkManager/system-connected and the file is still there (allthough without any device configuration options). Cockpit does not delete a configuration file.

Simplified Access to VMs Cockpit

Typically, a (physical) server does host not only one, but several or even a huge number of VMs. Each VM will have a running Cockpit Web Interface. To access it, you have to open a Browser window, connect and login to respective VM you want to work with separately - even if you only want to do the most minimalist of tasks. It’s certainly not difficult, but it is cumbersome.

For convenience, Cockpit provides the option to access the Cockpit instance of the VMs from the web interface of one instance, conveniently the host. The communication between the involved Cockpit instances is not handled via HTTP protocol but via SSH. Therefore, authentication can also be done via a key file. This is more secure and also more convenient.

In the upper left corner you will see the name of the logged in user and the server, along with an expand icon. This opens a box where you can switch to another server or add a new one.

Cockpit select of add a Cockpit instance

The Add new host link opens a simple form to fill in hostname and user. And you can assign a color. If you connect the VM using a direct public interface (mac-vlan) you can’t use that interface because a direct access from host to vm via mac-vlan is not supported for technical reasons. You have to use the internal interface (usually libvirt provided virbr0). As a user, you should select the unprivileged administrative account that can be created during the VM installation by Anaconda.

Cockpit will then connect and ask for the user’s password.

Cockpit login to remote instance

On this occasion, select automatic login via ssh keyfile. Cockpit will create one for you if none exists or otherwise uses an existing one created previously for another Cockpit connection. Cockpit installs the public key in the VM, too. You don’t have to bother with all these details.

You can also log in as root. However, the default configuration does block a password-supported root ssh login. You have to generate a keyfile pair manually in advance and also install the public part in the VM. You can then select it for a root connection.

Security Considerations

A Fedora installation makes several efforts to minimize public exposure. One of these is to prevent password-based root login via public interface. At the same time, Fedora Server installs by default Cockpit and opens the firewall accordingly. That way it introduces a password-based login again, including terminal access. That counteracts in a way the ssh measure.

It’s OK to make the post installation tasks as easy and painless as possible. But securing Cockpit access has to be one of the post installation tasks. All the more so as Cockpit is usually not used on a daily basis, but runs largely unused and unattended in the background.

There are several options.

  1. The easiest way is to close the Cockpit firewall port.

    […]# firewall-cmd  --permanent --remove-service=cockpit
    […]# firewall-cmd  --reload

    To use Cockpit you have to ssh into the server and temporarily open the port.

    […]# firewall-cmd   --add-service=cockpit
  2. Since you log in via ssh anyway, you can also initialize an ssh tunnel in pone go and use it for Cockpit if needed. As described above you have the Cockpit service from the firewall removed.

    To use Cockpit log into the server and temporarily open the port.

    […]# ssh $user@example.com -i $key  -L 9090:example.com:9090

    You can access the Cockpit interface using https://localhost:9090 in your favorite browser.

  3. You can install Cockpit on your local Fedora workstation or on a lab server shielded by a firewall. Configure this instance to access any of your Cockpit instances as described in the previous section. Use this shielded instance as a proxy to your public servers. As described,it uses a protected ssh connection in the public network. Of course, you have the Cockpit service removed from the firewall as well.

Concluding remarks

The examples here cover only the most common administrative tasks in virtual machine management. Currently, Cockpit can not completely replace the terminal window CLI. But the software is regularly updated and extended. A task that is currently not supported might be included in the next release. It’s worth getting into the habit of briefly checking Cockpit’s support for each new task.