Fedora CoreOS Frequently Asked Questions

If you have other questions than are mentioned here or want to discuss further, join us in our Libera.Chat IRC channel, #fedora-coreos, or on our discussion board. Please refer back here as some questions and answers will likely get updated.

What is Fedora CoreOS?

Fedora CoreOS is an automatically updating, minimal, monolithic, container-focused operating system, designed for clusters but also operable standalone, optimized for Kubernetes but also great without it. It aims to combine the best of both CoreOS Container Linux and Fedora Atomic Host, integrating technology like Ignition from Container Linux with rpm-ostree and SELinux hardening from Project Atomic. Its goal is to provide the best container host to run containerized workloads securely and at scale.

How does Fedora CoreOS relate to Red Hat CoreOS?

Fedora CoreOS is a freely available, community distribution that is the upstream basis for Red Hat CoreOS. While Fedora CoreOS embraces a variety of containerized use cases, Red Hat CoreOS provides a focused OS for OpenShift, released and life-cycled in tandem with the platform.

Does Fedora CoreOS replace Container Linux? What happens to CL?

Fedora CoreOS is the official successor to CoreOS Container Linux, but it is not a drop-in replacement. CoreOS Container Linux will reach its end of life on May 26, 2020, and will no longer receive updates after that date. For notes on migrating from Container Linux to Fedora CoreOS, see the migration guide.

Does Fedora CoreOS replace Fedora Atomic Host? What happens to Fedora Atomic Host and CentOS Atomic Host?

Fedora CoreOS is the official successor to Fedora Atomic Host. The last Fedora Atomic Host release was version 29, which has now reached end-of-life.

CentOS Atomic Host will continue producing downstream rebuilds of RHEL Atomic Host and will align with the end-of-life. The Fedora CoreOS project will be the consolidation point for the community distributions. Users are encouraged to move there in the future.

What happens to Project Atomic?

Project Atomic is an umbrella project consisting of two flavors of Atomic Host (Fedora and CentOS) as well as various other container-related projects. Project Atomic as a project name will be sunset by the end of 2018 with a stronger individual focus on its successful projects such as Buildah and Cockpit. This merges the community side of the operating system more effectively with Fedora and allows for a clearer communication for other community-supported projects, specifically the well-adopted #nobigfatdaemons approach of Buildah and the versatile GUI server manager Cockpit.

What are the communication channels around Fedora CoreOS?

We have the following new communication channels around Fedora CoreOS:

There is a community meeting that happens every week. See the Fedora CoreOS fedocal for the most up-to-date information.

If you think you have found a problem with Fedora CoreOS, file an issue in our issue tracker.

Technical FAQ

Where can I download Fedora CoreOS?

Fedora CoreOS artifacts are available at getfedora.org.

Does Fedora CoreOS embrace the Container Linux Update Philosophy?

Yes, Fedora CoreOS comes with automatic updates and regular releases. Multiple update channels are provided catering to different users' needs. It introduces a new node-update service based on rpm-ostree technologies, with a server component that can be optionally self-hosted. Failures that prevent an update from booting will automatically be reverted.

How are Fedora CoreOS nodes provisioned? Can I re-use existing cloud-init configurations?

Fedora CoreOS is provisioned with Ignition. However, existing Ignition configurations will require changes, as the OS configuration will be different from Container Linux. Existing cloud-init configurations are not supported and will need to be migrated into their Ignition equivalent.

What data persists across upgrades and reboots?

The directories /etc and /var are mounted as read-write which lets users write and modify files.

The directory /etc may be changed by deployments, but will not override user made changes. The content under /var is left untouched by rpm-ostree when applying upgrades or rollbacks. For more information, refer to the Mounted Filesystems section.

How do I migrate from Container Linux to Fedora CoreOS?

Migrations can be accomplished by re-provisioning the machine with Fedora CoreOS. For notes on migrating from Container Linux to Fedora CoreOS, see the migration guide.

How do I migrate from Fedora Atomic Host to Fedora CoreOS?

As with Container Linux, the best practice is to re-provision the node, due to the cloud-init/Ignition transition at least. Since Fedora CoreOS uses rpm-ostree technology, it may be possible to rebase from Fedora Atomic Host to Fedora CoreOS, but it is not recommended. It’s preferable to gain experience deploying systems using Ignition so that they can be re-provisioned easily if needed. For notes on migrating from Atomic Host to Fedora CoreOS, see the migration guide.

Which container runtimes are available on Fedora CoreOS?

Fedora CoreOS includes Docker and podman by default. Based on community engagement and support this list could change over time.

Which platforms does Fedora CoreOS support?

Fedora CoreOS runs on at least

  • Alibaba Cloud,

  • AWS,

  • Azure,

  • GCP,

  • OpenStack,

  • QEMU,

  • VMware,

  • and bare-metal systems if installed to disk or network-booted.

Can I run Kubernetes on Fedora CoreOS?

Yes. However, we envision Fedora CoreOS as not including a specific container orchestrator (or version of Kubernetes) by default — just like Container Linux and Atomic Host. We will work with the upstream Kubernetes community on tools (e.g. kubeadm) and best practices for installing Kubernetes on Fedora CoreOS.

How do I run custom applications on Fedora CoreOS?

On Fedora CoreOS, containers are the way to install and configure any software not provided by the base operating system. The package layering mechanism provided by rpm-ostree will continue to exist for use in debugging a Fedora CoreOS machine, but we strongly discourage its use. For more about this, please refer to documentation.

Where is my preferred tool for troubleshooting?

The FCOS image is kept minimal by design. Not every troubleshooting tool are included by default. Instead, it is recommended to use the toolbox utility.

How do I coordinate cluster-wide OS updates? Is locksmith or the Container Linux Update Operator available for Fedora CoreOS?

The etcd-lock feature from locksmith has been directly ported to Zincati, as a lock-based updates strategy. It has also been augmented to support multiple backends, not being anymore constrained to etcd2 only.

The capabilities of Container Linux Update Operator (CLUO) have been embedded into the Machine Config Operator (MCO), which is a core component of OKD. The MCO additionally covers reconciliation of machine configuration changes.

How do I upload Fedora CoreOS to private AWS EC2 regions?

Fedora CoreOS today is only uploaded to the standard AWS regions. For regions in other AWS partitions like GovCloud and AWS China, you must upload the images yourself.

Note that Fedora CoreOS uses a unified BIOS/UEFI partition layout. As such, it is not compatible with the aws ec2 import-image API (for more information, see related discussions). Instead, you must use aws ec2 import-snapshot combined with aws ec2 register-image.

To learn more about these APIs, see the AWS documentation for importing snapshots and creating EBS-backed AMIs.

Can I run containers via docker and podman at the same time?

No. Running containers via docker and podman at the same time can cause issues and unexpected behavior. We highly recommend against trying to use them both at the same time.

It is worth noting that in Fedora CoreOS we have docker.service disabled by default but it is easily started if anything communicates with the /var/run/docker.sock because docker.socket is enabled by default. This means that if a user runs any docker command (via sudo docker) then the daemon will be activated. We did this to make the transition easier for users of Container Linux.

In coreos/fedora-coreos-tracker#408 it was pointed out that because of socket activation users who are using podman for containers could unintentionally start the docker daemon. This could weaken the security of the system because of the interaction of both container runtimes with the firewall on the system. To prevent making this mistake you can disable docker completely by masking the docker.service systemd unit.

Example Butane config for disabling docker.service
variant: fcos
version: 1.3.0
systemd:
  units:
    - name: docker.service
      mask: true

Are Fedora CoreOS x86_64 disk images hybrid BIOS+UEFI bootable?

The x86_64 images we provide can be used for either BIOS (legacy) boot or UEFI boot. They contain a hybrid BIOS/UEFI partition setup that allows them to be used for either. The exception to that is the metal4k 4k native image, which is targeted at disks with 4k sectors and does not have a BIOS boot partition because 4k native disks are only supported with UEFI.

What’s the difference between Ignition and Butane configurations?

Ignition configuration is a low-level interface used to define the whole set of customizations for an instance. It is primarily meant as a machine-friendly interface, with content encoded as JSON and a fixed structure defined via JSON Schema. This JSON configuration is processed by each FCOS instance upon first boot.

Many high-level tools exist that can produce an Ignition configuration starting from their own specific input formats, such as terraform, matchbox, openshift-installer, and Butane.

Butane is one such high-level tool. It is primarily meant as a human-friendly interface, thus defining its own richer configuration entries and using YAML documents as input. This YAML configuration is never directly processed by FCOS instances (only the resulting Ignition configuration is).

Although similar, Ignition configurations and Butane ones do not have the same structure; thus, converting between them is not just a direct YAML-to-JSON translation, but it involves additional logic. Butane exposes several customization helpers (e.g. distribution specific entries and common abstractions) that are not present in Ignition and make the formats not interchangeable. Additionally, the different formats (YAML for Butane, JSON for Ignition) help to avoid mixing up inputs by mistake.

What is the format of the version number?

This is covered in detail in the design docs.

The summary is that Fedora CoreOS uses the format X.Y.Z.A

  • X is the Fedora major version (i.e. 32)

  • Y is the datestamp that the package set was snapshotted from Fedora (i.e. 20200715)

  • Z is a code number used by official builds

    • 1 for the next stream

    • 2 for the testing stream

    • 3 for the stable stream

  • A is a revision number that is incremented for each new build with the same X.Y.Z parameters

The version numbering scheme is subject to change and is not intended to be parsed by machine.

Why is the dnsmasq.service systemd unit masked?

We have found that the dnsmasq binary can be used for several host applications, including podman and NetworkManager. For this reason we include the dnsmasq package in the base OSTree layer, but we discourage the use of the dnsmasq.service in the host by masking it with systemctl mask dnsmasq.service.

"Why do you mask the service?"

dnsmasq is useful for running a DHCP/DNS/TFTP server for external clients (i.e. not local to the host), too, but that is something we’d prefer users to do in a container. Putting the service in a container insulates the hosted service from breakage as a result of host layer changes. For example, if NetworkManager and podman stopped using dnsmasq, we would remove it from the host and the service you depend on would cease to work.

"But, I really want to use it!"

We don’t recommend it, but if you really want to use it you can just unmask and enable it:

Example Butane config for unmasking dnsmasq.service
variant: fcos
version: 1.3.0
systemd:
  units:
    - name: dnsmasq.service
      mask: false
      enabled: true

For more information see the tracker issue discussion.

Why does SSH stop working after upgrading to Fedora 33?

In Fedora 33 there was a change to implement stronger crypto defaults. Part of this included taking the advice of OpenSSH upstream and disabling the use of the ssh-rsa public key signature algorithm.

You may hit issues if you use RSA keys and:

  • use an old version of the SSH client

  • use tooling/software libraries that don’t support using RSA SHA2 public key signatures

For example, Go has an open issue to solve this problem in its SSH implementation, but has yet to resolve it. This has been hit and worked around by the FCOS community in our build tooling and also our higher level projects:

If you run into this problem and need to work around the issue, you have a few options:

  • Switch to a newer non-RSA key type.

  • Provide a configuration to your machine that re-enables the insecure key signatures:

Example Butane config for re-enabling SSH RSA SHA1 key signatures
variant: fcos
version: 1.3.0
storage:
  files:
    - path: /etc/ssh/sshd_config.d/10-insecure-rsa-keysig.conf
      mode: 0600
      contents:
        inline: |
          PubkeyAcceptedKeyTypes=+ssh-rsa

Why do I get SELinux denials after updates if I have local policy modifications?

Currently the OSTree and SELinux tooling conflict a bit. If you have permanently applied local policy modifications then policy updates delivered by the OS will no longer apply; your policy stays frozen. This means any policy "fixes" needed to enable new functionality will not get applied. See coreos/fedora-coreos-tracker#701 for more details.

This means you may see denials like the following, which can take down critical parts of a system like in coreos/fedora-coreos-tracker#700:

Example SELinux denial
systemd-resolved[755]: Failed to symlink /run/systemd/resolve/stub-resolv.conf: Permission denied
audit[755]: AVC avc:  denied  { create } for  pid=755 comm="systemd-resolve" name=".#stub-resolv.confc418434d59d7d93a" scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:systemd_resolved_var_run_t:s0 tclass=lnk_file permissive=0

To see if your system currently has local policy modifications you can run ostree admin config-diff. The following system has a modified policy:

Example system with a modified SELinux policy
$ sudo ostree admin config-diff | grep selinux/targeted/policy
M    selinux/targeted/policy/policy.32

To work around this incompatibility, please attempt to apply policy modifications dynamically. For example, for an SELinux boolean you can use the following systemd unit that executes on every boot:

Example Butane config for dynamically applying SELinux boolean
variant: fcos
version: 1.3.0
systemd:
  units:
    - name: setsebool.service
      enabled: true
      contents: |
        [Service]
        Type=oneshot
        ExecStart=setsebool container_manage_cgroup true
        RemainAfterExit=yes
        [Install]
        WantedBy=multi-user.target

If your system’s basic functionality has stopped working because of SELinux denials check to see if your system currently has local policy modifications. You can check with ostree admin config-diff:

Example system with a modified SELinux policy
$ sudo ostree admin config-diff | grep selinux/targeted/policy
M    selinux/targeted/policy/policy.32

If your system is in this state you have two options:

  • Re-deploy starting with the latest image artifacts.

    • This means you start with the latest policy.

  • Follow the workaround in coreos/fedora-coreos-tracker#701 to restore the base policy.

Why is the systemd-repart.service systemd unit masked?

system-repart is a tool to grow and add partitions to a partition table. On Fedora CoreOS, we only support using Ignition to create partitions, filesystems and mount points, thus systemd-repart is masked by default.

Ignition runs on first boot in the initramfs and is aware of Fedora CoreOS specific disk layout. It is also capable of reconfiguring the root filesystem (from xfs to ext4 for example), setting up LUKS, etc…​ See the Configuring Storage page for examples.

See the Why is the dnsmasq.service systemd unit masked entry for an example config to unmask this unit.