Fedora CoreOS Cincinnati SOP

Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines. This SOP describes how to access and how to troubleshoot it.

Contact Information

Source

https://github.com/coreos/fedora-coreos-cincinnati

Playbook

Location

Project

Deployment

Containers

Routes

Each FCOS Cincinnati service exposes live metrics in Prometheus format:

FCOS Cincinnati is built as container image directly from source, referencing a pinned git commit.

In order to build and deploy a new version, you will first have to find the relevant commit (i.e. the latest on the main branch) at https://github.com/coreos/fedora-coreos-cincinnati .

Once you have identified the target commit, these are the steps to build a new container image:

update the fcos_cincinnati_git_sha playbook variable in roles/openshift-apps/coreos-cincinnati/vars/staging.yml
update the fcos_cincinnati_git_sha playbook variable in roles/openshift-apps/coreos-cincinnati/vars/production.yml
commit and push the update to the fedora-infra/ansible repository
SSH to batcave01.rdu3.fedoraproject.org
run sudo rbac-playbook openshift-apps/coreos-cincinnati.yml using your FAS password and your second-factor OTP
schedule a new build by running sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.yml
Wait for the coreos-cincinnati-img:latest tag to be updated

Issues in the underlying OpenShift cluster may result in builds being permanently stuck.

If a build does not complete within a reasonable amount of time (i.e. 15 minutes):

go to the build overview at https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/builds
click on the build
cancel it through the "Cancel Build" button
go back to the build overview page
schedule a new build through the "Start Build" button