cloud-image-uploader SOP

Upload Cloud images to public clouds after they are built in Koji.

Source code: https://pagure.io/cloud-image-uploader

Contact Information

Owner

Cloud SIG, Jeremy Cline (jcline)

Contact

#cloud:fedoraproject.org (Matrix)

Servers
Purpose

Upload Cloud images to public clouds.

Description

cloud-image-uploader is an AMQP message consumer (run via fedora-messaging consume) that processes Pungi compose messages published on the org.fedoraproject.*.pungi.compose.status.change AMQP topic. When a compose enters the FINISHED or FINISHED_INCOMPLETE states, the service downloads any images in the compose and uploads it to the relevant cloud provider by running an Ansible playbook. Consult the playbooks directory in the source repository or Python package to see the playbooks.

The service does not accept any incoming connections and only depends on the RabbitMQ message broker and the relevant cloud provider’s APIs.

It requires a few gigabytes of temporary space to download the images before uploading them to the cloud provider. It is heavily I/O bound and the most computationally expensive thing it does is decompress the images.

General Configuration

The Fedora Ansible repository contains the OpenShift application definition. The playbook to create the OpenShift application is located at playbooks/openshift-apps/cloud-image-uploader.yml.

Within the container image, configuration is provided via /etc/fedora-messaging/config.toml. Additionally, secrets may be provided via environment variables and are noted in the relevant cloud sections.

Deploying

The service contains a single image and one pod in its deployment configuration.

Staging

The staging BuildConfig builds a container from the main branch. You need to trigger a build manually, either from the web UI or the CLI.

Production

The staging BuildConfig builds a container from the prod branch. Just like staging, you need to trigger a build manually. After deploying to staging, the main branch can be merged into the production branch to "promote" it:

$ git checkout prod && git merge --ff-only main

Azure

Images are uploaded whenever a compose that contains vhd-compressed images. Images are first uploaded to a container in the storage account and then imported into an Image Gallery.

Credentials for Azure are provided using environment variables. The credentials are used by the Azure Ansible collection.

Image Cleanup

Image clean-up is automated.

The storage account is configured to delete any blob in the container older than 1 week and should require no manual attention. Nothing in the container is required after the VHD is imported to the Image Gallery.

Images in the Gallery are cleaned up by the image uploader after a new image has been uploaded. For complete details on the image cleanup policy refer to the consumer code, but at the time of this writing the policy is as follows:

  • Any image that has an end-of-life field that is in the past is removed.

  • Only the latest 7 images that are marked as "excluded from latest = True" within an image definition are retained. When an image is marked as "exclude from latest = False", new virtual machines that don’t reference an explicit image version will boot using the newest image (following semver). All images are uploaded with "excluded from latest = True" and are only marked as "excluded from latest = False" after testing.

  • Only the latest 7 images in the Rawhide image definitions are retained, regardless of whether they are marked "excluded from latest = False".

At the moment, testing and promotion to "excluded from latest = False" is a manual process, but in the future will be automated to happen regularly (weekly, perhaps).

Authentication

The following environment variables are used:

AZURE_SUBSCRIPTION_ID - Identifies the subscription within an Azure tenant (our tenant only has 1)
AZURE_CLIENT_ID - The application ID used during authentication.
AZURE_SECRET - The application secret used during authentication.
AZURE_TENANT - Identifies the Azure tenant.

If you have access to the Fedora Project tenant, these values are available in the web portal under the Microsoft Entra ID service in the "App registrations" tab. To manage things via the CLI you can do dnf install azure-cli. All commands below assume you’ve logged in with az login.

There are two app registrations, fedora-cloud-image-uploader and fedora-cloud-image-uploader-staging. These were created by running:

$ az ad app create --display-name fedora-cloud-image-uploader

Authorization

Images are placed in two resource groups (containers for arbitrary resources). fedora-cloud-staging is used for the staging deployment, and fedora-cloud is used for the production deployment.

The app registrations are granted access to their respective resource group by assigning them a role on the resource group. The role definition can be seen with:

$ az role definition list --name "Image Uploader"

This role is then assigned to the app registration with

$ az role assignment create --assignee "fedora-cloud-image-uploader" \
    --role "Image Uploader" \
    --scope "/subscriptions/{subscription_id}/resourceGroups/fedora-cloud"

In the event that additional permissions are required, the role can be updated with additional permission.

Credential rotation

At the moment, credentials are set to expire and will need to be periodically rotated. To do so via the CLI:

$ az ad app list -o table  # Find the application to issue new secrets for and set CLIENT_ID to its "Id" field
$ touch azure_secret
$ chmod 600 azure_secret
$ SECRET_NAME="Some useful name for the secret"
$ az ad app credential reset --id $CLIENT_ID --append --display-name $SECRET_NAME --years 1 --query password --output tsv > azure_secret

Want to help? Learn how to contribute to Fedora Docs ›