Renew API certs for Openshift

This SOP should be followed in the following scenarios:

  • When the API certs are due to expire (this is tracked in Zabbix)

Overview

The API certs are provided by LetsEncrypt, via a DNS challenge. Therefore, the process is to: - use certbot to generate a challenge - commit a DNS record that matches - obtain the cert - upload to OCP

Notes: - If working on the staging cluster, replace api.ocp with api.ocp.stg as required

Steps

  1. On any machine (even your own laptop), install and run certbot

    sudo dnf install certbot           #if required
mkdir ~/certbot && cd ~/certbot    #if required
certbot certonly --expand --keep --manual \
  --preferred-challenges=dns -d "api.ocp.fedoraproject.org" \
  --config-dir ./config \
  --work-dir ./config \
  --logs-dir ./logs

    This will generate a DNS challenge, which needs to be committed as a TXT record.

  2. Follow DNS process for a new host/record

    The record will be for _acme-challenge.api.ocp.fedoraproject.org and you can use git log / git blame to find prior examples.

    Allow 20mins for propagation, or use the update_dns.yml playbook as noted in the DNS process to push it faster. You can test it via dig -t TXT _acme-challenge.api.ocp.fedoraproject.org and check it returns the right string.

  3. Complete the certbot process

    Return to the certbot process, and press Enter to tell certbot to complete the challenge. It should pass and write the files to ./config/live/api.ocp.fedoraproject.org

  4. Copy the files to Openshift control node

    DATE=$(date '+%Y-%m-%d')
scp ./config/live/api.ocp.fedoraproject.org/privkey.pem os-control01.rdu3.fedoraproject.org:/tmp/api.ocp.fedoraproject.org.key-$DATE
scp ./config/live/api.ocp.fedoraproject.org/fullchain.pem os-control01.rdu3.fedoraproject.org:/tmp/api.ocp.fedoraproject.org.cert-$DATE

    Change hostname to .stg.rdu3 as required.

  5. Log in to os-control01.rdu3.fedoraproject.org (or stg, as required)

    ssh os-control01.rdu3.fedoraproject.org
sudo -i
DATE=$(date '+%Y-%m-%d')
mv /tmp/api.ocp.fedoraproject.org.key-$DATE /tmp/api.ocp.fedoraproject.org.cert-$DATE /root

  6. Delete and replace the cert secret in Openshift

    oc delete secret api.ocp.fedoraproject.org -n openshift-config
sleep 3
DATE=$(date '+%Y-%m-%d')
oc create secret tls api.ocp.fedoraproject.org \
  --cert=api.ocp.fedoraproject.org.cert-$DATE \
  --key=api.ocp.fedoraproject.org.key-$DATE \
  -n openshift-config

  7. Wait a few mins and then check the API cert

    You can verify the certificate by checking the API endpoint or using OpenShift CLI commands.

Want to help? Learn how to contribute to Fedora Docs ›