Anitya Infrastructure SOP
Anitya is used by Fedora to track upstream project releases and maps them to downstream distribution packages, including (but not limited to) Fedora.
Anitya staging instance: https://stg.release-monitoring.org
Anitya production instance: https://release-monitoring.org
Anitya project page: https://github.com/fedora-infra/anitya
Fedora Infrastructure Team
os-master01.iad2.fedoraproject.org + Staging +
Map upstream releases to Fedora packages.
The current deployment is made up of release-monitoring OpenShift namespace.
This OpenShift namespace runs following pods:
The apache/mod_wsgi application for release-monitoring.org
A libraries.io SSE client
A service checking for new releases
This OpenShift project relies on:
A postgres db server running in OpenShift
Lots of external third-party services. The anitya webapp can scrape pypi, rubygems.org, sourceforge and many others on command.
Lots of external third-party services. The check service makes all kinds of requests out to the Internet that can fail in various ways.
Fedora messaging RabbitMQ hub for publishing messages
Things that rely on this host:
hotness-sopis a fedora messaging consumer running in Fedora Infra in OpenShift. It listens for Anitya messages from here and performs actions on koji and bugzilla.
The release process is described in Anitya documentation.
Staging deployment of Anitya is deployed in OpenShift on os-master01.stg.iad2.fedoraproject.org.
To deploy staging instance of Anitya you need to push changes to staging branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on staging.
Production deployment of Anitya is deployed in OpenShift on os-master01.iad2.fedoraproject.org.
To deploy production instance of Anitya you need to push changes to production branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on production.
All the following commands should be run from batcave01.
First, ensure there are no configuration changes required for the new update. If there are, update the Ansible anitya role(s) and optionally run the playbook:
$ sudo rbac-playbook openshift-apps/release-monitoring.yml
The configuration changes could be limited to staging only using:
$ sudo rbac-playbook openshift-apps/release-monitoring.yml -l staging
This is recommended for testing new configuration changes.
To deploy new version of Anitya you need to push changes to staging branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on staging.
To deploy new version of Anitya you need to push changes to production branch on Anitya GitHub. GitHub webhook will then automatically deploy a new version of Anitya on production.
Congratulations! The new version should now be deployed.
Anitya web application offers some functionality to administer itself.
User admin status is tracked in Anitya database. Admin users can grant or revoke admin priviledges to users in the users tab.
Admin users have additional functionality available in web interface. In particular, admins can view flagged projects, remove projects and remove package mappings etc.
For more information see Admin user guide in Anitya documentation.
Anitya lets users flag projects for administrator attention. This is accessible to administrators in the flags tab.
This section contains various issues encountered during deployment or configuration changes and possible solutions.
Issue: Fedmsg messages aren’t sent.
Solution: Set USER environment variable in pod.
Explanation: Fedmsg is using USER env variable as a username inside messages. Without USER env set it just crashes and didn’t send anything.
Issue: Cronjob pod is crashing on start, even after configuration change that should fix the behavior.
Solution: Restart the cronjob. This could be done by OPS.
Explanation: Every time the cronjob is executed after crash it is trying to actually reuse the pod with bad configuration instead of creating a new one with new configuration.
Issue: Database migration is taking few hours to complete.
Solution: Stop every pod and cronjob before migration.
Explanation: When creating new index or doing some other complex operation on database, the migration script needs exclusive access to the database.
Issue: The pod is deployed with old version of Anitya, but it says that it was triggered by correct commit.
Solution: Set dockerStrategy in buildconfig.yml to noCache.
Explanation: The OpenShift is by default caching the layers of docker containers, so if there is no change in Dockerfile it will just use the cached version and don’t run the commands again.