Koschei is a continuous integration system for RPM packages. Koschei runs package scratch builds after dependency change or after time elapse and reports package buildability status to interested parties.
continuous integration system
Koschei deployment is managed by two Ansible playbooks:
sudo rbac-playbook groups/koschei-backend.yml sudo rbac-playbook groups/koschei-web.yml
Koschei is deployed on two separate machines -
koschei-web) is a Flask WSGi application running with httpd.
It displays information to users and allows editing package groups and
koschei-backend) consists of multiple services:
koschei-watcher- listens to fedmsg events for complete builds and changes build states in the database
koschei-repo-resolver- resolves package dependencies in given repo using hawkey and compares them with previous iteration to get a dependency diff. It resolves all packages in the newest repo available in Koji. The output is a base for scheduling new builds
koschei-build-resolver- resolves complete builds in the repo in which they were done in Koji. Produces the dependency differences visible in the frontend
koschei-scheduler- schedules new builds based on multiple criteria:
dependency priority - dependency changes since last build valued by their distance in the dependency graph
manual and static priorities - set manually in the frontend. Manual priority is reset after each build, static priority persists
time priority - time elapsed since the last build
koschei-polling- polls the same types of events as koschei-watcher without reliance on fedmsg. Additionaly takes care of package list synchronization and other regularly executed tasks
Koschei configuration is in
/etc/koschei/config-frontend.cfg, and is merged with the default
/usr/share/koschei/config.cfg (the ones in
overrides the defaults in
/usr). Note the merge is recursive. The
configuration contains all configurable items for all Koschei services
and the frontend. The alterations to configuration that aren’t temporary
should be done through ansible playbook. Configuration changes have no
effect on already running services — they need to be restarted, which
happens automatically when using the playbook.
Koschei doesn’t keep on disk anything that couldn’t be recreated easily - all important data is stored in PostgreSQL database, configuration is managed by Ansible, code installed by RPM and so on.
To speed up operation and reduce load on external servers, Koschei
caches some data obtained from services it integrates with. Most
notably, YUM repositories downloaded from Koji are kept in
/var/cache/koschei/repodata. Each repository takes about 100 MB of
disk space. Maximal number of repositories kept at time is controlled by
cache_l2_capacity parameter in
config-backend.cfg.j2 in Ansible). If repodata cache starts to
consume too much disk space, that value can be decreased - after
koschei-*-resolver will remove least recently used cache
entries to respect configured cache capacity.
Koschei needs to connect to a PostgreSQL database, other database
systems are not supported. Database connection is specified in the
configuration under the
database_config key that can contain the
username, password, host, port, database.
After an update of koschei, the database needs to be migrated to new schema. This happens automatically when using the upgrade playbook. Alternatively, it can be executed manulally using:
koschei-admin alembic upgrade head
The backend services need to be stopped during the migration.
Koschei services are systemd units managed through
systemctl. They can
be started and stopped independently in any order. The frontend is run
For stopping builds from being scheduled, stopping the
koschei-scheduler service is enough. For planned Koji outages, it’s
recommended to stop
koschei-scheduler. It is not necessary, as koschei
can recover from Koji errors and network errors automatically, but when
Koji builders are stopped, it may cause unexpected build failures that
would be reported to users. Other services can be left running as they
automatically restart themselves on Koji and network errors.
Koschei is by default limited to 30 concurrently running builds. This
limit can be changed in the configuration under
key. There’s also Koji load monitoring, that prevents builds from being
scheduled when Koji load is higher that certain threshold. That should
prevent scheduling builds during mass rebuilds, so it’s not necessary to
stop scheduling during those.
Koschei optionally supports sending fedmsg notifications for package
state changes. The fedmsg dispatch can be turned on and off in the
fedmsg-publisher.enabled). Koschei doesn’t supply
configuration for fedmsg, it lets the library to load it’s own (in
Koschei can display announcement in web UI. This is mostly useful to inform users about outages or other problems.
To set announcement, run as koschei user:
koschei-admin set-notice "Koschei operation is currently suspended due to scheduled Koji outage"
koschei-admin set-notice "Sumbitting scratch builds by Koschei is currently disabled due to Fedora 23 mass rebuild"
To clear announcement, run as koschei user:
Packages can be added to one or more group.
To add new group named
mynewgroup, run as
koschei-admin add-group mynewgroup
To add new group named
mynewgroup and populate it with some packages,
koschei-admin add-group mynewgroup pkg1 pkg2 pkg3
Some packages are more or less important and can have higher or lower priority. Any user can change manual priority, which is reset after package is rebuilt. Admins can additionally set static priority, which is not affected by package rebuilds.
To set static priority of package
foo to value
100, run as
koschei-admin --collection f27 set-priority --static foo 100
After branching occurs and Koji build targets have been created, Koschei should be updated to reflect the new state. There is a special admin command for this purpose, which takes care of copying the configuration and also last builds from the history.
To branch the collection from Fedora 27 to Fedora 28, use the following:
koschei-admin branch-collection f27 f28 -d 'Fedora 27' -t f28 --bugzilla-version 27
Then you can optionally verify that the collection configuration is correct by visiting https://apps.fedoraproject.org/koschei/collections and examining the configuration of the newly branched collection.