Retrace server - provides complete tracebacks for unhandled crashes and show aggregated information for developers.
Fedora QA Devel, Fedora Infrastructure Team, ABRT team
#abrt, #fedora-admin, #fedora-noc
Provides complete tracebacks for unhandled crashes and show aggregated information for developers.
The physical server (retrace.fedoraproject.org) runs two main services: retrace-server and FAF.
The upstream for retrace server lives at:
When a user has the ABRT client installed and a process crashes with an unhandled exception (e.g., traceback or core dump), the user can send a request to retrace-server. The server will install the same set of packages plus debuginfo, and will return a traceback to the user that includes function names instead of plain pointers. This information is useful for debugging.
The upstream retrace-server allows users to upload coredumps through a web interface, but the Fedora instance disables this feature.
When a user decides to report a crash, data is sent to FAF. ABRT can also be configured to send microreports automatically, if desired.
FAF can aggregate similar reports into one entity (called a Problem). FAF provides a nice web interface for developers, allowing them to see crashes of their packages. It lives at:
The playbook is split into several roles. There are two main roles
These roles are copied from upstream. You should never update it directly. The new version can be fetched from upstream using:
cd ansible/abrt rm -rf faf retrace ansible-galaxy install -f -r requirements.yml --ignore-errors -p ./
You should review the new differences, and commit and push.
Then there are some roles which are local for our instance:
abrt/faf-local - This is run before abrt/faf.
abrt/retrace-local - This is run after abrt/retrace.
abrt/retrace-local-pre - This is run before abrt/retrace.
FAF and retrace-server each have cron tasks. They are not installed
/etc/cron* but are installed as user cron jobs for the 'faf' and
You can list those crons using:
sudo -u faf crontab -l sudo -u retrace crontab -l
All cronjobs should be Ansible managed. Just make sure if you delete some cron from Ansible that it does not remain on the server (not always possible with state=absent).
/srv/ssd- fast disk, used for PostgreSQL storage
/srv- big fat disk, used for storing packages. Mainly:
/srv/faf/db-backup/- Daily backups of DB. No rotating yet. Needs to be manually deleted occasionally.
/srv/faf/lob/InvalidUReport/- Invalid reports, can be pretty big. No automatic removal too. Need to be purged manually occasionally.
The main web page is handled by the abrt-server-info-page package, which can be controlled using: