datanommer SOP
Consume fedmsg bus activity and stuff it in a postgresql db.
Contact Information
- Owner
-
Messaging SIG, Fedora Infrastructure Team
- Contact
-
#fedora-apps, #fedora-fedmsg, #fedora-admin, #fedora-noc
- Servers
-
busgateway01
- Purpose
-
Save fedmsg bus activity
Description
datanommer is a set of three modules:
- python-datanommer-models
-
Schema definition and API for storing new items and querying existing items
- python-datanommer-consumer
-
A plugin for the fedmsg-hub that actively listens to the bus and stores events.
- datanommer-commands
-
A set of CLI tools for querying the DB.
datanommer will one day serve as a backend for future web services like datagrepper and dataviewer.
CLI tools
Dump the db into a file as json:
$ datanommer-dump > datanommer-dump.json
When was the last bodhi message?:
$ # It was 678 seconds ago
$ datanommer-latest --category bodhi --timesince
[678]
When was the last bodhi message in more readable terms?:
$ # It was 12 minutes and 43 seconds ago
$ datanommer-latest --category bodhi --timesince --human
[0:12:43.087949]
What was that last bodhi message?:
$ datanommer-latest --category bodhi
[{"bodhi": {
"topic": "org.fedoraproject.stg.bodhi.update.comment",
"msg": {
"comment": {
"group": null,
"author": "ralph",
"text": "Testing for latest datanommer.",
"karma": 0,
"anonymous": false,
"timestamp": 1360349639.0,
"update_title": "xmonad-0.10-10.fc17"
},
"agent": "ralph"
},
}}]
Show me stats on datanommer messages by topic:
$ datanommer-stats --topic
org.fedoraproject.stg.fas.group.member.remove has 10 entries
org.fedoraproject.stg.logger.log has 76 entries
org.fedoraproject.stg.bodhi.update.comment has 5 entries
org.fedoraproject.stg.busmon.colorized-messages has 10 entries
org.fedoraproject.stg.fas.user.update has 10 entries
org.fedoraproject.stg.wiki.article.edit has 106 entries
org.fedoraproject.stg.fas.user.create has 3 entries
org.fedoraproject.stg.bodhitest.testing has 4 entries
org.fedoraproject.stg.fedoratagger.tag.create has 9 entries
org.fedoraproject.stg.fedoratagger.user.rank.update has 5 entries
org.fedoraproject.stg.wiki.upload.complete has 1 entries
org.fedoraproject.stg.fas.group.member.sponsor has 6 entries
org.fedoraproject.stg.fedoratagger.tag.update has 1 entries
org.fedoraproject.stg.fas.group.member.apply has 17 entries
org.fedoraproject.stg.__main__.testing has 1 entries
Upgrading the DB Schema
datanommer uses "python-alembic" to manage its schema. When developers want to add new columns or features, these should/must be tracked in alembic and shipped with the RPM.
In order to run upgrades on our stg/prod dbs:
-
ssh to busgateway01\{.stg}
-
cd /usr/share/datanommer.models/
-
Run:
$ alembic upgrade +1
Over and over again until the db is fully upgraded.
Want to help? Learn how to contribute to Fedora Docs ›