SOP Day-to-Day Zabbix usage for Infrastructure

Overview

This SOP documents the steps to handle day-to-day operation of Zabbix

Contact Information

Owner

Fedora Infrastructure Team

Contact

#fedora-admin, sysadmin-main, sysadmin-noc

Purpose

Ensure monitoring config is stored in configuration management for later (re)use.

Overview

Zabbix is the primary monitoring tool for Fedora Infrastructure. The layout is:

  • We have a Prod server for the main infra

    • The URL is [1]

    • Notifications are sent to the UI (for Warning level or lower)

    • Notifications are also sent to #noc:fedoraproject.org (for Average level or above)

  • We have a Staging server for testing things on

    • The URL is [2]

    • Notifications are sent to the UI (for Warning level or lower)

    • Notifications are also sent to #fedora-zodbot:fedora.im (for Average level or above)

For temporary changes, we use the Zabbix UI - this doc covers the main reasons for that in day-to-day operations.

For permanent changes, we use Ansible to record/deploy them - see SOP Add Zabbix template to Ansible for details on permanent changes.

Common Tasks for Infrastructure

Acknowledging/closing an alert

This takes three possible paths. In all cases, get to the Update form by either:

  • clicking the Update button next to the alert on the Dashboard page, or on the Monitoring > Problems page.

  • clicking the alert in Matrix and then Update on the Event Details page.

Once you have the Update form open, you have three options:

  1. If the alert is known and requires the team to take action (e.g. a disk has failed, or similar) check the Acknowledge box.

  2. If it’s something we expect to go away, but would like to be re-alerted if it doesn’t (e.g. load on a proxy), check the Suppress box and pick a duration.

  3. If it’s dealt with already (e.g. number of packages changed on a host), check the Close Problem box.

In all cases, a description is useful for the rest of the team.

Common manual interventions

There are some frequent alerts that need to be handled manually:

  • Datanommer: FMN queue is stale:

    • This monitors the system at https://notifications.fedoraproject.org, and it does so by checking when a rule was last edited. Create/delete a rule for yourself to trigger an update and clear the alert.

  • Datanommer: Bugzilla queue is stale:

    • The connection to the Stomp queue fails from time to time, it needs to be prodded. Restart the pod in fedmsg2bugzilla project in OpenShift

  • Datanommer: RPM Sign queue is stale:

Disabling/enabling a host

We sometimes need to remove a host due to problems with the machine. To do this:

  1. Search for the host in the search sidebar.

  2. Click the hostname itself (in the Hosts column of the search results).

  3. Scroll to the end of the Host form, and uncheck the Enabled field.

  4. Save the host.

To bring the host back, edit again and check Enabled again.

No alerts will be recorded, nor will there be data collection. If data collection is wanted during the downtime, see maintenance windows below.

Maintenance Windows

Zabbix can schedule time periods in which alerts are paused, and optionally data collection too, for one or more hosts. This can be scheduled in advance (e.g. for a migration of hosts to new hardware), or done ad-hoc, and can be one host or everything.

Generally, you can create a maintenance window in the Zabbix UI like so:

  1. Go to Data Collection > Maintenance

  2. Click Create Maintenance Period (top-right)

  3. Give it a name, type, start/end times, and an optional description

  4. For hosts/hostgroups pick what you need

  5. Save the maintenance window

Not that triggers still fire if even one host in the trigger is not in maintenance. This does not apply to many of our triggers currently (nearly all are single-host triggers), but worth knowing.

Single host shush playbook

To put a single host into maintenance, there is a playbook for that:

  • ansible-playbook /srv/web/infra/ansible/playbooks/zabbix_shush_host.yml -e "target=sundries02.stg.rdu3.fedoraproject.org"

It defaults to a 1 hour window, you can add '-e "time=120"' to alter that. You can also add '-e "state=absent"' if you want to remove a window early.

If you want to quickly add more hosts to the shush, it’s often easier to go find the created window (Data Collection > Maintenance) and edit your window, then add more host names or host groups to the required fields.

All hosts shush playbook/script

During a large outage, it might be needed to shush everything. There are scripts in Ansible (and a playbook) which you can use to disable all hosts for alerts (but continue data colection), and restore it later. See:

Last resort shush

If there is still notification storms going on, then as a last resort you can disable the media types:

  1. Go to Alerts > Media types

  2. Where it says "Enabled" next to (e.g.) Matrix Webhook, click it

  3. It should change to Disabled

Be sure to change it back after the storm has passed.

Graphing data

Often we want to look at the history for a given host or even a set of items across a group. We can get this data in several ways, here is just one:

  1. Go to Monitoring > Latest Data

  2. Search for a host of interest

  3. Scroll down to find the item name (or use the item filters on name/tag)

  4. Click the "Graph" button on the right

  5. Select the desired time range

For a group of hosts, the UI often fails to show everything when searching for hostgroups, so the process is:

  1. Go to Monitoring > Latest Data

  2. Search for one hostname in the group (e.g. proxy01)

  3. Find the item of interest (e.g Load average)

  4. Copy the item to the Name search box, and change the Hosts search to be a Hostgroup search instead (e.g. Proxies)

  5. Check the hosts you want to compare, and click "Display Graph" or "Display Stacked Graph" depending on how you want to see it

If you’re investigating an alert, you can get a graph of the relevant item directly from the Problem (on the Dashboard or Event Detail page). Click the problem, hover over History, and then select the item, and you’ll be taken directly to the graph for that problem.

Resources

Want to help? Learn how to contribute to Fedora Docs ›