Log Infrastructure SOP

Logs are centrally referred to our loghost and managed from there by rsyslog to create several log outputs.

Epylog provides twice-daily log reports of activities on our systems. It runs on our central loghost and generates reports on all systems centrally logging.

Contact Information

Owner: Fedora Infrastructure Team
Contact: #fedora-admin, sysadmin-main
Servers: log01.iad2.fedoraproject.org
Purpose: Provides our central logs and reporting

Essential data/locations:

Logs compiled using rsyslog on log01 into a single set of logs for all systems:
```
/var/log/merged/
```
These logs are rotated every day and kept for only 2 days. This set of logs is only used for immediate analysis and more trivial 'tailing' of the log file to watch for events.
Logs for each system separately in /var/log/hosts

These logs are maintained forever, practically, or for as long as we possibly can. They are broken out into a $hostname/$YEAR/$MON/$DAY directory structure so we can locate a specific day’s log immediately.
Log reports generated by epylog: Log reports generated by epylog are outputted to /srv/web/epylog/merged

This path requires a username and a password to access. To add your username and password you must first join the sysadmin-logs group then login to log01.iad2.fedoraproject.org and run this command:
```
htpasswd -m /srv/web/epylog/.htpasswd $your_username
```
when prompted for a password please input a password which is NOT YOUR FEDORA ACCOUNT SYSTEM PASSWORD.

Let’s say that again to be sure you got it:

DO NOT HAVE THIS BE THE SAME AS YOUR FAS PASSWORD

Configs

Epylog configs are controlled by ansible - please see the ansible epylog module for more details. Specifically the files in roles/epylog/files/merged/

Generating a one-off epylog report

If you wish to generate a specific log report you will need to run the following command on log01:

sudo /usr/sbin/epylog -c /etc/epylog/merged/epylog.conf --last 5h

You can replace '5h' with other time measurements to control the amount of time you want to view from the merged logs. This will mail a report notification to all the people in the sysadmin-logs group.

Audit logs, centrally

We’ve taken the audit logs and enabled our rsyslogd on the hosts to relay the audit log contents to our central log server.

Here’s how we did that:

modify the selinux policy so that rsyslogd can read the file(s) in /var/log/audit/audit.log

BEGIN Selinux policy module:

module audit_via_syslog 1.0;

require {
    type syslogd_t;
    type auditd_log_t;
    class dir { search };
    class file { getattr read open };

}

#============= syslogd_t ==============
allow syslogd_t auditd_log_t:dir search;
allow syslogd_t auditd_log_t:file { getattr read open };

END selinux policy module

add config to rsyslog on the clients to repeatedly send all changes to their audit.log file to the central syslog server as local6:

# monitor auditd log and send out over local6 to central loghost
$ModLoad imfile.so

# auditd audit.log
$InputFileName /var/log/audit/audit.log
$InputFileTag tag_audit_log:
$InputFileStateFile audit_log
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor

then modify your emitter to the syslog server to send local6.* there

on the syslog server - setup log destinations for:
- merged audit logs of all hosts explicitly drop any non-AVC audit message here) magic exclude line is:
  :msg, !contains, "type=AVC" ~
  that line must be directly above the log entry you want to filter and it has a cascade effect on everything below it unless you disable the filter
  per-host audit logs - this is everything from audit.log
On the syslog server - we can run audit2allow/audit2why on the audit logs sent there by doing this:
```
grep 'hostname' /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow
```
the sed is to remove the log prefix garbage from syslog transferring the msg

Future

additional log reports for errors from http processes or servers
SEC Simple Event Coordinator to report, immediately, on events from a log stream - available in fedora/epel.
New report modules within epylog

Want to help? Learn how to contribute to Fedora Docs ›