OProfile

There are two mutually exclusive methods for collecting profiling data with OProfile. You can either use the newer and preferred operf or the opcontrol tool.

This is the recommended mode for profiling. The operf tool uses the Linux Performance Events Subsystem, and therefore does not require the oprofile kernel driver. The operf tool allows you to target your profiling more precisely, as a single process or system-wide, and also allows OProfile to co-exist better with other tools using the performance monitoring hardware on your system. Unlike opcontrol, it can be used without the root privileges. However, operf is also capable of system-wide operations with use of the --system-wide option, where root authority is required.

With operf, there is no initial setup needed. You can invoke operf with command-line options to specify your profiling settings. After that, you can run the OProfile post-processing tools described in Analyzing the Data. See Using operf for further information.

This mode consists of the opcontrol shell script, the oprofiled daemon, and several post-processing tools. The opcontrol command is used for configuring, starting, and stopping a profiling session. An OProfile kernel driver, usually built as a kernel module, is used for collecting samples, which are then recorded into sample files by oprofiled. You can use legacy mode only if you have root privileges. In certain cases, such as when you need to sample areas with disabled interrupt request (IRQ), this is a better alternative.

Before OProfile can be run in legacy mode, it must be configured as shown in Configuring OProfile Using Legacy Mode. These settings are then applied when starting OProfile (Starting and Stopping OProfile Using Legacy Mode).

To monitor the kernel, execute the following command:

With this option, you can specify a path to a vmlinux file that matches the running kernel. Kernel samples will be attributed to this binary, allowing post-processing tools to attribute samples to the appropriate kernel symbols. If this option is not specified, all kernel samples will be attributed to a pseudo binary named “no-vmlinux”.

Most processors contain counters, which are used by OProfile to monitor specific events. As shown in OProfile Processors and Counters, the number of counters available depends on the processor.

The events for each counter can be configured via the command line or with a graphical interface. For more information on the graphical interface, see Graphical Interface. If the counter cannot be set to a specific event, an error message is displayed.

To set the event for each configurable counter via the command line, use:

Here, pass a comma-separated list of event specifications for profiling. Each event specification is a colon-separated list of attributes in the following form:

Event Specifications summarizes these options. The last three values are optional, if you omit them, they will be set to their default values. Note that certain events do require a unit mask.

The events available vary depending on the processor type. When no event specification is given, the default event for the running processor type will be used for profiling. See Default Events for a list of these default events. To determine the events available for profiling, use the ophelp command.

The --separate-thread option categorizes samples by thread group ID (tgid) and thread ID (tid). This is useful for seeing per-thread samples in multi-threaded applications. When used in conjunction with the --system-wide option, --separate-thread is also useful for seeing per-process (i.e., per-thread group) samples for the case where multiple processes are executing the same program during a profiling run.

The --separate-cpu option categorizes samples by CPU.

First, configure whether OProfile should monitor the kernel. This is the only configuration option that is required before starting OProfile. All others are optional.

To monitor the kernel, execute the following command as root:

To configure OProfile not to monitor the kernel, execute the following command as root:

This command also loads the oprofile kernel module, if it is not already loaded, and creates the /dev/oprofile/ directory, if it does not already exist. See Understanding the /dev/oprofile/ directory for details about this directory.

Setting whether samples should be collected within the kernel only changes what data is collected, not how or where the collected data is stored. To generate different sample files for the kernel and application libraries, see Separating Kernel and User-space Profiles.

Most processors contain counters, which are used by OProfile to monitor specific events. As shown in OProfile Processors and Counters, the number of counters available depends on the processor.

Use OProfile Processors and Counters to determine the number of events that can be monitored simultaneously for your CPU type. If the processor does not have supported performance monitoring hardware, the timer is used as the processor type.

If timer is used, events cannot be set for any processor because the hardware does not have support for hardware performance counters. Instead, the timer interrupt is used for profiling.

If timer is not used as the processor type, the events monitored can be changed, and counter 0 for the processor is set to a time-based event by default. If more than one counter exists on the processor, the counters other than 0 are not set to an event by default. The default events monitored are shown in Default Events.

The number of events that can be monitored at one time is determined by the number of counters for the processor. However, it is not a one-to-one correlation; on some processors, certain events must be mapped to specific counters. To determine the number of counters available, execute the following command:

The events available vary depending on the processor type. To determine the events available for profiling, execute the following command as root (the list is specific to the system’s processor type):

The events for each counter can be configured via the command line or with a graphical interface. For more information on the graphical interface, see Graphical Interface. If the counter cannot be set to a specific event, an error message is displayed.

To set the event for each configurable counter via the command line, use opcontrol:

Replace event-name with the exact name of the event from ophelp, and replace sample-rate with the number of events between samples.

By default, kernel mode and user mode information is gathered for each event. To configure OProfile to ignore events in kernel mode for a specific counter, execute the following command:

Execute the following command to start profiling kernel mode for the counter again:

To configure OProfile to ignore events in user mode for a specific counter, execute the following command:

Execute the following command to start profiling user mode for the counter again:

When the OProfile daemon writes the profile data to sample files, it can separate the kernel and library profile data into separate sample files. To configure how the daemon writes to sample files, execute the following command as root:

The choice argument can be one of the following:

If --separate=library is used, the sample file name includes the name of the executable as well as the name of the library.

The opreport tool provides an overview of all the executables being profiled. The following is part of a sample output from the opreport command:

Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is the name of the executable.

See the opreport(1) manual page for a list of available command-line options, such as the -r option used to sort the output from the executable with the smallest number of samples to the one with the largest number of samples. You can also use the -t or --threshold option to trim the output of opcontrol.

To retrieve more detailed profiled information about a specific executable, use the opreport command:

Replace executable with the full path to the executable to be analyzed. mode stands for one of the following options:

OProfile collects data on a system-wide basis for kernel- and user-space code running on the machine. However, once a module is loaded into the kernel, the information about the origin of the kernel module is lost. The module could come from the initrd file on boot up, the directory with the various kernel modules, or a locally created kernel module. As a result, when OProfile records samples for a module, it just lists the samples for the modules for an executable in the root directory, but this is unlikely to be the place with the actual code for the module. You will need to take some steps to make sure that analysis tools get the proper executable.

To get a more detailed view of the actions of the module, you will need to either have the module “unstripped” (that is installed from a custom build) or have the debuginfo package installed for the kernel.

Find out which kernel is running with the uname -a command, obtain the appropriate debuginfo package and install it on the machine.

Then proceed with clearing out the samples from previous runs with the following command:

To start the monitoring process, for example, on a machine with Westmere processor, run the following command:

Then the detailed information, for instance, for the ext4 module can be obtained with:

The opannotate tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting generated files should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.

For this utility to work, the appropriate debuginfo package for the executable must be installed on the system. On Fedora, the debuginfo packages are not automatically installed with the corresponding packages that contain the executable. You have to obtain and install them separately.

The general syntax for opannotate is as follows:

These command-line options are mandatory. Replace src-dir with a path to the directory containing the source code and specify the executable to be analyzed. See the opannotate(1) manual page for a list of additional command line options.

To profile JIT code from the Java Virtual Machine with the JVMTI agent, add the following to the JVM startup parameters:

Where jvmti_oprofile is a path to the OProfile agent. For 64-bit JVM, the path looks as follows:

Currently, you can add one command-line option: --debug, which enables debugging mode.

Depending on the JVM that you are using, you may have to install the debuginfo package for the JVM. For OpenJDK, this package is required, there is no debuginfo package for Oracle JDK. To keep the debug information packages synchronized with their respective non-debug packages, you also need to install the yum-plugin-auto-update-debug-info plug-in. This plug-in searches the debug information repository for corresponding updates.

After successful setup, you can apply the standard profiling and analyzing tools described in previous sections

To learn more about Java support in OProfile, see the OProfile Manual, which is linked from Additional Resources.