Product SiteDocumentation Site

第22章 The kdump Crash Recovery Service

22.1. Configuring the kdump Service
22.1.1. Using the Kernel Dump Configuration Utility
22.1.2. Configuring kdump on the Command Line
22.1.3. Testing the Configuration
22.2. Analyzing the Core Dump
22.2.1. Running the crash Utility
22.2.2. Displaying the Message Buffer
22.2.3. Displaying a Backtrace
22.2.4. Displaying a Process Status
22.2.5. Displaying Virtual Memory Information
22.2.6. Displaying Open Files
22.2.7. Exiting the Utility
22.3. その他のリソース
22.3.1. インストールされているドキュメント
22.3.2. 役に立つ Web サイト
kdump is an advanced crash dumping mechanism. When enabled, the system is booted from the context of another kernel. This second kernel reserves a small amount of memory, and its only purpose is to capture the core dump image in case the system crashes. Since being able to analyze the core dump helps significantly to determine the exact cause of the system failure, it is strongly recommended to have this feature enabled.
This chapter explains how to configure, test, and use the kdump service in Fedora, and provides a brief overview of how to analyze the resulting core dump using the crash debugging utility.

22.1. Configuring the kdump Service

This section covers two common means of configuring the kdump service: using the Kernel Dump Configuration graphical utility, and doing so manually on the command line. It also describes how to test the configuration to verify that everything works as expected.

Disable IOMMU on Intel chipsets

A limitation in the current implementation of the Intel IOMMU driver can occasionally prevent the kdump service from capturing the core dump image. To use kdump on Intel architectures reliably, it is advised that the IOMMU support is disabled.

Make sure you have relevant packages installed

To use the kdump service, you must have the kexec-tools and system-config-kdump packages installed. To do so, type the following at a shell prompt as root:
yum install kexec-tools system-config-kdump
For more information on how to install new packages in Fedora, refer to 「パッケージのインストール」.

22.1.1. Using the Kernel Dump Configuration Utility

To start the Kernel Dump Configuration utility, select ApplicationsOtherKernel crash dumps from the Activities menu, or type system-config-kdump at a shell prompt. You will be presented with a window as shown in 図22.1「Basic Settings」.
The utility allows you to configure kdump as well as to enable or disable starting the service at boot time. When you are done, click Apply to save the changes. The system reboot will be requested, and unless you are already authenticated, you will be prompted to enter the superuser password.

Make sure the system has enough memory

Unless the system has enough memory, the utility will not start, and you will be presented with the following error message:
The not enough memory error
For the information on minimum memory requirements, refer to the Hardware Overview section of the Fedora 15 Release Notes. Note that when the kdump crash recovery is enabled, the minimum memory requirements increase by the amount of memory reserved for it. This value is determined by a user, and defaults to 128MB.

22.1.1.1. Enabling the Service

To start the kdump daemon at boot time, click the Enable button on the toolbar. This will enable the service for runlevels 2, 3, 4, and 5, and start it for the current session. Similarly, clicking the Disable button will disable it for all runlevels and stop the service immediately.
For more information on runlevels and configuring services in general, refer to 7章サービスおよびデーモン.

22.1.1.2. The Basic Settings Tab

The Basic Settings tab enables you to configure the amount of memory that is reserved for the kdump kernel. To do so, select the Manual kdump memory settings radio button, and click the up and down arrow buttons next to the New kdump Memory field to increase or decrease the value. Notice that the Usable Memory field changes accordingly showing you the remaining memory that will be available to the system.
Basic Settings
Basic Settings
図22.1 Basic Settings

22.1.1.3. The Target Settings Tab

The Target Settings tab enables you to specify the target location for the vmcore dump. It can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol.
Target Settings
Target Settings
図22.2 Target Settings

To save the dump to the local file system, select the Local filesystem radio button. Optionally, you can customize the settings by choosing a different partition from the Partition, and a target directory from the Path pulldown lists.
To write the dump directly to a device, select the Raw device radio button, and choose the desired target device from the pulldown list next to it.
To store the dump to a remote machine, select the Network radio button. To use the NFS protocol, select the NFS radio button, and fill the Server name and Path to directory fields. To use the SSH protocol, select the SSH radio button, and fill the Server name, Path to directory, and User name fields with the remote server address, target directory, and a valid remote user name respectively. Refer to 9章OpenSSH for information on how to configure an SSH server, and how to set up a key-based authentication.
For a complete list of currently supported targets, see 表22.1「Supported kdump targets」.
表22.1 Supported kdump targets
Type Supported Targets Unsupported Targets
Raw device All locally attached raw disks and partitions.
Local file system ext2, ext3, ext4, minix file systems on directly attached disk drives, hardware RAID logical drives, LVM devices, and mdraid arrays. The eCryptfs file system.
Remote directory Remote directories accessed using the NFS or SSH protocol over IPv4. Remote directories on the rootfs file system accessed using the NFS protocol.
Remote directories accessed using the iSCSI protocol over hardware initiators. Remote directories accessed using the iSCSI protocol over software initiators.
Remote directories accessed over IPv6.
Remote directories accessed using the SMB/CIFS protocol.
Remote directories accessed using the FCoE (Fibre Channel over Ethernet) protocol.
Remote directories accessed using wireless network interfaces.
Multipath-based storages.

Using the hpsa driver for a storage

Due to known issue with the hpsa driver, kdump is unable to save the dump to a storage that uses this driver for HP Smart Array Controllers. If this applies to your machine, it is advised that you save the dump to a remote system using the NFS or SSH protocol instead.

22.1.1.4. The Filtering Settings Tab

The Filtering Settings tab enables you to select the filtering level for the vmcore dump.
Filtering Settings
Filtering Settings
図22.3 Filtering Settings

To exclude the zero page, cache page, cache private, user data, or free page from the dump, select the check box next to the appropriate label.

22.1.1.5. The Expert Settings Tab

The Expert Settings tab enables you to choose which kernel and initial RAM disk to use, as well as to customize the options that are passed to the kernel and the core collector program.
Expert Settings
Expert Settings
図22.4 Expert Settings

To use a different initial RAM disk, select the Custom initrd radio button, and choose the desired RAM disk from the pulldown list next to it.
To capture a different kernel, select the Custom kernel radio button, and choose the desired kernel image from the pulldown list on the right.
To adjust the list of options that are passed to the kernel at boot time, edit the content of the Edited text field. Note that you can always revert your changes by clicking the Refresh button.
To choose what steps should be taken when the kernel crash is captured, select the appropriate option from the Default action pulldown list. Available options are mount rootfs and run /sbin/init (the default action), reboot (to reboot the system), shell (to present a user with an interactive shell prompt), halt (to halt the system), and poweroff (to power the system off).
To customize the options that are passed to the makedumpfile core collector, edit the Core collector text field; see 「Configuring the Core Collector」 for more information.

22.1.2. Configuring kdump on the Command Line

To perform actions described in this section, you have to be logged in as root. To do so, run the following command:
su -

22.1.2.1. Configuring the Memory Usage

To configure the amount of memory that is reserved for the kdump kernel, open the /boot/grub/grub.conf file in a text editor such as vi or nano, and add the crashkernel=<size>M parameter to the list of kernel options as shown in 例22.1「A sample /boot/grub/grub.conf file」.
例22.1 A sample /boot/grub/grub.conf file
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/sda3
#          initrd /initrd
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux (2.6.32-54.el6.i686)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-54.el6.i686 root=/dev/sda3 ro crashkernel=128M
        initrd /initramfs-2.6.32-54.el6.i686.img

Make sure the system has enough memory

When the kdump crash recovery is enabled, the minimum memory requirements increase by the amount of memory reserved for it. This value is determined by a user, and defaults to 128 MB, as lower values proved to be unreliable. For the information on minimum memory requirements, refer to the Hardware Overview section of the Fedora 15 Release Notes.

22.1.2.2. Configuring the Target Type

When a kernel crash is captured, the core dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol. Note that only one of these options can be set at the moment. The default option is to store the vmcore file in the /var/crash/ directory of the local file system. To change this, open the /etc/kdump.conf configuration file in a text editor such as vi or nano, and edit the options as described below.
To change the local directory in which the core dump is to be saved, remove the hash sign (#) from the beginning of the #path /var/crash line, and replace the value with a desired directory path. Optionally, if you wish to write the file to a different partition, follow the same procedure with the #ext4 /dev/sda3 line as well, and change both the file system type and the device (a device name, a file system label, and UUID are all supported) accordingly. For example:
ext3 /dev/sda4
path /usr/local/cores
To write the dump directly to a device, remove the hash sign (#) from the beginning of the #raw /dev/sda5 line, and replace the value with a desired device name. For example:
raw /dev/sdb1
To store the dump to a remote machine using the NFS protocol, remove the hash sign (#) from the beginning of the #net my.server.com:/export/tmp line, and replace the value with a valid hostname and directory path. For example:
net penguin.example.com:/export/cores
To store the dump to a remote machine using the SSH protocol, remove the hash sign (#) from the beginning of the #net user@my.server.com line, and replace the value with a valid username and hostname. For example:
net john@penguin.example.com
Refer to 9章OpenSSH for information on how to configure an SSH server, and how to set up a key-based authentication.
For a complete list of currently supported targets, see 表22.1「Supported kdump targets」.

Using the hpsa driver for a storage

Due to known issue with the hpsa driver, kdump is unable to save the dump to a storage that uses this driver for HP Smart Array Controllers. If this applies to your machine, it is advised that you save the dump to a remote system using the NFS or SSH protocol instead.

22.1.2.3. Configuring the Core Collector

To reduce the size of the vmcore dump file, kdump allows you to specify an external application (that is, a core collector) to compress the data, and optionally leave out all irrelevant information. Currently, the only fully supported core collector is makedumpfile.
To enable the core collector, open the /etc/kdump.conf configuration file in a text editor such as vi or nano, remove the hash sign (#) from the beginning of the #core_collector makedumpfile -c --message-level 1 -d 31 line, and edit the command line options as described below.
To enable the dump file compression, add the -c parameter. For example:
core_collector makedumpfile -c
To remove certain pages from the dump, add the -d value parameter, where value is a sum of values of pages you want to omit as described in 表22.2「Supported filtering levels」. For example, to remove both zero and free pages, use the following:
core_collector makedumpfile -d 17 -c
Refer to the manual page for makedumpfile for a complete list of available options.
表22.2 Supported filtering levels
オプション 説明
1 Zero pages
2 Cache pages
4 Cache private
8 User pages
16 Free pages

22.1.2.4. Changing the Default Action

By default, when the kernel crash is captured, the root file system is mounted, and /sbin/init is run. To change this behavior, open the /etc/kdump.conf configuration file in a text editor such as vi or nano, remove the hash sign (#) from the beginning of the #default shell line, and replace the value with a desired action as described in 表22.3「Supported actions」. For example:
default halt
表22.3 Supported actions
オプション 説明
reboot Reboot the system, losing the core in the process.
halt After attempting to capture a core, halt the system no matter if it succeeded.
poweroff Power off the system.
shell Run the msh session from within the initramfs, allowing a user to record the core manually.

22.1.2.5. Enabling the Service

To start the kdump daemon at boot time, type the following at a shell prompt:
systemctl enable kdump.service
Similarly, typing systemctl disable kdump.service will disable it. To start the service in the current session, use the following command:
systemctl start kdump.service
For more information on runlevels and configuring services in general, refer to 7章サービスおよびデーモン.

22.1.3. Testing the Configuration

Be careful when using these commands

The commands below will cause the kernel to crash. Use caution when following these steps, and by no means use them on a production machine.
To test the configuration, reboot the system with kdump enabled, and make sure that the service is running (refer to 「サービスの実行」 for more information on how to run a service in Fedora):
systemctl is-active kdump.service
Then type the following commands at a shell prompt:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
This will force the Linux kernel to crash, and the address-YYYY-MM-DD-HH:MM:SS/vmcore file will be copied to the location you have selected in the configuration (that is, to /var/crash/ by default).
例22.2 Listing a content of /var/crash/ after a crash
~]# tree --charset=ascii /var/crash
/var/crash
`-- 127.0.0.1-2010-08-25-08:45:02
    `-- vmcore

1 directory, 1 file