Kernel Logging, Log Ring Buffer, printk(), syslog-ng (转载)

共两篇，第一篇：

from: http://oguzhanozmen.blogspot.com/2008/09/kernel-log-buffering-printk-syslog-ng.html

Tuesday, September 16, 2008
Kernel Logging, Log Ring Buffer, printk(), syslog-ng
Kernel Log (Ring) Buffer

Linux kernel generates log messages using printk(). These messages are stored in a "ring buffer". The size of this buffer is controlled by a kernel configuration parameter:

CONFIG_LOG_BUF_SHIFT

The default value of this parameter is 14, which means 2^14 bytes, thus 16KB. The size of the buffer can not be changed online so it should be modified (if you'd like to have a larger or smaller buffer) before compiling the kernel (more information on kernel compilation). (1)

printk() - print() function at the Kernel level

printk() is used to print messages at the kernel level. The size of the message can not be larger than 1KB. Below is a sample printk() statement:

printk( KERN_INFO "message\n");

One can classify kernel messages according to their importance/priorities. Priority of a print statement is given by a log level macro. There are 8 priority levels defined in the kernel:

* KERN_EMERG - emergency the highest level
* KERN_ALERT, KERN_CRIT, KERN_ERR, KERN_WARNING, KERN_NOTICE, KERN_INFO, and
* KERN_DEBUG - debugging messages with the lowest priority

For more information on printk(): Linux Device Drivers, 3rd Edition.

How are Kernel Log Messages Exposed to User Space?

The log buffer is exposed to user through /proc/kmsg file. If the file is read one can catch the kernel log messages. In fact, there are available programs to display the ingredients of the file /proc/kmsg, and log the content of this file in a static file (e.g., klogd and syslogd deamons, and syslog-ng logging facility). /proc is a memory filesystem (it contains virtual files which reveals the current state of the running Linux kernel), and the content of /proc/kmsg are being overwritten. In addition, reading from /proc/kmsg is destructive; that's, once you read a line/message it's removed from the file.(2)

"syslog-ng" (system log new generation) is a widely used logging application in Linux systems. It can directly read from /proc/kmsg and log the messages into static file(s). "syslog-ng" can be regarded as the upgraded version of the old kernel deamon "syslogd".

syslog-ng

syslog-ng can be configured so that messages can be directed based on their priorities. That means you direct different level of messages to different files.

syslog-ng can be configured using the configuration file:

* /etc/syslog-ng/syslog-ng.conf (in SuSE Linux, you need to change syslog-ng.conf.in file which is used to generate syslog-ng.conf automatically by SuSE config).

In this configuration file, you define:

* the sources syslog-ng is using: for example, /proc/kmsg, /dev/log, etc.
* filters to identify the priority of a message or the facility from which a message is originated (news, mail, etc.)
* destination files to direct the messages belonging to certain group (based on filters)
* Finally, with "log" statements, you combine {source, filter, destination} to specify where to log which messages.

You can find some sample configuration files here, and some more information on logging with syslog-ng here.

Tuning syslog-ng for Performance

In "Monitoring Block I/O at Linux FileSystem Level" project, I am logging information on each block I/O using printk() function and syslog-ng logging facility. In our experiments, we can produce a trace file (i.e., a log file) of size 1 GB within couple of hours. As a result, to be able to configure HOW we log kernel messages into a static trace file is critical, we don't want to hurt the performance by producing too many I/O operations at a high rate.

However, using syslog-ng, you can control how you log the messages into a file. While defining a destionation file, you can set

* log_fifo_size: log buffer size (in terms of number of messages)
* fsynch(no): by saying "no", syslog-ng will not issue fflush() for each of the message received from the source; otherwise, it'd be catastrophic for the performance.
* flush_lines & flush_timeout: syslog-ng will flush either flush_lines many message are collected in the log buffer of the destination or flush_timeout is passed since the last flushing.

As a result, you can control the rate at which you write the syslog-ng destination files.

Example

Below is a simple example for a syslog-ng configuration file:

# defining a source: /proc/kmsg -> kernel messages
source my_source { file("/proc/kmsg" log_msg_size(1024)); };

# defining a filter: kernel messages with the level of KERN_DEBUG
filter f_myfilter { facility(kern) and priority(debug); };

# defining a destination file: it defines a log file (i.e., a.txt) which will have its
# own buffer with a size of 10000 messages. Messages are flushed from buffer
# to the log file if 8000 messages are collected in the buffer of 5 second is past
# since the last flushing.
destination my_destination { file("/home/user/a.txt" log_fifo_size(10000) fsync(no) flush_lines(8000) flush_timeout(5000) );};

# Finally, a logging point is defined using above source, filter, and destination:
log { source(my_source); filter(f_myfilter); destination(my_destination); };

Foot Notes:

(1) Some information may be kernel version dependant. I've considered Linux kernel 2.6.21 in this document.
(2) dmesg which also examines the kernel ring buffer is on the other hand non-destructive.


Written by Oguzhan Ozmen
Labels: Linux


第二篇：
新手学堂：摸清Linux日志处理的来龙去脉
from：
http://linux.banma.com/club/readsubart.php?b=linux&sid=1492

每个使用UNIX/LINUX的人都知道日志的用处,那你是否清楚LINUX这些日志信息处理的来龙去脉呢？

我们可以看到LINUX系统信息日志的途径基本有以下2种:

（1）dmesg查看----这个命令比较常见

（2）/var/log/下的文件

那下面我们就从这个2个途径着手,一步步的走下去.

（一）

首先,我们来看dmesg这个常见的命令背后隐藏的是什么!!

（1）先让我们来MAN一下这个家伙

-------------man dmesg--------------------------
NAME
dmesg - print or control the kernel ring buffer
SYNOPSIS
dmesg [ -c ] [ -n level ] [ -s bufsize ]
DESCRIPTION
dmesg is used to examine or control the kernel ring
buffer.
The program helps users to print out their bootup mes-
sages. Instead of copying the messages by hand, the user
need only:
dmesg > boot.messages
and mail the boot.messages file to whoever can debug their
problem.
OPTIONS
-c Clear the ring buffer contents after printing.
-sbufsize
Use a buffer of size bufsize to query the kernel
ring buffer. This is 16392 by default. (The
default kernel syslog buffer size was 4096 at
first, 8192 since 1.3.54, 16384 since 2.1.113.) If
you have set the kernel buffer to be larger than
the default then this option can be used to view
the entire buffer.
-nlevel
Set the level at which logging of messages is done
to the console. For example, -n 1 prevents all
messages, expect panic messages, from appearing on
the console. All levels of messages are still
written to /proc/kmsg, so syslogd(8) can still be
used to control exactly where kernel messages
appear. When the -n option is used, dmesg will not
print or clear the kernel ring buffer.
When both options are used, only the last option on
the command line will have an effect.

从LINUX提供的手册,我们可以得知一条最重要的信息dmesg是从kernel 的ring buffer(环缓冲区)中读取信息的.

（2）那什么是ring buffer呢?

在LINUX 中,所有的系统信息(包内核信息)都会传送到ring buffer中.而内核产生的信息由printk()打印出来。系统启动时所看到的信息都是由该函数打印到屏幕中。 printk（）打出的信息往往以＜0＞＜2＞...这的数字表明消息的重要级别。高于一定的优先级别会打印到屏幕上， 否则只会保留在系统的缓冲区中(ring buffer)。

至于dmesg具体是如何从ring buffer中读取的,大家可以看dmesg.c源代码.很短,比较容易读懂.

（二）

dmesg怎么搞的大家应该很明白了吧.至于/var/log/下的文件更是大家熟悉得不能再熟悉了!

（1）/var/log/..下为什么有这么多文件呢?
一句话解释: 是syslogd这个守护进程根据/etc/syslog.conf,将不同的服务产生的Log记录到不同的文件中.这里的/etc/syslog.conf我就不细说了,很多这方面的信息(去查吧).

（2）既然知道了,/var/log/..是由syslogd这个守护进程产生的.那就再顺着这条线走下去.

LINUX系统启动后，由/etc/init.d/sysklogd先后启动klogd,syslogd两个守护进程。
其中klogd会通过syslog()系统调用或者读取proc文件系统来从系统缓冲区(ring buffer)中得到由内核printk()发出的信息.而syslogd是通过klogd来读取系统内核信息.
我想至此,大家心理应该对log产生,读取等一系列的动作有所感觉.

总结
（1）所有系统信息是输出到ring buffer中去的.dmesg所显示的内容也是从ring buffer中读取的.
（2）LINUX系统中/etc/init.d/sysklogd会启动2个守护进程:Klogd&&Syslogd
（3）klogd是负责读取内核信息的,有2种方式:
syslog()系统调用(这个函数用法比较全,大家去MAN一下看看)
直接的对/proc/kmsg进行读取(再这提一下,/proc/kmsg是专门输出内核信息的地方)
（4）Klogd的输出结果会传送给syslogd进行处理,syslogd会根据/etc/syslog.conf的配置把log
信息输出到/var/log/下的不同文件中。

NB:根据我的理解，log ring buffer 就是 /proc/kmsg