🔹Logging
Centralized Logging
Important Linux Log Files
journalctl
System messages captured by journald are stored in the /run directory. rsyslog can process these messages and store them in traditional log files or forward them to a remote syslog server. You can also access the logs directly with the journalctl command.
displays all log entries:
View journal logs runtime:
Check Service Logs
show systemd logs for a service
To show the most recent 100 journal entries from a specific binary:
default journal configuration file is /etc/systemd/journald.conf; however, this file is not intended to be edited directly.Instead, add your customized configurations to the /etc/systemd/journald.conf.d directory.
The Storage option controls whether to save the journal to disk. The possible val- ues are somewhat confusing:
• volatile stores the journal in memory only.
• persistent saves the journal in /var/log/journal/, creating the directory if it doesn’t already exist.
• auto saves the journal in /var/log/journal/ but does not create the direc- tory. This is the default value.
• none discards all log data.
Filter Logs based on Process PID
To view systemd logs based on process PID we can use _PID as shown below:
Filter Logs based on Priority
Filter output by message priorities or priority ranges. Takes either a single numeric or textual log level (i.e. between 0/"emerg" and 7/"debug"), or a range of numeric/text log levels in the form FROM..TO. The log levels are the usual syslog log levels i.e. "emerg" (0), "alert" (1), "crit" (2), "err" (3), "warning" (4), "notice" (5), "info" (6), "debug" (7).
filter logs for "emerg" priority:
Here we are filtering logs for a range between emerg(0) and critical(2):
View logs in verbose mode
Persistent Journald
Most Linux distributions default to the value auto and do not come with a /var/log/journal directory. Hence, the journal is not saved between reboots by default, which is unfortunate. You can modify this behavior either by creating the /var/log/journal directory or by updating the journal to use persistent storage and restarting systemd-journald:
shows the size of the journal on disk:
Check Boot Logs
shows a sequential list of system boots with numerical identifiers:
You can use the -b option to restrict the log display to a particular boot session. For example, to view logs generated by SSH during the current session:
Check Logs Based on Time
To show all the messages from yesterday at midnight until now:
Filter systemd logs based on timestamp:
You can view systemd logs based on timestamp. There are various arguments to filter such messages. Some of them are shown below.
Check Logs with Details
Filter Kernel Messages
you can use (-k). It is equivalent to (--dmesg). To some extent you can also view Linux boot messages from kernel.
Or alternatively you can also use _TRANSPORT where all the logs with "kernel" match will be filtered
Check the disk or memory used by journal logs
If you are using persistent storage then the below output shows the amount of disk used and if using non-persistent storage then this command will show the amount of memory used for systemd logs.
Perform journal log files cleanup
You can use --vaccum-size which removes archived journal files until the disk space they use falls below the specified size (specified with the usual "K", "M", "G", "T" suffixes),
We will reduce our journal file usage to 200MB using below command:
Now check the disk/memory usage journal logs
Alternatively you can also use --vacuum-time or you can use both --vaccum-size and --vacuum-time together to enforce both a size and time limit on the archived journal files
--vaccum-time is specified with the usual "s", "min", "h", "days", "months", "weeks", "years" suffixes
syslog
Syslog, originally written by Eric Allman, is a comprehensive logging system and IETF-standard logging protocol.
log files are stored at /var/log/syslog .
config file is at /etc/rsyslog.conf .
Syslog generally will log all messages of the priority you specify plus all messages of higher priority levels. However, some newer Syslog agents may allow you to specify facility/priority pairs like "authpriv.=info" which mean "log authpriv.info messages only" (though this is not standard behavior for traditional Unix Syslog implementations).
Log messages can be sent to a local file or another host (note that you can use host names or IP addresses). It is perfectly OK to send the same logging information to multiple destinations. In fact, it is considered good practice to have a copy of your logs on another system to help thwart attacks that tamper with your local log files. Also collecting all of your system logs on a central log server makes analyzing logs more convenient.
It is vitally important to use tabs in /etc/syslog.conf or the file will not be parsed properly. Again, newer syslog agents may allow you to use other whitespace, but traditional Syslog insists on tab characters.
Log Rotation
moving the old log file out of the way and starting a new log file so that the log files doesn't grow without bound and consume the entire filesystem. the Syslog daemon that ships with Linux actually will create new log files on the fly.
There are a number of "free" log-rotating scripts available on the internet—a Google search for "rotate unix logs" comes back with millions of hits.
Manual Rotation
If you end up rolling your own log rotation script, you should know that there is a "correct" way to rotate a log file. You're supposed to rename the old log file with the "mv" command so that syslogd can continue writing to the file (assuming you’re moving files within the same partition, mv just changes the filename but not the inode number on the file), then create a new file with the canonical log filename, and then HUP or restart syslogd to get it to close the old file and open the new file you just created. In this way, no logging information is lost.
Last updated
