From the course: Ubuntu Linux: Essential Commands and System Administration

Explore system log files

From the course: Ubuntu Linux: Essential Commands and System Administration

Start my 1-month free trial Buy for my team

Explore system log files

“

It's useful to have a record of what activities are going on on a system in order to perform troubleshooting, to verify that things are working correctly, and to audit access and security. On Ubuntu, system logging is handled by systemd-journald, a component of systemd. We can view the journal with a journalctl command, which shows us a fire hose of all the entries. That's a little hard to work with, so we'll often want to look at the logs for a specific unit with journalctl -u and a service name like SSH. Or we can use -S or --since, and use a timestamp or the terms yesterday or today. We can also use the -b command to show logs from a specific boot. Each time the system boots, systemd generates a unique ID for that boot, and we can use that, or we can use numbers like minus 1 for the previous boot, minus 2 for the one before that, and so on. Or -b by itself will show the current boot. I encourage you to spend some time with the man page for journalctl to explore what it can do. For convenience and for various historical reasons, the system also writes certain information to various specific log files that can be useful too. Generally speaking, there's a system log and a kernel log, and many other applications and services have their own logs as well. So let's explore a few of the general-purpose system logs. System logs are stored in /var/log. And here on Ubuntu, the system log is named syslog. Let's take a look at that one first. I could use cat to take a look through the file, but these tend to get long, so let's use less instead. I'll type less /var/log/syslog. We've used less before, and we can scroll up and down and search for text, too, which is helpful if we're looking for something in particular. I'd recommend taking a minute or so just to explore this file to get a sense of what yours contains. The log will be different on every system, though it'll have some common themes and types of messages. Every line has a specific format, starting with the date and time that an entry happened in the log or when some process wrote information into the log. After that, is the hostname where the entry came from. Mine here are all the same, the hostname of my machine. But we have this field here because the system logger, whether that's journald or the older, our syslog, can be set up to accept logs from other machines too. And we need a way to tell those entries apart from each other and our own. After that is the name of the process that wrote to the log and its Process ID. This is helpful when we have many processes of the same software running. Then after the colon, there's a message that the software sent to the log. They'll often wrap, but you can scroll right and left with the arrows to see all the information, or you can try to zoom out if you want to make them fit on your screen. The format of what each process writes to the log varies with what kind of information it logs and how the developers or the community need to use that information. Logs tend to get long, so if you're looking for specific information, it's helpful to use some other tools to help narrow down what you have to look at. grep is handy for this, especially if we're looking at simple strings like process names or IDs. Because software constantly writes to them, the log files can get fairly large. By default, every week, the files are rotated and the system compresses an old log file and adds a number to it and creates a new blank log file to receive input. This helps conserve disk space, especially because text files compress extremely well. But it does mean that we don't have one long continuous file to search through. That's helpful when we're trying to wrangle text, but it's not as helpful if we want to look at historical information or see if a certain tool or application has been sending messages of interest for longer than a week. Our regular tools, cat, grep, and less won't work on these files because these files are G-zipped. So instead, we can use zcat, zgrep, and zless, which are designed to work with these compressed text files. I'll move into my /var/log folder and take a look around. We installed the system only recently, so it doesn't have too many archived logs, but here's a couple from the kernel. I'll use the commands that I mentioned just like their counterparts. I'll write zLess and provide the path to one of these compressed logs. And there we go. We can browse it just like a regular text file. Other than starting with the letter Z, these commands work exactly the same as their normal counterparts. You can control how often the logs rotate and how many old logs to keep by editing the /etc/logrotate.conf file. Let's take a look at that with less /etc/logrotate.conf. Here toward the top of the file, I can see that I'm rotating logs weekly and that I'm keeping four weeks' worth of backlogs. Another log that's interesting to look at is the kernel log, which contains messages that the kernel shares about the startup process and hardware information. This information appears in the syslog and in kern.log, but it can also be useful to have the information separately as well. To see the kernel log, we can use the dmesg command. The output of dmesg shows the time since system startup and a message from components of the system. We can switch this output to a human-readable version with dmesg -H. This shows the time since the previous event in a different way. Again, this information is available both in the current. log file and in the syslog, but it can be helpful to have separate access to it as well. The system logs capture a large amount of information, and it can be overwhelming to start exploring them to find what you need. But as you spend time with these files, you'll start to recognize things like errors and hardware events and other useful pieces of information that can point you towards solving a problem.

Contents