journalctl: Filtering and Formatting systemd Logs
Logs disappeared. Server rebooted, and the familiar less /var/log/syslog returns nothing. On modern distros with systemd, logs are collected by journald and read with journalctl. Without knowing its filters, system debugging turns into guesswork.
Why Logs Disappear After Reboot
By default, journal stores data in /run/log/journal/ — a tmpfs that wipes on reboot. To make logs survive reboots, create the directory:
Then restart systemd-journald:
Check current location and size:
Fresh CentOS/RHEL 8+ and Fedora create /var/log/journal automatically. Debian and Ubuntu typically do not.
Filtering by Unit and Time Range
The most common case — logs for a specific service:
Combine multiple units by repeating the flag:
Time filters are for incident debugging:
Night crash? Look at logs from that period, not the entire buffer.
If a time filter returns empty output, check the timezone. journalctl stores timestamps in UTC, but --since interprets local time.
Filtering by Priority
Log levels match syslog:
| Level | Number | Description |
|---|---|---|
| emerg | 0 | System unusable |
| alert | 1 | Immediate action required |
| crit | 2 | Critical condition |
| err | 3 | Error |
| warning | 4 | Warning |
| notice | 5 | Normal but significant |
| info | 6 | Informational |
| debug | 7 | Debug-level messages |
The -l flag shows full hostnames instead of truncated ones.
Kernel and Boot Logs
The kernel sends its messages separately. The -k flag replaces dmesg:
List all boots:
Output:
Select a specific boot:
For boot analysis, use systemd-analyze:
Regex Search
Piping journalctl output to grep loses metadata. Use -g (–grep) instead:
The -g flag supports basic regex. For complex conditions, combine with --since:
This keeps the unit and time selection intact, then filters by pattern.
Follow Mode (-f)
Analogous to tail -f for journald. Unlike watching a log file, follow works with any filter:
Ctrl+C stops follow in a terminal. From scripts, wrap with timeout or send a signal.
Run -f in a separate tmux/screen pane. If the pane closes, logs keep going to journald — you won’t lose data.
Flags combine with AND: -u nginx -p err shows errors from nginx only. For OR across units, use journal fields:
Other useful fields:
Output Formats
By default, journalctl paginates output. For scripts and piping to jq, you need machine-readable format:
| Flag | Description | Use Case |
|---|---|---|
-o short | Classic syslog | Default |
-o short-iso | ISO 8601 timestamps | SIEM logging |
-o short-precise | Millisecond precision | Precise timing |
-o verbose | All fields | Maximum detail |
-o json | JSON Lines | jq, Splunk, ELK |
-o cat | MESSAGE field only | Minimal output |
-n 100 limits output to the last 100 lines. --no-pager disables pagination for scripts.
Cleanup and Size Management
journald rotates logs by size and time. Configure in /etc/systemd/journald.conf:
Apply without restart:
Free up space manually:
--vacuum-* only removes files exceeding the limit. To free space reliably, increase SystemMaxUse and restart journald.
Common Errors
journalctl: cannot open files — insufficient permissions. Add yourself to the systemd-journal group:
Logs empty after reboot — persistent storage not configured (see first section).
journalctl hangs — huge buffer. Start with -b or limit with --since.
No unit logs — check that the unit actually ran:
journalctl is built for fast searching. Don’t read logs manually — filter from the start.