# journalctl: Filtering and Formatting systemd Logs

LLMS index: [llms.txt](/en/llms.txt)

---

Logs disappeared. Server rebooted, and the familiar `less /var/log/syslog` returns nothing. On modern distros with systemd, logs are collected by journald and read with `journalctl`. Without knowing its filters, system debugging turns into guesswork.

## Why Logs Disappear After Reboot

By default, journal stores data in `/run/log/journal/` — a tmpfs that wipes on reboot. To make logs survive reboots, create the directory:

```bash
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
```

Then restart systemd-journald:

```bash
sudo systemctl restart systemd-journald
```

Check current location and size:

```bash
journalctl --disk-usage
```

> [!NOTE]
> Fresh CentOS/RHEL 8+ and Fedora create `/var/log/journal` automatically. Debian and Ubuntu typically do not.

## Filtering by Unit and Time Range

The most common case — logs for a specific service:

```bash
journalctl -u nginx.service
journalctl -u postgresql@main.service
```

Combine multiple units by repeating the flag:

```bash
journalctl -u nginx.service -u php-fpm.service
```

Time filters are for incident debugging:

```bash
# Last hour
journalctl --since "1 hour ago"

# Specific day
journalctl --since "2025-01-15" --until "2025-01-15 23:59:59"

# Last 24 hours
journalctl --since "yesterday"

# From 08:00 to 09:00
journalctl --since "today 08:00" --until "today 09:00"
```

Night crash? Look at logs from that period, not the entire buffer.

> [!WARNING]
> If a time filter returns empty output, check the timezone. journalctl stores timestamps in UTC, but `--since` interprets local time.

## Filtering by Priority

Log levels match syslog:

| Level | Number | Description |
|-------|--------|-------------|
| emerg | 0 | System unusable |
| alert | 1 | Immediate action required |
| crit | 2 | Critical condition |
| err | 3 | Error |
| warning | 4 | Warning |
| notice | 5 | Normal but significant |
| info | 6 | Informational |
| debug | 7 | Debug-level messages |

```bash
# Errors and critical only
journalctl -p err -l

# Warnings through errors
journalctl -p warning..err

# Everything from notice upward
journalctl -p notice
```

The `-l` flag shows full hostnames instead of truncated ones.

## Kernel and Boot Logs

The kernel sends its messages separately. The `-k` flag replaces `dmesg`:

```bash
# Kernel messages for current boot
journalctl -k

# Kernel messages for previous boot
journalctl -k -b -1
```

List all boots:

```bash
journalctl --list-boots
```

Output:

```
-2 5d3c1a9... Mon 2025-01-13 08:00:00 — Mon 2025-01-13 18:00:00
-1 a7b2d8f... Mon 2025-01-13 18:05:00 — Tue 2025-01-14 08:00:00
 0 c9e1f3a... Tue 2025-01-14 08:05:00 — currently running
```

Select a specific boot:

```bash
journalctl -b 5d3c1a9...
```

For boot analysis, use `systemd-analyze`:

```bash
systemd-analyze blame | head -20
systemd-analyze critical-chain nginx.service
```

## Regex Search

Piping journalctl output to grep loses metadata. Use `-g` (--grep) instead:

```bash
# Search for DB connection failures
journalctl -g "connection.*failed" -u myapp.service

# Authentication errors
journalctl -g "auth.*fail" -p err
```

The `-g` flag supports basic regex. For complex conditions, combine with `--since`:

```bash
journalctl -u nginx.service --since "1 hour ago" | grep -E "(timeout|502|503)"
```

This keeps the unit and time selection intact, then filters by pattern.

## Follow Mode (-f)

Analogous to `tail -f` for journald. Unlike watching a log file, follow works with any filter:

```bash
journalctl -u nginx.service -f
journalctl -f -p err
journalctl -u nginx.service -p err -f
```

Ctrl+C stops follow in a terminal. From scripts, wrap with `timeout` or send a signal.

> [!TIP]
> Run `-f` in a separate tmux/screen pane. If the pane closes, logs keep going to journald — you won't lose data.

Flags combine with AND: `-u nginx -p err` shows errors from nginx only. For OR across units, use journal fields:

```bash
journalctl --no-pager _SYSTEMD_UNIT=nginx.service OR _SYSTEMD_UNIT=php-fpm.service -p err
```

Other useful fields:

```bash
# By UID
journalctl --no-pager _UID=1000

# By executable
journalctl --no-pager _EXE=/usr/sbin/nginx

# List values seen for a field
journalctl --no-pager -F _SYSTEMD_UNIT
```

## Output Formats

By default, journalctl paginates output. For scripts and piping to jq, you need machine-readable format:

```bash
# JSON Lines (jq-friendly)
journalctl -u nginx -n 50 -o json

# JSON with pretty structure
journalctl -u nginx -n 50 -o json-pretty
```

| Flag | Description | Use Case |
|------|-------------|----------|
| `-o short` | Classic syslog | Default |
| `-o short-iso` | ISO 8601 timestamps | SIEM logging |
| `-o short-precise` | Millisecond precision | Precise timing |
| `-o verbose` | All fields | Maximum detail |
| `-o json` | JSON Lines | jq, Splunk, ELK |
| `-o cat` | MESSAGE field only | Minimal output |

```bash
# Messages only, no metadata — equivalent to tail -f /var/log/app.log
journalctl -u myapp -f -o cat
```

> [!TIP]
> `-n 100` limits output to the last 100 lines. `--no-pager` disables pagination for scripts.

## Cleanup and Size Management

journald rotates logs by size and time. Configure in `/etc/systemd/journald.conf`:

```ini
[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=30day
```

Apply without restart:

```bash
sudo systemd-tmpfiles --create /etc/tmpfiles.d/journald.conf
sudo killall -USR1 systemd-journald
```

Free up space manually:

```bash
# Show disk usage
journalctl --disk-usage

# Delete logs older than N days
sudo journalctl --vacuum-time=7days

# Delete logs, keeping last N megabytes
sudo journalctl --vacuum-size=200M

# Delete old journal files (not current)
sudo journalctl --vacuum-files=5
```

> [!WARNING]
> `--vacuum-*` only removes files exceeding the limit. To free space reliably, increase `SystemMaxUse` and restart journald.

## Common Errors

**journalctl: cannot open files** — insufficient permissions. Add yourself to the `systemd-journal` group:

```bash
sudo usermod -aG systemd-journal $USER
# re-login
```

**Logs empty after reboot** — persistent storage not configured (see first section).

**journalctl hangs** — huge buffer. Start with `-b` or limit with `--since`.

**No unit logs** — check that the unit actually ran:

```bash
systemctl status nginx
journalctl -u nginx --no-pager -n 20
```

journalctl is built for fast searching. Don't read logs manually — filter from the start.