# strace: System Call Tracing for Diagnosing Hangs and Leaks

LLMS index: [llms.txt](/en/llms.txt)

---

When a service hangs, standard tools like top, htop, and ps show the state but not the cause. If a process is in state D (uninterruptible sleep), it's waiting on a syscall. strace attaches to a live process and outputs every system call in real time. This turns a mysterious hang into a specific syscall, its arguments, and return code.

> [!NOTE]
> strace uses ptrace, the kernel's debugging mechanism. On production, tracing slows a process by 2–10x. Use it sparingly, targeting a single PID.

## Basic Flags and Syntax

Installation:

```bash
# Debian/Ubuntu
apt install strace

# RHEL/CentOS
yum install strace

# Alpine
apk add strace
```

Running:

```bash
# Trace a new process
strace ls -la /tmp

# Attach to a running process
strace -p 12345

# Attach including subprocesses (fork/clone)
strace -fp 12345
```

Core flags for diagnostics:

| Flag | Purpose |
|------|---------|
| `-f` | Follow child processes |
| `-c` | Count calls and time (summary) |
| `-tt` | Microsecond timestamps |
| `-T` | Time spent in each syscall |
| `-e trace=openat,read,write` | Trace only specified calls |
| `-e write=1,2` | Trace writes to fd 1 and 2 only |
| `-o output.log` | Write output to file |
| `-s 1024` | Truncate strings longer than N characters |

## Diagnosing Blocking Calls

Scenario: process is hung, ps shows state D. Need to find what it's blocked on.

```bash
# Check process status
ps aux | grep nginx
# root     12345  0.0  0.1 ... S    pts/0    0:00 nginx: worker

# Attach for 5 seconds, output everything
strace -p 12345 -f -tt -T 2>&1 | head -50
```

Typical output when blocked on a file:

```
14:23:45.123456 read(15, "", 1024)  = 0 <2.345678>
14:23:47.469134 openat(AT_FDCWD, "/var/data/large-file.db", O_RDONLY) = 15 <0.000023>
14:23:47.469157 read(15, "", 1024)  = 0 <5.678901>
```

If you see `read(...) <time> = 0` with large time values, the process is waiting for data. If `<time>` is in seconds, you've found the bottleneck.

> [!TIP]
> Blocking on `epoll_wait`, `poll`, `select` is normal for an idle process. Look for `read`, `write`, `openat`, `sendto` with times exceeding 100ms.

For network sockets:

```bash
# Trace network calls only
strace -p 12345 -e trace=network,read,write -f
```

Hang on connect() to an unreachable host:

```
14:30:01.234 connect(14, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.100")}, 16) = -1 EINPROGRESS <3.456789>
```

EINPROGRESS means non-blocking socket, but long time indicates a network issue or timeout.

## Finding File Descriptor Leaks

Scenario: process won't open files, error "too many open files". Need to find who's holding the descriptors.

```bash
# Check limits and current usage
ps aux | grep myservice
# 12345  5123  0.0  /opt/myservice

cat /proc/12345/limits | grep "Max open files"
# Max open files            1024                 1024                 files

# Count open fd
ls /proc/12345/fd | wc -l
# 987
```

Attach strace with a filter on file operations:

```bash
strace -p 12345 -e trace=openat,open,close,clone -f 2>&1 | tee /tmp/strace.log
```

Analyze the output:

```bash
# What was opened but not closed
grep -E "openat|close" /tmp/strace.log | awk '{print $2}' | sort | uniq -c | sort -rn | head -20
```

Alternative — summary mode:

```bash
# Stats over 30 seconds
timeout 30 strace -p 12345 -c -f 2>&1
```

```
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- -------
 45.23    1.234567          123       1000         openat
 30.12    0.823456           45      18234       close
 20.45    0.559123         8905        63         read
------ ----------- ----------- --------- --------- -------
100.00    2.617146               19297        63 total
```

If close is called less than openat — you found a leak.

> [!WARNING]
> With high call frequency (thousands per second), strace generates enormous output. Limit time with `-tt` and filter by syscall via `-e trace=`.

## Analyzing Slow Requests

Scenario: API endpoint responds in 5 seconds instead of 200ms. Need to find where time is lost.

```bash
# Start process with tracing and timings
strace -f -tt -T -o /tmp/slow.log ./myservice

# Or attach to running process
strace -p 12345 -f -tt -T 2>&1 | tee /tmp/live.log
```

Find calls with high execution time:

```bash
# Highlight syscalls taking more than 500ms
grep -E "\> [0-9]\.[0-9]{3}" /tmp/live.log | sort -t '>' -k2 -rn | head -20
```

```
14:45:23.123456 write(7, "HTTP/1.1 200 OK\r\n"..., 512) = 512 <0.000045>
14:45:23.890123 openat(AT_FDCWD, "/opt/app/cache.json", O_RDONLY) = 12 <0.523456>
14:45:24.413679 read(12, "{\"key\":\"value\"}"..., 4096) = 4096 <0.000089>
```

openat with 523ms — investigate: file on NFS, missing permissions, remote filesystem.

For SQL-like queries (PostgreSQL, MySQL), trace the socket:

```bash
# Find socket fd of process
ls -la /proc/12345/fd | grep socket
# lr-x 14 -> socket:[1234567]

# Trace specific fd
strace -p 12345 -e write=14 -f -tt -T
```

Slow database query looks like a series of write/read with long time between them:

```
14:50:01.123 write(14, "SELECT * FROM orders"..., 45) = 45 <0.000234>
14:50:05.890 read(14, "", 4096)          = 2048 <4.766890>
```

4.7 seconds between sending the query and receiving data — problem is on the database side or network path to it.

## Summary

strace turns a hang with no visible cause into a specific syscall. Attach to PID, filter calls via `-e trace=`, check execution time with `-T`. For leaks — compare open/close counts in summary mode. For slow requests — find calls with time exceeding 100ms.

> [!TIP]
> On production use `-e trace=write,read,openat` instead of tracing all calls — reduces overhead by 3–5x.