Posts

• 1: fixing memory leaks in python services: diagnostics and dump collection
• 2: SSH key best practices
• 3: kubectl cheatsheet: essential commands for Kubernetes
• 4: Creating User and Role in Kubernetes and Binding Them via RBAC
• 5: Git Tag: Marking Releases and Bookmarks in History
• 6: Deploying with a post-receive Git Hook
• 7: ulimit and systemd LimitNOFILE — why ulimit -n inside a unit doesn't stick
• 8: coredumpctl: Finding a Binary Crash
• 9: curl --resolve and SNI: Testing Virtual Hosts Without /etc/hosts
• 10: Docker logs and journald: choosing a logging driver
• 11: scp — Secure Copy Over SSH
• 12: Sudoers: NOPASSWD Without Holes
• 13: ThinLinc: Remote Access to Linux Desktops
• 14: Chrony Instead of ntpd
• 15: fail2ban: SSH Jail Configuration
• 16: journalctl: filters and follow
• 17: nftables: Basic Rule Set
• 18: Debian — The Swiss Army Knife of Linux
• 19: pipx: Isolated Python CLI Tools Without the Mess
• 20: uv — Fast Python Package Manager
• 21: logrotate for Custom Daemons
• 22: Rsync: Backing Up a Directory Over SSH
• 23: Setting Up Your Own SSH Bastion Server
• 24: tmux on Prod After Screen
• 25: bpftrace: one-liners that replace strace in production
• 26: ip: Network Setup and Diagnostics in CLI
• 27: nslookup and drill: DNS resolution in terminal
• 28: journalctl: Filtering and Formatting systemd Logs
• 29: OpenSSL: TLS Certificate Verification and Parsing in CLI
• 30: ProxyJump and bastion hosts via ~/.ssh/config
• 31: auditd: file access and syscall logging
• 32: logrotate: automatic log rotation and archiving
• 33: sshd_config: baseline for a test stand
• 34: systemd-run: Run Services Without Unit Files
• 35: curl: HTTP Debugging in CLI
• 36: ethtool: network interface diagnostics and tuning
• 37: lsof: which processes listen on port and hold file
• 38: mc — MinIO Client S3 CLI
• 39: nftables: Modern Linux Firewall
• 40: ss: socket statistics instead of deprecated netstat
• 41: SSH Config: Wildcards and Dynamic Variable Substitution
• 42: strace: System Call Tracing for Diagnosing Hangs and Leaks
• 43: tcpdump and tshark: Packet Capture in CLI
• 44: CasaOS: Web Dashboard for Home Lab
• 45: cockpit-ufw-module: Uncomplicated Firewall in Cockpit
• 46: Creating a Custom Systemd Service
• 47: kubectl whoami and Service Account Permission Checks
• 48: ngrep: grep for Network Packets in Real Time
• 49: socat: Forwarding Unix Sockets Over TCP
• 50: SSH Escape Sequences: Reviving a Frozen Terminal
• 51: What is Self-Hosted and Why It's So Popular
• 52: cockpit-modules: web panels for day-to-day operations
• 53: dig: DNS Query Debugging in CLI
• 54: MkDocs: Project Documentation from Markdown
• 55: Squid: Internet Forwarding to Remote VM
• 56: SSH certificates instead of authorized_keys
• 57: systemd-timer: scheduling instead of cron
• 58: Cron setup: a practical walkthrough
• 59: GNU Screen: sessions that survive an SSH drop
• 60: Kafka: Cluster Health Check
• 61: kind: Local Kubernetes in Docker
• 62: Too many authentication failures: SSH ran out of tries
• 63: Trusting a custom CA: system store, browsers, and CLI
• 64: ssh-connection-manager: a TUI for hosts in ~/.ssh/config

1 - fixing memory leaks in python services: diagnostics and dump collection

Python services under load can silently consume memory until the cgroup limit triggers an OOM kill. Without systematic dump collection and introspection, root-cause analysis devolves into hypothesis spinning. Below is a practical set of commands and scripts for diagnostics, heap dump collection, and leak mitigation.

1. Memory Consumption Diagnosis Commands

Basic level — psutil. Installed in one line and works without process restart.

pip install psutil

Current process consumption:

import psutil, os
p = psutil.Process(os.getpid())
info = p.memory_info()
print(f"RSS: {info.rss}  VMS: {info.vms}")

Extended structure in one command:

full = p.memory_full_info()
print(full)
# Attributes: python, rss, vms, shared, text, lib, data, dt

Key attributes psutil.Process.memory_full_info() quick reference table:

For tracking growth dynamics over a short interval, psutil in a loop or watch -n 1 psutil ... can be used, but in production tracemalloc, built into CPython, is more common.

import tracemalloc
tracemalloc.start()
# ... service work ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
    print(stat)

2. Tools for Tracking Growth and Dump Collection

When RSS begins creeping upward unnoticed, deeper inspection is required. The toolset depends on debugger availability and ptrace permissions.

gdb + gcore — classic method to extract a full process dump without stopping it (provided coredump is enabled).

# 1. Find the PID
pgrep -f your_service

# 2. Attach gdb and dump the core
gdb -p <PID>
# inside gdb:
(gdb) gcore /tmp/heap_dump.core
(gdb) quit

The resulting gcore file is binary and can be analyzed locally:

# Show segment info
gdb -ex "info files" -ex "quit" /tmp/heap_dump.core
# Or via gdb python plugins (see further)

objgraph — quick answer to “who is holding this object”.

pip install objgraph

import objgraph
# Most frequent object types in memory
objgraph.most_common_types(limit=20)
# Find reference chains
objgraph.find_backref_chains(some_object, 'owner')

tracemalloc + heapdump — Python 3.4+ native mechanism can save a heap snapshot to a file.

import tracemalloc, sys
tracemalloc.start()
# ... ...
snapshot = tracemalloc.take_snapshot()
snapshot.dump('/tmp/tracemalloc.dump')

The dump can be opened in a visualizer, but for deep analysis gdb remains more convenient.

3. Common Anti-patterns and What to Avoid

4. OOM Incident Response Scenario

1. Confirmation — check the cluster event: kubectl get events -n <ns> | grep OOM or dmesg | grep out of memory. Verify the process terminated with code 137.
2. Quick RSS glance — if the service is still alive, run one command:

python3 -c "import psutil, os; p=psutil.Process(os.getpid()); print(p.memory_info().rss // 1024, 'KB')"

3. Dump collection — choose the method based on tool availability:
   • If gdb is available with rights: gdb -p <PID> -batch -ex gcore -ex quit /tmp/oom_dump.core
   • If a quick snapshot is needed: python3 -c "import tracemalloc; tracemalloc.start(); ..." followed by tracemalloc.take_snapshot().statistics('traceback')
4. Analysis — use objgraph.most_common_types() or gdb commands info files, bt for call stacks. Look for unexpected dict, list, or domain class object counts.
5. Fix — eliminate the root cause: add maxsize to caches, remove circular imports, replace global lists with bounded loops, adopt weakref for observers.
6. Prevention — add an alert at rss > 80% of limit in Prometheus/Grafana and document the dump-collection command in the runbook.

Completing diagnostics at this stage either restores the service or gathers sufficient data for a tracker ticket with specific object types and call stacks. Here’s a thinking process:

1. Analyze the Request:

• Topic: Fixing memory leaks in Python services: diagnostics and dump collection
• Role: Practicing Lead DevOps
• Format: IT notes for a Hugo blog (Mana theme)
• Style: Short paragraphs, concrete commands, flag tables, code with fences, no fluff, no marketing, no emojis, no author signature at end
• Allowed: GitHub admonitions (> [!NOTE], > [!TIP], > [!WARNING])
• Prohibited: Made-up flags, URLs, versions; uncertain facts → cautious tone; no YAML/TOML front matter; no wrapping in ```; start with a lid (2-4 sentences); sections with ##; practical commands, tables if needed; 800-1600 words; end on last substantive section; don’t repeat rules, don’t write “User wants”, "
• Key constraint: Write the English article as a parallel original, not a word-for-word translation. Same structure and facts as the Russian draft.

2 - SSH key best practices

SSH keys are the de facto standard for authenticating to infrastructure, but poor management turns every deployment into a potential vulnerability. This note collects proven practices: from key generation to revocation and rotation without service downtime.

Introduction

SSH keys function as long‑lived credentials, and their lifecycle directly impacts supply‑chain security. Unlike passwords, keys are often created once and forgotten, leading to an accumulation of “zombie” keys with privileges that exceed current needs. Proper generation, binding to an agent, and regular rotation minimize the attack surface and enable auditing of changes. The following sections describe concrete commands and configurations used in operations.

Key Generation

Modern OpenSSH defaults recommend the Ed25519 algorithm because of its short key length (256 bits) and high resistance to attacks. RSA still appears in legacy systems but requires increasing the length to at least 4096 bits for acceptable security.

The primary generation command is:

ssh-keygen -t ed25519 -C "user@infrastructure" -f ~/.ssh/id_ed25519

Flag explanations:

• -t – cryptographic algorithm type (ed25519, rsa, ecdsa, sk-ed25519@openssh.com for Security Key).
• -C – comment, usually an email or service name; it is stored in the key but not used for authentication.
• -f – path to the key file; if the directory does not exist, it will be created with correct permissions.

For fully automated scenarios (CI/CD, scripts) you can pass an empty passphrase via -N "", but understand the risk: a passwordless key can be used by any process running as the user. It is preferable to use ssh-agent with an unlock session or a key store (Keychain, ssh-agent -s).

Table: Comparison of common key types

When generating keys for different services, it is convenient to split them by files (e.g., id_ed25519_work, id_ed25519_cloud) to simplify revocation and rotation without affecting other contexts.

Using ssh-agent

The SSH agent holds the decrypted key in memory of the current session, avoiding password entry on each connection. Starting the agent and adding a key:

# Start agent (output environment variables)
eval $(ssh-agent -s)

# Add key to agent
ssh-add ~/.ssh/id_ed25519

In newer OpenSSH versions (8.2+) the agent can be started in config mode:

ssh-agent -c

The ~/.ssh/config file allows automatically loading keys for specific hosts:

Host *.github.com
    AddKeysToAgent yes
    UseKeychain yes
    IdentityFile ~/.ssh/id_ed25519_github

Host *.gitlab.com
    AddKeysToAgent yes
    IdentityFile ~/.ssh/id_ed25519_gitlab

Flags in IdentityFile and AddKeysToAgent guarantee that the key is loaded once on the first connection to a host and remains available for subsequent sessions within the agent’s lifetime. On macOS the flag UseKeychain yes integrates the key with Keychain, locking it when the screen is locked and unlocking when unlocked.

Revocation and Rotation

Keys should not live forever. Rotation policy depends on the infrastructure risk profile, but minimal hygiene includes regular audits and a revocation procedure when personnel change or responsibility shifts.

Revocation procedure:

1. Remove the public key from the remote server:

   ssh-keygen -f ~/.ssh/known_hosts -R "hostname_or_ip"

   or manually edit ~/.ssh/authorized_keys deleting the line corresponding to the revoked key.

2. Locally delete the key and generate a new one:

   shred -u ~/.ssh/id_ed25519  # secure deletion (if util-linux installed)
   rm -rf ~/.ssh/id_ed25519*
   ssh-keygen -t ed25519 -C "user@infrastructure" -f ~/.ssh/id_ed25519

3. Distribute the new public key via IaC or a configuration manager (Ansible, Terraform) to avoid manual edits.

Automated rotation:

For large fleets you can use scripts that check the key creation date (comment -C or metadata inside the key) and automatically revoke keys older than N days. It is important to retain key history for at least 30–90 days before full deletion to avoid breaking services that have not yet updated their configuration.

We need to translate the Russian content into English, preserving the same details, commands, tables, admonitions. Ensure we keep the same practical tone, short paragraphs, specific commands, tables, code fences with language spec. Use GitHub admonitions as in Russian: > [!TIP], > [!WARNING]. Also maybe > [!NOTE] if needed but Russian didn’t have note. We’ll keep tip and warning.

We need to produce

3 - kubectl cheatsheet: essential commands for Kubernetes

kubectl is the primary interface to a Kubernetes cluster. This cheatheet covers routine operations — from context setup to pod debugging and namespace switching. All commands are verified against current kubectl versions (1.28+).

Installation and Context Setup

Installation depends on your OS. On Linux, use the package manager or the binary directly:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
kubectl version --client

Configuration lives in ~/.kube/config. A context defines the cluster, user, and default namespace.

kubectl config current-context
kubectl config use-context production-cluster
kubectl config view --minify   # show current context
kubectl config get-contexts    # list all contexts

Working with Pods

Basic operations:

kubectl get pods                          # all pods in the current namespace
kubectl get pods -A                       # all namespaces
kubectl describe pod <pod-name>           # details and events
kubectl logs <pod-name>                   # container logs
kubectl logs <pod-name> -c <container>    # logs for a specific container in a multi-container pod
kubectl exec -it <pod-name> -- /bin/sh    # enter a pod
kubectl delete pod <pod-name>             # delete a pod

Deployments, StatefulSets, and DaemonSets

kubectl get deployments,statefulsets,daemonsets
kubectl rollout status deployment/<name>       # update status
kubectl rollout history deployment/<name>      # revision history
kubectl rollout undo deployment/<name>         # rollback to previous revision
kubectl rollout undo deployment/<name> --to-revision=2   # rollback to a specific revision
kubectl scale deployment/<name> --replicas=5   # scale replicas

For StatefulSets, startup order and stable identities are critical. For DaemonSets, a guaranteed instance runs on every node.

Services, Ingress, and ConfigMap

kubectl get svc                      # services
kubectl expose deployment/<name> --port=80 --type=NodePort   # create svc from a deployment
kubectl get ingress                  # ingress resources
kubectl describe ingress <name>      # rules and events
kubectl apply -f ingress.yaml        # apply ingress from a file

ConfigMap and Secret for configuration:

kubectl create configmap app-config --from-file=config.yaml
kubectl get configmap app-config -o yaml
kubectl create secret generic db-creds --from-literal=password='s3cr3t'
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d

Debugging and Diagnostics

When a pod is not working, follow this sequence:

kubectl get pods -o wide             # status and node
kubectl describe pod <pod-name>      # events, reason for CrashLoopBackOff
kubectl logs <pod-name> --previous   # logs from a crashed container
kubectl top pod <pod-name>           # CPU/memory consumption (requires metrics-server)
kubectl attach -it <pod-name> -- /bin/sh   # alternative to exec

To test network connectivity from inside the cluster:

kubectl run debug-pod --image=busybox --rm -it --restart=Never -- wget -O- http://<svc-name>.<namespace>.svc.cluster.local:80

Labels, Selectors, and Output Formatting

Labels let you group resources and select them in bulk:

kubectl get pods --show-labels
kubectl label pods <pod-name> app=frontend tier=web
kubectl get pods -l app=frontend       # label selector
kubectl get pods -l 'app in (frontend,backend)'
kubectl get pods -l '!tier=web'        # negative selector

Output formatting:

kubectl get pods -o wide
kubectl get pods -o json               # full JSON
kubectl get pods -o jsonpath='{.items[*].metadata.name}'
kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase

Managing Namespaces and Switching Contexts

kubectl get namespace
kubectl create namespace staging
kubectl delete namespace staging        # deletion is asynchronous
kubectl config set-context --current --namespace=staging   # default namespace in context

For convenience, create a shell alias or function:

# ~/.bashrc or ~/.zshrc
kswitch() { kubectl config use-context "$1" && kubectl config set-context --current --namespace="$2"; }
# usage: kswitch production-cluster default

kubectl is more than a CLI — it is an abstraction layer over the Kubernetes API. Knowing formatting flags and selectors cuts diagnostic time significantly. Keep this cheatheet handy and update it as new versions ship.

4 - Creating User and Role in Kubernetes and Binding Them via RBAC

In Kubernetes there are no “users” in the traditional sense — there are ServiceAccounts and certificates, bound to roles through RBAC. Without proper configuration, anyone holding a kubeconfig gets full access to the cluster. Below is the complete cycle: create a ServiceAccount, define permissions, bind them, and verify.

Creating ServiceAccount and Generating kubeconfig

Start by creating a ServiceAccount in the target namespace:

kubectl create serviceaccount devops-sa -n staging

To generate a kubeconfig, extract the token from secrets and build the config file:

SA_NAME=devops-sa
SA_NS=staging

TOKEN=$(kubectl -n $SA_NS get secret $(kubectl -n $SA_NS get sa $SA_NAME -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode)

kubectl config set-credentials $SA_NAME --token=$TOKEN
kubectl config set-context devops-staging --cluster=$(kubectl config current-context) --user=$SA_NAME --namespace=$SA_NS
kubectl config use-context devops-staging

Defining Role and ClusterRole

A Role is namespace-scoped; a ClusterRole is cluster-scoped. The difference is critical: a Role grants no rights outside its namespace, a ClusterRole does.

Example Role allowing read and create operations on pods in the staging namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: staging
  name: pod-reader-writer
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "create", "delete"]

Example ClusterRole for managing ingresses across the entire cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-manager
rules:
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Creating RoleBinding and ClusterRoleBinding

A RoleBinding binds a Role to a subject within a namespace. A ClusterRoleBinding binds a ClusterRole cluster-wide.

Binding a Role to a ServiceAccount in the staging namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: devops-pod-access
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: devops-sa
    namespace: staging
roleRef:
  kind: Role
  name: pod-reader-writer
  apiGroup: rbac.authorization.k8s.io

Binding a ClusterRole to the same SA (now with cluster-wide ingress rights):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: devops-ingress-access
subjects:
  - kind: ServiceAccount
    name: devops-sa
    namespace: staging
roleRef:
  kind: ClusterRole
  name: ingress-manager
  apiGroup: rbac.authorization.k8s.io

Verifying and Debugging Access Rights

After applying the YAML, confirm the SA actually received the intended permissions:

kubectl auth can-i get pods --as=system:serviceaccount:staging:devops-sa -n staging
kubectl auth can-i create pods --as=system:serviceaccount:staging:devops-sa -n staging
kubectl auth can-i get ingresses --as=system:serviceaccount:staging:devops-sa

The last command checks via ClusterRoleBinding — it returns yes if the binding is correct.

If something doesn’t work, check the audit log or use kubectl auth reconcile with --dry-run=server for a preview:

kubectl auth reconcile role-binding.yaml --dry-run=server

Another useful trick is to find which RoleBindings are attached to a specific SA:

kubectl get rolebindings,clusterrolebindings --all-namespaces -o json | \
  jq '.items[] | select(.subjects[]?.name=="devops-sa") | {name: .metadata.name, kind: .kind, namespace: .metadata.namespace}'

Bottom line: RBAC in Kubernetes works as a chain — SA → Role/ClusterRole → Binding. Each link can be verified independently, which greatly simplifies debugging. Start with minimum privileges and expand as needed; never grant cluster-admin without a compelling reason.

5 - Git Tag: Marking Releases and Bookmarks in History

Git Tag: Marking Releases and Bookmarks in History

Tags are Git’s mechanism for assigning meaningful labels to specific commits. Unlike branches, tags don’t move — they’re pinned to a commit and serve as anchors for releases, versions, and critical checkpoints. Without tags, release history devolves into a hash search — and that’s a direct path to deployment errors.

Types of Tags: Lightweight vs Annotated

There are two types of tags. Lightweight is just a name attached to a commit, with no additional metadata. Annotated is a full Git object with author, date, message, and signing capability.

Creating, Viewing, and Deleting Tags

Create an annotated tag:

git tag -a v1.2.0 -m "Release 1.2.0: API stabilization"

Create a lightweight tag:

git tag v1.2.0-rc1

List all tags:

git tag

Show details of a specific annotated tag:

git show v1.2.0

Delete a local tag:

git tag -d v1.2.0-rc1

List tags matching a pattern:

git tag -l "v1.*"

Pushing Tags to Remote

By default, git push does not send tags. This is a common reason why teammates can’t see a release tag on the remote repository.

Push a single tag:

git push origin v1.2.0

Push all local tags:

git push origin --tags

Delete a tag on remote:

git push origin --delete v1.2.0

Local deletion and remote deletion are two separate operations. Forgetting the second is a typical mistake during a release rollback.

GPG-Signing Tags

Annotated tags can be signed with a GPG key. This guarantees the tag was created by a specific author and hasn’t been tampered with.

Ensure your GPG key is configured first:

gpg --list-secret-keys --keyid-format=long
git config --global user.signingkey <KEY_ID>
git config --global gpg.format openpgp

Create a signed tag:

git tag -s v1.2.0 -m "Signed release 1.2.0"

Verify the signature:

git tag -v v1.2.0

Working with Tags in CI/CD Pipelines

In pipelines, tags are the primary trigger for releases. Most CI/CD systems allow filtering by branches and tags.

Example for GitHub Actions:

on:
  push:
    tags:
      - 'v*'

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-tags: true
      - run: echo "Building for $(git describe --tags)"

Example for GitLab CI:

deploy:
  script:
    - echo "Deploying tag $CI_COMMIT_TAG"
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'

Useful commands inside a pipeline:

# Current tag if the commit is tagged
git describe --tags --exact-match HEAD

# Last tag before the current commit
git describe --tags --abbrev=0 HEAD

# All tags sorted by creation date (newest first)
git tag --sort=-creatordate

Tags are a minimal tool with maximum impact. Annotated with GPG signatures, --tags on release push, and v* filtering in CI — that set is enough to keep release history readable and verifiable.

6 - Deploying with a post-receive Git Hook

Deploying through CI is great, but sometimes you just need to push code to a server with a single git push. The post-receive hook in a bare repository handles this without extra dependencies: push to the server, and the hook automatically checks out files into the working directory.

Schema: bare repo as a deploy trigger

The logic is straightforward:

1. A bare repository is created on the server (e.g., /srv/deploy/app.git).
2. The developer adds it as a remote and runs git push origin main.
3. Git receives the data and triggers hooks/post-receive.
4. The hook script runs git --work-tree=/var/www/app --git-dir=/srv/deploy/app.git checkout -f main.

This is not CI — it is a direct trigger at the Git level. No pipelines, artifacts, or queues. It fits small services, VPS instances, and internal tools well.

Creating a bare repository on the server

SSH into the server and create the repo:

mkdir -p /srv/deploy
cd /srv/deploy
git init --bare app.git

The resulting structure is standard: hooks/, objects/, refs/, HEAD, config. Default hooks live in /srv/deploy/app.git/hooks/ with a .sample suffix — rename them or create your own.

Writing the post-receive hook

Create /srv/deploy/app.git/hooks/post-receive:

#!/usr/bin/env bash
set -euo pipefail

REPO_DIR="/srv/deploy/app.git"
WORK_TREE="/var/www/app"
BRANCH="main"

while read oldrev newrev refname; do
  if [ "$refname" = "refs/heads/$BRANCH" ]; then
    echo "Deploying $BRANCH to $WORK_TREE..."
    git --work-tree="$WORK_TREE" --git-dir="$REPO_DIR" checkout -f "$BRANCH"
    echo "Deployment complete."
  fi
done

Key elements:

Setting up permissions and access

The most common issue is permissions. The Git user you push as must be able to write to WORK_TREE and execute commands from the hook.

Setup options:

• Single user: Git user and web user are the same identity. Simply chown -R deploy:deploy /srv/deploy /var/www/app.
• Group: Add the Git user to the web server group. usermod -aG www-data deploy, then chmod -R g+w /var/www/app.
• SSH key: Push over SSH; the key authenticates as the correct user.

# Example: deploy user, web root /var/www/app
sudo useradd -m deploy
sudo mkdir -p /var/www/app
sudo chown -R deploy:deploy /var/www/app
sudo chmod -R 755 /var/www/app

Do not forget to make the hook executable:

chmod +x /srv/deploy/app.git/hooks/post-receive

Pushing from the client and verifying

On the developer machine, add the remote and push:

git remote add production deploy@server:/srv/deploy/app.git
git push production main

On the server, the hook log will show:

Deploying main to /var/www/app...
Deployment complete.

Verify files on the server:

ls -la /var/www/app
git --git-dir=/srv/deploy/app.git --work-tree=/var/www/app status

That is it. No additional tools — just Git and shell. For high-load production, this is obviously not a replacement for a proper CI pipeline, but for quick deploys to one or two servers it works reliably without unnecessary complexity.

7 - ulimit and systemd LimitNOFILE — why ulimit -n inside a unit doesn't stick

What is nofile and where it lives

nofile is the maximum number of open file descriptors per process. That’s not just regular files — it covers sockets, pipes, stdin/stdout/stderr, logs shipped through journald — everything counts. When nginx or a Go app crashes with too many open files, this is the limit to blame.

Limits live at three levels:

LimitNOFILE in a systemd unit

In a unit file the directive looks like this:

[Service]
LimitNOFILE=65536

You can set both soft and hard limits at once, separated by a space:

LimitNOFILE=65536:1048576

The first value is the soft limit, the second is the hard limit. If you specify only one, it becomes the soft limit and the hard limit is taken from the system maximum.

Why ulimit -n in ExecStart doesn’t work

The typical mistake is trying to set the limit directly in the launch command:

[Service]
ExecStart=/bin/sh -c 'ulimit -n 65536 && exec /usr/bin/myapp'

ulimit -n inside ExecStart does not take effect on the process systemd tracks as MainPID. systemd sets limits before ExecStart runs, and the shell wrapper already operates in a context where the limit is fixed. Worse, ulimit -n may fail with Operation not permitted if the requested value exceeds the hard limit systemd assigned.

If you need to change the limit, use the [Service] directive — not a shell wrapper.

How to check applied limits

Three ways, from simple to authoritative:

# 1. From inside the process
cat /proc/self/limits | grep "Max open files"

# 2. For a specific PID
cat /proc/<pid>/limits | grep "Max open files"

# 3. What systemd sees for the unit
systemctl show myservice.service -p LimitNOFILE

systemctl show reports exactly what systemd applied at fork() — that’s the authoritative source. /proc/<pid>/limits is what the kernel sees for the process. If they disagree, the problem is in an intermediate layer (PAM, container runtime, sudo).

For debugging ExecStart, you can add:

[Service]
ExecStartPre=/bin/sh -c 'cat /proc/self/limits | grep "Max open files"'

This shows the limits before the main process starts — what systemd set for the service.

PAM limits vs systemd limits

On most modern distros (RHEL 8+, Ubuntu 20.04+, Debian 11+) systemd does not invoke pam_limits.so for system services. Limits from /etc/security/limits.conf are not applied to unit files. They only work for login sessions (SSH, local login, su).

If a service isn’t started through systemd (e.g., via supervisor, docker --ulimit, or directly), LimitNOFILE in the unit file doesn’t apply at all. In Docker that’s --ulimit nofile=65536:1048576, in supervisor the stdout_maxbytes and similar options aren’t related to nofile directly — you configure it at the OS or container level.

8 - coredumpctl: Finding a Binary Crash

What is coredumpctl and how it works

When a binary crashes with SEGV, the kernel can save a core dump — a snapshot of process memory at the moment of the crash. In systemd-based distributions, coredumpctl handles collecting, storing, and searching these dumps. It’s a wrapper around systemd-coredump, which stores dumps in /var/lib/systemd/coredump/ and indexes metadata through journald.

Installation is trivial for most distributions:

# Debian/Ubuntu
sudo apt install systemd-coredump

# RHEL/Fedora
sudo dnf install systemd-coredump

After installation, verify that kernel.core_pattern points to a pipe into systemd-coredump:

cat /proc/sys/kernel/core_pattern
# expected: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

If it shows core.%e.%p, dumps are written to files and coredumpctl won’t see them.

Listing dumps: coredumpctl list

The base command prints all saved crashes:

coredumpctl list

Output contains columns: MESSAGE ID, TIMESTAMP, PID, UID, GID, COMM, EXE, COREFILE.

Key flags for list:

Crash details: coredumpctl info

For a single dump, info gives everything needed before pulling the file:

coredumpctl info 1234

Where 1234 is the process PID or number from list. Output includes:

• the signal that caused the crash (SIGSEGV, SIGABRT, etc.)
• timestamp
• executable path
• core file size
• MESSAGE ID — correlation with journald

# Example output (key lines)
         PID: 1234 (myapp)
     UID:GID: 1000:1000
      Signal: 11 (SIGSEGV)
    Timestamp: Mon 2025-01-06 14:23:01 UTC
     Command Line: /opt/myapp/bin/myapp --config prod.yaml
     Executable: /opt/myapp/bin/myapp
       Core File: /var/lib/systemd/coredump/myapp.1234.abc123.core

Extracting the core file: coredumpctl dump

The extracted file can be piped directly into gdb or saved to disk:

# Save to current directory
coredumpctl dump 1234 --output=myapp.core

# Pipe directly into gdb
coredumpctl dump 1234 -o - | gdb /opt/myapp/bin/myapp -

Filtering and searching by binary, PID, time

In production, dumps accumulate by the dozens, and searching by PID is impractical. coredumpctl supports multiple filters at once:

# All crashes of a specific binary
coredumpctl list --exe /opt/myapp/bin/myapp

# Crashes in the last hour
coredumpctl list --since "1 hour ago"

# By specific user and binary
coredumpctl list --exe /usr/bin/python3 --uid 1000

# Only SIGSEGV
coredumpctl list --signal 11

For scripts and automation, JSON output is convenient:

coredumpctl list --json=short | jq '.[] | select(.exe == "/opt/myapp/bin/myapp") | .pid'

Practical examples for debugging crashes

A typical debugging cycle looks like this:

# 1. Find the latest crash of myapp
coredumpctl list --exe /opt/myapp/bin/myapp -n 1

# 2. Check details — what signal and where
coredumpctl info <PID>

# 3. Pull the core and launch gdb
coredumpctl dump <PID> --output=/tmp/myapp.core
gdb /opt/myapp/bin/myapp /tmp/myapp.core

# 4. In gdb:
(gdb) bt full
(gdb) info registers
(gdb) x/16i $pc

If dumps don’t appear, check coredumpctl list and journalctl -u systemd-coredump. A common cause: LimitCORE in the systemd unit is set to 0, or kernel.core_pattern isn’t configured as a pipe.

[Service]
LimitCORE=infinity

9 - curl --resolve and SNI: Testing Virtual Hosts Without /etc/hosts

When you need to test a virtual host on a specific IP but don’t want to edit /etc/hosts — whether due to permissions, conflicts with other services, or just the habit of keeping the file clean — curl --resolve solves both problems at once: it overrides DNS resolution and sends the correct SNI in the TLS handshake.

Problem: virtual host without editing /etc/hosts

Multiple virtual hosts can live on a single IP, and the server picks the right one based on the Host header (HTTP/1.1) and SNI (TLS). Without an entry in /etc/hosts, curl first tries to resolve the name through DNS — getting the wrong IP, or no response at all.

Editing /etc/hosts works, but it requires sudo, clutters the file, and can break other services that depend on the same record.

curl –resolve: syntax and example

The --resolve flag intercepts name resolution at the curl level and substitutes it:

curl --resolve HOST:PORT:ADDR URL

Example:

curl --resolve example.com:443:203.0.113.50 https://example.com/health

curl sends the request to 203.0.113.50:443, but the HTTP Host header will still be example.com.

SNI in combination with –resolve

This is the key point. If the URL contains https://example.com, curl sends example.com as SNI in the TLS ClientHello — even though the IP is overridden via --resolve. The server uses SNI to select the correct certificate and virtual host.

To verify that SNI is actually being sent, use openssl:

openssl s_client -connect 203.0.113.50:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer

If SNI is empty or wrong, the server returns the default certificate — and curl produces an error like SSL: certificate subject name does not match.

Practical example: checking a virtual host

Suppose several virtual hosts run on server 198.51.100.10, and you need to check app.local without touching /etc/hosts:

curl -v --resolve app.local:443:198.51.100.10 https://app.local/api/status

In the -v output you will see:

• Connected to 198.51.100.10 (198.51.100.10) port 443 — connection to the correct IP.
• > Host: app.local — correct Host header.
• TLS SNI extension: "app.local" — SNI sent correctly.

For bulk-checking multiple hosts on the same IP:

for host in app.local api.local admin.local; do
  echo "=== $host ==="
  curl --resolve "$host:443:198.51.100.10" "https://$host/health" -s -o /dev/null -w "%{http_code}\n"
done

If you need the override to work for all tools in the terminal, not just curl, then nsupdate or a local DNS resolver (dnsmasq, stubby) is a better fit. But for a one-off virtual host check, --resolve is faster and safer.

10 - Docker logs and journald: choosing a logging driver

When a container crashes, logs are the first thing you need to see. docker logs looks simple, but under the hood different logging drivers are at work, and the choice affects how logs are stored, rotated, and accessed. Here is what you should know before trusting the default.

How docker logs works

The docker logs <container> command reads the container’s stdout/stderr stream and outputs it to the terminal. Behind this sits a logging driver — a component that determines where the data actually goes. By default it is json-file: each container gets a JSON file on the host into which every output line is written.

The driver is configured at the Docker daemon level or per container. The choice affects log rotation, access via journalctl, and integration with centralized collection systems.

Driver json-file (default)

json-file is the built-in driver with no external dependencies. Each container creates a file at /var/lib/docker/containers/<container-id>/<container-id>-json.log. The format is JSON lines: each entry contains log, stream (stdout or stderr), and time.

Rotation is controlled by two flags:

Without these flags, log files grow without limits. In production this is a direct path to filling the disk.

docker run --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 nginx

Logs can be read via docker logs or directly from the file path on the host, but the latter is not recommended because files may be held open by the daemon.

Driver journald

journald sends container logs into the systemd journal. This means logs are accessible through journalctl, all of journald’s rotation and compression mechanisms apply, and there are no separate JSON files growing on disk.

This requires systemd and the systemd-journal-remote package (on some distributions). The container must be started with the driver specified:

docker run --log-driver journald --log-opt tag={{.Name}} nginx

The tag flag sets the identifier in journald — without it the tag is an empty string and finding the right container becomes difficult. The template {{.Name}} substitutes the container name.

Reading logs:

journalctl -u docker --grep="nginx"
journalctl --user-console -t docker --since "1 hour ago"

More precisely, through journald filters:

journalctl -t docker -g "nginx" --since "2024-01-01"

Actual filtering depends on which metadata Docker passes into journald. Check journalctl -o verbose for a specific container to see what fields are available.

Comparing json-file and journald

Configuring the driver in daemon.json

The global setting goes in /etc/docker/daemon.json:

{
  "log-driver": "journald",
  "log-opts": {
    "tag": "{{.Name}}"
  }
}

After changing it, restart Docker:

sudo systemctl restart docker

Per-container override is possible via --log-driver and --log-opt at startup — this takes precedence over daemon settings.

To check the current driver for a specific container:

docker inspect --format='{{.HostConfig.LogConfig.Type}}' <container>

Practical recommendations

For local development, json-file with explicit max-size and max-file is sufficient and simple to use. For production with dozens of containers on a single host, journald is preferable: a unified search space, built-in compression, and integration with systemd and monitoring.

For centralized collection, both drivers support log forwarding through intermediate drivers (fluentd, gelf, splunk). But journald adds an extra step: first into journald, then the forwarder. For simple cases, direct json-file + fluentd may be a shorter path.

Monitor disk space regularly regardless of the driver. journalctl --disk-usage and du -sh /var/lib/docker/containers/*/ are the minimum set for this.

11 - scp — Secure Copy Over SSH

scp — a utility for copying files over SSH using the SSH protocol. It works from the terminal, requires no extra server setup — just a running sshd and working authentication. In an era of rsync and bat, SCP survives as a simple tool for one-off transfers when you don’t want to deal with daemons or configuration files.

Syntax and Basic Scenarios

General form:

scp [flags] source destination

Source and destination can be local paths or remote addresses in the format user@host:path.

# Local file to a remote machine
scp ./deploy.tar.gz deploy@10.0.2.15:/opt/app/

# Remote file to the local machine
scp deploy@10.0.2.15:/opt/app/deploy.tar.gz ./

# Between two remote hosts (via the local machine)
scp user@host1:/data/backup.sql user@host2:/data/restore/

Recursive Directory Copying

To copy a directory, the -r flag is required. Without it, scp refuses to transfer a directory and prints an error.

scp -r ./project/ dev@10.0.2.15:/home/dev/projects/

Useful Flags

scp -r -p -C -i ~/.ssh/deploy_key.pem -P 2222 ./build/ deploy@10.0.2.15:/var/www/

Typical Transfer Patterns

Deploying artifacts:

scp ./release.tar.gz deploy@prod:/tmp/releases/

Fetching a log from a remote server:

scp admin@10.0.2.15:/var/log/app/error.log ./logs/

Transferring multiple files at once:

scp config.yaml secrets.env deploy@10.0.2.15:/opt/app/

Direct transfer between two servers (both source and destination are remote):

scp -3 user@host1:/data/file.csv user@host2:/data/import/

The -3 flag routes traffic through the local machine. Without it, scp attempts a direct connection between the hosts, which usually fails.

Limitations and Alternatives

scp does not support resuming interrupted transfers — if the connection drops halfway through a multi-gigabyte file, you start over. There is no incremental sync, no checksum verification. Transfers are sequential, with no built-in parallel file transfer.

For everyday tasks, rsync solves these problems:

rsync -avz -e "ssh -p 2222" ./build/ deploy@10.0.2.15:/var/www/

rsync can resume broken transfers, skip already-copied files, and work incrementally. For one-off transfers of a few small files, scp remains convenient — fewer parameters, faster to type.

12 - Sudoers: NOPASSWD Without Holes

Unrestricted NOPASSWD in sudoers is a misconfiguration that grants root access without a password, turning any user script or library vulnerability into a full system compromise. The correct approach limits NOPASSWD to specific commands only.

Why NOPASSWD + ALL Is a Hole, Not a Solution

%admin ALL=(ALL) NOPASSWD: ALL — the most common sudoers error. The user receives unlimited root access without a password. Any script, any utility, any vulnerability in the user’s environment becomes a direct path to full machine control. NOPASSWD without command restrictions is not convenience; it is a backdoor in plain sight.

The proper method: allow specific commands via Cmnd_Alias and attach NOPASSWD only to them. Then the user can restart a service but cannot read /etc/shadow or run su.

visudo: The Only Safe Way to Edit sudoers

Editing /etc/sudoers directly via vi or nano is a path to locking yourself and the entire team out. visudo locks the file, validates syntax before saving, and rejects invalid entries.

# Correct path:
sudo visudo

# If the default editor is inconvenient:
sudo EDITOR=nano visudo

# For separate files in /etc/sudoers.d/:
sudo visudo -f /etc/sudoers.d/deployer

Cmnd_Alias: Grouping Commands Instead of Allowing Everything

Cmnd_Alias lets you create a named group of commands. You then reference the name — readable and easy to change.

Cmnd_Alias RESTART_WEB = /usr/bin/systemctl restart nginx, /usr/bin/systemctl reload nginx
Cmnd_Alias RESTART_DB = /usr/bin/systemctl restart postgresql
Cmnd_Alias PACKAGE_MGMT = /usr/bin/apt, /usr/bin/yum, /usr/bin/dnf
Cmnd_Alias LOG_VIEW = /usr/bin/tail, /usr/bin/journalctl

Alias syntax: name in uppercase, comma-separated absolute paths. The path is mandatory — systemctl without /usr/bin/ will not work.

Example: Limited NOPASSWD for Specific Tasks

Real scenario: a deployer user needs to restart nginx and view logs, nothing more.

Cmnd_Alias RESTART_WEB = /usr/bin/systemctl restart nginx, /usr/bin/systemctl reload nginx
Cmnd_Alias LOG_VIEW = /usr/bin/journalctl, /usr/bin/tail

deployer ALL=(ALL) NOPASSWD: RESTART_WEB, LOG_VIEW

Now deployer can:

sudo systemctl restart nginx    # without password
sudo journalctl -u nginx        # without password
sudo apt update                 # denied
sudo su                         # denied

To allow a single user one binary:

monitor ALL=(ALL) NOPASSWD: /usr/bin/tail /var/log/syslog

What to Avoid: Common sudoers Mistakes

Another frequent trap — spaces in Cmnd_Alias. A comma and space after it are mandatory:

# Correct:
Cmnd_Alias WEB = /usr/bin/systemctl restart nginx, /usr/bin/systemctl reload nginx

# Incorrect (space replaces comma, parsing breaks):
Cmnd_Alias WEB = /usr/bin/systemctl restart nginx /usr/bin/systemctl reload nginx

Verify a rule without root privileges:

sudo -l -U deployer

The output lists permitted commands and their flags. An empty list means the rules did not apply — check syntax and paths. Here’s a thinking process:

1. Analyze User Input:

• Role: Practicing Lead DevOps writing IT notes for a Hugo blog (Mana theme).
• Style: Short paragraphs, specific commands, tables of flags, code with language fences.
• Constraints: No water, no “in this article we’ll explore”, no marketing, no emojis, no author signature at the end.
• Allowed: GitHub admonitions: > [!NOTE], > [!TIP], > [!WARNING].
• Fact Constraint: Don’t make up non-existent flags, URLs, or versions. If fact is unclear, write cautiously as in operations.
• Task: Write the English article as a parallel original, not a word-for-word translation.
• Structure & Facts: Must match the Russian draft exactly.
• Format:

13 - ThinLinc: Remote Access to Linux Desktops

ThinLinc: Remote Access to Linux Desktops

In enterprise environments, remote access to Linux desktops often comes down to VNC with flaky encryption or RDP proxies held together with workarounds. ThinLinc by Cendio is a full-featured remote desktop solution that runs on top of VNC, supports RDP clients, and provides a web-based administration interface without the usual hassle with certificates and firewalls.

What Is ThinLinc

ThinLinc is a remote desktop access solution with a server-plus-clients architecture. The server runs VNC sessions on the backend, and clients connect through a web browser or native ThinLinc clients. The protocol between client and server is tunneled over SSH, which solves the encryption problem out of the box.

Key features:

• Built-in web server for browser-based desktop access
• Native clients for Linux, Windows, and macOS
• Load balancing across multiple servers
• Single sign-on (SSO) via Kerberos and LDAP
• Session management through web panel and CLI

Installing the Server

ThinLinc ships as packages for RHEL/CentOS 7/8/9 and Ubuntu/Debian. For installation on a RHEL system:

# Add the Cendio repository
sudo yum install -y https://www.cendio.com/downloads/thinlinc/rpm/cendio-release-latest.noarch.rpm

# Install the server
sudo yum install -y thinlinc-server

# Start services
sudo systemctl start tlwebd
sudo systemctl start vncserver@:1

For Ubuntu/Debian:

sudo dpkg -i thinlinc-server-*.deb
sudo systemctl start tlwebd
sudo systemctl start vncserver@:1

After installation, the web panel is available at https://<host>:1443.

Server Configuration

The main configuration lives in /etc/thinlinc/. Key files:

Basic configuration via tlconfig:

# Set default screen size
sudo tlconfig --set vncserver.default_screen_width 1920
sudo tlconfig --set vncserver.default_screen_height 1080

# Set color depth
sudo tlconfig --set vncserver.default_color_depth 24

# Set home directory for sessions
sudo tlconfig --set vncserver.home_dir /var/lib/thinlinc

For VNC parameter tuning:

# Edit the VNC configuration
sudo tlconfig --edit vncserver-config-defaults

Client Connections

ThinLinc provides several connection methods:

1. Web browser — navigate to https://<host>:1443, enter credentials. Works on any device with a modern browser.
2. Native client — download from the same URL. Clients are available for Linux, Windows, and macOS.
3. RDP client — ThinLinc supports RDP proxying, allowing connections via standard rdesktop or freerdp.

Connecting via native client:

# Linux
thinlinc-client

# Windows (PowerShell)
Start-Process "C:\Program Files\ThinLinc\Client\tlclient.exe"

Manual SSH tunnel connection:

ssh -L 5901:localhost:5901 user@thinlinc-host
vncviewer localhost:5901

Administration and Session Management

All active sessions can be viewed through the web panel (https://<host>:1443/admin) or via CLI:

# List all sessions
sudo /opt/thinlinc/bin/vncserver -list

# Terminate a specific session
sudo /opt/thinlinc/bin/vncserver -kill :<display_number>

# Restart a specific session
sudo /opt/thinlinc/bin/vncserver -restart :<display_number>

For bulk operations:

# Terminate all sessions for a user
sudo /opt/thinlinc/bin/vncserver -kill -u username

# Limit sessions per user
sudo tlconfig --set vncserver.max_sessions_per_user 3

The admin web panel allows:

• Viewing session lists and their status
• Sending messages to users
• Forcefully terminating sessions
• Viewing logs and usage metrics

Security and Integration

ThinLinc uses SSH for encrypting traffic between client and server. Web server certificates can be replaced with your own:

# Replace the self-signed certificate
sudo cp server.crt /etc/thinlinc/ssl/
sudo cp server.key /etc/thinlinc/ssl/
sudo systemctl restart tlwebd

LDAP/Kerberos integration:

# Enable LDAP authentication
sudo tlconfig --set authentication.ldap.enabled true
sudo tlconfig --set authentication.ldap.server ldap://ldap.example.com
sudo tlconfig --set authentication.ldap.base_dn "dc=example,dc=com"

# Enable Kerberos
sudo tlconfig --set authentication.krb5.enabled true
sudo tlconfig --set authentication.krb5.realm EXAMPLE.COM

For PAM integration:

# Use system PAM authentication
sudo tlconfig --set authentication.pam.enabled true

ThinLinc is a ready-made solution that eliminates the manual assembly of VNC infrastructure with firewalls and certificates. For environments that need remote access to Linux desktops without security compromises, it is one of the most straightforward paths available.

14 - Chrony Instead of ntpd

Chrony has replaced ntpd as the default NTP client in most modern Linux distributions. It converges to accurate time faster, handles intermittent network connections better, and consumes fewer resources. If your machine still runs ntpd, switching takes only a few minutes.

Installation

On RHEL-based systems:

sudo dnf install chrony -y
sudo systemctl enable --now chronyd

On Debian/Ubuntu:

sudo apt install chrony -y
sudo systemctl enable --now chronyd

If ntpd was running on this machine before, stop and disable it to avoid port conflicts:

sudo systemctl stop ntpd
sudo systemctl disable ntpd

makestep Configuration

The key directive in /etc/chrony/chrony.conf (or /etc/chrony.conf on RHEL) is makestep. It controls how chronyd behaves at startup — whether to correct time gradually or in a single step.

makestep 1.0 3

A common mistake is setting makestep -1 1 and expecting a server with a large initial offset to correct instantly. In practice, negative values only work under specific conditions. For reliable startup, use a positive number and limit the number of steps.

allow Directive

By default, chronyd operates only as a client. To let other hosts synchronize through this server, add allow:

allow 10.0.0.0/24

To block a specific host, use deny — it takes effect after allow and overrides it:

allow 10.0.0.0/24
deny 10.0.0.42

After changing the configuration, restart the service:

sudo systemctl restart chronyd

Verification with timedatectl

timedatectl shows the current synchronization state and time source:

timedatectl

Example output:

               Local time: Wed 2025-01-15 14:23:01 MSK
           Universal time: Wed 2025-01-15 11:23:01 UTC
                 RTC time: Wed 2025-01-15 11:23:01
                Time zone: Europe/Moscow (MSK, +0300)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Key fields for diagnostics:

For detailed information about current sources:

chronyc sources -v

If NTP service: active and System clock synchronized: yes, everything is working. If not, check sudo systemctl status chronyd and network access to NTP servers (UDP port 123).

15 - fail2ban: SSH Jail Configuration

Securing SSH against brute-force attacks is one of the first steps in hardening any server. fail2ban scans logs, detects repeated failed login attempts, and blocks the source via iptables or nftables. This note covers the sshd jail — from installation to fine-tuning ban durations.

Installation and Basic Configuration

Install from the standard repository:

# Debian/Ubuntu
apt install fail2ban

# RHEL/CentOS
yum install fail2ban

Enable and start the service:

systemctl enable --now fail2ban
systemctl status fail2ban

Create a local overrides file:

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Or, preferably, create a minimal jail.local with only the parameters you need — fail2ban stacks jail.local on top of jail.conf, so overriding specific keys is sufficient.

Filter for sshd

A ready-made filter ships with the package: /etc/fail2ban/filter.d/sshd.conf. It scans /var/log/auth.log (or /var/log/secure on RHEL) for patterns like Failed password for invalid user and Connection closed by authenticating user.

Verify the filter parses your logs correctly:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

If the output shows a high match rate, the filter works. If not, check the logpath parameter in the jail and the logpath value inside the [Definition] section of the filter.

bantime and findtime Parameters

Three keys define the blocking logic:

A typical production configuration:

[sshd]
bantime  = 3600
findtime = 600
maxretry = 3

This means: three failed logins within ten minutes triggers a one-hour ban. For critical servers, set bantime = -1 (permanent ban) and unblock manually with fail2ban-client set sshd unbanip <IP>.

Activating the Jail

By default, the [sshd] section is commented out in jail.local. Enable it:

[sshd]
enabled = true
port    = ssh
filter  = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime  = 3600
findtime = 600

On RHEL/CentOS, logpath is typically /var/log/secure. Verify the log path on your distribution.

Restart fail2ban and check the status:

systemctl restart fail2ban
fail2ban-client status
fail2ban-client status sshd

The output of fail2ban-client status sshd shows the current number of banned IPs and active filters.

Manual block and unblock:

fail2ban-client set sshd banip 203.0.113.50
fail2ban-client set sshd unbanip 203.0.113.50

Check fail2ban logs (/var/log/fail2ban.log) on first startup — they show whether the jail picked up the log file and whether filters are firing.

16 - journalctl: filters and follow

Systemd’s journal is the first place to look when a service crashes or a node starts burning CPU. journalctl does far more than dump the entire log in sequence: it can filter by units, priorities, time windows, and stream in real time. Below is the working set I use daily.

Follow in real time

Behavior similar to tail -f, but aware of journald’s structured format:

journalctl -f

The -f flag (short for --follow) streams new entries as they appear. By default it shows all units — handy when you don’t know where the fire is.

journalctl -f --no-pager

Filter by unit

The unit is the most common filter. One -u key and the specific service name:

journalctl -u nginx.service
journalctl -u docker.service

Multiple units can be passed — journalctl will show entries from all specified ones:

journalctl -u nginx.service -u postgresql.service

Priority and time filters

Priority filtering is set via -p (or --priority). Levels range from 0 to 7:

Show only errors and critical:

journalctl -p err

Time windows use --since and --until. Both absolute dates and relative expressions are supported:

journalctl --since "1 hour ago"
journalctl --since today --until "2 hours ago"
journalctl --since "2025-01-15 08:00:00" --until "2025-01-15 12:00:00"

Combining flags

In practice, filters are combined. A typical request: watch errors from a specific service over the last hour in real time:

journalctl -u nginx.service -p err --since "1 hour ago" -f --no-pager

Another common pattern — show the last N lines for a unit with a priority filter:

journalctl -u postgresql.service -p warning -n 100 --no-pager

Here -n 100 limits output to the last 100 entries.

Quick reference for everyday flags:

17 - nftables: Basic Rule Set

nftables replaced iptables, but documentation for a basic rule set is often scattered. Here is the reference I use when bringing up a firewall on a new host.

Creating the inet filter table

The inet family table unifies IPv4 and IPv6 under a single namespace. This is the preferred approach when both stacks are active on the host.

nft add table inet filter

If the table already exists the command returns an error. To avoid duplication when a script is re-run:

nft 'add table inet filter' 2>/dev/null || true

To wipe the current rule set before loading your own:

nft flush ruleset

Input and forward chains

Chains bind to a table and define the interception point for traffic. A basic firewall needs input (traffic destined for the host itself) and forward (traffic passing through the host).

nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }
nft add chain inet filter forward { type filter hook forward priority 0 \; policy drop \; }

Key components inside the curly braces:

The syntax requires escaping semicolons inside the string or using single quotes as shown above.

Basic rules for input

With a chain set to policy drop, you must explicitly permit the traffic you need. A typical minimum:

# Allow loopback
nft add rule inet filter input iif lo accept

# Allow established and related connections
nft add rule inet filter input ct state established,related accept

# Allow SSH (port 22)
nft add rule inet filter input tcp dport 22 accept

# Allow ping (ICMP echo request)
nft add rule inet filter input ip protocol icmp icmp type echo-request accept
nft add rule inet filter input ip6 nexthdr icmpv6 icmpv6 type echo-request accept

Each rule appends to the end of the chain. Order matters: accept rules for loopback and established traffic should come before rules with narrower criteria.

To log dropped packets before the drop policy (optional but useful for diagnostics):

nft add rule inet filter input log prefix "nft-drop: " level warn

Basic rules for forward

The forward chain is needed when the host acts as a router or NAT gateway. A minimum setup:

# Allow established connections
nft add rule inet filter forward ct state established,related accept

# Allow forwarding between specific interfaces (example)
nft add rule inet filter forward iifname "eth0" oifname "eth1" accept

If the host does not perform routing, leave forward with policy drop and no additional rules.

To enable IP forwarding at the kernel level (if not already done):

sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1

Viewing and managing rules

After setup, verify the current configuration:

nft list table inet filter
nft list chain inet filter input
nft list ruleset

list ruleset outputs the full config, which you can save and reuse as a boot script.

Deleting a specific rule by handle within a chain:

nft delete rule inet filter input handle <handle-number>

The handle number appears in the output of nft list ruleset -a.

To delete an entire chain:

nft delete chain inet filter input

A chain can only be deleted when it is empty. To remove a table completely, delete all chains inside it first.

Saving rules to a file for boot-time loading:

nft list ruleset > /etc/nftables.conf

On systems with systemd, enable auto-start:

systemctl enable nftables
systemctl start nftables

18 - Debian — The Swiss Army Knife of Linux

Debian isn’t the flashiest distribution, but it’s the most reliable foundation in the Linux world. Behind it stands the largest community of volunteer developers, and behind its track record are decades of uninterrupted operation on servers, embedded systems, and cloud infrastructure. If you manage Linux machines in production, Debian (or one of its derivatives) is already in your stack — whether you’ve acknowledged it or not.

Managing Packages: apt and dpkg

Two tools form the core of Debian’s packaging system. apt is the high-level interface for working with repositories, resolving dependencies, and upgrading the system. dpkg is the low-level engine that installs, removes, and inspects individual .deb files without contacting repositories.

# Update index and upgrade system
apt update
apt upgrade -y
apt full-upgrade -y

# Install and remove
apt install -y nginx
apt remove --purge nginx
apt autoremove -y

# Search and inspect
apt search nginx
apt show nginx
apt list --installed | grep nginx

# dpkg for direct .deb manipulation
dpkg -i package.deb
dpkg -l | grep nginx
dpkg -L nginx
dpkg -S /usr/bin/nginx

A common mistake is mixing apt and dpkg during partial installations. If dpkg -i fails with a dependency error, run apt --fix-broken install and complete the operation through apt.

Stability Model and Release Branches

Debian is built on three branches that determine the aggressiveness of updates and the level of testing.

In sources.list this looks like:

# Stable — recommended for production
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware

Moving between branches is apt dist-upgrade with a swapped sources.list. Do this only on test benches — binary compatibility between testing and stable is not guaranteed for all packages.

Architecture Support

Debian officially supports more hardware platforms than any other distribution. This is critical for embedded and IoT projects.

Multi-architecture support allows installing packages for foreign platforms:

dpkg --add-architecture arm64
apt update
apt install -y package:arm64

Debian as a Base for Other Distributions

Nearly every major distribution you use is built on Debian or inherits its packaging model.

• Ubuntu — takes the testing branch, adds its own PPAs and proprietary drivers.
• Linux Mint — based on Ubuntu, and therefore on Debian packages.
• Proxmox VE — a modified Debian 12 with added enterprise repositories.
• Kali Linux — Debian unstable with a suite of pentesting tools.
• Raspberry Pi OS (formerly Raspbian) — Debian armhf/arm64 with Pi-specific patches.
• MX Linux, antiX — Debian stable with lightweight desktop environments.

echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/sources.list
apt update
apt install -t bookworm-backports linux-image-amd64

Practical DevOps Scenarios

In day-to-day work, the Debian approach manifests in several key patterns.

Base container images. Official Docker images like debian:bookworm, debian:slim are the starting point for hundreds of minimalist containers. debian:slim weighs around 70 MB compared to 200+ MB for ubuntu:latest.

FROM debian:slim
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

VM templates. Proxmox, which runs on Debian, uses debootstrap to create LXC container templates. It is the same tool that underlies a fresh system installation.

Configuration management. Ansible, Puppet, Chef — all three work directly with Debian packages. The apt module in Ansible is one of the most-used modules in any playbook.

CI/CD pipelines. Building .deb packages in GitLab CI or GitHub Actions is a standard pattern. debhelper, dh_make, pbuilder, or sbuild provide reproducible builds in a clean chroot.

Key Commands for Daily Work

Bookmark this table — it covers 90% of tasks on a Debian server.

Debian isn’t the fastest at first glance — no rolling releases and no fresh software versions out of the box. But that predictability is exactly what makes it the foundation for production environments running millions of servers. If you want reliability over trending versions, Debian won’t let you down.

19 - pipx: Isolated Python CLI Tools Without the Mess

pipx solves a simple but chronic problem: you need to run a Python utility once or occasionally, and pip install pollutes the global environment or leaves behind a virtual environment you forget to clean up. pipx creates an isolated venv for each utility, installs dependencies there, and makes the binary available in $PATH. One command and the tool works without conflicting with anything.

What Is pipx and Why You Need It

pipx installs and runs Python applications in isolated virtual environments. Each utility lives in its own venv under ~/.local/pipx/venvs/, and its console-scripts are symlinked into ~/.local/bin/.

Problems it solves:

• Version conflicts between projects: black==23 and black==24 can’t coexist in one environment, but in pipx they can.
• Global pip install pollutes system Python and can break apt on Debian-based systems.
• Forgotten venvs after one-off usage.

Installing pipx

The most reliable path is through pip in user mode or through your system package manager.

# Option 1: via pip (Python 3.6+)
python3 -m pip install --user pipx
python3 -m pipx ensurepath

# Option 2: via apt (Debian/Ubuntu; version may be older)
sudo apt install pipx

# Option 3: via brew (macOS)
brew install pipx

After installation, verify ~/.local/bin is in $PATH:

echo $PATH | grep -q "$HOME/.local/bin" && echo "OK" || echo "add to PATH"

Basic Commands: install, run, list

Three commands cover 90% of use cases.

# Install a utility globally (creates venv, symlinks binary)
pipx install black

# Run a utility without installing it (download + run in a temporary venv)
pipx run httpie https://api.example.com/health

# List all installed utilities
pipx list

Flags worth remembering:

Managing Dependencies and Reinstallation

After installation you can upgrade, remove, and inspect dependencies.

# Upgrade a single utility
pipx upgrade black

# Upgrade all installed utilities
pipx upgrade-all

# Remove a utility and its venv entirely
pipx uninstall black

# Remove everything except pipx itself
pipx uninstall-all

# Inspect dependencies of an installed package
pipx list --verbose

If something breaks, reinstallation takes seconds:

pipx reinstall black
# or with a specific interpreter
pipx reinstall --python python3.12 black

Typical DevOps Scenarios

pipx fits several working patterns where a full project with requirements.txt is overkill.

1. One-off utilities in CI/CD. Instead of installing into a Docker image or globally:

# In a Dockerfile or entrypoint script
pipx run --spec https://pypi.org/project/aws-nuke/ aws-nuke --force --account-id $AWS_ACCOUNT_ID

2. Parallel versions of the same tool. Useful during project migrations:

pipx install black --suffix==23
pipx install black --suffix==24
black==23 --version
black==24 --version

3. Local dev environment without privileges. Install ansible, pre-commit, and similar tools without sudo and without affecting system Python.

4. Quick package evaluation before integration. Test a utility without adding it to requirements.txt:

pipx run httpx https://example.com
# If it works — install permanently
pipx install httpx

pipx doesn’t try to be a package manager for all of Python. It does one thing — isolated CLI utility installation — and does it without the noise. For a Lead DevOps, that means less time fighting dependency conflicts and more time on architecture.

20 - uv — Fast Python Package Manager

What is uv

uv is a Python package manager written in Rust. It solves one problem: the standard pip is slow at resolving dependencies, and poetry adds its own project model on top of PEP 621. uv works with pyproject.toml, is compatible with PEP 621 and PEP 508, and does it significantly faster.

Under the hood is a caching resolver written in Rust that reuses data from pip-compatible indexes (PyPI by default). uv can replace pip, pip-tools, virtualenv, and poetry in a single tool.

Installation

The fastest path is the curl script:

curl -LsSf https://astral.sh/uv/install.sh | sh

This places the binary in ~/.local/bin. On systems without curl:

pip install uv

Or download the standalone binary from GitHub releases.

After installation, verify the version:

uv --version

Key commands

The project workflow:

uv init myproject          # creates pyproject.toml with PEP 621 metadata
cd myproject
uv add requests            # adds a dependency and updates uv.lock
uv add --group dev pytest  # adds a dependency to a named group
uv sync                    # synchronizes the environment with the lock file
uv run pytest              # runs a command inside the project's .venv

Key flags for daily work:

Managing interpreters:

uv python list             # shows discovered CPython installations
uv python install 3.12     # downloads and caches CPython 3.12
uv python find             # shows the path to the active Python

For package-level work without creating a project:

uv pip install requests    # pip equivalent, but faster
uv pip freeze              # lists installed packages
uv pip uninstall requests  # removes a package

Comparison with pip and poetry

uv wins on resolution speed — in astral-sh benchmarks it is 10–50x faster than pip on large dependency graphs. At the same time, uv pip is fully compatible with pip workflows: requirements.txt, --index-url, --find-links.

A common mistake is confusing uv add with uv pip install. The former writes the dependency into pyproject.toml and updates uv.lock; the latter works like pip directly. For projects with pyproject.toml, use uv add. For one-off scripts and Dockerfiles, use uv pip install.

Another frequent issue: uv cannot find the interpreter if Python is installed in a non-standard path. The fix is uv python install --force or explicit specification via --python /path/to/python.

21 - logrotate for Custom Daemons

Log rotation is missing for your daemon, the log file has grown to dozens of gigabytes, the disk is full, and monitoring is screaming. systemd-journald and syslog-ng rotate on their own, but if your custom daemon writes directly to a file, rotation falls to logrotate. Here is how to configure it for a specific service.

Why write a custom logrotate config

Packages from the repository usually drop their config into /etc/logrotate.d/, but for self-built daemons or those compiled from source, there is none. Without a config the file grows without limits. logrotate runs via a systemd timer (logrotate.timer) or cron and reads all files from /etc/logrotate.d/. Creating a single file is enough to start the rotation cycle.

copytruncate vs create — when to use which

These are two fundamentally different approaches to renaming and creating a new file.

If the daemon can receive signals, use create and restart it via postrotate.

delaycompress and how it affects the archive chain

By default logrotate compresses the rotated file in the same cycle. The problem: if the daemon is still writing to the old file (or has not yet re-opened the new one), compression breaks everything.

delaycompress defers compression by one cycle. The chain looks like this:

app.log          ← current
app.log.1        ← rotated, not yet compressed
app.log.2.gz     ← compressed, two cycles ago
app.log.3.gz     ← compressed, three cycles ago

Without delaycompress the transition is harsher: app.log immediately becomes app.log.1.gz, and if the daemon is still writing to app.log through its descriptor, data goes into the compressed archive — or is lost entirely.

Integration with systemd notify

If the daemon supports sd_notify(3), you do not need to rely on postrotate with a manual restart. systemd can restart the service based on a signal from logrotate.

In the logrotate config, specify:

postrotate
    systemctl kill -s HUP my-daemon.service
endscript

Or, if the daemon listens on NOTIFY_SOCKET:

postrotate
    systemctl notify-reload my-daemon.service
endscript

For copytruncate, postrotate is usually unnecessary — truncating the file in place does not require daemon involvement.

Example working config

Suppose daemon my-app writes to /var/log/my-app/app.log and supports SIGHUP and sd_notify.

/var/log/my-app/app.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    postrotate
        systemctl notify-reload my-app.service >/dev/null 2>&1 || true
    endscript
}

Flags:

To test without actually rotating:

logrotate -d /etc/logrotate.d/my-app

The -d flag runs in debug mode — it shows what would be done, without modifying any files.

22 - Rsync: Backing Up a Directory Over SSH

Rsync: Backing Up a Directory Over SSH

The classic way to copy a directory to a remote machine is rsync over SSH. No extra ports to open, traffic is encrypted, and the tool itself handles incremental transfers and metadata preservation. One command and the backup is ready.

Basic rsync Command Over SSH

The minimal invocation to copy a local directory to a remote host:

rsync -avz /path/to/source/ user@host:/path/to/dest/

The trailing slash on source matters: without it, rsync creates a source/ subdirectory on the remote side; with it, the contents go directly into dest/.

If SSH listens on a non-standard port:

rsync -avz -e 'ssh -p 2222' /path/to/source/ user@host:/path/to/dest/

Archive and Sync Flags

-a (archive) is the key flag. It bundles several options into one: recursive traversal, preservation of permissions, ownership, timestamps, symlinks, and empty directories.

Full backup example with bandwidth limit and progress:

rsync -avzP --bwlimit=10000 -e 'ssh -p 2222' \
  /data/backup/ user@backupserver:/mnt/backup/

File and Directory Exclusions

Use --exclude to omit specific paths. Patterns are relative to the source:

rsync -avz --exclude='*.log' --exclude='cache/' \
  /data/ user@host:/data/

When there are many exclusions, a file list is more convenient:

# exclude.txt
*.tmp
*.bak
cache/
lost+found/

rsync -avz --exclude-from='exclude.txt' /data/ user@host:/data/

Dry-Run Before Launch

Always run with --dry-run (or -n) first. rsync shows what would be copied or deleted without touching any files:

rsync -avzn --delete --exclude-from='exclude.txt' \
  /data/ user@host:/data/

Compare the output against the expected list. If it matches, remove -n and run the real sync.

For a cron job with email notifications:

#!/bin/bash
rsync -avz --delete --exclude-from='/etc/rsync-exclude.txt' \
  /data/ user@host:/data/ 2>&1 | mail -s "Rsync backup report" admin@example.com

Summary

rsync over SSH covers 90% of backup tasks without additional agents on the receiver side. The essential order of operations: first --dry-run, then --exclude, then --delete if exact mirroring is needed. Everything else is host- and port-specific tuning.

23 - Setting Up Your Own SSH Bastion Server

Why You Need a Bastion and Where It Lives

A bastion is the single entry point into a private network segment. Instead of exposing SSH on every server to the internet, you funnel traffic through one hardened host with a strict access policy. Typical layout: internet → bastion (public IP) → internal servers (only private subnet, SSH listening on 127.0.0.1 or a private interface).

The bastion sits in a demilitarized zone (DMZ) or a public subnet provided by your hosting platform. Internal machines have no route to the internet through the bastion — return traffic flows only over established connections. This is a baseline model you can deploy on any VPS in about 15 minutes.

Choosing an OS and Basic Setup

The bastion doesn’t need to be heavy. Debian, Ubuntu Server, or AlmaLinux — all work. I usually grab a minimal Ubuntu 22.04 LTS install and bring it to working state by hand.

# Update and install the minimum utility set
sudo apt update && sudo apt upgrade -y
sudo apt install -y fail2ban ufw curl htop

Create a non-privileged user for daily work:

sudo adduser deploy
sudo usermod -aG sudo deploy

SSH Configuration: Keys, Port, Disable Passwords

Generate a key on your workstation if you don’t have one yet:

ssh-keygen -t ed25519 -C "bastion-access"
ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@bastion_ip

On the bastion, edit /etc/ssh/sshd_config:

Port 22220
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
MaxSessions 5
AllowUsers deploy

Validate the config and restart:

sudo sshd -t
sudo systemctl restart sshd

sshd_config keys and what they do:

Firewall and Access Restrictions

UFW is the simplest way to close everything unnecessary:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22220/tcp comment "SSH bastion"
sudo ufw enable

If the bastion is only for your IP, restrict it further:

sudo ufw allow from YOUR_IP to any port 22220 proto tcp

For internal traffic, add a rule on the bastion to allow packet forwarding:

sudo sysctl -w net.ipv4.ip_forward=1

Add net.ipv4.ip_forward = 1 to /etc/sysctl.conf so the setting survives a reboot.

Fail2ban and Brute-Force Protection

Install and configure Fail2ban to protect against password guessing:

sudo apt install fail2ban -y
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

In /etc/fail2ban/jail.local:

[sshd]
enabled = true
port = 22220
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

Restart:

sudo systemctl restart fail2ban
sudo systemctl enable fail2ban
sudo fail2ban-client status sshd

Logging and Auditing

The bastion must log everything. On Debian/Ubuntu, SSH logs go to /var/log/auth.log. For centralized collection, configure rsyslog to ship to a separate SIEM or at least a second server:

# Add to /etc/rsyslog.conf:
*.* @logserver_ip:514

For user action auditing, attach auditd:

sudo apt install auditd -y
sudo auditctl -w /etc/ssh/sshd_config -p wa -k ssh_config
sudo auditctl -w /home -p r -k user_files

View events:

sudo ausearch -k ssh_config
sudo ausearch -k user_files

Port Forwarding and Tunnels Through the Bastion

The bastion’s main job is to give access to internal machines without exposing their ports. Three ways to do it:

1. Port forwarding via SSH tunnel:

ssh -L 5432:10.0.1.5:5432 deploy@bastion_ip -p 22220

Now local port 5432 on your machine proxies to PostgreSQL on 10.0.1.5.

2. SOCKS proxy for access to all internal hosts:

ssh -D 1080 deploy@bastion_ip -p 22220

Point your browser or proxychains at 127.0.0.1:1080.

3. Reverse tunnel for accessing your local machine from the bastion network:

ssh -R 9090:localhost:8080 deploy@bastion_ip -p 22220

This lets someone reach a local service on port 9090 via the bastion.

For persistent tunnels, use autossh or configure ~/.ssh/config:

Host bastion
    HostName bastion_ip
    Port 22220
    User deploy
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 60
    ServerAliveCountMax 3

Now ssh bastion is all you need to connect, and tunnels are built with a single command.

24 - tmux on Prod After Screen

Why We Switched from Screen to tmux

Screen was our primary tool for about five years. After migrating the cluster to new servers it became obvious: screen drops sessions on SSH disconnect when hardstatus isn’t configured, and screen -r recovery sometimes hits a race condition when multiple admins connect simultaneously. tmux solves both problems out of the box — sessions live in server memory, are bound to a socket, and reconnection doesn’t depend on the TCP connection state.

The migration took half a day: we set up a shared config, distributed keybindings, and validated on staging. We went to production a week later — after tmux survived two incidents where screen would have lost context.

Key Differences from GNU Screen

The main difference is architectural. tmux uses a client-server model with a separate server process per session. Screen is also client-server, but its session is bound to the terminal less reliably and is lost more often on disconnect.

Another thing — tmux has a proper copy-mode. In Screen you had to fight with text selection through escape sequences. In tmux, Ctrl+B [ drops you into scrollback with search.

When to Use tmux, When Not To

tmux makes sense when:

• multiple admins work on the same server simultaneously;
• sessions are long-running (monitoring, deploys, debugging);
• you need reliable copy-mode and scrollback;
• automation through the tmux CLI (CI/CD scripts that send commands into a session).

tmux is not needed when:

• one admin per server, short sessions;
• memory-constrained systems — a tmux server consumes more RAM than screen (though on modern machines this is negligible);
• you’re in a containerized environment with an ephemeral filesystem — the tmux config won’t survive container recreation without a volume.

Production Commands

The default prefix is Ctrl+B. All commands follow it.

Creating and attaching:

tmux new -s prod-db
tmux attach -t prod-db
tmux list-sessions

Managing windows and panes:

Ctrl+B c          # new window
Ctrl+B &          # kill window
Ctrl+B %          # split vertically
Ctrl+B "          # split horizontally
Ctrl+B o          # switch between panes
Ctrl+B z          # toggle fullscreen for current pane

Working with copy:

Ctrl+B [          # enter copy-mode
Space             # start selection
Enter             # copy
Ctrl+B ]          # paste clipboard buffer

Scripted invocation (for automation):

tmux send-keys -t prod-db "systemctl restart api" C-m
tmux split-window -t prod-db -h "tail -f /var/log/api.log"

Session persistence across reboot:

Minimal ~/.tmux.conf for production:

set -g default-terminal "screen-256color"
set -g base-index 1
set -g pane-base-index 1
set -g allow-rename off
set -g set-titles on
set -g set-titles-string "#S:#I.#P #W"
set -g status-keys vi
set -g mouse on
bind -n M-1 select-pane -t 0
bind -n M-2 select-pane -t 1
bind -n M-3 select-pane -t 2

After editing the config: tmux source-file ~/.tmux.conf.

Bottom Line

tmux replaced Screen on production not because it’s “newer,” but because its sessions don’t get lost on disconnect, copy works without workarounds, and CLI scripting gives predictability for automation. The trade-off is slightly higher memory usage and the need to maintain a single config across all servers. For our stack of 12 production servers with three admins each, it paid off in the first week. The task is to write an English IT blog post for “Lead DevOps” blog, parallel to the Russian draft provided. The English version should be the original article (not a word-for-word translation), same structure and facts, practical tone, same commands and tables.

Key constraints:

25 - bpftrace: one-liners that replace strace in production

strace halts a process on every syscall. On a live server at 2000 RPS, that means timeouts and alerts. bpftrace runs through eBPF in the kernel — tracing happens in parallel, without stopping anything. The overhead difference is orders of magnitude.

Installation

# Debian / Ubuntu
sudo apt install bpftrace

# RHEL / CentOS / Fedora
sudo dnf install bpftrace

# Arch
sudo pacman -S bpftrace

# Verify
sudo bpftrace -V

Full probe coverage requires debug symbols:

# Debian
sudo apt install linux-image-$(uname -r)-dbg
sudo apt install systemtap-sdt-dev

Check available probes:

sudo bpftrace -l | grep sched_process
# sched:sched_process_exec
# sched:sched_process_fork
# sched:sched_process_exit

bpftrace Syntax in 60 Seconds

One-liner format:

sudo bpftrace -e 'probe { action }'

Structure: what (probe) and what to do (action). Probe types:

Built-in variables available in actions:

pid          # process ID
tid          # thread ID
comm         # process name
nsecs        # nanoseconds timestamp
curtask      # current task_struct
args         # probe arguments (if available)

Example — all execve calls:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { join(args->argv); }'

exec: Who Is Launching Processes

Want to understand which process triggers fork/exec across the system:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_execve {
        time("%H:%M:%S ");
        printf("%s (PID %d) exec: %s\n", comm, pid, args->argv[0]);
    }
'

Output during 10 seconds of monitoring:

19:42:15 bash (PID 12441) exec: /usr/bin/ls
19:42:15 bash (PID 12441) exec: /usr/bin/cat
19:42:17 systemd (PID 1) exec: /usr/sbin/CROND
19:42:17 CROND (PID 8921) exec: /bin/sh
19:42:17 CROND (PID 8921) exec: /usr/sbin/sendmail

Track a specific process and its children:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_execve /pid == 1234/ {
        printf("child exec: %s\n", args->argv[0]);
    }
'

The filter /pid == 1234/ uses standard syntax. Without it, bpftrace catches everything.

open: Which Files a Process Opens

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_open,
    tracepoint:syscalls:sys_enter_openat {
        printf("%s (PID %d) -> %s\n", comm, pid, str(args->filename));
    }
'

Filter by process name:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_openat /comm == "nginx"/ {
        @[str(args->filename)] = count();
    }
'

This is aggregation — counting how many times each file was opened. @ is the built-in variable for maps. Output after Ctrl+C shows a sorted table.

Monitor open failures (ENOENT, EACCES):

sudo bpftrace -e '
    tracepoint:syscalls:sys_exit_openat {
        if (args->ret < 0) {
            printf("%s error %d on %s\n", comm, args->ret, str(args->filename));
        }
    }
'

Network: Connections and Dropped Packets

Monitor outbound connections:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_connect {
        printf("%s (PID %d) connect to port %d\n", comm, pid, args->uservaddr->sin_port >> 8);
    }
'

Dropped iptables packets:

sudo bpftrace -e '
    kprobe:nf_hook_slow {
        @drops[comm] = count();
    }
'

Aggregation by port — a common task:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_connect {
        @port = count();
    }
' 2>/dev/null | sort -rn | head -20

Errors: getpid Does Not Exist

Familiar functions may be absent in bpftrace. This is not bash — different rules apply.

Calling getpid() inside bpftrace produces a compilation error — the BPF program has no access to libc.

Error output — use strerror():

sudo bpftrace -e '
    tracepoint:syscalls:sys_exit_openat {
        if (args->ret < 0) {
            printf("%s: %s\n", str(args->filename), strerror(-args->ret));
        }
    }
'

bpftrace vs strace: Overhead Comparison

strace uses ptrace(PTRACE_SYSCALL). On every syscall, the kernel stops the process, copies data to userspace, then resumes. This is synchronous.

bpftrace compiles a BPF program and loads it into the kernel. Tracing happens in kernel context without stopping the process. Data accumulates in a ring buffer and is read asynchronously.

Comparison on nginx, 5000 RPS:

When bpftrace Is Enough vs When You Need strace

bpftrace — for a system-wide view. Monitoring all processes, aggregation, heat maps, catching anomalies without affecting production.

strace — for deep-diving a specific request. Detailed log of every syscall with arguments and returns to reproduce a problem.

# bpftrace: aggregation — who opens the most files
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

# strace: detailed log of a single request
strace -f -e openat -s 200 curl localhost/api/endpoint

Three rules:

1. Don’t know the process — use bpftrace.
2. Know the PID and need detailed logs — use strace -p PID.
3. On production under load — bpftrace only.

One-liners as aliases:

echo 'alias bt="sudo bpftrace"' >> ~/.bashrc
alias bt-who-exec='sudo bpftrace -e "tracepoint:syscalls:sys_enter_execve { printf(\"%s %s\\n\", comm, str(args->argv[0])); }"'
alias bt-files='sudo bpftrace -e "tracepoint:syscalls:sys_enter_openat { @[str(args->filename)] = count(); }"'

bpftrace goes further — kernel memory, CPU profiling, allocator debugging. For a basic start, these four one-liners cover most of what you need.

26 - ip: Network Setup and Diagnostics in CLI

When ifconfig returns nothing and configuring a route requires a separate command, that’s not a system bug. It’s iproute2 — the package that replaced net-tools in modern Linux distributions. The ip utility from iproute2 is the standard interface for managing the Linux network stack. It covers interfaces, addresses, routes, ARP cache, routing policies, and namespace isolation.

Why iproute2 replaced net-tools

net-tools (ifconfig, route, arp, netstat, nameif) originated in BSD and migrated to Linux in the 1990s. By the 2000s it became clear: they cannot handle VLAN, IPsec, QoS, multicast routing, or Policy Routing. Each task required a separate command with unrelated syntax.

iproute2 consolidated everything into one ip utility with subcommands. The Linux kernel communicates with the network subsystem via netlink sockets — ip talks to them directly, while ifconfig parses /proc/net/. In systemd-based distributions (RHEL 7+, Ubuntu 16.04+, Debian 9+) ip is installed by default. net-tools remains in repositories for compatibility, but kernel developers haven’t added new functionality since 2001.

Install if needed: apt install iproute2 or yum install iproute.

ip link: interface state and management

ip link operates at L2 — listing and controlling interface state.

ip link show
ip link show eth0
ip link show type bridge

Output shows index, name, MAC address, MTU, state (UP/DOWN), and error/packet counters.

Bring an interface up or down:

ip link set eth0 up
ip link set eth0 down

Set MTU, change MAC, or rename:

ip link set eth0 mtu 9000
ip link set eth0 address 02:42:ac:11:00:02
ip link set eth0 name enp0s3

Create virtual interfaces (VETH pair for namespace or bridge):

ip link add veth0 type veth peer name veth1
ip link add br0 type bridge
ip link set veth0 master br0

Delete an interface:

ip link del veth0

ip addr: address binding and diagnostics

ip addr manages IP addresses (L3).

ip addr show
ip addr show eth0

Add an address:

ip addr add 192.168.1.10/24 dev eth0

Add a secondary address (alias) on the same interface:

ip addr add 192.168.1.11/24 dev eth0

Remove an address:

ip addr del 192.168.1.10/24 dev eth0

Flush all addresses from an interface:

ip addr flush dev eth0

Useful during reconfiguration: clear old addresses and assign new ones without restarting the service.

Specify scope and label:

ip addr add 10.0.0.5/8 dev eth0 scope host label eth0:internal

scope host — address only for local sockets; scope global — routable.

ip route: default and static routes

ip route works with the routing table.

ip route show

Add a default route (gateway):

ip route add default via 192.168.1.1 dev eth0

Add a specific route:

ip route add 10.20.0.0/16 via 192.168.1.254 dev eth0

Route to a host via direct ARP (no route, L2 only):

ip route add 192.168.1.50/32 dev eth0

Remove a route:

ip route del default via 192.168.1.1

Replace a route (updates if exists, creates if not):

ip route replace default via 10.0.0.1 dev eth0

Get the route the kernel will choose for an address:

ip route get 8.8.8.8

Add a route to a different table (default is table 254):

ip route add default via 10.0.0.1 dev eth0 table 100

ip neigh: ARP/NDP cache

ip neigh manages the neighbour table — ARP for IPv4, NDP for IPv6.

ip neigh show
ip neigh show dev eth0

Add a static ARP entry:

ip neigh add 192.168.1.1 lladdr 00:11:22:33:44:55 dev eth0 nud permanent

nud (Neighbour Unreachability Detection) defines the state:

• permanent — entry never expires
• noarp — managed by protocol but not removed
• reachable / stale / delay / probe — automatic states

Remove an entry:

ip neigh del 192.168.1.1 dev eth0

Flush all neighbours on an interface:

ip neigh flush dev eth0

ip rule: routing policies

ip rule determines which routing table is used for a packet.

ip rule show

Standard output:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Add a rule for source IP:

ip rule add from 10.0.0.5 table 100

Rule for incoming interface:

ip rule add iif eth0 table 100

Remove a rule:

ip rule del from 10.0.0.5 table 100

ip maddr: multicast addresses

ip maddr displays and manages multicast groups on an interface.

ip maddr show eth0

Add an interface to a multicast group:

ip maddr add 239.0.0.1 dev eth0

Remove:

ip maddr del 239.0.0.1 dev eth0

Unlike unicast, multicast addressing is used in broadcast domains, routing protocols (OSPF, RIP), discovery services, and streaming. In most tasks this command is unnecessary, but when configuring clusters or monitoring via specific protocols — it will be required.

ip netns: network stack isolation

ip netns creates isolated network namespaces. Each namespace has its own interfaces, addresses, routes, ARP table, and rules.

Create a namespace:

ip netns add testns

Run a command inside a namespace:

ip netns exec testns ip link show

Bring up an interface in a namespace:

ip netns exec testns ip link set lo up
ip netns exec testns ip addr add 127.0.0.1/8 dev lo

Move a VETH interface into a namespace:

ip link set veth1 netns testns

Delete a namespace:

ip netns del testns

List namespaces:

ip netns list

ifconfig, arp, route — legacy compatibility

net-tools is formally available in all major distribution repositories. The source code is unmaintained, but packages persist for compatibility.

ss from iproute2 replaces netstat — faster and provides more socket information.

If ifconfig is missing in a fresh distribution — that’s expected. Configuration via ip covers all current scenarios: from address assignment to complex routing policies and service isolation via namespaces.

27 - nslookup and drill: DNS resolution in terminal

The server won’t resolve a domain, but pings fly through. No familiar dig at hand — the BIOS is already loading a minimal busybox. Or on a host without bind-tools. nslookup and drill fill this gap: the first one is built into almost everything, the second gives more context when debugging.

nslookup: interactive and one-liner modes

nslookup ships with bind-utils and isc-dhcp-client. It works in two modes.

One-liner query:

nslookup example.com
nslookup example.com 8.8.8.8

Interactive mode starts with no arguments. Typical session:

$ nslookup
> server 1.1.1.1
Default server: 1.1.1.1
Address: 1.1.1.1#53
> set type=MX
> example.com
Server:         1.1.1.1
Address:        1.1.1.1#53

example.com     mail exchanger = 10 mx1.example.com.
> exit

Switching servers inside a session only changes the resolver for that query. If you need a permanent resolver — edit /etc/resolv.conf.

DNS record types in queries

By default nslookup queries A records. For other types use set type=:

One-liner equivalent — -type= flag:

nslookup -type=ANY example.com
nslookup -type=MX github.com

drill: output with Resource Record type

drill is part of ldns. It returns results in classic DNS format with ANSWER, AUTHORITY, ADDITIONAL sections:

drill A example.com
drill MX github.com @1.1.1.1

drill output is more readable when tracing a CNAME chain:

$ drill CNAME www.cloudflare.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDRIBUTE: 0

;; QUESTION SECTION:
;www.cloudflare.com.   IN   CNAME

;; ANSWER SECTION:
www.cloudflare.com.  300  IN  CNAME  cloudflare.com.

;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:

Without the @server flag, drill reads the resolver from /etc/resolv.conf.

DNSSEC validation

drill checks the DNSSEC trust chain:

drill -S _dmarc.example.com TXT @1.1.1.1

The -S flag requests the DS record higher in the chain and validates the signature. On an invalid chain:

drill: RRSIG validation failed: Signature has expired

nslookup does not validate DNSSEC — it only sends queries with the DO flag (include RRSIG in the response). For full validation you need drill or delv.

NXDOMAIN and SERVFAIL: reading response codes

First step on any error — look at the response code.

NXDOMAIN (code 3) — the domain does not exist. Source: authoritative server for the zone. If dig +short returns nothing, and nslookup says ** server can't find example.invalid, that’s NXDOMAIN. Causes: typo in the domain, stale CNAME, deleted zone.

SERVFAIL (code 2) — the resolver couldn’t answer. Causes: broken DNSSEC validation, exceeded timeout, circular reference in NS records, overloaded authoritative server. nslookup shows ** server can't find example.com: Server failed.

REFUSED (code 5) — the recursor refused to answer. Usually ACL on the DNS server or rate limiting.

nslookup example.com 10.0.0.1
# Server:  10.0.0.1
# Address: 10.0.0.1#53
# ** server can't find example.com: Server failed