The server that died every 5.4 days

Problem

Generic WorkFlow runs as a Docker Compose stack on a single VPS: the Spring Boot backend, PostgreSQL, Kafka and an Nginx-served frontend. In September 2026 the whole machine started becoming unresponsive every five to six days.

It wasn't one service failing. SSH dropped with 'Connection closed'. The backend returned 500s because it couldn't create threads or open database connections. Docker health checks failed with 'OCI runtime exec failed: resource temporarily unavailable'. The hosting control panel stopped responding.

Worse, SAR and cron stopped collecting data too, so the monitoring went blind at the exact moment it was needed. A reboot fixed everything, wiped whatever evidence there was, and about five days later it happened again.

Approach

01Read the symptoms as one failure

Every symptom had the same shape: something failed when it tried to start a new process or thread. SSH needs to fork a session, the JVM needs a native thread for a connection, and Docker needs a process for each health check. 'Resource temporarily unavailable' is EAGAIN, the kernel refusing to create a task.

The cadence was the other clue. A failure that arrives on a steady schedule and resets on reboot is usually a leak with a constant rate, filling a fixed-size table. Here the table was the kernel's task table, capped by threads-max at 31,069 on this VPS.

Kernel tasks over timeRate and limit real · cycle timing schematic

240 leaked tasks an hour against a 31,069 limit: about 129 hours, or 5.4 days, from boot to a host where nothing can fork.

02Find what was filling the table

The task table was filling with zombies: exited processes whose parent never collected their exit status. They were leftovers of the Kafka container's health check, a small TCP probe Docker ran inside the container every 15 seconds.

Inside a container, PID 1 inherits any orphaned process and is responsible for reaping it with wait(). In the Kafka container PID 1 was the JVM itself, and the JVM never reaps children it didn't start. Each probe left one zombie behind: 4 a minute, 240 an hour.

That makes the outage pure arithmetic. 31,069 slots divided by 240 leaked per hour is about 129 hours, or 5.4 days of uptime, before every fork() on the host fails.

The health check, run every 15s inside the Kafka containeryaml

kafka:
  image: apache/kafka:3.7.0
  healthcheck:
    test: ["CMD-SHELL", "timeout 5 bash -c 'exec 3<>/dev/tcp/127.0.0.1/9092'"]
    interval: 15s

03Give the container a real init

The fix is one line. With init: true, Docker starts tini as the container's PID 1 and runs the JVM under it. tini's whole job is to reap orphaned children, so health-check processes are collected the moment they exit.

The fix: tini as PID 1 reaps health-check childrenyaml

kafka:
  image: apache/kafka:3.7.0
  # Use tini as PID 1 so health-check child processes are reaped.
  # Without it, each 15s probe leaves a zombie under the Kafka JVM.
  init: true

04Watch the resource that actually ran out

The exhausted resource was one that usual host dashboards don't graph, so I added a small monitor for it: a systemd timer that runs every 10 minutes and records the task count against threads-max, the zombie count and the load average.

It warns at 80% of threads-max, escalates at 95%, and also alerts on a zombie spike. Alerts fire once per state change rather than every run, go to syslog and optionally to a webhook, and the script always exits 0 so the timer itself never flaps.

The first version had its own bug: it parsed the wrong field of /proc/loadavg. The fourth field is running/total, and only the total is the number that matters. Testing the monitor against the real host caught it the same day.

Every 10 minutes

systemd oneshot + timer appends a history line: tasks, zombies, load.

80% / 95%

WARNING and CRITICAL thresholds against the host's own threads-max.

Once per transition

State files in /run mean one alert per change, not one per run.

Webhook optional

Slack, Discord or ntfy push; syslog and a log file either way.

Excerpt from the monitor (simplified)bash

THREADS_MAX="$(cat /proc/sys/kernel/threads-max)"
# /proc/loadavg field 4 is "running/total" (e.g. "9/412") - we want the total.
TASKS="$(awk '{split($4,a,"/"); print a[2]}' /proc/loadavg)"
ZOMBIES="$(ps -eo stat | grep -c '^Z')"

WARN=$(( THREADS_MAX * 80 / 100 ))
CRIT=$(( THREADS_MAX * 95 / 100 ))

if [ "$TASKS" -ge "$CRIT" ] && [ ! -f "$STATE/crit" ]; then
  touch "$STATE/crit"
  alert CRITICAL "Kernel task count $TASKS >= $CRIT - fork() will fail host-wide"
fi

Outcome

With tini reaping health-check children the leak stopped, and the server no longer has a countdown on it. The monitor's history line now gives a cheap, continuous record of task-table pressure, so a leak like this would show up as a rising line days before it became an outage.

Lessons

PID 1 has a job. If the process in that slot isn't an init, anything that orphans children inside the container leaks.
Health checks are production code. One that runs every 15 seconds runs 5,760 times a day, so any per-run leak compounds fast.
A failure on a steady schedule is a rate problem. Find the table that's filling and divide.
Monitor the resource that actually runs out, not only the ones that are easy to graph.