My websites kept going down for two or three minutes at a time, and then fixing themselves. No pattern. No warning. By the time I logged in over SSH, the server looked perfectly healthy. Load was back to normal, PHP workers had exited, and there was nothing left to look at. Just a few angry […]

The post I Built a Black Box for My Linux Server — Because My Load Hit 20 While the CPU Sat at 3% first appeared on Internet Security Blog - Hackology.

My websites kept going down for two or three minutes at a time, and then fixing themselves.

No pattern. No warning. By the time I logged in over SSH, the server looked perfectly healthy. Load was back to normal, PHP workers had exited, and there was nothing left to look at. Just a few angry messages from visitors and an uptime monitor saying the site was down at 3:41 AM.

This post is the story of how I chased that problem, the tool I ended up writing, and what it can do today. If your server randomly slows down and you can never catch it in the act, I think you will recognize a lot of this.

The tool is free and open source: Server Forensics Recorder on GitHub.

The problem: high load, almost no CPU usage

The first real clue I got was a load average of over 20 on a 4-core box, while the CPU was only about 3% busy.

That combination makes no sense if you think load means “CPU is busy”. It doesn’t. Load counts processes that are waiting as well as running. So my server was not working hard. It was stuck, with dozens of processes queued up behind something.

MariaDB looked fine. Apache was responsive. Memory was okay. The only odd thing was a pile of lsphp workers and a very high iowait number.

And every single time, by the time I got there, the evidence had already disappeared.

Why monitoring was not enough

I already ran Netdata, and I want to be clear: Netdata is great.

But monitoring answered a question I already knew the answer to. It told me that my server had a problem, at 3:41 AM, with a nice red graph. What I needed to know was why — which process was stuck, what it was waiting for, and who or what started it.

That’s a different job. Monitoring samples numbers for dashboards. I needed something that would grab deep evidence at the exact moment things went bad, and keep it for me to read in the morning.

So I thought about it like an aeroplane. A plane doesn’t just have gauges in the cockpit. It has a black box that records everything, so that after something goes wrong, you can go back and find out what happened.

That was the idea: a black box for my web server.

The first version was simple (and not very smart)

The plan was basic, and I kept it basic on purpose:

1. Every minute, take a cheap sample — load, CPU, memory, swap, PHP worker count, MariaDB threads, mail queue, TCP states, stuck processes. Almost all of it read from /proc, so it costs nothing.


2. If a number crosses a threshold, go into panic mode and capture the expensive stuff: full process tree, iostat, vmstat, open files, disk stats, kernel messages, MySQL process list.


3. Keep capturing until the server recovers, then write a summary and stop.

I had one hard rule from day one: the tool must never make an outage worse. A forensic recorder that hangs the server while trying to investigate it is useless. So everything runs with timeouts, the heavy commands only run during an incident, and package managers like dnf are detected but never invoked — calling one during a disk stall could block forever.

That first version worked. It caught incidents. And then it started confidently telling me things that were completely wrong.

Where it got interesting: every wrong answer taught me something

This is the part I actually want to share, because the tool only became useful after it was wrong a few times. I keep a file in the repo called docs/decisions.md listing every approach I had to throw away. Here are the ones worth telling.

Mistake 1: it blamed whatever was running

My early version scored causes by looking for process names. If it saw backup or security or maintenance processes in the list, it gave points to “maintenance activity”.

One incident got reported as “Maintenance (84%)”. It was nonsense. My server runs Imunify360, and those daemons are running 24 hours a day. Their presence during an incident tells you absolutely nothing. Worse, the tool had also matched its own timeout wrapper and counted that as evidence.

What I learned: presence is not evidence. Now a process being there scores almost nothing. It only becomes real evidence if that same process also shows up as the measured top consumer of CPU or disk. If nothing is actually consuming anything, the report says so out loud.

Mistake 2: it named a process using 9% of the CPU

Another incident had the CPU at 93% on a 4-core box. The tool proudly named the biggest process… which was using 9%.

The real answer was fourteen short-lived PHP workers at about 26% each. Ranking by single process can never express “PHP as a whole took the box”.

What I learned: group by program, not by process ID. Now reports combine workers into one row, with the count, the peak, and the average. And if total CPU is high while no single process is over 15%, the report prints a warning that the cause is spread out.

Mistake 3: it was always measuring the recovery, not the event

This one was the most annoying, and the most important.

Load average over one minute is a lagging number. It peaks after the real work is done. In one incident the load went from 0.63 to 22.46 between two samples, 69 seconds apart. The trigger fired, panic mode started, and the first measurement landed exactly as the workers were exiting.

In other words, the design was structurally guaranteed to miss any burst shorter than the sampling interval. Making the trigger slightly faster would not have fixed it. The ordering itself was wrong.

What I learned: if the evidence expires, you cannot wait for a trigger to go and fetch it. So I added a ring buffer that samples continuously in the background and keeps a rolling window of recent history. When an incident fires, the tool already has the minutes before the trigger. That’s the --runup command: what was building up before everything went bad.

Mistake 4: the sampler was 29 seconds late

For a while my reports kept saying “wait channels: unavailable”. I assumed the capture was broken.

It wasn’t. It was late. Panic mode collected things in a nice readable order — general system state first, then the detailed stuck-process forensics. But those early commands took time: one of them (ss -antp) took 17 seconds by itself. By the time the important capture ran, 29 seconds after the trigger, every stuck process had cleared.

Eight attempts across four incidents. Zero stuck tasks captured. The tool worked perfectly and recorded nothing, which is the worst kind of bug, because it looks like a missing feature instead of a measuring error.

What I learned: collect in order of how fast the evidence disappears, not in the order that reads nicely. Sub-second kernel reads first, slow sampling commands last. And anything truly perishable moved into the always-running ring buffer.

Mistake 5: one incident stayed open for three and a half hours

A crash in the panic script meant nothing closed one incident. It stayed open for 3 hours 27 minutes, until a later unrelated event got pulled into it.

So the report described a disk stall at 18:32, but its only snapshot was taken at 21:59 during an unrelated nightly update. The verdict? “Package manager (38%)” — with “corroborating evidence” that the three-hour gap had invented all by itself.

What I learned: evidence has an age, and age matters. Now incidents get closed as soon as a healthy sample arrives, every capture records how far it was from the trigger, and a stale capture can no longer be used to name a cause.

The payoff: the day it finally proved the real cause

After all that, the tool caught what I had been chasing for months. Not guessed — proved, from the kernel’s own records.

The stuck processes were all sitting in the same place:

jbd2_log_wait_commit   [jbd2]
ext4_sync_file         [ext4]
do_fsync
__x64_sys_fsync

In plain English: every process trying to safely save a file to disk was queued behind the filesystem journal, and the journal could not commit because the virtual disk was not completing its writes. One stuck write freezes everything that touches the filesystem. That is exactly how you get load 20 at 3% CPU.

Then the disk numbers backed it up. During two incidents the recorder caught:

• the disk at 100% utilisation while moving roughly zero data


• service time up to 1001 ms per request


• one read that took 7849 ms


• 15 requests stuck in flight, never completing


• compared to a normal all-day average of about 11% utilisation

A disk reporting “completely busy” while transferring nothing is not a disk being overworked by my websites. It’s a disk that isn’t being served by the host underneath it. This is a Namecheap VPS, and that is the conclusion the evidence points to: the guest was asking politely and the host wasn’t answering.

And the best part: the processes doing all that saving were not my websites at all. It was server housekeeping — cPanel tailwatchd, quota checks, security file-integrity scans, the mail queue runner, cron — all calling fsync at the same time on a disk that could not absorb it. My biggest actual disk user was moving a pathetic 96 to 232 KB per second.

I sent Namecheap the timestamps and the figures — two exact incidents, nine consecutive one-second disk samples each, with the service times and the stuck request count. That was only possible because something had been recording while I was asleep. Without it, the whole conversation would have been me saying “my server feels slow sometimes” and them saying “we see no problem”.

I’ll be honest about scope, because that honesty is the whole point of the tool: the dramatic zero-throughput numbers appeared in 2 of 9 incidents. In the other 7 the stall was still proven, just by a different kernel measurement (PSI), because the disk sampler happened to miss the worst second. The reports say which is which rather than pretending every incident is the same.

What Server Forensics Recorder can do today

So that’s how Server Forensics Recorder ended up the way it is. It’s on GitHub, free and MIT licensed, because this problem is clearly not unique to my server. Everything is plain Bash — there’s no daemon, no web interface and no database, and every report is text you can read over SSH.

It records continuously, cheaply. One lightweight sample per minute, plus a background ring buffer. Normal overhead is tiny because almost everything comes from /proc.

It captures deep evidence only during an incident — process trees, disk statistics, open files, kernel messages, pressure stall information, MySQL state — ordered so the most perishable evidence is grabbed first.

It writes its own root-cause analysis. When an incident ends, you get a report that separates observed facts (measurements, no opinion) from inference (reasoning) from proven (direct kernel evidence). It ranks the possible causes with a confidence score, and here’s the bit I’m most pleased with: the confidence is capped when the decisive evidence is missing, and the report tells you it was capped and why. It never claims 100%. If nothing clears the noise floor, it says Inconclusive instead of inventing a cause.

It names the URL and the visitor, not just the layer. “PHP was busy” is useless. At incident close, the recorder reads the domain access logs for the incident window and reports the actual pages hit per domain, the top client IPs, and counts for commonly abused WordPress endpoints like xmlrpc.php, wp-login.php and admin-ajax.php. It’s Cloudflare-aware, so it points you at the real visitor IP instead of blaming the CDN, and it labels the server’s own loopback requests so wp-cron never looks like an attacker.

That feature found a genuine attack. A single IP — 96.126.176.53 — was responsible for 22,542 of 25,802 requests, about 87% of everything in the window, flooding two WordPress admin endpoints behind a fake Chrome user agent. It came in through Cloudflare, which is exactly why a server-level firewall ban would have been useless: the connection my server sees is a Cloudflare edge, not the attacker. I blocked it at Cloudflare instead, and it stopped in minutes.

It’s just as useful in reverse. Another night I was sure I was under attack — and the report showed 11 requests in the whole window, from different IPs, on normal mobile browsers. Not an attack. Just my slow disk again. That saved me from blocking innocent visitors and adding firewall rules for nothing.

It records CPU steal. That’s time my server was ready to run but the host gave the physical core to somebody else. It’s the one number in a hosting dispute that cannot be blamed on my own workload.

The commands I use most:

Command	What it tells me
server-forensics --incidents	Every recorded incident, newest first, with the trigger reason
server-forensics --last-analysis	The automatic root-cause verdict for the latest incident
server-forensics --offenders	Which program actually caused the spike, grouped properly
server-forensics --io	Full disk evidence — latency, utilisation, per-process I/O
server-forensics --requests	Which URLs and client IPs drove the load
server-forensics --runup	What was building up before the trigger fired
server-forensics --health	Is the recorder itself running properly

Who this is for

I wrote this for my own cPanel server, but it will help you if:

• Your site freezes for a minute or two and then recovers, and you can never catch it live. This is the exact case it was built for.


• You see high load with low CPU and nothing you read online explains it.


• You’re on a VPS and suspect the host. Suspecting is cheap. Having timestamps, latency figures and kernel stack traces to send to support is what actually moves a ticket forward.


• You run WordPress sites and need to know if it’s an attack or not. The request report answers that in seconds, so you don’t block real customers out of panic.


• You manage servers for clients. Instead of “it was probably a traffic spike”, you can show what was stuck and what it was waiting for.


• You’re learning Linux performance. Honestly, reading these reports taught me more about load average, iowait, the ext4 journal and fsync than any article did — because it was my own server, with real numbers.

It is not a replacement for monitoring. Keep Netdata or Prometheus for live graphs and alerts. This sits next to them and answers the question they can’t.

Install it

It’s built for AlmaLinux 8 with cPanel, CloudLinux, Apache with mod_lsapi, MariaDB and Exim, but the collector is generic enough for most Linux web servers.

git clone https://github.com/Dr-Hack/Server-Forensics-Recorder.git
cd Server-Forensics-Recorder
sudo bash install.sh

Then check it:

sudo server-forensics --doctor
server-forensics --health

It installs a systemd timer that runs once a minute. Reports land in /var/log/server-forensics/. Set your thresholds using your own server’s normal numbers — start conservative and tighten later.

MIT licensed. Issues and pull requests are welcome.

FAQ

Why is my server load high when CPU usage is low?

Because load average counts processes that are waiting, not just processes that are running. If load is high while the CPU is idle, your processes are blocked on something — usually disk I/O, sometimes memory pressure or a lock. High iowait alongside high load points at storage.

Does this replace Netdata, Prometheus or Grafana?

No. Those tell you when something happened. This tells you why. Run both.

Will it slow down my server?

Normal operation is one cheap sample a minute, mostly /proc reads, plus a light background sampler. Expensive commands only run during an incident, each with a timeout and an output limit, so the recorder can’t fill the disk it’s already waiting on.

Does it work without cPanel?

The per-minute collector and the analysis work on most Linux web servers. The deeper snapshots are tuned for cPanel-style stacks, and the request attribution expects cPanel-style domain log paths, though you can point it elsewhere.

Can I use its output to raise a ticket with my host?

That’s exactly why the disk latency and CPU steal numbers are recorded. Quote the timestamps and the measurements. It’s much harder to wave away than “my site feels slow”.

Is it safe to run during an outage?

That was the first design rule. Timeouts everywhere, bounded output, and package managers are detected but never run, because calling one during a disk stall could block indefinitely.

---

Have you hit the high-load-low-CPU problem? Tell me what it turned out to be on your server — I’m collecting these cases, and the wait-channel maps in the tool get better every time I see a new one.

The post I Built a Black Box for My Linux Server — Because My Load Hit 20 While the CPU Sat at 3% first appeared on Internet Security Blog - Hackology.

Source: Hackology
Source Link: https://blog.drhack.net/high-server-load-low-cpu-server-forensics-recorder/