Skip to content

Build1 publisher3 min readPublished

Do Not Reboot The Box: A Five-Minute Triage Order That Survives A 3 AM Page

A sysadmin's incident writeup argues the reboot reflex destroys the evidence you need. Its fixed check order is the useful part, though only two of the five steps survive in the published text.

The Engineer · Build desk

Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction

What happened

  • The dev.to post '5 Things I Check First When a Linux Server Goes Down' is authored by Asep Sayyad (handle asepsayyad007) and opens with an urgent Prometheus alert received at 3:15 AM on a Saturday.
  • In the incident described, the primary application server had stopped replying to health check probes, HTTP requests were timing out, API clients were dropping connections, and Slack was lighting up with red automated incident notifications.
  • The author writes that the first instinct on being woken by such an alert is panic, wanting to rush in and hit the reboot button and restart every service in sight.
  • The author states that rebooting and restarting services on reflex is the biggest mistake you can make.
  • Rebooting an unresponsive server without inspecting its state destroys debugging evidence: ephemeral kernel buffers, process core dumps, open file handles and volatile memory allocations disappear the moment power is cut.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

A writeup on dev.to by Asep Sayyad opens with a 3:15 AM Saturday Prometheus alert: the primary application server had stopped replying to health check probes, HTTP requests were timing out, API clients were dropping connections, and Slack was filling with automated incident notifications [1][2]. The interesting claim is not in the commands that follow but in the first instruction, which is to do nothing to the machine's state: the author argues that the instinct to hit reboot and restart every service in sight is the biggest mistake available at that hour [3][4].

The reasoning is specific about what you lose. Ephemeral kernel buffers, process core dumps, open file handles and volatile memory allocations all disappear the moment power is cut [5]. The consequence is a familiar shape of incident: the reboot fixes the symptom for ten minutes, and the same crash returns during peak business hours [6]. That is the trade being made at 3 AM without anyone noticing it, a minute of diagnosis exchanged for a repeat performance in front of customers.

Step one is not a command, it is a question: did the operating system crash, or is the box merely cut off. A server can be running perfectly while looking completely dead from outside if a default gateway dropped, a firewall rule blocked inbound traffic, or an interface lost its IP [7]. Four ICMP packets from a workstation or jump host answer the first half [8]. Then `ssh -vvv`, where the diagnosis is the point at which the connection stalls [9]. Connection refused means the path works and `sshd` is stopped or crashed [10]. Connection timed out means packets are being dropped by a network firewall, a cloud security group, or a local iptables rule [11]. A banner that appears and then freezes means the daemon is accepting connections but the system is too resource-starved to spawn a login shell [12]. That third signature is the one worth memorising, because it hands you the next step before you have run a single command on the host.

If SSH is gone entirely, the article routes you out of band: an EC2 serial console, a DigitalOcean web console, or hardware IPMI, iLO or KVM [13]. From the virtual tty the sequence is mechanical. Check interface and link state with `ip addr show` and `ip link show eth0`, and bring the link back with `ip link set eth0 up` plus a `systemd-networkd` restart [14]. Check for a default route with `ip route show` and re-add it temporarily if missing [15]. Then check local filter rules with `iptables -L -n -v --line-numbers` and `ufw status verbose`, disabling the firewall temporarily if a recent script or bad deployment installed an accidental drop-all [16]. Worth noting what that last check implies: your packet filter is a deployment artifact, and it fails like one.

Step two starts where step one's frozen banner left off. If you can log in but commands take thirty seconds, suspect resource starvation; at 100 percent CPU, memory or process table capacity Linux slows to a crawl, and in severe cases the kernel triggers emergency mechanisms that terminate critical processes [17][18]. The first command is `uptime` or `w` [19].

Here the honesty ends, because the published text available breaks off partway through step two, so only two of the advertised five steps can actually be read [20][1]. The author's headline promise, that the full checklist runs in under five minutes on almost any modern distribution, is not checkable from what is there [21]. What is checkable is the ordering logic, and it holds: reachability before resources, because the cheapest test also tells you which layer to stop looking at.

Worth checking against your own runbook: whether it names an order at all, and whether serial or IPMI console access is configured now rather than requested during the incident [13].

Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories