This box already builds and qemu-boots the kernel packages. Running one on the metal is a different proposition from the MicroPC lane, because this machine has no console at all: no serial port and no display. Everything below follows from that.
Read micropc-testing.md first if you have not. Where the two lanes agree, this document says so and does not repeat the reasoning.
| Role | x86_64 build box; also the x86_64 hardware-boot target |
| OS | Fedora Linux 44 (Workstation Edition), multi-user.target |
| Running kernel | the distribution’s own, currently 7.1.9-200.fc44.x86_64 |
| Stock kernels installed | three (7.1.7, 7.1.8, 7.1.9) |
| Firmware | UEFI, Secure Boot disabled, so an unsigned kernel boots |
| Bootloader | GRUB 2 with GRUB_DEFAULT=saved, GRUB_TIMEOUT=5 |
| Root | ext4 on /dev/sda2, 817 GB free; ESP is a separate 2 GB /dev/sda1 |
| Network | one wired interface, static lease on the lab network |
| Watchdogs | intel_oc_wdt and iTCO_wdt, both loaded |
| Serial | a 16550A exists (ttyS0, type=4, 0x3f8, IRQ 4) but reaches no accessible connector: the board ships no DB-9, and any PCB header is not reachable. Unusable as a console |
| Display | none attached. All three DRM connectors report disconnected |
The root filesystem is ext4, so there is no filesystem snapshot to roll back to. Rollback here means boot a different kernel, and nothing else.
There is a UART on this board – sysfs reports ttyS0 as type=4
(PORT_16550A) at 0x3f8 IRQ 4, a port the driver probed rather than a
phantom node – but it reaches no connector anyone can attach a cable to. So
there is no serial console, and with no monitor attached there is no console
at all.
A kernel that fails before the network comes up therefore produces no output anywhere: nothing to watch, no scrollback to read, no shell to log into. The machine is either reachable over ssh or it is a black box. Everything below follows from that: the visibility has to be arranged in advance, and the recovery has to be automatic, because there will be nobody at a console to intervene.
(An earlier revision of this document asserted the machine had no serial port
at all. That came from reading /proc/tty/driver/serial, which is root-only
and had returned empty because the command failed, not because the file was
empty. The conclusion held; the evidence for it did not.)
Four independent layers, in the order they take effect. Any one of them alone recovers the machine; they are stacked because the cheap ones fail silently.
GRUB_DEFAULT=saved is already set, so grub2-reboot selects an entry for
the next boot only. The saved default is untouched.
# select the badc entry for exactly one boot
sudo grub2-reboot "$(sudo grubby --info=ALL | awk -F= '/^title=/ {print $2}' | grep badc | head -1)"
sudo systemctl reboot
If that kernel does not reach userspace, the following boot is the stock
default again – no intervention, no console needed. Do not run
grub2-set-default or grubby --set-default against a badc entry.
Put these on the badc entry’s command line only, not in /etc/default/grub:
panic=30 oops=panic nmi_watchdog=panic softlockup_panic=1
A panic or oops then reboots after 30 s, and combined with layer 1 that reboot lands on a stock kernel. Thirty seconds is long enough for netconsole to flush and for pstore to write, and short enough not to matter.
The last two are what make a detected lockup recoverable. Both kernels run
with hardlockup_panic=0 and softlockup_panic=0 – Fedora’s default, and the
badc config leaves BOOTPARAM_HARDLOCKUP_PANIC unset – so the NMI watchdog
detecting a lockup wrote a warning to a console this box does not have and did
nothing else. With them it panics instead, which printk.always_kmsg_dump=1
and efi_pstore record and panic=30 reboots out of.
Two spellings matter here, and both fail silently. panic_on_oops and
hardlockup_panic are sysctl names; the kernel does not take either on the
command line and passes them to init as environment instead, printing one
Unknown kernel command line parameters line. oops=panic and
nmi_watchdog=panic are the boot-parameter forms of the same two settings.
softlockup_panic=1 is a parameter in its own right.
hwprep.py entry puts all four on the badc entry; they are not options.
A hang that no detector catches – no panic, no oops, and nothing the NMI
watchdog sees – is the case layers 1 and 2 do not cover, because nothing ever
decides to reboot. iTCO_wdt can, but only once something opens
/dev/watchdog, so it protects the window from systemd onwards and not
before:
# /etc/systemd/system.conf.d/watchdog.conf
[Manager]
RuntimeWatchdogSec=60
RebootWatchdogSec=120
A hang before systemd starts is not covered by anything here and needs the power button. That gap is real; see “What is still uncovered”.
Do not run dnf remove kernel, and keep installonly_limit at three or more.
The badc package installs under its own version string (7.1.10), so it is an
addition and not a replacement. Verify before rebooting:
rpm -q kernel | wc -l # expect >= 3
sudo grubby --info=ALL | grep -c '^title='
netconsole.ko ships with the distribution kernel and the badc one, and both
build it as a module (CONFIG_NETCONSOLE=m). That decides where the target
goes, and getting it wrong is silent: netconsole= on the kernel command line
is a parameter only a builtin registers, so on these kernels it is rejected –
the boot prints one Unknown kernel command line parameters line and carries
on with no remote log at all. hwprep.py entry reads CONFIG_NETCONSOLE
from the config of the kernel it is preparing – not from the running one,
which can answer differently – and for a module writes
/etc/modprobe.d/badc-netconsole.conf options netconsole netconsole=<spec>
/etc/udev/rules.d/99-badc-netconsole.rules ACTION=="add|move", SUBSYSTEM=="net",
ENV{INTERFACE}=="<iface>",
RUN+="/usr/sbin/modprobe netconsole"
or puts it on the command line for a builtin. The spec is
netconsole=6666@<box-ip>/<iface>,6666@<collector-ip>/<collector-mac>
The load has to wait for the interface. A modules-load.d entry does
not: systemd-modules-load runs before the network driver has probed, netpoll
finds nothing to bind to, and the target is dropped for the rest of the boot.
That is what this box did, loading at 5.54 s against an interface that
appeared at 6.32 s:
[ 5.544195] netpoll: netconsole: enp5s0 doesn't exist, aborting
[ 5.544232] netconsole: Not enabling netconsole for cmdline0. Netpoll setup failed
[ 5.544253] netconsole: network logging started
[ 6.319026] alx 0000:05:00.0 enp5s0: renamed from eth0
The last two lines are the ones to read: netconsole announces that logging started whether or not any target set up, so the boot reports itself armed and sends nothing. The collector saw the pre-boot probe and nothing after it.
The interface’s own udev event is the earliest trigger available. RUN
executes in the udev worker that handled the event, which is before systemd is
told the device exists, so it is ahead of anything ordered after
sys-subsystem-net-devices-<iface>.device and well ahead of
network-online.target, which this box reaches at 18.7 s. The rule matches
add|move because udev applies rules before it renames an interface: the add
event still carries eth0 and the rename that follows emits a move event
carrying enp5s0. modprobe on a loaded module changes nothing, so matching
both costs nothing.
The rule is not specific to the badc entry – it loads netconsole on every
boot, stock ones included. That is what makes it checkable in advance: after
any boot, hwprep.py check reports the target carrying or refuses.
As a module it still cannot cover the window before the interface exists –
driver probe, mount, switch_root. pstore is the only record for that window,
which is why both are armed.
Both IP addresses and the collector’s MAC are site-specific and deliberately not recorded here. Collect on the other machine with:
nc -u -l -k 6666 | tee "netconsole-$(date +%Y%m%dT%H%M%S).log"
This is the primary window. It covers the interface’s appearance on: the rest of the boot, systemd, and userspace.
How much of it arrives is a live setting. netconsole is a console, so it
carries what console_loglevel admits at the time, and that is not fixed for
the boot. The stock entries here carry quiet, which sets it to
CONFIG_CONSOLE_LOGLEVEL_QUIET – 3 in the Fedora configuration, and
/proc/sys/kernel/printk reads 3 4 1 7 on 7.1.12 – so KERN_WARNING and
below do not leave the box. hwprep.py entry removes rhgb quiet from the
badc entry, which starts it at CONFIG_CONSOLE_LOGLEVEL_DEFAULT, 7; userspace
can lower it after that. Read /proc/sys/kernel/printk on the machine rather
than assuming the level a boot ran at.
Prove the path carries before a boot depends on it, because a netconsole that does not arrive is indistinguishable from a kernel that produced no output. Start the collector, then send from the box over the same route the kernel will use:
# on the collector
nc -u -l 6666
# on the box, from bash -- the login shell may be zsh, which has no /dev/udp
bash -c 'echo probe > /dev/udp/<collector-ip>/6666'
The probe arriving confirms the addressing, the route and that nothing between
the two machines drops the port. It does not confirm the MAC in the
netconsole= line, which the kernel uses directly rather than resolving by
ARP: get that wrong and the frames are emitted and silently not delivered. Read
it from the collector’s own interface, and re-read it if the collector’s
hardware or network changes.
The module is present but disabled (pstore_disable=Y), so today a panic
leaves nothing behind. Enable it:
# /etc/modprobe.d/pstore.conf
options efi_pstore pstore_disable=0
and add printk.always_kmsg_dump=1 to the badc entry so the dump also runs on
a clean-ish shutdown path. After the machine comes back on a stock kernel:
ls /sys/fs/pstore/ # dmesg-efi-* records, newest first
sudo cat /sys/fs/pstore/dmesg-efi-*
sudo rm /sys/fs/pstore/dmesg-efi-* # clear before the next attempt
efi_pstore is builtin on the Fedora kernels this box runs
(modinfo efi_pstore reports filename: (builtin)), and it ships with
pstore_disable=Y. A builtin takes its parameters from the kernel command line,
not from modprobe.d: a modprobe.d drop-in for it is read by nothing and
changes nothing. The parameter therefore goes on the badc entry’s command line,
as efi_pstore.pstore_disable=0, where it applies to the kernel whose death is
being recorded and to no other.
The ESP has 2 GB free and EFI variable space is small; clearing records between runs keeps the variable store from filling.
rhgb quietThe stock command line carries rhgb quiet, which suppresses exactly the
messages worth having. Drop both from the badc entry. Leave
/etc/default/grub alone so the stock entries keep their normal behaviour.
Each step is reversible and the undo is recorded at the end of this document.
Every step below needs root; the operator account has a passwordless sudo
rule, so they can be driven over ssh. hwprep.py applies them, records every
change it makes, and replays the record backwards on rollback. Run it on
the box:
scp demos/linux/hwprep.py <box>: # from the repo
# 1. Record the state to return to. Refuses nothing, changes nothing.
sudo python3 hwprep.py record
# 2-3. Post-mortem capture and the watchdog.
sudo python3 hwprep.py arm
# 4. Install the badc package. It adds a version, it replaces none.
sudo python3 hwprep.py install kernel-7.1.10-*.x86_64.rpm
# 5. Give that entry its own arguments, and no other entry any: panic=30
# and oops=panic, the lockup pair, pstore, and the netconsole target.
sudo python3 hwprep.py entry --kernel 7.1.10 \
--netconsole '6666@<box-ip>/<iface>,6666@<collector-ip>/<collector-mac>'
# 6. Confirm the machine can still recover. This is the step not to skip.
sudo python3 hwprep.py check
check is the gate. It reports READY only when the default boot entry is a
stock kernel, at least one stock kernel remains installed to fall back to, and
the recovery configuration is in effect – reading the watchdog’s live timeout
from systemd and pstore’s state from the running kernel, rather than the
presence of the files that were meant to set them. It also reports
whether the badc entry carries the lockup pair, so an entry written by an
earlier run is visible rather than assumed. The arm step relies on
that distinction: on this machine it removes its own modprobe.d drop-in once
it sees efi_pstore is builtin, because that file could not have worked.
Every step is idempotent, so a re-run after an interruption is safe, and each
prints what it changed. --dry-run prints without changing anything.
Then, and only after check reports READY:
sudo python3 hwprep.py boot --kernel 7.1.10 # one boot, then back to stock
sudo systemctl reboot
boot selects the entry through grub2-reboot, which GRUB consumes on the
next start. It does not change GRUB_DEFAULT, so a kernel that panics, hangs
or does not reach userspace is followed by a stock boot without anyone touching
the machine.
sudo rm -f /sys/fs/pstore/*.grub2-reboot, then reboot.Linux version 7.1.10 ... (badc 0.4.2 (gcc-compatible, GNU C 4.3.0), GNU ld (badc 0.4.2) ...)uname -r,
/proc/sys/kernel/tainted, systemctl is-system-running, lsmod | wc -l,
the root disk driver chain, and demos/linux/exercise.py – so the hardware
result is comparable to the qemu result rather than a separate vocabulary./sys/fs/pstore/.| failure | netconsole | pstore | recovers by itself |
|---|---|---|---|
| panic or oops after the NIC probes | yes | yes | yes, panic=30 then stock |
| panic before the NIC probes | no | yes | yes, panic=30 then stock |
| lockup the NMI watchdog detects | up to the lockup | yes | yes, panic=30 then stock |
| hang no detector catches, after systemd starts | up to the hang | no | yes, watchdog |
| hang no detector catches, before systemd starts | up to the hang | no | no – power button |
| hang before the NIC probes | nothing at all | no | no – power button |
The third row is new. Both stops of a badc kernel on this box ended without
it: the machine stopped logging and the hardware watchdog reset it a minute or
two later. After the second, /sys/fs/pstore was empty on the stock boot that
followed, although efi_pstore.pstore_disable=0 and
printk.always_kmsg_dump=1 were on the entry and the EFI variable store was
writable – no panic path had run, which is what hardlockup_panic=0 means. A
lockup the NMI watchdog detects now panics and leaves that record. A hang it
cannot detect still leaves only what the collector saw, and still needs the
hardware watchdog to end it.
The last row is the honest limit of this lane. There is no way to observe or recover from it remotely on this hardware.
earlycon=uart8250 covers; there is no equivalent here. Mitigations: only
boot kernels that already boot in qemu on this same box, and keep the
one-shot selection so a dark failure still self-recovers on the next power
cycle.earlycon=efifb gives output from very early on.
Worth attaching for a first bring-up even if it is removed afterwards.Eight items, and nothing else. hwprep.py status prints the recorded ones at
any time; the table gives the manual undo for each, should the record be lost.
| # | Change | Where it lives | Outlives a reboot | Undo |
|---|---|---|---|---|
| 1 | The rollback snapshot | /var/lib/badc-hwprep/ |
yes | sudo rm -rf /var/lib/badc-hwprep – it only records |
| 2 | Watchdog drop-in | /etc/systemd/system.conf.d/badc-watchdog.conf |
yes | sudo rm it, then sudo systemctl daemon-reexec |
| 3 | Kernel package | rpm database, /boot, /lib/modules |
yes | sudo rpm -e kernel-<version> – takes its BLS entry with it |
| 3a | The default entry, moved by the package | grubenv | yes | sudo grubby --set-default=/boot/vmlinuz-<stock> – install does this itself |
| 4 | Arguments on the badc entry | that entry’s BLS file only | yes | sudo grubby --update-kernel=/boot/vmlinuz-<version> --remove-args="..." |
| 4a | netconsole target and its load trigger | /etc/modprobe.d/badc-netconsole.conf, /etc/udev/rules.d/99-badc-netconsole.rules |
yes | sudo rm both |
| 5 | One-shot boot selection | next_entry in the grubenv |
no, one boot | sudo grub2-editenv - unset next_entry |
| 6 | pstore records left by a crash | EFI variable store, via /sys/fs/pstore |
yes | sudo rm -f /sys/fs/pstore/* |
| 7 | Initramfs rebuild | /boot/initramfs-<running>.img |
yes | sudo dracut -f regenerates it |
Items 2 and 4a are the ones that change how the machine behaves outside the
badc entry. The udev rule loads netconsole on every boot, so the kernel log of
a stock boot goes to the collector as well – which is what lets the route be
proved before a badc kernel depends on it. After arm, systemd pets a
hardware watchdog with a one-minute timeout on every boot, stock kernels
included. A stock system that wedges hard enough to stop systemd from petting
it will therefore reset itself rather than sit there. That is the intended behaviour – it is what makes an unattended
badc boot recoverable – but it applies machine-wide, and it is live from the
moment arm runs, not from the first badc boot.
Item 3a is not something the preparation asks for. Fedora’s kernel package
makes the kernel it just installed the default, so rpm -i alone leaves the
machine one reboot away from starting a kernel that may not come back –
without anyone having chosen that. install captures the default beforehand,
notices when the package has moved it to a kernel this tool installed, and puts
it back, reporting both. The invariant check that follows would catch it
regardless, but detecting a hazard whose window is a reboot wide is not as good
as not opening it.
Items 3 through 6 otherwise touch the badc entry alone. Nothing in this lane modifies
/etc/default/grub, the stock kernels, their command lines, the default boot
entry, or the root filesystem.
sudo python3 hwprep.py rollback # --keep-kernels leaves the packages
sudo python3 hwprep.py status # what remains, and what is default
rollback replays the recorded changes newest-first: it strips the arguments
it added from the entries it added them to, removes the kernel packages it
installed, and restores or deletes each file it wrote according to whether
that file existed beforehand. It then re-checks the invariant and compares the
installed kernel set against the one record captured, reporting any
difference in either direction rather than reporting success on the strength
of having run.
It does not clear items 5 and 6 – a pending one-shot selection is consumed by the next boot whether or not anyone clears it, and the pstore records are the evidence a failed boot was run to collect. Clear those by hand when done:
sudo grub2-editenv - unset next_entry
sudo rm -f /sys/fs/pstore/*
To confirm the machine is where it started:
sudo python3 hwprep.py status # 0 recorded changes
diff <(sudo grubby --info=ALL) /var/lib/badc-hwprep/before/grubby-info-all.txt
rpm -q kernel # the stock set, unchanged