Recovering from a Kernel Panic - Not Syncing: Fatal Exception

When a customer system became unresponsive and failed to boot, we quickly uncovered the cause: a fatal exception that sent our core operating system into an unrecoverable state. As operating system providers, that's our signal to stop everything and act because when the core system breaks, every workload built on top fails. In this post, I'll walk through how we diagnosed the issue, traced it back to a drifted kernel version, and what we've done since to lock critical packages and prevent this failure from reoccurring.

An introduction to kernel panics

A kernel or system panic is a safety measure the operating system takes when it runs into an unrecoverable error or a fatal exception. When our operating system enters a state of panic, our options as operators are severely limited. There is no running system to execute commands on, to run diagnostic tools, and no operable console to view logs. To make matters worse, remote access is impossible without a working internet connection in the cluster.

We rely on a serial console connection to troubleshoot a panicked remote cluster. This out-of-band management channel is provided through BMC/IPMI on enterprise-grade servers, giving access to system controls and a virtual serial console. On hardware without BMC or IPMP support, a network accessible IP-VGA or DIY console server (with 1-to-many RS232 interfaces) can serve the same purpose. As a last resort, you are hopefully stuck on site before the KVM monitor without clients watching over your shoulder.

Drivers, especially filesystem drivers, cause most kernel panics. This is critical because the Linux root (/) directory, its subdirectories, and files reside on a physical disk's formatted partition. When the power button is pressed, the BIOS or UEFI initiates the boot process, loading and executing the operating system according to the configured boot order. The kernel must load the appropriate drivers to recognize the filesystem on the disk partition. At this early stage, the system runs from the initramfs, a temporary root filesystem loaded into memory.

Things are not as bad when the initramfs executes successfully. We can debug the bootstrap process from there by appending or modifying the Linux kernel parameters. This works much like the Linux runlevels, allowing us to step through and investigate where the failure occurred.

Key checkpoints to verify include:

1. Are any drivers missing?
2. Are all required drivers loaded?
3. Are the paths correct?
4. Are the hard drives detected?
5. Are the filesystems recognized?
6. Are the folders mounted with valid contents?
7. Are environmental variables present with expected values?
8. Was switchroot successful?

Today, I will share a case where all the above checkpoints fail.

The inexplicable kernel panic case study

Today’s issue was a kernel panic that occurred after initramfs
was unpacked. The log listed the drivers I had built into it, but gave no indication of why the panic happened. At least none that was immediately obvious.

Image source: Bigstack CubeCOS
Image Source: Bigstack CubeCOS

To troubleshoot, I began reviewing the Linux boot parameters, checking initramfs_size, ACPI status, and a few more parameters with no luck. I then ruled out hardware issues such as faulty memory, drives, or the motherboard since the existing operating system worked. Thanks to our internal CI/CD pipeline, I could also confirm that the latest firmware was validated and working. I compared the changesets between the last known good build and today's build, but found nothing noticeable. Digging deeper, I unpacked the firmware and noticed a difference in the kernel versions. The kernel version has shifted from kernel-4.18.0-348 to kernel-4.18.0-373. At first, the version difference seemed trivial, just a few sub-releases apart. But it highlighted a risk in our build process: relying on RPM to resolve dependencies directly from upstream repositories.

Our build process relies on the RPM package manager, which resolves dependencies and installs packages from the provided repositories. This approach offers us the advantages of staying current with the latest and greatest packages from upstream sources and avoiding the overhead of maintaining our own RPM repositories. However, it also introduces challenges when applying patches, hot fixes, or fix packs on deployed systems. Each build can differ, and in this case, the small difference triggered the kernel panic during firmware installation.

Upon identifying the root cause of the kernel panic, we decided to lock kernel packages which are too critical to be unmanaged. Pinning and installing the 4.18.0-348 kernel on the panicked host.

sh-4.4# dnf install -y 'dnf-command(versionlock)'
sh-4.4# dnf -y install kernel-core-4.18.0-348.el8.x86_64 kernel-modules-4.18.0-348.el8.x86_64 kernel-headers-4.18.0-348.el8.x86_64 kernel-4.18.0-348.el8.x86_64 kernel-modules-extra-4.18.0-348.el8.x86_64

Once the correct kernel version is verified and installed through dnf list installed, I could proceed to validate the lock and deploy the new image.

    sh-4.4# dnf install -y 'dnf-command(versionlock)'
    sh-4.4# dnf -y install kernel-core-4.18.0-348.el8.x86_64 kernel-modules-4.18.0-348.el8.x86_64 kernel-headers-4.18.0-348.el8.x86_64 kernel-4.18.0-348.el8.x86_64 kernel-modules-extra-4.18.0-348.el8.x86_64

    sh-4.4# dnf list installed | grep kernel | tr -s ' '
    kernel.x86_64 4.18.0-348.el8 @baseos
    kernel-core.x86_64 4.18.0-348.el8 @baseos
    kernel-headers.x86_64 4.18.0-348.el8 @baseos
    kernel-modules.x86_64 4.18.0-348.el8 @baseos
    kernel-modules-extra.x86_64 4.18.0-348.el8 @baseos

I added the kernel packages to the lock list with:

dnf versionlock add <package-name>

After verifying that the versions are properly locked by viewing the lock list, we can move on to installing two major dependencies.

sh-4.4# dnf versionlock list
Last metadata expiration check: 0:28:02 ago on Thu 21 Apr 2022 06:06:00 PM CEST.
kernel-0:4.18.0-348.el8.*
kernel-core-0:4.18.0-348.el8.*
kernel-modules-0:4.18.0-348.el8.*
kernel-modules-extra-0:4.18.0-348.el8.*
kernel-headers-0:4.18.0-348.el8.*

The two major dependencies provided the daemon, rados-ng/rados-kv, and config files.

sh-4.4# dnf -y install nfs-ganesha-ceph.x86_64 nfs-ganesha-rados-grace

Once  the original kernel version and the critical dependencies were installed, I proceeded to write the new firmware to a USB disk.

ship$ sudo dd if=CUBE_2.2.4_20220421-1616_55765df7.img of=/dev/sda bs=4M
    1459+1 records in
    1459+1 records out
    6122962944 bytes (6.1 GB, 5.7 GiB) copied, 505.694 s, 12.1 MB/s

Now insert the USB that contains our newly-built firmware and power-cycle the server to boot off the USB drive. The system now passes the panic point, successfully completes the boot process, completes the recovery processes and restores services to normal levels.

Image Source: Bigstack CubeCOS