The hunt for a kernel bug, part 3: compiling a kernel

Compiling a Linux kernel sounds scary and complicated, but I found out it actually isn’t.

The first thing to do, is to install some prerequisites:

$ sudo apt install --yes asciidoc binutils-dev bison build-essential ccache \
    crash dwarves fakeroot flex git git-core git-doc git-email kernel-package \
    kernel-wedge kexec-tools libelf-dev libncurses5 libncurses5-dev libssl-dev \
    makedumpfile zstd
$ sudo apt-get --yes build-dep linux

Next I cloned the Ubuntu Impish repository. This takes a while…

$ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-impish.git
$ cd ubuntu-impish

Now let’s see which versions are in the repository:

$ git tag --list
Ubuntu-5.11.0-16.17
Ubuntu-5.11.0-18.19+21.10.1
Ubuntu-5.11.0-20.21+21.10.1
Ubuntu-5.13.0-11.11
Ubuntu-5.13.0-12.12
Ubuntu-5.13.0-13.13
Ubuntu-5.13.0-14.14
Ubuntu-5.13.0-15.15
Ubuntu-5.13.0-16.16
Ubuntu-5.13.0-17.17
Ubuntu-5.13.0-18.18
Ubuntu-5.13.0-19.19
Ubuntu-5.13.0-20.20
Ubuntu-5.13.0-21.21
Ubuntu-5.13.0-22.22
Ubuntu-5.13.0-23.23
Ubuntu-5.13.0-24.24
Ubuntu-5.13.0-25.26
Ubuntu-5.13.0-26.27
Ubuntu-5.13.0-27.29
Ubuntu-5.13.0-28.31
Ubuntu-5.13.0-29.32
Ubuntu-5.13.0-30.33
Ubuntu-5.13.0-31.34
Ubuntu-5.13.0-32.35
freeze-20211018
freeze-20211108
freeze-20220131
freeze-20220221
v5.11
v5.13

The two tags that interest me, are Ubuntu-5.13.0-22.22 and Ubuntu-5.13.0-23.23. I’m starting with the former.

git checkout Ubuntu-5.13.0-22.22

First I copy the configuration of the current running kernel to the working directory:

$ cp /boot/config-$(uname --kernel-release) .config

I don’t want or need full debugging. That makes an enormous kernel and it takes twice as long to compile, so I turn debugging off:

$ scripts/config --disable DEBUG_INFO

I need to disable certificate stuff:

$ scripts/config --disable SYSTEM_TRUSTED_KEYS
$ scripts/config --disable SYSTEM_REVOCATION_KEYS

Next: update the kernel config and set all new symbols to their default value.

$ make olddefconfig

Then the most exciting thing can start: actually compiling the kernel!

$ make clean
$ time make --jobs=$(getconf _NPROCESSORS_ONLN) bindeb-pkg \
    LOCALVERSION=-$(git describe --long | tr '[:upper:]' '[:lower:]')

• time is to see how long the compilation took.
• getconf _NPROCESSORS_ONLN queries the number of processors on the computer. make will then try to run that many jobs in parallel.
• bindeb-pkg will create .deb packages in the directory above.
• LOCALVERSION appends a string to the kernel name.
• git describe --long shows how far after a tag a certain commit is. In this case: Ubuntu-5.13.0-22.22-0-g3ab15e228151
  • Ubuntu-5.13.0-22.22 is the tag.
  • 0 is how many commits after the tag. In this case it’s the tag itself.
  • 3ab15e228151 is the abbreviated hash of the current commit.
• tr '[:upper:]' '[:lower:]' is needed because .deb packages can’t contain upper case letters (I found out the hard way).

Now go grab a coffee, tea or chai latte. Compilation took 22 minutes on my computer.

When the compilation is done, there are 3 .deb packages in the directory above:

$ ls -1 ../*.deb
../linux-headers-5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151_5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151-21_amd64.deb
../linux-image-5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151_5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151-21_amd64.deb
../linux-libc-dev_5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151-21_amd64.deb

Install the linux-headers and the linux-image packages, you don’t need the libc-dev package.

$ sudo dpkg --install \
    ../linux-{headers,image}-*$(git describe --long | tr '[:upper:]' '[:lower:]')*.deb

The kernel is now installed in the /boot directory, and it’s available in the Grub menu after reboot.

$ ls -1 /boot/*$(git describe --long | tr '[:upper:]' '[:lower:]')*
/boot/config-5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151
/boot/initrd.img-5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151
/boot/System.map-5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151
/boot/vmlinuz-5.13.19-ubuntu-5.13.0-22.22-0-g3ab15e228151

Kernel ubuntu-5.13.0-22.22-0-g3ab15e228151 is, for all intents and purposes, the same as kernel 5.13.0-22-generic, so I expected it to be a “good” kernel, and it was.

For kernel Ubuntu-5.13.0-23.23 I did the same thing: starting from the git checkout. I skipped copying and editing the config file, because between minor releases I don’t expect there to be much change. I did run make olddefconfig for good measure, though. As expected, the keyboard and mouse didn’t work with the compiled ...-23 kernel.

Next up: using git bisect to find the exact commit where it went wrong. It’s got to be somewhere between ...-22 and ...-23!