The hunt for a kernel bug, part 2: an easy way to install mainline kernels

As I wrote previously, I’m suspecting a Linux kernel bug somewhere between versions 5.13.0-22 and 5.13.0-23, in the Ubuntu kernels. I wanted to know if the issue only surfaced in Ubuntu-flavored kernels, or also in the upstream (mainline) kernels from

There is an Ubuntu Mainline PPA with all the upstream kernels, but I found it a bit too opaque to use. Fortunately I found the Ubuntu Mainline Kernel Installer (UMKI), a tool for installing the latest Linux kernels on Ubuntu-based distributions.

Ubuntu Mainline Kernel Installer (UMKI)

The UMKI is pretty straightforward. It fetches a list of kernels from the Ubuntu Mainline PPA and a GUI displays available and installed kernels, regardless of how they were installed. It installs the kernel, headers and modules. There is also a CLI client.

To install the UMKI:

sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline

With that out of the way, there’s the matter of deciding which kernels to try. The “interesting” Ubuntu kernels are 5.13.0-22 and 5.13.0-23, so the mainline kernels I definitely want to test, are around those versions. That means 5.13.0 and 5.13.1. I also want to try the latest 5.13.x kernel, so that’s 5.13.19, and the most recent stable kernel, 5.16.11 (as of 2022-03-01).

To summarize, I have tested these mainline kernels:

  • 5.13.0
  • 5.13.1
  • 5.13.19
  • 5.16.11

The result (after several reboots)? With all of them, my keyboard and mouse worked without a hitch. That means the issue most likely doesn’t occur in (stable) mainline kernels, only in kernels with additional patches from Ubuntu.

Up next: compiling kernels from source.

Lasciate ogne speranza, voi ch’intrate.

Dante Alighieri

The hunt for a kernel bug, part 1

The operating system on my computer is Ubuntu Linux, version 21.10 (Impish Indri). Recently I had an issue that, after a kernel update (and reboot), my USB keyboard and mouse didn’t work any more in the login screen. Huh, that’s unexpected.
The issue was:

  • At the Grub boot menu, the keyboard works: I can use the keys, the numlock led lights up, the LCD of the Logitech G19 displays a logo.
  • At the Ubuntu login screen, the keyboard (and the mouse) went dark: no backlight of the keys, no numlock led, no logo on the display. And the mouse cursor didn’t move on screen.

Must be a problem at my end, I initially thought, because surely, something so essential as input devices wouldn’t break by a simple kernel update? So I did some basic troubleshooting:

  • Have you tried to turn it off and on again?
Have you tried to turn it off and on again?
Have you tried to turn it off and on again?
  • Plug the keyboard in another USB port.
  • Try a different keyboard.
  • Start with the older kernel, which was still in the Grub menu. And indeed, this gave me back control over my input devices!

So if the only thing I changed was the kernel, then maybe it’s a kernel bug after all?

I know that Ubuntu 21.10 uses kernel 5.something, and I know that I use the generic kernels. So which kernels are we talking about, actually?

$ apt-cache show linux-image-5*-generic | grep Package: | sed 's/Package: //g'

9 kernels, that’s not too bad. All of them 5.13.0-XX-generic. So I just installed all the kernels:

$ sudo apt install --yes \
One Eternity Later

My /boot directory is quite busy now:

$  ls -hl /boot
total 1,2G
drwxr-xr-x  4 root root  12K mrt  1 18:11 .
drwxr-xr-x 20 root root 4,0K mrt  1 18:11 ..
-rw-r--r--  1 root root 252K okt  7 11:09 config-5.13.0-19-generic
-rw-r--r--  1 root root 252K okt 15 15:53 config-5.13.0-20-generic
-rw-r--r--  1 root root 252K okt 19 10:41 config-5.13.0-21-generic
-rw-r--r--  1 root root 252K nov  5 10:21 config-5.13.0-22-generic
-rw-r--r--  1 root root 252K nov 26 12:14 config-5.13.0-23-generic
-rw-r--r--  1 root root 252K jan  7 16:16 config-5.13.0-25-generic
-rw-r--r--  1 root root 252K jan 12 15:43 config-5.13.0-27-generic
-rw-r--r--  1 root root 252K jan 13 18:13 config-5.13.0-28-generic
-rw-r--r--  1 root root 252K feb  4 17:40 config-5.13.0-30-generic
drwx------  4 root root 4,0K jan  1  1970 efi
drwxr-xr-x  5 root root 4,0K mrt  1 18:11 grub
lrwxrwxrwx  1 root root   28 feb 28 04:26 initrd.img -> initrd.img-5.13.0-22-generic
-rw-r--r--  1 root root  40M mrt  1 16:02 initrd.img-5.13.0-19-generic
-rw-r--r--  1 root root  40M mrt  1 17:39 initrd.img-5.13.0-20-generic
-rw-r--r--  1 root root  40M mrt  1 17:38 initrd.img-5.13.0-21-generic
-rw-r--r--  1 root root  40M feb 26 13:55 initrd.img-5.13.0-22-generic
-rw-r--r--  1 root root  40M mrt  1 17:40 initrd.img-5.13.0-23-generic
-rw-r--r--  1 root root  40M mrt  1 17:40 initrd.img-5.13.0-25-generic
-rw-r--r--  1 root root  40M mrt  1 17:41 initrd.img-5.13.0-27-generic
-rw-r--r--  1 root root  40M mrt  1 17:41 initrd.img-5.13.0-28-generic
-rw-r--r--  1 root root  40M mrt  1 17:38 initrd.img-5.13.0-30-generic
-rw-------  1 root root 5,7M okt  7 11:09
-rw-------  1 root root 5,7M okt 15 15:53
-rw-------  1 root root 5,7M okt 19 10:41
-rw-------  1 root root 5,7M nov  5 10:21
-rw-------  1 root root 5,7M nov 26 12:14
-rw-------  1 root root 5,7M jan  7 16:16
-rw-------  1 root root 5,7M jan 12 15:43
-rw-------  1 root root 5,7M jan 13 18:13
-rw-------  1 root root 5,7M feb  4 17:40
lrwxrwxrwx  1 root root   25 feb 28 04:27 vmlinuz -> vmlinuz-5.13.0-22-generic
-rw-------  1 root root 9,8M okt  7 19:37 vmlinuz-5.13.0-19-generic
-rw-------  1 root root 9,8M okt 15 15:56 vmlinuz-5.13.0-20-generic
-rw-------  1 root root 9,8M okt 19 10:43 vmlinuz-5.13.0-21-generic
-rw-------  1 root root 9,8M nov  5 13:51 vmlinuz-5.13.0-22-generic
-rw-------  1 root root 9,8M nov 26 11:52 vmlinuz-5.13.0-23-generic
-rw-------  1 root root 9,8M jan  7 16:19 vmlinuz-5.13.0-25-generic
-rw-------  1 root root 9,8M jan 12 16:19 vmlinuz-5.13.0-27-generic
-rw-------  1 root root 9,8M jan 13 18:10 vmlinuz-5.13.0-28-generic
-rw-------  1 root root 9,8M feb  4 17:46 vmlinuz-5.13.0-30-generic

I tried all these kernels. The last kernel where my input devices still worked, was 5.13.0-22-generic, and the first where they stopped working, was 5.13.0-23-generic. Which leads me to assume that some unintended change was introduced between those two versions, and it hasn’t been fixed since.

For now, I’m telling Ubuntu to keep kernel 5.13.0-22-generic and not upgrade to a more recent version.

$ sudo apt-mark hold linux-image-5.13.0-22-generic
linux-image-5.13.0-22-generic set on hold.

I also want Grub to show me the known working kernel as the default change. To do that, I’ve put this in /etc/default/grub:

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.13.0-22-generic"

followed by sudo update-grub.

I’ll do the following things next, to get to the bottom of this:

A small rant about dependencies (and a promise)

Every now and then I run into some awesome open source project on GitHub, that is written in some cool programming language, and it assumes that the development tools for that language are already installed. My assumption is that they have a specific target audience in mind: an already existing developer community around that specific language. People who already have those tools installed.

The annoying thing is when someone like me, who doesn’t really need to know if a thing is written in Python or Ruby or JavaScript or whatever, tries to follow instructions like these:

$ pip install foo
Command 'pip' not found
$ gem install bar
Command 'gem' not found
$ yarn install baz
Command 'yarn' not found
$ ./configure && make && sudo make install
Command 'make' not found

By now, I already know that I first need to do sudo apt install python3-pip (or the equivalent installation commands for RubyGems, Yarn, build-essential,…). I also understand that, within the context of a specific developer community, this is so obvious that it is often assumed. That being said, I am making a promise:

For every open source project that I will henceforth publish online (on Github or any other code sharing platforms), I promise to do the following things:
(1) Test the installation on at least one clean installed operating system – which will be documented.
(2) Include full installation steps in the documentation, including all frameworks, development tools, etc. that would otherwise be assumed.
(3) Where possible and useful, provide an installation script.

The operating system I’m currently targeting, is Ubuntu, which means I’ll include apt commands. I’m counting on Continuous Integration to help me test on other operating systems that I don’t personally use.

Living without email for a month

Remember when my webserver was acting up? Well, I was so fed up with it, that I took a preconfigured Bitnami WordPress image and ran that on AWS. I don’t care how Bitnami configured it, as long as it works.

As a minor detail, postfix/procmail/dovecot were of course not installed or configured. Meh. This annoyed the Mrs. a bit because she didn’t get her newsletters. But I was so fed up with all the technical problems, that I waited a month to do anything about it.

Doing sudo apt-get -y install postfix procmail dovecot-pop3d and copying over the configs from the old server solved that.

Did I miss email during that month? Not at all. People were able to contact met through Twitter, Facebook, Telegram and all the other social networks. And I had an entire month without spam. Wonderful!

Creating and publishing a NuGet package on Linux

Suppose you have a couple of .dll files that were built on a TeamCity server and you want to bundle them into a NuGet package and publish them on, how would you do that if you were a Linux user? Is that even possible??? Let’s find out!

  1. Preparation

    First things first, lets create a clean working environment:

    mkdir -p ~/repos/qa-nugetlinux
    cd qa-nugetlinux
    git init
    gi linux,vagrant >> .gitignore
    git add .gitignore
    git commit -m ".gitignore created by <a href=",vagrant">,vagrant</a>"
    vagrant init --minimal ubuntu/yakkety64
    git add Vagrantfile
    git commit -m "Add Vagrantfile"
    vagrant up --provider virtualbox

    This creates a Vagrant box where I will conduct my experiments. Let’s dive in and make sure that everything is up-to-date inside:

    vagrant ssh
    sudo apt-get update
    sudo apt-get -y dist-upgrade
    sudo apt-get -y autoremove
  2. Installing NuGet

    Now let’s get this party going!

    cd ~/vagrant
    wget <a href=""></a>
    chmod +x nuget.exe
    -bash: ./nuget.exe: cannot execute binary file: Exec format error

    Computer says no…
    Why not?

    file nuget.exe
    nuget.exe: PE32 executable (console) Intel 80386 Mono/.Net assembly, for MS Windows

    Oops, silly me. It’s a Mono executable.

    mono nuget.exe
    The program 'mono' is currently not installed. You can install it by typing:
    sudo apt install mono-runtime

    Thank you for that helpful message, Ubuntu!

    sudo apt-get -y install mono-runtime

    16 MiB later, I try again:

    mono nuget.exe
    Unhandled Exception:
    System.IO.FileNotFoundException: Could not load file or assembly 'System.Core, Version=, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
    File name: 'System.Core, Version=, Culture=neutral, PublicKeyToken=b77a5c561934e089'
      at NuGet.CommandLine.Program.Main (System.String[] args)  in :0 
    [ERROR] FATAL UNHANDLED EXCEPTION: System.IO.FileNotFoundException: Could not load file or assembly 'System.Core, Version=, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
    File name: 'System.Core, Version=, Culture=neutral, PublicKeyToken=b77a5c561934e089'
      at NuGet.CommandLine.Program.Main (System.String[] args)  in :0

    System.Core is missing? OK let’s install that.

    sudo apt-get -y install libmono-system-*

    And try again:

    mono nuget.exe
    Could not load file or assembly or one of its dependencies.

    Sigh. Ok, let’s use a cannon to shoot a mosquito:

    sudo apt-get -y install mono-complete

    Does it work now?

    mono nuget.exe
    NuGet Version:
    usage: NuGet  [args] [options] 
    Type 'NuGet help ' for help on a specific command.
    Available commands:
     add         Adds the given package to a hierarchical source. http sources are not supported. For more info, goto
     config      Gets or sets NuGet config values.
     delete      Deletes a package from the server.
     help (?)    Displays general help information and help information about other commands.
     init        Adds all the packages from the  to the hierarchical . http feeds are not supported. For more info, goto
     install     Installs a package using the specified sources. If no sources are specified, all sources defined in the NuGet configuration file are used. If the configuration file specifies no sources, uses the default NuGet feed.
     list        Displays a list of packages from a given source. If no sources are specified, all sources defined in %AppData%\NuGet\NuGet.config are used. If NuGet.config specifies no sources, uses the default NuGet feed.
     locals      Clears or lists local NuGet resources such as http requests cache, packages cache or machine-wide global packages folder.
     pack        Creates a NuGet package based on the specified nuspec or project file.
     push        Pushes a package to the server and publishes it.
                 NuGet's default configuration is obtained by loading %AppData%\NuGet\NuGet.config, then loading any nuget.config or .nuget\nuget.config starting from root of drive and ending in current directory.
     restore     Restores NuGet packages.
     setApiKey   Saves an API key for a given server URL. When no URL is provided API key is saved for the NuGet gallery.
     sources     Provides the ability to manage list of sources located in %AppData%\NuGet\NuGet.config
     spec        Generates a nuspec for a new package. If this command is run in the same folder as a project file (.csproj, .vbproj, .fsproj), it will create a tokenized nuspec file.
     update      Update packages to latest available versions. This command also updates NuGet.exe itself.
    For more information, visit

    And there was much rejoicing (Monty Python And The Holy Grail)

  3. Creating the .nuspec file

    1. Trying the easy way, and failing miserably

      According to some Idiot’s Guide to Creating and Publishing a NuGet package I found, I should be able to create a .nuspec file by running NuGet in the same directory as a .csproj file. Let’s try that:

      cd ~/vagrant/itextcore-dotnet/itext/itext.barcodes/
      mono ~/vagrant/nuget.exe pack itext.barcodes.csproj -verbosity detailed
      Attempting to build package from 'itext.barcodes.csproj'.
      MSBuild auto-detection: using msbuild version '4.0' from '/usr/lib/mono/4.5'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
      System.NotImplementedException: The method or operation is not implemented.
        at (wrapper dynamic-method) System.Object:CallSite.Target (System.Runtime.CompilerServices.Closure,System.Runtime.CompilerServices.CallSite,object)
        at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid1[T0] (System.Runtime.CompilerServices.CallSite site, System.Dynamic.T0 arg0)  in :0 
        at NuGet.CommandLine.ProjectFactory.ResolveTargetPath ()  in :0 
        at NuGet.CommandLine.ProjectFactory.BuildProject ()  in :0 
        at NuGet.CommandLine.ProjectFactory.CreateBuilder (System.String basePath)  in :0 
        at NuGet.CommandLine.PackCommand.BuildFromProjectFile (System.String path)  in :0 
        at NuGet.CommandLine.PackCommand.BuildPackage (System.String path)  in :0 
        at NuGet.CommandLine.PackCommand.ExecuteCommand ()  in :0 
        at NuGet.CommandLine.Command.ExecuteCommandAsync ()  in :0 
        at NuGet.CommandLine.Command.Execute ()  in :0 
        at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args)  in :0

      That seems like a big ball of NOPE to me… According to this GitHub comment from a NuGet member, this is to be expected.

    2. Hand Crank the .nuspec File

      So it’s going to be the hard way.

      This blog post was a draft, and I decided to publish whatever I had already, and if anyone is ever interested, I may or may not finish it. ¯\_(ツ)_/¯

How big is a clean install of Ubuntu Jammy Jellyfish (22.04)?

Because curiosity killed the cat, not because it’s useful! 😀

Start with a clean install in a virtual machine

I start with a simple Vagrantfile:

Vagrant.configure("2") do |config| = "ubuntu/jammy64"
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "playbook.yml"

This Ansible playbook updates all packages to the latest version and removes unused packages.

- name: Update all packages to the latest version
  hosts: all
  remote_user: ubuntu
  become: yes


  - name: Update apt cache
      update_cache: yes
      cache_valid_time: 3600
      force_apt_get: yes

  - name: Upgrade all apt packages
      force_apt_get: yes
      upgrade: dist

  - name: Check if a reboot is needed for Ubuntu boxes
    register: reboot_required_file
    stat: path=/var/run/reboot-required get_md5=no

  - name: Reboot the Ubuntu box
      msg: "Reboot initiated by Ansible due to kernel updates"
      connect_timeout: 5
      reboot_timeout: 300
      pre_reboot_delay: 0
      post_reboot_delay: 30
      test_command: uptime
    when: reboot_required_file.stat.exists

  - name: Remove unused packages
      autoremove: yes
      purge: yes
      force_apt_get: yes

Then bring up the virtual machine with vagrant up --provision.

Get the installation size

I ssh into the box (vagrant ssh) and run a couple of commands to get some numbers.

Number of installed packages:

$ dpkg-query --show | wc --lines

Size of the installed packages:

$ dpkg-query --show --showformat '${Installed-size}\n' | awk '{s+=$1*1024} END {print s}' | numfmt --to=iec-i --format='%.2fB'

I need to multiply the package size with 1024 because dpkg-query outputs size in kilobytes.

Total size:

$ sudo du --summarize --human-readable --one-file-system /
1.9G	/

Get the installation size using Ansible

Of course, I can also add this to my Ansible playbook, and then I don’t have to ssh into the virtual machine.

  - name: Get the number of installed packages
    shell: dpkg-query --show | wc --lines
    register: package_count
    changed_when: false
    failed_when: false
  - debug: msg="{{ package_count.stdout }}"

  - name: Get the size of installed packages
    shell: >
      dpkg-query --show --showformat '${Installed-size}\n' 
      | awk '{s+=$1*1024} END {print s}' 
      | numfmt --to=iec-i --format='%.2fB'
    register: package_size
    changed_when: false
    failed_when: false
  - debug: msg="{{ package_size.stdout }}"

  - name: Get the disk size with du
    shell: >
      du --summarize --one-file-system /
      | numfmt --to=iec-i --format='%.2fB'
    register: du_used
    changed_when: false
    failed_when: false
  - debug: msg="{{ du_used.stdout }}"

The output is then:

TASK [Get the number of installed packages] ************************************
ok: [default]

TASK [debug] *******************************************************************
ok: [default] => {
    "msg": "592"

TASK [Get the size of installed packages] **************************************
ok: [default]

TASK [debug] *******************************************************************
ok: [default] => {
    "msg": "1.14GiB"

TASK [Get the disk size with du] ***********************************************
ok: [default]

TASK [debug] *******************************************************************
ok: [default] => {
    "msg": "1.82MiB /"

screenshots van de framebuffer

Soms moet ne mens al eens iets speciaals doen, zoals het nemen van een screenshot op een toestel dat wel Linux draait, maar geen X. Oink? Volgens StackExchange zou ik fbgrab of fbdump moeten gebruiken, maar dat is in dit concrete geval niet mogelijk because reasons.

In dit concrete geval is er een toepassing die rechtstreeks naar de framebuffer beelden stuurt. Bon, alles is een file onder Linux, dus ik ging eens piepen wat er dan eigenlijk in dat framebuffer device zat:

$ cp /dev/fb0 /tmp/
$ head -c 64 /tmp/

Tiens, dat zag er verdacht regelmatig uit, telkens in groepjes van 4 bytes. “k” heeft ASCII waarde 107, of 6B hexadecimaal, en #6B6B6B is een grijstint. Ik had voorlopig nog geen enkel idee wat die “�” betekende, maar ik wist dat ik iets op het spoor was!

Ik heb dan gekopieerd naar een pc met daarop Gimp. (referentie naar Contact invoegen)

Bluetooth werkt eindelijk naar behoren in Feisty

Ik heb dus een Logitech Bluetooth Desktop MX5000. Al van in het begin heb ik daar problemen mee gehad in Ubuntu. Ofwel werken muis&toetsenbord, maar kon ik geen foto’s van mijn Nokia 6680 naar de pc sturen, ofwel omgekeerd. Lastig…

Gisteren was er een langverwachte update (3.9-0ubuntu2) van bluetooth in Ubuntu Feisty. Resultaat: muis en gsm werken, maar het toetsenbord niet. Dat is nog altijd lastig, maar het is een interessante wijziging!

Ik heb dan een oud PS/2-toetsenbord vanonder het stof gehaald, daarmee de gebruikelijke mantra ingetypt om verbinding te maken met een bluetooth device: sudo hidd --connect 00:07:61:XX:XX:XX terwijl ik tegelijkertijd op de connect-knopjes drukte op het toetsenbord en de bluetooth dongle. Ik moest wel héél snel zijn en een aantal keer opnieuw proberen, maar… (tromgeroffel) mijn toetsenbord werkt nu!

Beer++ voor de bluetooth-developers.

Ubuntu Code of Conduct ondertekend

Ik gebruik al sinds 2005 Ubuntu, maar vandaag heb ik eindelijk ook de moeite genomen om de Ubuntu Code of Conduct te ondertekenen. Waarom? Omdat ik wel al eens wat bijdragen lever aan de mailinglijsten en dat kan daar wel eens een krabbenmand zijn. *kuch* (understatement) Ik probeer naar eer en geweten te handelen in al de Linux- en andere Free Software communities waarbinnen ik actief ben, en door de CoC te ondertekenen, leg ik dat commitment vast.

Het heeft wel wat voeten in de aarde gehad want je moet dan een document ondertekenen met pgp en dat is, om het beleefd te zeggen, niet echt een uitblinker qua gebruiksgemak. Ik vermoed dat ik pgp totaal verkeerd gebruik, maar iedere keer dat ik iets van pgp-keys nodig heb, verwijder ik al mijn vorige keys en maak ik er totaal nieuwe aan. Soit. Het hele proces bestaat uit 3 delen:

OpenPGP key aanmaken

gpg --gen-key
gpg --list-keys
gpg --keyserver --send-keys 82BA02FC
gpg --fingerprint

Dan de fingerprint copy/pasten in Launchpad. Even later krijg je een encrypted mail van Launchpad. Kopieer de inhoud van de mail naar een tekstfile en doe:
gpg -d launchpad.txt
Op het einde van te tekst staat een link om de OpenPGP key te bevestigen.

CoC ondertekenen

gpg --clearsign UbuntuCodeofConduct-2.0.txt

en dan de inhoud van UbuntuCodeofConduct-2.0.txt.asc copy/pasten op


De OpenPGP key heb ik nu niet meer nodig, dus alles mag weg:
rm -rf ~/.gnupg UbuntuCodeofConduct*

Het resultaat is te bewonderen op