Christian Werner e4cab679b5 first try

2026-06-18 22:25:42 +02:00

38 KiB

Raw Blame History

Installing and Verifying a Precompiled Out-of-Tree Kernel Module on Raspberry Pi OS

Research tier: deep dive · 2026-06-18

Question

How do you deploy, load, and verify a precompiled .ko kernel module on a Raspberry Pi Zero 2 W running 64-bit Raspberry Pi OS Bookworm (kernel 6.6.x, -v8 arm64 flavour)? The module (iec_listener.ko) is built by an emulated-arm64 Docker host, creates /dev/iec0, accepts address=4, and exposes an IEC_IOC_SELFTEST ioctl. An optional iec-overlay.dtbo reserves GPIO pins.

Summary

A precompiled arm64 module built against linux-headers-rpi-v8 in an emulated-arm64 Docker container will load cleanly on the matching Pi with sudo insmod ./iec_listener.ko address=4, provided the vermagic string in the .ko exactly matches the running kernel's vermagic (kernel release string + SMP preempt mod_unload modversions aarch64). Stock Raspberry Pi OS Bookworm does not enforce module signing (CONFIG_MODULE_SIG_FORCE is not set); this is confirmed by checking the kernel config. The character device /dev/iec0 is created automatically by udev when the driver calls class_create/device_create — no mknod is needed. The device-tree overlay goes in /boot/firmware/overlays/ (Bookworm path — not /boot/overlays/) and is activated via a dtoverlay=iec line in /boot/firmware/config.txt. To survive apt kernel upgrades, pin the kernel packages with apt-mark hold immediately after first load.

Notes on `docs/kernel-notes.md` — discrepancies and gaps found

Issue	Location in notes	Correct value for Bookworm
`sudo cp iec-overlay.dtbo /boot/overlays/`	§"Build & deploy"	Must be `/boot/firmware/overlays/` on Bookworm
`echo "dtoverlay=iec" \| sudo tee -a /boot/config.txt`	(derived from prior research doc §4.1)	File is `/boot/firmware/config.txt` on Bookworm
`/boot/cmdline.txt` mentioned for `isolcpus`	§1.4 of prior research	File is `/boot/firmware/cmdline.txt` on Bookworm
Module signing stated as "not required" but not verified	§Decisions table	Correct — confirmed below in §6 with verification commands
No `depmod -a` step after permanent install	§"Build & deploy"	Required before `modprobe` works

Findings

1. Getting the `.ko` onto the Pi

Run on the build host (not the Pi):

# Preferred: rsync preserves timestamps, skips unchanged
rsync -avz --progress kernel/iec_listener.ko pi@raspberrypi.local:/home/pi/iec/

# Alternative: plain scp
scp kernel/iec_listener.ko pi@raspberrypi.local:/home/pi/iec/

# If you also built the overlay:
scp dts/iec-overlay.dtbo pi@raspberrypi.local:/home/pi/iec/

Stage under ~/iec/ (or any writable directory). Do not copy directly to /lib/modules/ yet — test with insmod from the staging directory first.

2. Pre-load verification: does the module match the running kernel?

This step prevents a wasted insmod attempt. Run on the Pi.

2.1 Read the module's vermagic

modinfo ~/iec/iec_listener.ko

Expected output (values vary with exact kernel point release):

filename:       /home/pi/iec/iec_listener.ko
description:    Commodore IEC bus listener
author:         Christian Werner
license:        GPL
version:        0.1
srcversion:     A3F9C1D8E4B2071F5C6D890
depends:
name:           iec_listener
vermagic:       6.6.51+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64
parm:           address:IEC device address (int)

The vermagic line is the critical field. [1][2]

2.2 Read the running kernel's vermagic string

uname -r
# e.g.:  6.6.51+rpt-rpi-v8

# The kernel stores its own vermagic in any already-loaded module.
# The fastest cross-check: read one in-tree module.
modinfo /lib/modules/$(uname -r)/kernel/drivers/char/random.ko | grep vermagic
# OR
modinfo $(find /lib/modules/$(uname -r)/ -name "*.ko.xz" | head -1) | grep vermagic

Alternatively, check Module.symvers which is produced by the same kernel build and encodes the same release string:

head -1 /lib/modules/$(uname -r)/build/Module.symvers
# or from the installed headers:
head -1 /usr/src/linux-headers-$(uname -r)/Module.symvers

2.3 Fields that must match exactly

Field	Example value	Where to find
Kernel release string	`6.6.51+rpt-rpi-v8`	`uname -r`
`SMP`	always present on Pi Zero 2 W	vermagic
`preempt`	present (Bookworm stock)	vermagic
`mod_unload`	present	vermagic
`modversions`	present (CRC checking enabled)	vermagic
`aarch64`	present for 64-bit arm64 `-v8`	vermagic

A match looks like:

# uname -r output:
6.6.51+rpt-rpi-v8

# modinfo vermagic line (must be identical):
vermagic:  6.6.51+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64

2.4 `CONFIG_MODVERSIONS` and symbol CRC checking

The modversions word in the vermagic string signals that CONFIG_MODVERSIONS=y is set in this kernel. This adds per-symbol CRC checksums (stored in Module.symvers) to every exported kernel symbol. When insmod loads your module it checks the CRC of every symbol the module imports against the kernel's own CRC table. If any CRC diverges — even if the release string matches — you get Unknown symbol in module (err -22).

This means: building against the correct headers package is not sufficient if those headers were not produced by the same kernel build (i.e., the same Module.symvers). The Docker build strategy (install linux-headers-rpi-v8 via apt inside the container) works only when the apt archive serves the same kernel version that is running on the Pi. That is precisely why kernel-notes.md recommends keeping the Pi on the latest release and building with the default (latest) headers. [2][3]

2.5 Also check the ELF architecture

file ~/iec/iec_listener.ko
# Expected:
# iec_listener.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), not stripped

Any 32-bit or ARM, EABI5 in the output means the module was built with the wrong headers (e.g., -v7 armhf instead of -v8 arm64) and will fail with Exec format error. [4]

3. Loading the module

3.1 `insmod` — use during bring-up

# On the Pi, from the staging directory:
sudo insmod ~/iec/iec_listener.ko address=4

What insmod does:

Takes the full path to the .ko file.
Passes it directly to the kernel via finit_module(2) or init_module(2).
Resolves no dependencies — fails hard if any MODULE_IMPORT symbol is missing.
Does not read /etc/modprobe.d/; parameters must be on the command line.
Does not require the module to be installed under /lib/modules/.

Use insmod during development and initial bring-up. It is simple, direct, and gives you the raw kernel error if something is wrong. [5]

3.2 `modprobe` — use for production / permanent install

# Requires the module to be installed first (see §8):
sudo modprobe iec_listener address=4

What modprobe does:

Looks up the module by name (not path) in /lib/modules/$(uname -r)/modules.dep (built by depmod).
Automatically loads any modules listed as dependencies in that dep file.
Reads /etc/modprobe.d/*.conf for default options (so address=4 can be made permanent there rather than on the command line).
If the module is not in the dep database → FATAL: Module iec_listener not found.

modprobe therefore requires two setup steps before it works for an out-of-tree module:

Copy the .ko to /lib/modules/$(uname -r)/extra/iec_listener.ko
Run sudo depmod -a

Use modprobe only after the permanent install steps in §8. [5][6]

3.3 Passing `address=4` in each case

# insmod — on the command line:
sudo insmod ~/iec/iec_listener.ko address=4

# modprobe — on the command line (temporary override):
sudo modprobe iec_listener address=4

# modprobe — permanent (persists across reboots; see §8):
echo "options iec_listener address=4" | sudo tee /etc/modprobe.d/iec_listener.conf
sudo modprobe iec_listener      # now uses address=4 from the conf file

4. Verifying a successful load

Run all commands on the Pi immediately after insmod:

4.1 `lsmod` — is the module listed?

lsmod | grep iec_listener

Expected:

iec_listener           24576  0

Columns: module name | size in bytes | use count (0 = no open file descriptors or dependent modules). [7]

4.2 `/sys/module/iec_listener/` — sysfs subtree

ls /sys/module/iec_listener/
# Expected directories/files: holders/ initstate parameters/ refcnt srcversion

# Confirm the address parameter was accepted:
cat /sys/module/iec_listener/parameters/address
# Expected: 4

# Module use count (should be 0 when no fd is open):
cat /sys/module/iec_listener/refcnt
# Expected: 0

The parameters/ directory exposes every module_param() as a file whose content reflects the runtime value. If address reads 4 the kernel accepted the parameter correctly. [8]

4.3 `dmesg` — read the init message

# Most recent kernel messages (shows init printk output):
dmesg | tail -20

# Or with timestamps since last boot:
sudo journalctl -k --since "1 minute ago"

# Filter to the module name:
dmesg | grep -i iec

Expected healthy output:

[  123.456789] iec_listener: IEC listener v0.1 loaded, address=4
[  123.456801] iec_listener: registered char device major=240 minor=0
[  123.456815] iec_listener: /dev/iec0 created

Anything with BUG:, WARNING:, kernel panic, or NULL pointer dereference in the module's output is a real problem — unload immediately with sudo rmmod iec_listener. [5]

4.4 `/dev/iec0` — does the device node exist?

ls -l /dev/iec0

Expected:

crw------- 1 root root 240, 0 Jun 18 14:35 /dev/iec0

Fields: c = character device, major 240 (dynamically allocated; your number may differ), minor 0.

How the node appears automatically: When the module calls class_create() followed by device_create(), the kernel creates a sysfs entry at /sys/class/iec/iec0/dev containing the major:minor pair. The udev daemon (running as a service — systemctl status udev) watches sysfs for new entries and creates the /dev/iec0 character device node automatically. No manual mknod is needed on any modern Linux with udev. [9][10]

Verify via sysfs:

cat /sys/class/iec/iec0/dev      # prints e.g. "240:0"
ls -la /sys/class/iec/iec0/      # shows the full sysfs entry

Permissions: udev defaults to root:root with mode 0600 (owner read/write only). During bring-up this is fine — run tests as root. For non-root access, add a udev rule:

# /etc/udev/rules.d/99-iec.rules
KERNEL=="iec0", GROUP="dialout", MODE="0660"

Then reload: sudo udevadm control --reload-rules && sudo udevadm trigger [10]

Check major/minor with stat:

stat /dev/iec0
# File: /dev/iec0
# Size: 0    Blocks: 0    IO Block: 4096  character special file
# Device: 5h/5d    Inode: 1357    Links: 1    Device type: f0,0
# (f0 hex = 240 decimal = major number)

5. The "is it actually working" check short of attaching the C64

5.1 Confirm the device is openable

sudo python3 -c "
import os
fd = os.open('/dev/iec0', os.O_RDONLY | os.O_NONBLOCK)
print('open OK, fd =', fd)
os.close(fd)
print('close OK')
"

Expected: open OK, fd = 3 then close OK. If you get PermissionError run as root (sudo) or fix the udev rule in §4.4. If you get No such device, the module's cdev_add() failed — check dmesg. [9]

5.2 Issue the `IEC_IOC_SELFTEST` ioctl

The ioctl number is defined in the module header as _IO(IEC_IOC_MAGIC, IEC_IOC_NR_SELFTEST). Substitute the values from kernel/iec_listener.h (e.g. magic 'I'=0x49, nr 0 → ioctl number 0x4900):

#!/usr/bin/env python3
"""Quick userspace selftest for iec_listener.ko — run on the Pi as root."""
import fcntl, os, struct

# Reconstruct the ioctl number from the kernel header:
#   _IO(type, nr) = ((type) << 8) | (nr)
# Example: IEC_IOC_MAGIC = ord('I') = 0x49, IEC_IOC_NR_SELFTEST = 0
IEC_IOC_SELFTEST = (ord('I') << 8) | 0   # = 0x4900  (adjust to match header)

fd = os.open('/dev/iec0', os.O_RDWR)
try:
    ret = fcntl.ioctl(fd, IEC_IOC_SELFTEST, 0)
    print(f"SELFTEST ioctl returned {ret} — {'PASS' if ret == 0 else 'FAIL'}")
except OSError as e:
    print(f"ioctl failed: {e}")
finally:
    os.close(fd)

After the ioctl, check dmesg again:

dmesg | tail -5

Healthy output example:

[  145.882011] iec_listener: SELFTEST: driving DATA low (GPIO 18)
[  145.882025] iec_listener: SELFTEST: reading DATA back → 0 (PASS)
[  145.882031] iec_listener: SELFTEST: releasing DATA
[  145.882038] iec_listener: SELFTEST: reading DATA back → 1 (PASS)
[  145.882041] iec_listener: SELFTEST passed

If SELFTEST: reading DATA back → 0 expected 1 FAIL appears, the GPIO pin wiring has an issue — the bus is stuck low. This is the exact test to run before connecting the C64 (as recommended in kernel-notes.md §"Level-shifter / DATA-sensing note"). [11]

C snippet alternative (if Python is not available on the Pi):

/* selftest.c — compile with: gcc -o selftest selftest.c */
#include <stdio.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>

#define IEC_IOC_SELFTEST  0x4900  /* adjust to match iec_listener.h */

int main(void) {
    int fd = open("/dev/iec0", O_RDWR);
    if (fd < 0) { perror("open"); return 1; }
    int r = ioctl(fd, IEC_IOC_SELFTEST, 0);
    printf("ioctl returned %d\n", r);
    close(fd);
    return r ? 1 : 0;
}

6. Troubleshooting table

Error 1: `Invalid module format` — vermagic mismatch

insmod output:

insmod: ERROR: could not insert module ./iec_listener.ko: Invalid module format

dmesg (the diagnostic detail):

[  124.001] iec_listener: version magic '6.6.47+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64'
             should be '6.6.51+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64'

What it means: The module was compiled against headers for kernel 6.6.47+rpt-rpi-v8 but the Pi is now running 6.6.51+rpt-rpi-v8. A kernel upgrade happened after the module was built.

Fix:

# On the Pi — check current kernel:
uname -r
# On the build host — rebuild with matching headers:
KERNEL_VERSION=<exact-version-string> ./build-in-docker.sh
# Or: keep Pi current (apt full-upgrade) then rebuild with default headers.

The vermagic comparison is byte-for-byte including the +rpt suffix, any + git-dirty marker, and the SMP/preempt/modversions flags. Even a missing or extra + character fails the check. [1][2]

Error 2: `Exec format error` — wrong ELF architecture

insmod output:

insmod: ERROR: could not insert module ./iec_listener.ko: Exec format error

dmesg:

[  125.002] iec_listener: Invalid architecture in ELF header (value 40, expected 183)

(40 = EM_ARM 32-bit; 183 = EM_AARCH64 64-bit)

What it means: The .ko was compiled for 32-bit ARM (armhf, using linux-headers-rpi-v7 or an x86 host without the correct cross-compile toolchain).

Diagnosis:

file ~/iec/iec_listener.ko
# Wrong:   ELF 32-bit LSB relocatable, ARM, EABI5 ...
# Correct: ELF 64-bit LSB relocatable, ARM aarch64 ...

readelf -h ~/iec/iec_listener.ko | grep Machine
# Wrong:   Machine: ARM
# Correct: Machine: AArch64

Fix: Rebuild using linux-headers-rpi-v8 (the -v8 suffix is the 64-bit arm64 flavour) inside the emulated-arm64 container. Confirm HEADERS_PKG in the build script is linux-headers-rpi-v8, not -v7 or -v7l. [4][12]

Error 3: `Unknown symbol in module` — symbol CRC mismatch

insmod output:

insmod: ERROR: could not insert module ./iec_listener.ko: Unknown symbol in module

dmesg:

[  126.003] iec_listener: Unknown symbol kfifo_alloc (err -22)
[  126.003] iec_listener: Unknown symbol class_create (err 0)

err 0 = symbol not found at all; err -22 = symbol exists but CRC mismatch (EINVAL). [13]

What it means: The module's Module.symvers (baked in at build time) has different CRC values than the running kernel. Most common cause: building against headers from a different build of the same kernel version (e.g., the headers were from a kernel re-compiled with different CONFIG_* options than the official Pi kernel).

Diagnosis:

# Find the offending symbols:
dmesg | grep "Unknown symbol"

# Check if the symbol exists in the running kernel:
grep kfifo_alloc /proc/kallsyms | head -3
# No output → symbol genuinely missing from this kernel
# Output → symbol exists but CRC differs (build mismatch)

Fix: Ensure the Docker build uses headers installed from the same apt pool that produced the running Pi kernel. If the Pi's kernel came from apt full-upgrade, the container's linux-headers-rpi-v8 from the same Bookworm repo will have matching symvers. Avoid using rpi-source or manually downloaded headers unless you also have the Module.symvers from that exact kernel build. [3][13]

Error 4: `Required key not available` / `Key was rejected by service` — module signing enforced

insmod output:

insmod: ERROR: could not insert module ./iec_listener.ko: Required key not available

insmod: ERROR: could not insert module ./iec_listener.ko: Key was rejected by service

This should NOT happen on stock Raspberry Pi OS Bookworm. The RPi kernel does not set CONFIG_MODULE_SIG_FORCE=y and the Raspberry Pi does not use UEFI Secure Boot by default. Secure Boot on RPi requires explicit OTP key burning — it is opt-in and off by default.

Verify that signing is not enforced (run on the Pi):

# Method 1: check the running kernel's config (most reliable)
# The stock RPi kernel ships with CONFIG_IKCONFIG=y, so /proc/config.gz exists:
sudo modprobe configs 2>/dev/null; zcat /proc/config.gz | grep -E "CONFIG_MODULE_SIG|CONFIG_SECURITY_LOCKDOWN"

Expected output on stock Bookworm:

CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
# CONFIG_MODULE_SIG_ALL is not set
# CONFIG_SECURITY_LOCKDOWN_LSM is not set

CONFIG_MODULE_SIG=y alone means the signing infrastructure is compiled in (the kernel can check signatures) but CONFIG_MODULE_SIG_FORCE being absent means unsigned modules are permitted (kernel taints itself with E but loads the module). [14][15]

# Method 2: check the lockdown sysfs node (only present if LSM compiled in)
cat /sys/kernel/security/lockdown 2>/dev/null
# If the file does not exist → lockdown LSM is not compiled in → safe.
# If it exists: "none" = off; "integrity" or "confidentiality" = modules blocked.

# Method 3: check dmesg for lockdown messages
dmesg | grep -i lockdown
# On stock RPi: no output

# Method 4: check kernel cmdline for enforcement override
cat /proc/cmdline | grep -o "module.sig_enforce=[01]"
# No output → not set → default is permissive

If you somehow see this error (e.g., on a custom Pi OS build with CONFIG_MODULE_SIG_FORCE=y), you must sign the module with a MOK key and enroll it via mokutil — this is out of scope for the standard Pi setup. [14]

Error 5: Module taint flags — what is expected vs. a problem

After loading an out-of-tree unsigned module, the kernel taints itself:

cat /proc/sys/kernel/tainted
# Typical value: 12288 = 4096 (O) + 8192 (E) = out-of-tree + unsigned

Taint bit decoding:

Bit	Value	Flag	Meaning	Expected for our module?
12	4096	O	Out-of-tree module loaded	YES — normal
13	8192	E	Unsigned module	YES — normal on stock RPi OS
0	1	P	Proprietary (non-GPL) module	No — we use `MODULE_LICENSE("GPL")`
1	2	F	Module force-loaded	No — bad, means vermagic was overridden

A tainted value of 12288 is entirely expected and benign for our use case. [16]

# Read taint in human-readable form from dmesg:
dmesg | grep -i taint
# Expected: "iec_listener: loading out-of-tree module taints kernel."
# Or:       "iec_listener: module license 'GPL' taints kernel."
# (The second message appears when CONFIG_MODULE_SIG_FORCE is set, not our case.)

A taint value of 2 (F flag) means insmod --force or modprobe --force was used — this overrides the vermagic check and can cause silent memory corruption. Never use --force except as a last-resort diagnostic step. [16]

Error 6: `Operation not permitted`

insmod output:

insmod: ERROR: could not insert module ./iec_listener.ko: Operation not permitted

What it means (several sub-cases):

a) You forgot sudo. Always run insmod / rmmod as root.

b) A secureboot-related lockdown is active (see Error 4).

c) The module tries to map a physical address (our ioremap(0x3F200000, ...)) and the kernel has CONFIG_STRICT_DEVMEM=y plus the request overlaps a restricted region. This is unlikely on stock RPi but worth checking:

dmesg | grep -E "ioremap|DEVMEM|mem_encrypt"

Error 7: `Device or resource busy`

insmod output:

insmod: ERROR: could not insert module ./iec_listener.ko: Device or resource busy

dmesg:

[  128.007] iec_listener: GPIO 17 request failed: -16  (EBUSY)

What it means: One of the GPIOs the module tries to claim with gpio_request() or devm_gpiod_get() is already owned by another driver (e.g., a previous unclean unload left the GPIO locked, or a DT overlay already claimed it through pinctrl).

Fix:

# See which GPIOs are claimed:
cat /sys/kernel/debug/gpio        # requires root / debugfs mounted
# or:
sudo ls /sys/class/gpio/

# If a previous load left GPIO locked, reboot to clear the state.
# If the DT overlay is claiming the pins, ensure the overlay's GPIO
# reservation matches exactly what the module requests.

Error 8: `rmmod: ERROR: Module iec_listener is in use`

rmmod output:

rmmod: ERROR: Module iec_listener is in use.

What it means: refcnt is non-zero. Either a process has /dev/iec0 open (e.g., your Python test script didn't close the fd) or another module depends on iec_listener.

Fix:

# See who holds references:
lsmod | grep iec_listener
# Third column > 0 means it has dependents or open file descriptors.

# Find open file descriptors:
sudo lsof /dev/iec0
# Shows the process(es) with the device open.

# Kill or close the process, then retry rmmod.
sudo kill <PID>
sudo rmmod iec_listener

7. Installing the optional device-tree overlay

The iec-overlay.dtbo reserves GPIO pins (ATN, CLK, DATA, RESET) via the kernel pinctrl subsystem, preventing other drivers from claiming them.

7.1 Where the file goes on Bookworm

# ON THE PI — copy to the Bookworm overlay directory:
sudo cp ~/iec/iec-overlay.dtbo /boot/firmware/overlays/iec.dtbo

Bookworm path: /boot/firmware/overlays/ — not /boot/overlays/. In Bullseye and earlier the path was /boot/overlays/. On fresh Bookworm installs /boot/overlays/ may be a symlink or may not exist at all. Always use /boot/firmware/overlays/ on Bookworm. [17][18]

7.2 Activate in `config.txt`

# Edit the Bookworm config file:
echo "dtoverlay=iec" | sudo tee -a /boot/firmware/config.txt

# Verify it was added:
grep dtoverlay /boot/firmware/config.txt

The firmware strips the .dtbo extension automatically — dtoverlay=iec loads /boot/firmware/overlays/iec.dtbo. [17]

7.3 Reboot required

sudo reboot

Device-tree overlays are applied by the VideoCore firmware during boot, before the kernel starts. They cannot be applied without a reboot via config.txt. (The dtoverlay runtime command works for some overlays but is unreliable for GPIO pinctrl fragments.) [18]

7.4 Verify the overlay loaded after reboot

# Method 1: dtoverlay list (shows overlays loaded by the firmware at boot)
sudo dtoverlay -l
# Expected:
# Overlays (in load order):
# 0:  iec

# NOTE: dtoverlay -l only shows overlays loaded via config.txt at boot.
# It does NOT show overlays applied at runtime via the dtoverlay command.

# Method 2: inspect the live device tree for our GPIO reservation node
ls /proc/device-tree/
# Look for a node matching the overlay name or the GPIO label:
find /proc/device-tree/ -name "*iec*" 2>/dev/null

# Method 3: check GPIO allocation after module load
sudo cat /sys/kernel/debug/gpio | grep -A 3 "iec"
# Expected lines like:
# gpio-2   (iec_atn         ) in  lo

# Method 4: firmware log (available on older RPi firmware builds)
sudo vcdbg log msg 2>/dev/null | grep -i overlay
# On Bookworm with recent firmware this command may not be available.

# Method 5: dmesg for pinctrl messages on module load
dmesg | grep -E "pinctrl|iec_pins"

If dtoverlay -l shows no overlays and dmesg shows GPIO claim errors, the overlay did not load. Check that the .dtbo file is present in /boot/firmware/overlays/ and that the dtoverlay=iec line is in /boot/firmware/config.txt (not /boot/config.txt). [17][19]

8. Persistence across reboot — permanent install

Use this after the module is confirmed working via insmod.

8.1 Install the `.ko` into the modules tree

# Standard out-of-tree location:
KVER=$(uname -r)
sudo mkdir -p /lib/modules/${KVER}/extra/
sudo cp ~/iec/iec_listener.ko /lib/modules/${KVER}/extra/

# Regenerate the module dependency database:
sudo depmod -a

depmod -a scans all .ko files under /lib/modules/$(uname -r)/ and writes modules.dep, modules.alias, and related files. Without this step, modprobe iec_listener fails with FATAL: Module iec_listener not found. [5][6]

8.2 Verify `modprobe` can now find it

modinfo iec_listener
# Should print the same output as modinfo on the .ko file path.
# If it says "modinfo: ERROR: Module iec_listener not found" → depmod -a was not run
# or the file wasn't copied to the right directory.

8.3 Auto-load at boot via `modules-load.d`

echo "iec_listener" | sudo tee /etc/modules-load.d/iec_listener.conf

systemd-modules-load.service reads files under /etc/modules-load.d/ at boot and calls modprobe for each listed module name. [5]

8.4 Supply `address=4` permanently via `modprobe.d`

echo "options iec_listener address=4" | sudo tee /etc/modprobe.d/iec_listener.conf

This file is read by modprobe (and by systemd-modules-load) whenever the module is loaded, eliminating the need to pass address=4 on the command line. [5][6]

8.5 Verify after a reboot

sudo reboot
# ... after reboot:

lsmod | grep iec_listener
# Expected: iec_listener  24576  0

cat /sys/module/iec_listener/parameters/address
# Expected: 4

ls -l /dev/iec0
# Expected: crw------- 1 root root 240, 0 ...

dmesg | grep iec_listener
# Expected: init message printed during boot

9. Surviving `apt` kernel upgrades

9.1 Why a kernel upgrade breaks a manually-installed module

When sudo apt full-upgrade installs a new raspberrypi-kernel package:

A new kernel image (e.g., 6.6.62+rpt-rpi-v8) replaces the old one in /boot/firmware/.
A new /lib/modules/6.6.62+rpt-rpi-v8/ tree is created.
Your module file lives in /lib/modules/6.6.51+rpt-rpi-v8/extra/ — it is not moved or rebuilt.
On next boot the new kernel runs, systemd-modules-load calls modprobe iec_listener, modprobe looks in /lib/modules/6.6.62+rpt-rpi-v8/ — finds nothing — fails silently (or with a log message).
Even if you copy the .ko manually, the vermagic mismatch causes Invalid module format.

9.2 Detecting that the kernel moved out from under the module

# After a reboot where the module failed to load:
uname -r
# e.g.: 6.6.62+rpt-rpi-v8

ls /lib/modules/$(uname -r)/extra/
# "No such file or directory" → module was installed for a different kernel version.

dmesg | grep "iec_listener"
# "FATAL: Module iec_listener not found" or "Invalid module format" in systemd log.

sudo journalctl -b | grep "Failed to insert module"

9.3 Pinning the kernel with `apt-mark hold`

sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader

Record the pinned version in kernel-notes.md:

dpkg -l raspberrypi-kernel | awk 'NR==5{print $2, $3}'
# e.g.: raspberrypi-kernel 1:6.6.51-1+rpt3

To unhold (when you deliberately want to upgrade and rebuild):

sudo apt-mark unhold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader
sudo apt full-upgrade
# → rebuild the module against the new headers
# → re-run §8 (copy + depmod)
sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader

Check current hold status:

apt-mark showhold

9.4 DKMS (for completeness — out of scope for our project)

DKMS (Dynamic Kernel Module Support) automates the rebuild-on-upgrade cycle. It requires the source code and a dkms.conf on the Pi itself. Since our build strategy uses Docker on an x86 host, DKMS is not applicable. Mention it only as a pointer if the project ever shifts to building on the Pi directly. [20]

10. Clean unload / reload during iteration

The reload cycle during development:

# Step 1: close any open file descriptors (kill test scripts, close /dev/iec0)
sudo lsof /dev/iec0

# Step 2: unload
sudo rmmod iec_listener

# Step 3: confirm GPIO/device cleanup in dmesg
dmesg | tail -10
# Expected:
# [  200.112] iec_listener: freeing IRQs
# [  200.113] iec_listener: DATA line released (GPIO 18 low)
# [  200.114] iec_listener: /dev/iec0 destroyed
# [  200.115] iec_listener: unloaded

# Step 4: confirm /dev/iec0 is gone
ls /dev/iec0
# ls: cannot access '/dev/iec0': No such file or directory

# Step 5: reload with new parameters or updated binary
sudo insmod ~/iec/iec_listener.ko address=4

# Step 6: verify
lsmod | grep iec_listener && ls -l /dev/iec0 && dmesg | tail -5

If Step 2 hangs ("Module iec_listener is in use"):

# Find the holder:
sudo lsof /dev/iec0
# Kill it:
sudo kill -9 <PID>
# Retry:
sudo rmmod iec_listener

CRITICAL — what the module's exit function must do (already correct in iec_exit per kernel-notes.md, verifying here):

free_irq(irq_atn, NULL) — disarm ATN interrupt first
free_irq(irq_clk, NULL) — disarm CLK interrupt
Release DATA line — gpiod_set_value(gd_data_out, 0) — must happen or the IEC bus is left stuck LOW. A stuck DATA line will prevent the C64 from communicating with any device.
iounmap(gpio_regs) — unmap BCM register block
device_destroy() + class_destroy() + cdev_del() — tear down the character device (causes udev to remove /dev/iec0 automatically)
unregister_chrdev_region() — free the major/minor allocation [11]

Open questions / gaps

/proc/config.gz availability on stock Bookworm: The CONFIG_IKCONFIG_PROC option makes /proc/config.gz available after modprobe configs. Whether the stock RPi OS 6.6.x kernel has CONFIG_IKCONFIG=y is not confirmed by a live system check — only by inference from forum posts. The fallback is cat /boot/config-$(uname -r) which may also not exist on RPi. Confirm on first boot by running both commands.
dtoverlay -l behaviour on Bookworm: As noted in §7.4, boot-time overlays from config.txt may or may not appear in dtoverlay -l output depending on the firmware version. The reliable check is /proc/device-tree/ inspection or dmesg GPIO messages after module load.
udev permissions for /dev/iec0: Default 0600 root:root permissions require sudo for all userspace access. The udev rule in §4.4 broadens this to 0660 root:dialout. Whether the pi user is in the dialout group on a fresh Bookworm install needs to be confirmed: groups pi | grep dialout.
Module signing config on the exact running Pi: The analysis is based on forum reports and the known default RPi kernel defconfig. The commands in §6 Error 4 (zcat /proc/config.gz | grep CONFIG_MODULE_SIG) should be run on first boot to confirm. If CONFIG_IKCONFIG is not set, check /boot/config-$(uname -r) — this path is used on Debian-family kernels (the RPi may or may not ship it; community evidence suggests it does not on stock installs, but /proc/config.gz via modprobe configs is available).
apt full-upgrade and the linux-headers-rpi-v8 meta-package: The linux-headers-rpi-v8 package is a meta-package that always pulls the latest headers. Pinning the kernel without also pinning the headers meta-package could result in headers that don't match the pinned kernel. The apt-mark hold command in §9.3 covers both.

Sources

[1] Gateworks — Linux Kernel Modules (vermagic, modinfo, loading) — https://trac.gateworks.com/wiki/linux/kernel/modules (accessed 2026-06-18)

[2] Raspberry Pi Forums — "Invalid module format when running a cross-compiled linux kernel module" — https://forums.raspberrypi.com/viewtopic.php?t=369617 (accessed 2026-06-18)

[3] Linux Kernel Docs — Building External Modules (Module.symvers, MODVERSIONS) — https://docs.kernel.org/kbuild/modules.html (accessed 2026-06-18)

[4] Raspberry Pi Forums — "After apt upgrade my 32-bit OS runs on a 64-bit kernel and now I can't compile an external module" — https://forums.raspberrypi.com/viewtopic.php?t=349070 (accessed 2026-06-18)

[5] ArchWiki — Kernel module (insmod, modprobe, /etc/modules-load.d/, /etc/modprobe.d/) — https://wiki.archlinux.org/title/Kernel_module (accessed 2026-06-18)

[6] Red Hat Docs — Setting Module Parameters (/etc/modprobe.d/, depmod) — https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/deployment_guide/sec-setting_module_parameters (accessed 2026-06-18)

[7] Linuxize — lsmod command — https://linuxize.com/post/lsmod-command-in-linux/ (accessed 2026-06-18)

[8] Linux Kernel ABI — /sys/module structure — https://www.kernel.org/doc/Documentation/ABI/stable/sysfs-module (accessed 2026-06-18)

[9] embetronicx — Device File Creation for Character Drivers (class_create / device_create / udev) — https://embetronicx.com/tutorials/linux/device-drivers/device-file-creation-for-character-drivers/ (accessed 2026-06-18)

[10] GitHub Gist strezh — Automatic /dev file creation when driver module is loaded — https://gist.github.com/strezh/b01fcd50875c214e510a81c6aa6d2a2a (accessed 2026-06-18)

[11] This project — docs/kernel-notes.md and _plans/poc-listener-printer-PLAN.md (accessed 2026-06-18)

[12] Zak's Electronics Blog — RPi compiling a module for the 64-bit kernel — https://blog.zakkemble.net/rpi-compiling-a-module-for-the-64-bit-kernel/ (accessed 2026-06-18)

[13] linuxvox.com — How to Fix 'Unknown Symbol in Module' Error — https://linuxvox.com/blog/unknown-symbol-in-while-loading-a-kernel-module/ (accessed 2026-06-18)

[14] Linux Kernel Docs — Kernel module signing facility — https://www.kernel.org/doc/html/v4.15/admin-guide/module-signing.html (accessed 2026-06-18)

[15] Raspberry Pi Forums — "How to lockdown kernel / Disable kernel rewrite on RPi 4" — https://forums.raspberrypi.com/viewtopic.php?t=360877 (accessed 2026-06-18)

[16] Linux Kernel Docs — Tainted Kernels (taint flag table) — https://docs.kernel.org/admin-guide/tainted-kernels.html (accessed 2026-06-18)

[17] Raspberry Pi Documentation — config.txt: dtoverlay directive, overlay path on Bookworm — https://www.raspberrypi.com/documentation/computers/config_txt.html (accessed 2026-06-18)

[18] Bootlin Blog — Enabling new hardware on Raspberry Pi with Device Tree Overlays — https://bootlin.com/blog/enabling-new-hardware-on-raspberry-pi-with-device-tree-overlays/ (accessed 2026-06-18)

[19] Raspberry Pi Forums — "Bookworm - Device tree overlays not loading" — https://forums.raspberrypi.com/viewtopic.php?t=367942 (accessed 2026-06-18)

[20] Flogistoni/raspbiec — instdrv.sh installation script (insmod + chmod pattern) — https://github.com/Flogistoni/raspbiec/blob/development/instdrv.sh (accessed 2026-06-18)

[21] yeri.be — RPi kernels in Bookworm (package names: linux-headers-rpi-v8 etc.) — https://yeri.be/rpi-kernels-in-bookworm/ (accessed 2026-06-18)

[22] Raspberry Pi Forums — "Rpi OS Bookworm 64 bit: cannot install linux-headers-rpi-v8" — https://forums.raspberrypi.com/viewtopic.php?t=360082 (accessed 2026-06-18)

Recommended canonical install+verify sequence (< 10 lines)

# On the build host — transfer:
rsync -avz kernel/iec_listener.ko pi@raspberrypi.local:/home/pi/iec/

# On the Pi — pre-check:
modinfo ~/iec/iec_listener.ko | grep vermagic   # must match uname -r exactly

# Load and verify:
sudo insmod ~/iec/iec_listener.ko address=4
lsmod | grep iec_listener                        # module listed → OK
cat /sys/module/iec_listener/parameters/address  # prints 4 → OK
ls -l /dev/iec0                                  # crw------- → udev node created
dmesg | tail -5                                  # init messages → no BUG/WARNING

# Selftest before connecting C64:
sudo python3 ~/iec/selftest.py                   # IEC_IOC_SELFTEST → PASS

# Unload cleanly:
sudo rmmod iec_listener
dmesg | tail -5                                  # DATA released message → safe

Bookworm / Pi Zero 2 W gotchas:

All boot config paths have moved: /boot/firmware/config.txt, /boot/firmware/cmdline.txt, /boot/firmware/overlays/. Using the old /boot/ paths silently does nothing.
The vermagic for arm64 Bookworm ends in modversions aarch64 — the modversions token means symbol CRC checking is active. Headers and running kernel must come from the same apt transaction.
apt full-upgrade will break the module. Pin the kernel immediately: sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader.
/dev/iec0 is created automatically by udev — no mknod. If it doesn't appear within ~1 second of insmod, check dmesg for device_create errors and confirm udev is running (systemctl status udev).
Module signing is NOT enforced on stock RPi OS. A taint value of 12288 (flags O+E) after load is normal and harmless.

38 KiB Raw Blame History

Installing and Verifying a Precompiled Out-of-Tree Kernel Module on Raspberry Pi OS

Question

Summary

Notes on docs/kernel-notes.md — discrepancies and gaps found

Findings

1. Getting the .ko onto the Pi

2. Pre-load verification: does the module match the running kernel?

2.1 Read the module's vermagic

2.2 Read the running kernel's vermagic string

2.3 Fields that must match exactly

2.4 CONFIG_MODVERSIONS and symbol CRC checking

2.5 Also check the ELF architecture

3. Loading the module

3.1 insmod — use during bring-up

3.2 modprobe — use for production / permanent install

3.3 Passing address=4 in each case

4. Verifying a successful load

4.1 lsmod — is the module listed?

4.2 /sys/module/iec_listener/ — sysfs subtree

4.3 dmesg — read the init message

4.4 /dev/iec0 — does the device node exist?

5. The "is it actually working" check short of attaching the C64

5.1 Confirm the device is openable

5.2 Issue the IEC_IOC_SELFTEST ioctl

6. Troubleshooting table

Error 1: Invalid module format — vermagic mismatch

Error 2: Exec format error — wrong ELF architecture

Error 3: Unknown symbol in module — symbol CRC mismatch

Error 4: Required key not available / Key was rejected by service — module signing enforced

Error 5: Module taint flags — what is expected vs. a problem

Error 6: Operation not permitted

Error 7: Device or resource busy

Error 8: rmmod: ERROR: Module iec_listener is in use

7. Installing the optional device-tree overlay

7.1 Where the file goes on Bookworm

7.2 Activate in config.txt

7.3 Reboot required

7.4 Verify the overlay loaded after reboot

8. Persistence across reboot — permanent install

8.1 Install the .ko into the modules tree

8.2 Verify modprobe can now find it

8.3 Auto-load at boot via modules-load.d

8.4 Supply address=4 permanently via modprobe.d

8.5 Verify after a reboot

9. Surviving apt kernel upgrades

9.1 Why a kernel upgrade breaks a manually-installed module

9.2 Detecting that the kernel moved out from under the module

9.3 Pinning the kernel with apt-mark hold

9.4 DKMS (for completeness — out of scope for our project)

10. Clean unload / reload during iteration

Open questions / gaps

Sources

Recommended canonical install+verify sequence (< 10 lines)

38 KiB

Raw Blame History

Notes on `docs/kernel-notes.md` — discrepancies and gaps found

1. Getting the `.ko` onto the Pi

2.4 `CONFIG_MODVERSIONS` and symbol CRC checking

3.1 `insmod` — use during bring-up

3.2 `modprobe` — use for production / permanent install

3.3 Passing `address=4` in each case

4.1 `lsmod` — is the module listed?

4.2 `/sys/module/iec_listener/` — sysfs subtree

4.3 `dmesg` — read the init message

4.4 `/dev/iec0` — does the device node exist?

5.2 Issue the `IEC_IOC_SELFTEST` ioctl

Error 1: `Invalid module format` — vermagic mismatch

Error 2: `Exec format error` — wrong ELF architecture

Error 3: `Unknown symbol in module` — symbol CRC mismatch

Error 4: `Required key not available` / `Key was rejected by service` — module signing enforced

Error 6: `Operation not permitted`

Error 7: `Device or resource busy`

Error 8: `rmmod: ERROR: Module iec_listener is in use`

7.2 Activate in `config.txt`

8.1 Install the `.ko` into the modules tree

8.2 Verify `modprobe` can now find it

8.3 Auto-load at boot via `modules-load.d`

8.4 Supply `address=4` permanently via `modprobe.d`

9. Surviving `apt` kernel upgrades

9.3 Pinning the kernel with `apt-mark hold`