comodore-iec-emu/_research/pi-kernel-module-install-verify-2026-06-18.md

# Installing and Verifying a Precompiled Out-of-Tree Kernel Module on Raspberry Pi OS

> Research tier: deep dive · 2026-06-18

---

## Question

How do you deploy, load, and verify a precompiled `.ko` kernel module on a
Raspberry Pi Zero 2 W running 64-bit Raspberry Pi OS Bookworm (kernel 6.6.x,
`-v8` arm64 flavour)? The module (`iec_listener.ko`) is built by an
emulated-arm64 Docker host, creates `/dev/iec0`, accepts `address=4`, and
exposes an `IEC_IOC_SELFTEST` ioctl. An optional `iec-overlay.dtbo` reserves
GPIO pins.

---

## Summary

A precompiled arm64 module built against `linux-headers-rpi-v8` in an
emulated-arm64 Docker container will load cleanly on the matching Pi with
`sudo insmod ./iec_listener.ko address=4`, provided the vermagic string in the
`.ko` exactly matches the running kernel's vermagic (kernel release string +
`SMP preempt mod_unload modversions aarch64`). Stock Raspberry Pi OS Bookworm
does **not** enforce module signing (`CONFIG_MODULE_SIG_FORCE` is not set);
this is confirmed by checking the kernel config. The character device
`/dev/iec0` is created automatically by udev when the driver calls
`class_create`/`device_create` — no `mknod` is needed. The device-tree overlay
goes in `/boot/firmware/overlays/` (Bookworm path — **not** `/boot/overlays/`)
and is activated via a `dtoverlay=iec` line in `/boot/firmware/config.txt`. To
survive `apt` kernel upgrades, pin the kernel packages with `apt-mark hold`
immediately after first load.

---

## Notes on `docs/kernel-notes.md` — discrepancies and gaps found

| Issue | Location in notes | Correct value for Bookworm |
|-------|------------------|---------------------------|
| `sudo cp iec-overlay.dtbo /boot/overlays/` | §"Build & deploy" | Must be `/boot/firmware/overlays/` on Bookworm |
| `echo "dtoverlay=iec" \| sudo tee -a /boot/config.txt` | (derived from prior research doc §4.1) | File is `/boot/firmware/config.txt` on Bookworm |
| `/boot/cmdline.txt` mentioned for `isolcpus` | §1.4 of prior research | File is `/boot/firmware/cmdline.txt` on Bookworm |
| Module signing stated as "not required" but not verified | §Decisions table | Correct — confirmed below in §6 with verification commands |
| No `depmod -a` step after permanent install | §"Build & deploy" | Required before `modprobe` works |

---

## Findings

---

### 1. Getting the `.ko` onto the Pi

Run **on the build host** (not the Pi):

```bash
# Preferred: rsync preserves timestamps, skips unchanged
rsync -avz --progress kernel/iec_listener.ko pi@raspberrypi.local:/home/pi/iec/

# Alternative: plain scp
scp kernel/iec_listener.ko pi@raspberrypi.local:/home/pi/iec/

# If you also built the overlay:
scp dts/iec-overlay.dtbo pi@raspberrypi.local:/home/pi/iec/
```

Stage under `~/iec/` (or any writable directory). Do **not** copy directly to
`/lib/modules/` yet — test with `insmod` from the staging directory first.

---

### 2. Pre-load verification: does the module match the running kernel?

This step prevents a wasted `insmod` attempt. Run **on the Pi**.

#### 2.1 Read the module's vermagic

```bash
modinfo ~/iec/iec_listener.ko
```

Expected output (values vary with exact kernel point release):

```
filename:       /home/pi/iec/iec_listener.ko
description:    Commodore IEC bus listener
author:         Christian Werner
license:        GPL
version:        0.1
srcversion:     A3F9C1D8E4B2071F5C6D890
depends:
name:           iec_listener
vermagic:       6.6.51+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64
parm:           address:IEC device address (int)
```

The **`vermagic`** line is the critical field. [1][2]

#### 2.2 Read the running kernel's vermagic string

```bash
uname -r
# e.g.:  6.6.51+rpt-rpi-v8

# The kernel stores its own vermagic in any already-loaded module.
# The fastest cross-check: read one in-tree module.
modinfo /lib/modules/$(uname -r)/kernel/drivers/char/random.ko | grep vermagic
# OR
modinfo $(find /lib/modules/$(uname -r)/ -name "*.ko.xz" | head -1) | grep vermagic
```

Alternatively, check `Module.symvers` which is produced by the same kernel
build and encodes the same release string:

```bash
head -1 /lib/modules/$(uname -r)/build/Module.symvers
# or from the installed headers:
head -1 /usr/src/linux-headers-$(uname -r)/Module.symvers
```

#### 2.3 Fields that must match exactly

| Field | Example value | Where to find |
|-------|---------------|---------------|
| Kernel release string | `6.6.51+rpt-rpi-v8` | `uname -r` |
| `SMP` | always present on Pi Zero 2 W | vermagic |
| `preempt` | present (Bookworm stock) | vermagic |
| `mod_unload` | present | vermagic |
| `modversions` | present (CRC checking enabled) | vermagic |
| `aarch64` | present for 64-bit arm64 `-v8` | vermagic |

A **match** looks like:
```
# uname -r output:
6.6.51+rpt-rpi-v8

# modinfo vermagic line (must be identical):
vermagic:  6.6.51+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64
```

#### 2.4 `CONFIG_MODVERSIONS` and symbol CRC checking

The `modversions` word in the vermagic string signals that
`CONFIG_MODVERSIONS=y` is set in this kernel. This adds per-symbol CRC
checksums (stored in `Module.symvers`) to every exported kernel symbol.
When insmod loads your module it checks the CRC of every symbol the module
imports against the kernel's own CRC table. If **any** CRC diverges — even if
the release string matches — you get `Unknown symbol in module (err -22)`.

This means: building against the correct *headers package* is not sufficient
if those headers were not produced by the same kernel build (i.e., the same
`Module.symvers`). The Docker build strategy (install `linux-headers-rpi-v8`
via apt inside the container) works **only** when the apt archive serves the
same kernel version that is running on the Pi. That is precisely why
`kernel-notes.md` recommends keeping the Pi on the latest release and building
with the default (latest) headers. [2][3]

#### 2.5 Also check the ELF architecture

```bash
file ~/iec/iec_listener.ko
# Expected:
# iec_listener.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), not stripped
```

Any `32-bit` or `ARM, EABI5` in the output means the module was built with the
wrong headers (e.g., `-v7` armhf instead of `-v8` arm64) and will fail with
`Exec format error`. [4]

---

### 3. Loading the module

#### 3.1 `insmod` — use during bring-up

```bash
# On the Pi, from the staging directory:
sudo insmod ~/iec/iec_listener.ko address=4
```

What `insmod` does:
- Takes the **full path** to the `.ko` file.
- Passes it directly to the kernel via `finit_module(2)` or `init_module(2)`.
- Resolves no dependencies — fails hard if any `MODULE_IMPORT` symbol is
  missing.
- Does **not** read `/etc/modprobe.d/`; parameters must be on the command line.
- Does **not** require the module to be installed under `/lib/modules/`.

**Use `insmod` during development and initial bring-up.** It is simple, direct,
and gives you the raw kernel error if something is wrong. [5]

#### 3.2 `modprobe` — use for production / permanent install

```bash
# Requires the module to be installed first (see §8):
sudo modprobe iec_listener address=4
```

What `modprobe` does:
- Looks up the module **by name** (not path) in
  `/lib/modules/$(uname -r)/modules.dep` (built by `depmod`).
- Automatically loads any modules listed as dependencies in that dep file.
- Reads `/etc/modprobe.d/*.conf` for default options (so `address=4` can be
  made permanent there rather than on the command line).
- If the module is not in the dep database → `FATAL: Module iec_listener not found`.

`modprobe` therefore **requires** two setup steps before it works for an
out-of-tree module:
1. Copy the `.ko` to `/lib/modules/$(uname -r)/extra/iec_listener.ko`
2. Run `sudo depmod -a`

**Use `modprobe` only after the permanent install steps in §8.** [5][6]

#### 3.3 Passing `address=4` in each case

```bash
# insmod — on the command line:
sudo insmod ~/iec/iec_listener.ko address=4

# modprobe — on the command line (temporary override):
sudo modprobe iec_listener address=4

# modprobe — permanent (persists across reboots; see §8):
echo "options iec_listener address=4" | sudo tee /etc/modprobe.d/iec_listener.conf
sudo modprobe iec_listener      # now uses address=4 from the conf file
```

---

### 4. Verifying a successful load

Run all commands **on the Pi** immediately after `insmod`:

#### 4.1 `lsmod` — is the module listed?

```bash
lsmod | grep iec_listener
```

Expected:
```
iec_listener           24576  0
```

Columns: module name | size in bytes | use count (0 = no open file descriptors
or dependent modules). [7]

#### 4.2 `/sys/module/iec_listener/` — sysfs subtree

```bash
ls /sys/module/iec_listener/
# Expected directories/files: holders/ initstate parameters/ refcnt srcversion

# Confirm the address parameter was accepted:
cat /sys/module/iec_listener/parameters/address
# Expected: 4

# Module use count (should be 0 when no fd is open):
cat /sys/module/iec_listener/refcnt
# Expected: 0
```

The `parameters/` directory exposes every `module_param()` as a file whose
content reflects the runtime value. If `address` reads `4` the kernel accepted
the parameter correctly. [8]

#### 4.3 `dmesg` — read the init message

```bash
# Most recent kernel messages (shows init printk output):
dmesg | tail -20

# Or with timestamps since last boot:
sudo journalctl -k --since "1 minute ago"

# Filter to the module name:
dmesg | grep -i iec
```

Expected healthy output:
```
[  123.456789] iec_listener: IEC listener v0.1 loaded, address=4
[  123.456801] iec_listener: registered char device major=240 minor=0
[  123.456815] iec_listener: /dev/iec0 created
```

Anything with `BUG:`, `WARNING:`, `kernel panic`, or `NULL pointer dereference`
in the module's output is a real problem — unload immediately with `sudo rmmod
iec_listener`. [5]

#### 4.4 `/dev/iec0` — does the device node exist?

```bash
ls -l /dev/iec0
```

Expected:
```
crw------- 1 root root 240, 0 Jun 18 14:35 /dev/iec0
```

Fields: `c` = character device, major 240 (dynamically allocated; your number
may differ), minor 0.

**How the node appears automatically:** When the module calls `class_create()`
followed by `device_create()`, the kernel creates a sysfs entry at
`/sys/class/iec/iec0/dev` containing the `major:minor` pair. The `udev` daemon
(running as a service — `systemctl status udev`) watches sysfs for new entries
and creates the `/dev/iec0` character device node automatically. No manual
`mknod` is needed on any modern Linux with udev. [9][10]

**Verify via sysfs:**
```bash
cat /sys/class/iec/iec0/dev      # prints e.g. "240:0"
ls -la /sys/class/iec/iec0/      # shows the full sysfs entry
```

**Permissions:** udev defaults to `root:root` with mode `0600` (owner read/write
only). During bring-up this is fine — run tests as root. For non-root access,
add a udev rule:

```bash
# /etc/udev/rules.d/99-iec.rules
KERNEL=="iec0", GROUP="dialout", MODE="0660"
```

Then reload: `sudo udevadm control --reload-rules && sudo udevadm trigger` [10]

**Check major/minor with `stat`:**
```bash
stat /dev/iec0
# File: /dev/iec0
# Size: 0    Blocks: 0    IO Block: 4096  character special file
# Device: 5h/5d    Inode: 1357    Links: 1    Device type: f0,0
# (f0 hex = 240 decimal = major number)
```

---

### 5. The "is it actually working" check short of attaching the C64

#### 5.1 Confirm the device is openable

```bash
sudo python3 -c "
import os
fd = os.open('/dev/iec0', os.O_RDONLY | os.O_NONBLOCK)
print('open OK, fd =', fd)
os.close(fd)
print('close OK')
"
```

Expected: `open OK, fd = 3` then `close OK`. If you get `PermissionError` run as
root (`sudo`) or fix the udev rule in §4.4. If you get `No such device`, the
module's `cdev_add()` failed — check `dmesg`. [9]

#### 5.2 Issue the `IEC_IOC_SELFTEST` ioctl

The ioctl number is defined in the module header as
`_IO(IEC_IOC_MAGIC, IEC_IOC_NR_SELFTEST)`. Substitute the values from
`kernel/iec_listener.h` (e.g. magic `'I'`=0x49, nr 0 → ioctl number 0x4900):

```python
#!/usr/bin/env python3
"""Quick userspace selftest for iec_listener.ko — run on the Pi as root."""
import fcntl, os, struct

# Reconstruct the ioctl number from the kernel header:
#   _IO(type, nr) = ((type) << 8) | (nr)
# Example: IEC_IOC_MAGIC = ord('I') = 0x49, IEC_IOC_NR_SELFTEST = 0
IEC_IOC_SELFTEST = (ord('I') << 8) | 0   # = 0x4900  (adjust to match header)

fd = os.open('/dev/iec0', os.O_RDWR)
try:
    ret = fcntl.ioctl(fd, IEC_IOC_SELFTEST, 0)
    print(f"SELFTEST ioctl returned {ret} — {'PASS' if ret == 0 else 'FAIL'}")
except OSError as e:
    print(f"ioctl failed: {e}")
finally:
    os.close(fd)
```

After the ioctl, check `dmesg` again:
```bash
dmesg | tail -5
```

Healthy output example:
```
[  145.882011] iec_listener: SELFTEST: driving DATA low (GPIO 18)
[  145.882025] iec_listener: SELFTEST: reading DATA back → 0 (PASS)
[  145.882031] iec_listener: SELFTEST: releasing DATA
[  145.882038] iec_listener: SELFTEST: reading DATA back → 1 (PASS)
[  145.882041] iec_listener: SELFTEST passed
```

If `SELFTEST: reading DATA back → 0 expected 1 FAIL` appears, the GPIO pin
wiring has an issue — the bus is stuck low. This is the exact test to run
before connecting the C64 (as recommended in `kernel-notes.md` §"Level-shifter
/ DATA-sensing note"). [11]

**C snippet alternative** (if Python is not available on the Pi):

```c
/* selftest.c — compile with: gcc -o selftest selftest.c */
#include <stdio.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>

#define IEC_IOC_SELFTEST  0x4900  /* adjust to match iec_listener.h */

int main(void) {
    int fd = open("/dev/iec0", O_RDWR);
    if (fd < 0) { perror("open"); return 1; }
    int r = ioctl(fd, IEC_IOC_SELFTEST, 0);
    printf("ioctl returned %d\n", r);
    close(fd);
    return r ? 1 : 0;
}
```

---

### 6. Troubleshooting table

#### Error 1: `Invalid module format` — vermagic mismatch

**insmod output:**
```
insmod: ERROR: could not insert module ./iec_listener.ko: Invalid module format
```

**dmesg (the diagnostic detail):**
```
[  124.001] iec_listener: version magic '6.6.47+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64'
             should be '6.6.51+rpt-rpi-v8 SMP preempt mod_unload modversions aarch64'
```

**What it means:** The module was compiled against headers for kernel
`6.6.47+rpt-rpi-v8` but the Pi is now running `6.6.51+rpt-rpi-v8`. A kernel
upgrade happened after the module was built.

**Fix:**
```bash
# On the Pi — check current kernel:
uname -r
# On the build host — rebuild with matching headers:
KERNEL_VERSION=<exact-version-string> ./build-in-docker.sh
# Or: keep Pi current (apt full-upgrade) then rebuild with default headers.
```

The vermagic comparison is **byte-for-byte** including the `+rpt` suffix, any
`+` git-dirty marker, and the SMP/preempt/modversions flags. Even a missing or
extra `+` character fails the check. [1][2]

---

#### Error 2: `Exec format error` — wrong ELF architecture

**insmod output:**
```
insmod: ERROR: could not insert module ./iec_listener.ko: Exec format error
```

**dmesg:**
```
[  125.002] iec_listener: Invalid architecture in ELF header (value 40, expected 183)
```
(`40` = `EM_ARM` 32-bit; `183` = `EM_AARCH64` 64-bit)

**What it means:** The `.ko` was compiled for 32-bit ARM (armhf, using
`linux-headers-rpi-v7` or an x86 host without the correct cross-compile
toolchain).

**Diagnosis:**
```bash
file ~/iec/iec_listener.ko
# Wrong:   ELF 32-bit LSB relocatable, ARM, EABI5 ...
# Correct: ELF 64-bit LSB relocatable, ARM aarch64 ...

readelf -h ~/iec/iec_listener.ko | grep Machine
# Wrong:   Machine: ARM
# Correct: Machine: AArch64
```

**Fix:** Rebuild using `linux-headers-rpi-v8` (the `-v8` suffix is the 64-bit
arm64 flavour) inside the emulated-arm64 container. Confirm `HEADERS_PKG` in
the build script is `linux-headers-rpi-v8`, not `-v7` or `-v7l`. [4][12]

---

#### Error 3: `Unknown symbol in module` — symbol CRC mismatch

**insmod output:**
```
insmod: ERROR: could not insert module ./iec_listener.ko: Unknown symbol in module
```

**dmesg:**
```
[  126.003] iec_listener: Unknown symbol kfifo_alloc (err -22)
[  126.003] iec_listener: Unknown symbol class_create (err 0)
```

`err 0` = symbol not found at all; `err -22` = symbol exists but CRC mismatch
(`EINVAL`). [13]

**What it means:** The module's `Module.symvers` (baked in at build time) has
different CRC values than the running kernel. Most common cause: building
against headers from a *different build* of the same kernel version (e.g., the
headers were from a kernel re-compiled with different `CONFIG_*` options than
the official Pi kernel).

**Diagnosis:**
```bash
# Find the offending symbols:
dmesg | grep "Unknown symbol"

# Check if the symbol exists in the running kernel:
grep kfifo_alloc /proc/kallsyms | head -3
# No output → symbol genuinely missing from this kernel
# Output → symbol exists but CRC differs (build mismatch)
```

**Fix:** Ensure the Docker build uses headers installed from the **same apt
pool** that produced the running Pi kernel. If the Pi's kernel came from
`apt full-upgrade`, the container's `linux-headers-rpi-v8` from the same
Bookworm repo will have matching symvers. Avoid using `rpi-source` or manually
downloaded headers unless you also have the `Module.symvers` from that exact
kernel build. [3][13]

---

#### Error 4: `Required key not available` / `Key was rejected by service` — module signing enforced

**insmod output:**
```
insmod: ERROR: could not insert module ./iec_listener.ko: Required key not available
```
or
```
insmod: ERROR: could not insert module ./iec_listener.ko: Key was rejected by service
```

**This should NOT happen on stock Raspberry Pi OS Bookworm.** The RPi kernel
does not set `CONFIG_MODULE_SIG_FORCE=y` and the Raspberry Pi does not use UEFI
Secure Boot by default. Secure Boot on RPi requires explicit OTP key burning —
it is opt-in and off by default.

**Verify that signing is not enforced (run on the Pi):**

```bash
# Method 1: check the running kernel's config (most reliable)
# The stock RPi kernel ships with CONFIG_IKCONFIG=y, so /proc/config.gz exists:
sudo modprobe configs 2>/dev/null; zcat /proc/config.gz | grep -E "CONFIG_MODULE_SIG|CONFIG_SECURITY_LOCKDOWN"
```

Expected output on stock Bookworm:
```
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
# CONFIG_MODULE_SIG_ALL is not set
# CONFIG_SECURITY_LOCKDOWN_LSM is not set
```

`CONFIG_MODULE_SIG=y` alone means the signing *infrastructure* is compiled in
(the kernel *can* check signatures) but `CONFIG_MODULE_SIG_FORCE` being absent
means unsigned modules are **permitted** (kernel taints itself with `E` but
loads the module). [14][15]

```bash
# Method 2: check the lockdown sysfs node (only present if LSM compiled in)
cat /sys/kernel/security/lockdown 2>/dev/null
# If the file does not exist → lockdown LSM is not compiled in → safe.
# If it exists: "none" = off; "integrity" or "confidentiality" = modules blocked.

# Method 3: check dmesg for lockdown messages
dmesg | grep -i lockdown
# On stock RPi: no output

# Method 4: check kernel cmdline for enforcement override
cat /proc/cmdline | grep -o "module.sig_enforce=[01]"
# No output → not set → default is permissive
```

**If you somehow see this error** (e.g., on a custom Pi OS build with
`CONFIG_MODULE_SIG_FORCE=y`), you must sign the module with a MOK key and
enroll it via `mokutil` — this is out of scope for the standard Pi setup. [14]

---

#### Error 5: Module taint flags — what is expected vs. a problem

After loading an out-of-tree unsigned module, the kernel taints itself:

```bash
cat /proc/sys/kernel/tainted
# Typical value: 12288 = 4096 (O) + 8192 (E) = out-of-tree + unsigned
```

**Taint bit decoding:**

| Bit | Value | Flag | Meaning | Expected for our module? |
|-----|-------|------|---------|--------------------------|
| 12 | 4096 | O | Out-of-tree module loaded | **YES — normal** |
| 13 | 8192 | E | Unsigned module | **YES — normal on stock RPi OS** |
| 0 | 1 | P | Proprietary (non-GPL) module | No — we use `MODULE_LICENSE("GPL")` |
| 1 | 2 | F | Module force-loaded | No — bad, means vermagic was overridden |

A `tainted` value of `12288` is **entirely expected and benign** for our
use case. [16]

```bash
# Read taint in human-readable form from dmesg:
dmesg | grep -i taint
# Expected: "iec_listener: loading out-of-tree module taints kernel."
# Or:       "iec_listener: module license 'GPL' taints kernel."
# (The second message appears when CONFIG_MODULE_SIG_FORCE is set, not our case.)
```

**A taint value of 2 (F flag) means `insmod --force` or `modprobe --force`
was used** — this overrides the vermagic check and can cause silent memory
corruption. Never use `--force` except as a last-resort diagnostic step. [16]

---

#### Error 6: `Operation not permitted`

**insmod output:**
```
insmod: ERROR: could not insert module ./iec_listener.ko: Operation not permitted
```

**What it means (several sub-cases):**

a) You forgot `sudo`. Always run `insmod` / `rmmod` as root.

b) A secureboot-related lockdown is active (see Error 4).

c) The module tries to map a physical address (our `ioremap(0x3F200000, ...)`)
   and the kernel has `CONFIG_STRICT_DEVMEM=y` plus the request overlaps a
   restricted region. This is unlikely on stock RPi but worth checking:
   ```bash
   dmesg | grep -E "ioremap|DEVMEM|mem_encrypt"
   ```

---

#### Error 7: `Device or resource busy`

**insmod output:**
```
insmod: ERROR: could not insert module ./iec_listener.ko: Device or resource busy
```

**dmesg:**
```
[  128.007] iec_listener: GPIO 17 request failed: -16  (EBUSY)
```

**What it means:** One of the GPIOs the module tries to claim with
`gpio_request()` or `devm_gpiod_get()` is already owned by another driver
(e.g., a previous unclean unload left the GPIO locked, or a DT overlay already
claimed it through pinctrl).

**Fix:**
```bash
# See which GPIOs are claimed:
cat /sys/kernel/debug/gpio        # requires root / debugfs mounted
# or:
sudo ls /sys/class/gpio/

# If a previous load left GPIO locked, reboot to clear the state.
# If the DT overlay is claiming the pins, ensure the overlay's GPIO
# reservation matches exactly what the module requests.
```

---

#### Error 8: `rmmod: ERROR: Module iec_listener is in use`

**rmmod output:**
```
rmmod: ERROR: Module iec_listener is in use.
```

**What it means:** `refcnt` is non-zero. Either a process has `/dev/iec0` open
(e.g., your Python test script didn't close the fd) or another module depends
on `iec_listener`.

**Fix:**
```bash
# See who holds references:
lsmod | grep iec_listener
# Third column > 0 means it has dependents or open file descriptors.

# Find open file descriptors:
sudo lsof /dev/iec0
# Shows the process(es) with the device open.

# Kill or close the process, then retry rmmod.
sudo kill <PID>
sudo rmmod iec_listener
```

---

### 7. Installing the optional device-tree overlay

The `iec-overlay.dtbo` reserves GPIO pins (ATN, CLK, DATA, RESET) via the
kernel pinctrl subsystem, preventing other drivers from claiming them.

#### 7.1 Where the file goes on Bookworm

```bash
# ON THE PI — copy to the Bookworm overlay directory:
sudo cp ~/iec/iec-overlay.dtbo /boot/firmware/overlays/iec.dtbo
```

**Bookworm path: `/boot/firmware/overlays/`** — not `/boot/overlays/`.
In Bullseye and earlier the path was `/boot/overlays/`. On fresh Bookworm
installs `/boot/overlays/` may be a symlink or may not exist at all. Always
use `/boot/firmware/overlays/` on Bookworm. [17][18]

#### 7.2 Activate in `config.txt`

```bash
# Edit the Bookworm config file:
echo "dtoverlay=iec" | sudo tee -a /boot/firmware/config.txt

# Verify it was added:
grep dtoverlay /boot/firmware/config.txt
```

The firmware strips the `.dtbo` extension automatically — `dtoverlay=iec` loads
`/boot/firmware/overlays/iec.dtbo`. [17]

#### 7.3 Reboot required

```bash
sudo reboot
```

Device-tree overlays are applied by the VideoCore firmware during boot, before
the kernel starts. They **cannot** be applied without a reboot via config.txt.
(The `dtoverlay` runtime command works for some overlays but is unreliable for
GPIO pinctrl fragments.) [18]

#### 7.4 Verify the overlay loaded after reboot

```bash
# Method 1: dtoverlay list (shows overlays loaded by the firmware at boot)
sudo dtoverlay -l
# Expected:
# Overlays (in load order):
# 0:  iec

# NOTE: dtoverlay -l only shows overlays loaded via config.txt at boot.
# It does NOT show overlays applied at runtime via the dtoverlay command.

# Method 2: inspect the live device tree for our GPIO reservation node
ls /proc/device-tree/
# Look for a node matching the overlay name or the GPIO label:
find /proc/device-tree/ -name "*iec*" 2>/dev/null

# Method 3: check GPIO allocation after module load
sudo cat /sys/kernel/debug/gpio | grep -A 3 "iec"
# Expected lines like:
# gpio-2   (iec_atn         ) in  lo

# Method 4: firmware log (available on older RPi firmware builds)
sudo vcdbg log msg 2>/dev/null | grep -i overlay
# On Bookworm with recent firmware this command may not be available.

# Method 5: dmesg for pinctrl messages on module load
dmesg | grep -E "pinctrl|iec_pins"
```

If `dtoverlay -l` shows no overlays and `dmesg` shows GPIO claim errors, the
overlay did not load. Check that the `.dtbo` file is present in
`/boot/firmware/overlays/` and that the `dtoverlay=iec` line is in
`/boot/firmware/config.txt` (not `/boot/config.txt`). [17][19]

---

### 8. Persistence across reboot — permanent install

Use this after the module is confirmed working via `insmod`.

#### 8.1 Install the `.ko` into the modules tree

```bash
# Standard out-of-tree location:
KVER=$(uname -r)
sudo mkdir -p /lib/modules/${KVER}/extra/
sudo cp ~/iec/iec_listener.ko /lib/modules/${KVER}/extra/

# Regenerate the module dependency database:
sudo depmod -a
```

`depmod -a` scans all `.ko` files under `/lib/modules/$(uname -r)/` and writes
`modules.dep`, `modules.alias`, and related files. Without this step, `modprobe
iec_listener` fails with `FATAL: Module iec_listener not found`. [5][6]

#### 8.2 Verify `modprobe` can now find it

```bash
modinfo iec_listener
# Should print the same output as modinfo on the .ko file path.
# If it says "modinfo: ERROR: Module iec_listener not found" → depmod -a was not run
# or the file wasn't copied to the right directory.
```

#### 8.3 Auto-load at boot via `modules-load.d`

```bash
echo "iec_listener" | sudo tee /etc/modules-load.d/iec_listener.conf
```

`systemd-modules-load.service` reads files under `/etc/modules-load.d/` at
boot and calls `modprobe` for each listed module name. [5]

#### 8.4 Supply `address=4` permanently via `modprobe.d`

```bash
echo "options iec_listener address=4" | sudo tee /etc/modprobe.d/iec_listener.conf
```

This file is read by `modprobe` (and by `systemd-modules-load`) whenever the
module is loaded, eliminating the need to pass `address=4` on the command line. [5][6]

#### 8.5 Verify after a reboot

```bash
sudo reboot
# ... after reboot:

lsmod | grep iec_listener
# Expected: iec_listener  24576  0

cat /sys/module/iec_listener/parameters/address
# Expected: 4

ls -l /dev/iec0
# Expected: crw------- 1 root root 240, 0 ...

dmesg | grep iec_listener
# Expected: init message printed during boot
```

---

### 9. Surviving `apt` kernel upgrades

#### 9.1 Why a kernel upgrade breaks a manually-installed module

When `sudo apt full-upgrade` installs a new `raspberrypi-kernel` package:
- A new kernel image (e.g., `6.6.62+rpt-rpi-v8`) replaces the old one in
  `/boot/firmware/`.
- A new `/lib/modules/6.6.62+rpt-rpi-v8/` tree is created.
- Your module file lives in `/lib/modules/6.6.51+rpt-rpi-v8/extra/` — it is
  **not moved or rebuilt**.
- On next boot the new kernel runs, `systemd-modules-load` calls `modprobe
  iec_listener`, modprobe looks in `/lib/modules/6.6.62+rpt-rpi-v8/` — finds
  nothing — fails silently (or with a log message).
- Even if you copy the `.ko` manually, the vermagic mismatch causes `Invalid
  module format`.

#### 9.2 Detecting that the kernel moved out from under the module

```bash
# After a reboot where the module failed to load:
uname -r
# e.g.: 6.6.62+rpt-rpi-v8

ls /lib/modules/$(uname -r)/extra/
# "No such file or directory" → module was installed for a different kernel version.

dmesg | grep "iec_listener"
# "FATAL: Module iec_listener not found" or "Invalid module format" in systemd log.

sudo journalctl -b | grep "Failed to insert module"
```

#### 9.3 Pinning the kernel with `apt-mark hold`

```bash
sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader
```

**Record the pinned version in `kernel-notes.md`:**
```bash
dpkg -l raspberrypi-kernel | awk 'NR==5{print $2, $3}'
# e.g.: raspberrypi-kernel 1:6.6.51-1+rpt3
```

To unhold (when you deliberately want to upgrade and rebuild):
```bash
sudo apt-mark unhold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader
sudo apt full-upgrade
# → rebuild the module against the new headers
# → re-run §8 (copy + depmod)
sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader
```

**Check current hold status:**
```bash
apt-mark showhold
```

#### 9.4 DKMS (for completeness — out of scope for our project)

DKMS (Dynamic Kernel Module Support) automates the rebuild-on-upgrade cycle.
It requires the *source code* and a `dkms.conf` on the Pi itself. Since our
build strategy uses Docker on an x86 host, DKMS is not applicable. Mention it
only as a pointer if the project ever shifts to building on the Pi directly. [20]

---

### 10. Clean unload / reload during iteration

**The reload cycle during development:**

```bash
# Step 1: close any open file descriptors (kill test scripts, close /dev/iec0)
sudo lsof /dev/iec0

# Step 2: unload
sudo rmmod iec_listener

# Step 3: confirm GPIO/device cleanup in dmesg
dmesg | tail -10
# Expected:
# [  200.112] iec_listener: freeing IRQs
# [  200.113] iec_listener: DATA line released (GPIO 18 low)
# [  200.114] iec_listener: /dev/iec0 destroyed
# [  200.115] iec_listener: unloaded

# Step 4: confirm /dev/iec0 is gone
ls /dev/iec0
# ls: cannot access '/dev/iec0': No such file or directory

# Step 5: reload with new parameters or updated binary
sudo insmod ~/iec/iec_listener.ko address=4

# Step 6: verify
lsmod | grep iec_listener && ls -l /dev/iec0 && dmesg | tail -5
```

**If Step 2 hangs ("Module iec_listener is in use"):**
```bash
# Find the holder:
sudo lsof /dev/iec0
# Kill it:
sudo kill -9 <PID>
# Retry:
sudo rmmod iec_listener
```

**CRITICAL — what the module's `exit` function must do** (already correct in
`iec_exit` per `kernel-notes.md`, verifying here):

1. `free_irq(irq_atn, NULL)` — disarm ATN interrupt first
2. `free_irq(irq_clk, NULL)` — disarm CLK interrupt
3. **Release DATA line** — `gpiod_set_value(gd_data_out, 0)` — **must happen or
   the IEC bus is left stuck LOW**. A stuck DATA line will prevent the C64 from
   communicating with any device.
4. `iounmap(gpio_regs)` — unmap BCM register block
5. `device_destroy()` + `class_destroy()` + `cdev_del()` — tear down the
   character device (causes udev to remove `/dev/iec0` automatically)
6. `unregister_chrdev_region()` — free the major/minor allocation [11]

---

## Open questions / gaps

1. **`/proc/config.gz` availability on stock Bookworm**: The `CONFIG_IKCONFIG_PROC`
   option makes `/proc/config.gz` available after `modprobe configs`. Whether
   the stock RPi OS 6.6.x kernel has `CONFIG_IKCONFIG=y` is not confirmed by
   a live system check — only by inference from forum posts. The fallback is
   `cat /boot/config-$(uname -r)` which may also not exist on RPi. Confirm on
   first boot by running both commands.

2. **`dtoverlay -l` behaviour on Bookworm**: As noted in §7.4, boot-time overlays
   from `config.txt` may or may not appear in `dtoverlay -l` output depending
   on the firmware version. The reliable check is `/proc/device-tree/` inspection
   or `dmesg` GPIO messages after module load.

3. **udev permissions for `/dev/iec0`**: Default `0600 root:root` permissions
   require `sudo` for all userspace access. The udev rule in §4.4 broadens this
   to `0660 root:dialout`. Whether the `pi` user is in the `dialout` group on
   a fresh Bookworm install needs to be confirmed: `groups pi | grep dialout`.

4. **Module signing config on the exact running Pi**: The analysis is based on
   forum reports and the known default RPi kernel defconfig. The commands in
   §6 Error 4 (`zcat /proc/config.gz | grep CONFIG_MODULE_SIG`) should be run
   on first boot to confirm. If `CONFIG_IKCONFIG` is not set, check
   `/boot/config-$(uname -r)` — this path is used on Debian-family kernels
   (the RPi may or may not ship it; community evidence suggests it does not
   on stock installs, but `/proc/config.gz` via `modprobe configs` is available).

5. **`apt full-upgrade` and the `linux-headers-rpi-v8` meta-package**: The
   `linux-headers-rpi-v8` package is a meta-package that always pulls the latest
   headers. Pinning the kernel without also pinning the headers meta-package
   could result in headers that don't match the pinned kernel. The `apt-mark
   hold` command in §9.3 covers both.

---

## Sources

[1] Gateworks — Linux Kernel Modules (vermagic, modinfo, loading) — https://trac.gateworks.com/wiki/linux/kernel/modules (accessed 2026-06-18)

[2] Raspberry Pi Forums — "Invalid module format when running a cross-compiled linux kernel module" — https://forums.raspberrypi.com/viewtopic.php?t=369617 (accessed 2026-06-18)

[3] Linux Kernel Docs — Building External Modules (Module.symvers, MODVERSIONS) — https://docs.kernel.org/kbuild/modules.html (accessed 2026-06-18)

[4] Raspberry Pi Forums — "After apt upgrade my 32-bit OS runs on a 64-bit kernel and now I can't compile an external module" — https://forums.raspberrypi.com/viewtopic.php?t=349070 (accessed 2026-06-18)

[5] ArchWiki — Kernel module (insmod, modprobe, /etc/modules-load.d/, /etc/modprobe.d/) — https://wiki.archlinux.org/title/Kernel_module (accessed 2026-06-18)

[6] Red Hat Docs — Setting Module Parameters (/etc/modprobe.d/, depmod) — https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/deployment_guide/sec-setting_module_parameters (accessed 2026-06-18)

[7] Linuxize — lsmod command — https://linuxize.com/post/lsmod-command-in-linux/ (accessed 2026-06-18)

[8] Linux Kernel ABI — /sys/module structure — https://www.kernel.org/doc/Documentation/ABI/stable/sysfs-module (accessed 2026-06-18)

[9] embetronicx — Device File Creation for Character Drivers (class_create / device_create / udev) — https://embetronicx.com/tutorials/linux/device-drivers/device-file-creation-for-character-drivers/ (accessed 2026-06-18)

[10] GitHub Gist strezh — Automatic /dev file creation when driver module is loaded — https://gist.github.com/strezh/b01fcd50875c214e510a81c6aa6d2a2a (accessed 2026-06-18)

[11] This project — docs/kernel-notes.md and _plans/poc-listener-printer-PLAN.md (accessed 2026-06-18)

[12] Zak's Electronics Blog — RPi compiling a module for the 64-bit kernel — https://blog.zakkemble.net/rpi-compiling-a-module-for-the-64-bit-kernel/ (accessed 2026-06-18)

[13] linuxvox.com — How to Fix 'Unknown Symbol in Module' Error — https://linuxvox.com/blog/unknown-symbol-in-while-loading-a-kernel-module/ (accessed 2026-06-18)

[14] Linux Kernel Docs — Kernel module signing facility — https://www.kernel.org/doc/html/v4.15/admin-guide/module-signing.html (accessed 2026-06-18)

[15] Raspberry Pi Forums — "How to lockdown kernel / Disable kernel rewrite on RPi 4" — https://forums.raspberrypi.com/viewtopic.php?t=360877 (accessed 2026-06-18)

[16] Linux Kernel Docs — Tainted Kernels (taint flag table) — https://docs.kernel.org/admin-guide/tainted-kernels.html (accessed 2026-06-18)

[17] Raspberry Pi Documentation — config.txt: dtoverlay directive, overlay path on Bookworm — https://www.raspberrypi.com/documentation/computers/config_txt.html (accessed 2026-06-18)

[18] Bootlin Blog — Enabling new hardware on Raspberry Pi with Device Tree Overlays — https://bootlin.com/blog/enabling-new-hardware-on-raspberry-pi-with-device-tree-overlays/ (accessed 2026-06-18)

[19] Raspberry Pi Forums — "Bookworm - Device tree overlays not loading" — https://forums.raspberrypi.com/viewtopic.php?t=367942 (accessed 2026-06-18)

[20] Flogistoni/raspbiec — instdrv.sh installation script (insmod + chmod pattern) — https://github.com/Flogistoni/raspbiec/blob/development/instdrv.sh (accessed 2026-06-18)

[21] yeri.be — RPi kernels in Bookworm (package names: linux-headers-rpi-v8 etc.) — https://yeri.be/rpi-kernels-in-bookworm/ (accessed 2026-06-18)

[22] Raspberry Pi Forums — "Rpi OS Bookworm 64 bit: cannot install linux-headers-rpi-v8" — https://forums.raspberrypi.com/viewtopic.php?t=360082 (accessed 2026-06-18)

---

## Recommended canonical install+verify sequence (< 10 lines)

```bash
# On the build host — transfer:
rsync -avz kernel/iec_listener.ko pi@raspberrypi.local:/home/pi/iec/

# On the Pi — pre-check:
modinfo ~/iec/iec_listener.ko | grep vermagic   # must match uname -r exactly

# Load and verify:
sudo insmod ~/iec/iec_listener.ko address=4
lsmod | grep iec_listener                        # module listed → OK
cat /sys/module/iec_listener/parameters/address  # prints 4 → OK
ls -l /dev/iec0                                  # crw------- → udev node created
dmesg | tail -5                                  # init messages → no BUG/WARNING

# Selftest before connecting C64:
sudo python3 ~/iec/selftest.py                   # IEC_IOC_SELFTEST → PASS

# Unload cleanly:
sudo rmmod iec_listener
dmesg | tail -5                                  # DATA released message → safe
```

**Bookworm / Pi Zero 2 W gotchas:**
- All boot config paths have moved: `/boot/firmware/config.txt`, `/boot/firmware/cmdline.txt`, `/boot/firmware/overlays/`. Using the old `/boot/` paths silently does nothing.
- The vermagic for arm64 Bookworm ends in `modversions aarch64` — the `modversions` token means symbol CRC checking is active. Headers and running kernel must come from the same apt transaction.
- `apt full-upgrade` will break the module. Pin the kernel immediately: `sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader`.
- `/dev/iec0` is created automatically by udev — no `mknod`. If it doesn't appear within ~1 second of insmod, check `dmesg` for `device_create` errors and confirm `udev` is running (`systemctl status udev`).
- Module signing is NOT enforced on stock RPi OS. A taint value of 12288 (flags O+E) after load is normal and harmless.