Netboot a Pi fleet from iSCSI

I have a small fleet of Raspberry Pi 4s in a rack. They exist for one reason: to run my CI-built custom OS images on actual ARM hardware before I ship them. A QEMU build catches some bugs; only a real Pi catches the ones that matter.

The flow used to look like this:

1. CI finishes a build, drops a .img.xz artifact.
2. I dd it onto an SD card.
3. Walk to the rack, pull the Pi, swap the card, push it back.
4. Power-cycle.
5. SSH in, run the test, read the result.
6. Walk back, pull the card.
7. Repeat for the next build.

The SD slot on a rackmounted Pi is on the wrong side of the chassis to reach without unmounting, so each step 3 / step 6 is unscrew, slide out, swap, slide in, screw back. Doable once. Eight times a day, no.

What I wanted: the Pi netboots from a fresh image every time CI finishes, runs the test, and the image goes away when it’s done. No SD card involved. PI4-iSCSI-shim is what I ended up building.

Why iSCSI specifically

Raspberry Pis happily PXE-boot — the bootloader will fetch a kernel + initrd over TFTP and hand control to it. The follow-up question is what does the kernel mount as root? The standard answers:

• NFS root. Works, but the NFS quirks (locking, fsync semantics, async-vs-sync) bite once you start running real workloads. CI tests that exercise filesystem behaviour pass on local disk and fail on NFS for irrelevant reasons.
• OverlayFS over a read-only NFS share. Same NFS surface, same problems, plus a layer of cache complications.
• iSCSI. A real block device. The kernel sees it as /dev/sda, the filesystem on top is whatever you put there, and the semantics match local-disk behaviour because at block level it is local-disk behaviour.

iSCSI also matches scsipub’s primitive perfectly — every CI build is just a fresh anonymous session with the right base image. The Pi disconnects, the overlay vanishes, the next build gets a clean slate.

The shim

PI4-iSCSI-shim is a small TFTP-served bootstrap. The Pi’s on-board firmware fetches a kernel + initramfs from the TFTP server; the initramfs’s only job is to bring up the iSCSI session and pivot to the iSCSI-mounted root.

Three modes, picked in the kernel command line:

# pivot_root mode: standard initramfs → iSCSI rootfs
scsipub.mode=pivot_root scsipub.iqn=iqn.2025-01.pub.scsipub:image.pi4-debian

# kexec mode: iSCSI image carries its own kernel, kexec into it
scsipub.mode=kexec scsipub.iqn=...

# usb-disk mode: present the iSCSI target as a virtual USB disk
# to the Pi's bootloader, useful when you want unmodified boot
# tooling
scsipub.mode=usb-disk scsipub.iqn=...

The pivot_root mode is the everyday case. kexec is for OS images that ship a known-good kernel and don’t want my shim’s. usb-disk is the “I want to use the standard rpi-imager-style flow but skip the SD card” mode.

The initramfs is small (~3 MB) and is bundled directly into the TFTP root the bootloader already expects. No DHCP changes on the network side.

CI side

A paying scsipub account gets a per-tier byte budget for private images plus an API key. CI hands a build URL to scsipub, scsipub fetches + checksums + registers it as the account’s image, and the shim points the Pi at the resulting IQN. The full request/response shapes for every endpoint below are in the API reference.

build:
  script:
    - ./scripts/build-pi-image.sh > /tmp/build.img.xz
    - sha256sum /tmp/build.img.xz | cut -d' ' -f1 > /tmp/sha256
    # Publish the artifact to anywhere scsipub can GET it —
    # GitLab job artifacts, R2, S3, your own static host.
    - scripts/publish-artifact /tmp/build.img.xz > /tmp/build_url

publish:
  needs: [build]
  script:
    # Tell scsipub to fetch + register the artifact under our
    # account. Synchronous: a 201 means the image is downloaded,
    # decompressed, checksummed, and ready to mount.
    - |
      curl -fsSL -X POST https://scsipub.com/api/images \
        -H "Authorization: Bearer $SCSIPUB_API_KEY" \
        -H "Content-Type: application/json" \
        -d "{
          \"name\":   \"ci-pi-${CI_COMMIT_SHORT_SHA}\",
          \"url\":    \"$(cat /tmp/build_url)\",
          \"sha256\": \"$(cat /tmp/sha256)\"
        }" \
        | tee /tmp/image.json

test:
  needs: [publish]
  script:
    - IQN=$(jq -r .iqn /tmp/image.json)
    - ssh pi-runner "scsipub-shim-set-target $IQN"
    - ssh pi-runner "sudo systemctl restart pi-power-cycle.service"
    - ./scripts/wait-for-pi.sh pi-runner
    - ssh pi-runner "/opt/runtests.sh"

cleanup:
  needs: [test]
  when: always
  script:
    - |
      curl -fsSL -X DELETE \
        "https://scsipub.com/api/images/ci-pi-${CI_COMMIT_SHORT_SHA}" \
        -H "Authorization: Bearer $SCSIPUB_API_KEY"

Each build gets its own image record and a unique IQN, so two in-flight CI runs don’t fight over the same target. Cleanup is its own job with when: always so a failed test still releases the storage quota.

Sending expected_sha256 is optional but worth doing — if the artifact storage gets weird (CDN caches a stale build, GitLab regenerates the page) the fetch fails fast with a checksum mismatch instead of silently shipping the wrong bytes to the Pi.

Build-to-test latency dropped from ~6 minutes per round trip to ~90 seconds, and I no longer have to walk to the rack at all.

What’s not solved

• Boot-firmware updates. The Pi’s on-board EEPROM firmware is what fetches the TFTP kernel. Updating that is still a “boot from real SD with the EEPROM updater image” one-shot. Once a Pi is on the right firmware it stays there; but bringing a fresh Pi onto the netboot pipeline still costs one SD-card cycle.
• Multiple Pis on the same target. scsipub’s anonymous sessions are per-TCP-connection, but the bootloader picks the IP. If two Pis power on simultaneously and try to iSCSI-login to the same target, both succeed (each gets its own ephemeral overlay) — which is what you want for isolated CI runs but means you can’t share state between test machines via the iSCSI image. Cross-machine state needs a separate persistent target.
• Pi 5. Not tested yet. The bootloader changed enough between 4 and 5 that I’d want to confirm the TFTP path works before claiming it does.

How scsipub fell out of this

I had the shim working against a target running on my desk within an afternoon. The harder problem turned out to be proving it works — which means handing other people a target they can boot against to try the shim themselves. I wanted to write “point your Pi at this address and follow the steps,” not “first install Linux, then install targetcli, then…”

There is no public iSCSI target on the internet. None. Everyone who runs iSCSI runs it inside their rack, and the few managed offerings are part of cloud provider portfolios with account creation, billing, and ten layers of consoles. Nothing where you can just open a terminal and type iscsiadm --login.

So I built one. scsipub was supposed to be a one-week side-project so the Pi shim could have a demo target to point at; it kept growing because every small thing that makes it “actually usable on the open internet” turned out to be a small project of its own. The post on the architecture behind it is the longer story.

See also

The shim’s source, build instructions, and TFTP layout are at PI4-iSCSI-shim. The companion project ESP32-iSCSI-USB is the adjacent “I have lab equipment that wants USB mass storage but lives on a network” problem.