Documentation › UPSTREAM_TOOLS

Official BusyBox and Storage Tools

Status and Scope

The production build now packages unmodified upstream BusyBox, util-linux, e2fsprogs, dosfstools and exfatprogs. Former private storage commands are no longer dispatched as BusyBox applets. The project-specific boot-payload copier is a separate script, explicitly not GNU grub-install.

Both production root targets have built successfully:

• out/x86_64/release/images/root.ext2: normal Live root.
• out/x86_64/release/images/installer-root.ext2: ext2 root containing the installer runtime root and the installed-system payload at /install/root (historically named root.fat).

Actual embedded tool/library bytes and compatibility links are checked with debugfs, not inferred from staging success. The seven previously failing guest checks now pass, with the expanded suite reporting 93 passing checks and zero failures for each root. This includes formatting and checking a disposable QEMU AHCI partition, not a host disk. Ordinary and installer ISOs are generated by run images-iso; VMDK and a complete graphical installation were not tested in this round. VMware remains unverified.

The earlier kernel compilation failure was resolved by replacing an undeclared memcpy with __builtin_memcpy, matching existing local code. The kernel and root-image builds now pass that compilation gate. Broader Linux ABI work remains paused. The existing storage change stops logging pending asynchronous DMA (-EAGAIN) as a device error and retains real errors; its host regression passes. This does not establish VMware transport stability.

Sources and Linkage

BusyBox

• Upstream BusyBox, tag 1_36_1, commit 1a64f6a20aaf6ea4dbba68bbfa8cc1ab7e5c57c4.
• The production recipe passes --official-source, exports committed bytes using git archive, and verifies the export before and after make. Dirty checkout files do not enter the official build; injected or changed cache files fail verification. No upstream source file is patched.
• The selected revision is the checkout's HEAD, recorded in the stamp; changing HEAD changes the input. This is not an independent signature-authentication mechanism. Build time is pinned to that commit via SOURCE_DATE_EPOCH and UTC, so separate root builds no longer differ only in BusyBox's timestamp banner.
• Private CONFIG_LEONOS_* and CONFIG_EXTRA_LDLIBS are filtered. Official mode neither injects private SDK headers nor links libleonos.a. The legacy adapted builder branch and old private source files remain in the repository, but the standard graph does not use them.
• BusyBox is static against the existing project musl runtime and its mimalloc startup object. Unmodified application sources do not imply an unmodified C runtime. This task does not replace musl or the allocator.
• build/userland/busybox.stamp records source, config, build flags, applets and ELF SHA256. Its conservative guest_verified: false is not promoted by a limited shell-pipeline test.
• Official ln, tar creation/GNU extensions, gzip, gunzip, zcat, and gzip decompression support are enabled in the production configuration. Their sources remain unmodified; tmpfs copy/link/rename/archive round trips are covered by the guest probe.

util-linux

• Official util-linux 2.41.6 archive.
• SHA256: e596083744e746be7d2823b62b43f4418dd7bf56303b4dc09e6fe8112fe3d7ed.
• --enable-fdisks=check builds fdisk and sfdisk; cfdisk is skipped without a terminal library. The profile also enables mount/umount, blkid, lsblk, fsck, libmount, libblkid, libfdisk, libsmartcols, libuuid, su and runuser.
• Builds are out-of-tree. Source verification was not relaxed for generated files. An earlier exploratory in-source config.log was moved to build/auth-upstream/diagnostic-source-config.log, outside the verified tree.
• 2.41.6's hook_idmap.c uses RESOLVE_NO_SYMLINKS without its defining header. CPPFLAGS=-include linux/openat2.h supplies the official target Linux UAPI definitions to all translation units without patching source or disabling idmapped mounts. The actual constant comes from Linux headers, not a private replacement header.
• The compiler places target musl libraries before libtool's install-time /usr/lib search. Tests inspect the installed ELFs to reject host glibc and private-library dependencies, not just the pre-install executables.
• Executables use /lib/ld-musl-x86_64.so.1; shared libraries live in /usr/lib. Build metadata: build/auth-upstream/util-linux-build.json.

Filesystem Tools

Pinned official archives, SHA256 values and checksum provenance are in configs/storage-upstream.json:

The builder verifies archives and extracted source trees, configures out-of-tree, and checks source integrity again after installation. No source patches apply. The six core tools are static target ELFs, using target util-linux UUID/blkid archives where needed. exfatprogs uses libtool -all-static and an explicit libblkid.a path to avoid installed .la host paths. Package licenses are staged under /usr/share/licenses; build commands and hashes are recorded under build/storage-upstream/, including root/.storage-package.json.

Command and Rootfs Ownership

tools/storage_tools.py declares package outputs and old /sbin aliases. tools/build/rootfs-stage.sh stages the layout and its compatibility links after the producers, replacing stale BusyBox storage links. mkfs.vfat/mkfs.fat32 resolve to mkfs.fat; fsck.vfat/fsck.fat32 resolve to fsck.fat. These are path aliases, not translations of the private CLI. Use upstream options: in particular FAT32 requires -F 32, including when using the mkfs.fat32 alias; dosfstools otherwise selects FAT width by volume size. The former private --force syntax is not emulated.

ChenPi cmd resolves storage commands directly to those external paths. /bin, /sbin, /lib, /usr/bin, /usr/sbin remain real separate directories; this is not a usr-merge. Normal, installer-runtime and installed-payload roots consume the same staging tree. Repeated storage staging replaces owned files before copying, preserving e2fsprogs' read-only archive modes without failing on the second build.

Before copying util-linux, staging checks every required command and library against the package root. A missing library or a link escaping that root fails before copying; an old staging library cannot satisfy the check.

Upstream BusyBox power commands normally signal PID 1. LeonOS init blocks and synchronously receives SIGUSR1 (halt), SIGUSR2 (poweroff) and SIGTERM (reboot), calls sync and the real reboot interface, and stays alive if that call fails. It also waits for SIGCHLD and reaps exited children. This preserves unmodified BusyBox command behavior without converting commands to forced reboot wrappers. This is not a service shutdown supervisor: orderly termination of every service before power transition remains unimplemented.

The boot copier validates its input files and destination directory, then copies EFI/BOOT/BOOTX64.EFI, loader.elf, leonos/kernel.sys and the complete grub directory. Default source is /install/esp; --source DIR supports an explicit prebuilt payload. It does not generate GRUB, format/mount an ESP, install boot sectors, or update NVRAM. The supplied destination must already be the intended mounted ESP. Tests cover spaces, exact bytes, absent destination, missing/partial source and nonzero failure reporting.

The graphical/TTY installer still uses its existing native shared block/format/ mount implementation. This task does not redesign or validate that workflow.

Verification

make iso
make installer
python3 tools/test_auth_source_integrity.py -v
python3 tools/test_storage_payload.py -v
python3 tools/test_upstream_tools_runtime.py -v
python3 tools/test_storage_upstream_runtime.py -v
LEONOS_STORAGE_TEST_ROOT=out/x86_64/release/stage/esp LEONOS_UPSTREAM_TEST_ROOT=out/x86_64/release/stage/esp \
  python3 tools/test_storage_upstream_runtime.py -v
python3 tools/test_storage_upstream_guest.py --root out/x86_64/release/images/installer-root.ext2 --smp 2
python3 tools/test_storage_upstream_guest.py --root out/x86_64/release/images/root.ext2 --smp 1
python3 tools/test_upstream_tools_images.py -v
python3 tools/test_regular_file_io.py
python3 tools/test_tmpfs.py
python3 tools/test_linux_memory.py
python3 tools/test_linux_threads.py
python3 tools/test_storage_mkdir_mount.py
python3 tools/test_storage_rename.py
python3 tools/test_storage_metadata.py
python3 tools/test_init_power.py
python3 tools/test_storage_upstream_guest.py --root out/x86_64/release/images/root.ext2 --power reboot
python3 tools/test_storage_upstream_guest.py --root out/x86_64/release/images/root.ext2 --power poweroff

Host reference tests execute the actual target ELFs with explicit target library search paths. They create GPT and ext2/FAT32/exFAT only on disposable regular files; checkers must reject unformatted files. Host mount tests are version and read-only listing tests. None of these tests writes a host block device.

Image tests compare 20 embedded commands/libraries at each of the three root locations and verify the declared compatibility links. The combined build log for this repair is build/storage-tools-build.log. Artifacts: build/images/leonos4.iso and build/images/leonos4-installer.iso. ISO and embedded payload verification is recorded in build/storage-tools-iso-evidence.json; obsolete hashes from the pre-fix images must not be used to identify this build.

Guest probes copy a production root, add only a test executable at the existing inventory autospawn slot, then build a diagnostic ISO. They do not patch the kernel or replace production tools. QEMU has no host block devices attached. The runner records base-image SHA256, invocation, complete serial output and failures under build/storage-upstream-guest/{install-smp2,live-smp1}/ and returns nonzero for failures; passing individual commands does not set package-wide acceptance.

The dedicated power probes pass with the rebuilt Live root: --power reboot receives SHUTDOWN reason=guest-reset, and --power poweroff receives SHUTDOWN reason=guest-shutdown. Both serial logs record the expected PID 1 reboot(2) command. Evidence is under the live-reboot and live-poweroff subdirectories. The host init harness separately covers halt/reboot/poweroff signal mappings, child reaping and failed reboot without invoking host power.

The inventory autospawn slot can execute before init finishes runtime startup. Power probes wait for the power signals to be blocked in /proc/1/stat before executing BusyBox. An earlier immediate request terminated PID 1 with SIGTERM: the kernel's missing early PID 1 default-signal protection remains a limitation. These passing post-initialization tests do not certify that early-boot case.

Repaired Guest Failures (2026-09-13)

The expanded 93-check suite also covers 1 MiB scalar/positional transfers, shared descriptor offsets, held-unlinked files, non-sector-aligned block I/O, end-of-device partial transfers, fsync/fdatasync, tmpfs permissions/quotas, read-only remount and official BusyBox file/archive workflows. The original 27 checks and their expectations remain present.

Tmpfs mmap now maps the inode's physical pages. Shared mappings and read/write observe the same bytes; private mappings use COW. Fork and partial munmap retain and release references. Truncate revokes shared and private PTEs before freeing pages, clears the final page tail, and repeated growth uses current inode size. Out-of-file and quota-exhausted page faults deliver SIGBUS/BUS_ADRERR. The 18-case portable mapping test passes on host Linux and in the guest, with nine additional guest checks for quota and mount ownership. msync implements tmpfs and private-map behavior, including zero flags/length and locked INVALIDATE rejection, without pretending that a separate cached copy was written back.

Reference: fixed Linux v6.12 mm/shmem.c (shmem_fault, shmem_setattr), mm/msync.c, fs/read_write.c, and block/fops.c under build/linux-6.12. Host tests exercise the real tmpfs/I/O/page/signal implementations with ASan/UBSan; interrupted partial I/O drains pending DMA before signal delivery. Existing storage mkdir/rename/metadata harnesses now link the new backend.

Remaining Limits

• The 2-vCPU QEMU configuration is a boot/scheduling smoke test. Production still has SMP_USER_SCHEDULER_ENABLED=0, so this is not concurrent multicore user-execution validation. TLB notifications reuse the existing core barrier; no NUMA or multi-socket work was added.
• Disk-file shared mmap writeback, unfaulted shared-anonymous pages across fork, complete mmap flags, tmpfs swapping/huge pages/xattrs/seals/FIFO/device nodes, quota-changing remount options, bind mounts and lazy unmount remain outside this repair. Native path/name and mount-count limits still apply.
• Read-only remount tracks live task descriptors and VMAs; Linux's full superblock writer accounting, including descriptors only in SCM queues, remains unverified.
• No LTP run, VMware verification, full graphical installer workflow or all-applet certification is claimed. Existing early PID 1 signal protection and orderly shutdown limitations above remain.
• test_linux_permissions.py passes the native permission checks but its old ACL stack harness still includes the removed leonos/auth_db.h; the aggregate command fails before that obsolete fixture compiles. Its expectations were not removed or weakened. Guest native tmpfs ownership/access checks pass.