Skip to content

Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) - #2174

Open
tlaurion wants to merge 4 commits into
linuxboot:masterfrom
tlaurion:reproducibility-fixes-v2
Open

Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump)#2174
tlaurion wants to merge 4 commits into
linuxboot:masterfrom
tlaurion:reproducibility-fixes-v2

Conversation

@tlaurion

@tlaurion tlaurion commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Fixes the build reproducibility regression introduced by PR #2136
(commit 3dfec51: musl-cross-make bump to 227df8b), which upgraded
binutils from 2.33.1 to 2.44.

Closes: #2168

Regression

PR #2136 bumped musl-cross-make for CVE fixes, upgrading binutils from
2.33.1 to 2.44. This introduced two independent reproducibility failures:

  1. Cross-compiler (affected gpg-agent): gas 2.44 section
    padding, config.guess host kernel probing, debug section compression, and
    DATE/TIME embedding caused tools.cpio binaries to build
    non-reproducibly between CI and local.

  2. Busybox linker (affects busybox only): ld.bfd 2.44's --gc-sections uses
    ASLR-influenced hash tables (no upstream fix through 2.47). Busybox's
    scripts/trylink is the only Heads package using this flag.

Fixes

modules/musl-cross-make

  • BUILD triplet pinned to x86_64-pc-linux-gnu (stops config.guess)
  • -Wa,--no-pad-sections (gas 2.44 section-end padding)
  • --with-debug-prefix-map=$(pwd)=. (build-path normalization)
  • --enable-compressed-debug-sections=no (zlib debug compression)
  • SOURCE_DATE_EPOCH from pinned commit epoch (GCC build determinism)
  • Idempotent config.mak append (guards against reconfiguration)

modules/busybox + patches/

  • Patch scripts/trylink to disable --gc-sections when SOURCE_DATE_EPOCH set
  • SOURCE_DATE_EPOCH=0 on busybox_target
  • Direct install.sh invocation (bypasses make install FORCE re-link)

Documentation

  • doc/reproducible-builds.md: practices applied and verification procedure
  • doc/index.md: regrouped by subject, missing entries added
  • doc/modules.md: toolchain module section
  • Cross-references added from docker.md, architecture.md, build-freshness.md
  • Removed decaying file:line references across all docs
  • Module comments documenting reproducibility choices

Verification

CI and local builds produce byte-identical hashes for all artifacts
(verified across clean builds and cache-reuse rebuilds at multiple commits).

Five changes to make musl-cross-make produce deterministic output:

1. Override BUILD triplet (x86_64-pc-linux-gnu) via config.mak to
   prevent config.guess from probing the Docker host kernel.
   Docker shares the host kernel across CI runners; different
   runners produce different build-system triplets.
   No prior reproducibility-motivated BUILD pinning found in
   other musl-cross-make deployments — this is Heads-original.

2. Pass -Wa,--no-pad-sections via CFLAGS to prevent gas from
   padding section ends to alignment boundaries (gas NEWS 2.27,
   2016).  No other project uses this flag for reproducibility;
   the need is specific to gas 2.44 behavior in musl-cross-make.

3. Pass --with-debug-prefix-map=$(pwd)=. to normalize the build
   directory path embedded in the cross-compiler's debug info.
   Same technique used by Buildroot (gcc.mk, BR2_REPRODUCIBLE),
   rust-musl-cross (config.mak), Debian dpkg (-ffile-prefix-map),
   and Yocto (DEBUG_PREFIX_MAP).
   Ref: https://reproducible-builds.org/docs/build-path/

4. Pass --enable-compressed-debug-sections=no to disable zlib
   debug-section compression (non-deterministic output).
   Used by Rust CI (crosstool-ng), Microsoft Azure Linux,
   Fedora binutils.spec, Chromium OS, and Frida.
   Ref: https://reproducible-builds.org/docs/deterministic-build-systems/

5. Export SOURCE_DATE_EPOCH from the pinned commit epoch to
   prevent __DATE__/__TIME__ embedding during the GCC build.
   Follows the Buildroot fakedate pattern and the spec:
   https://reproducible-builds.org/specs/source-date-epoch/

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
busybox's scripts/trylink passes -Wl,--gc-sections to ld.bfd.
In ld.bfd 2.44 (and through 2.47), the --gc-sections mark-phase
traversal depends on the ASLR layout of the linking process.
No upstream binutils fix exists.

Per the SOURCE_DATE_EPOCH convention, its presence signals
that a reproducible build is requested:
  https://reproducible-builds.org/specs/source-date-epoch/

Changes:
- patches/busybox-1.36.1/0004-trylink-reproducible.patch:
  Patch scripts/trylink to skip -Wl,--gc-sections when
  SOURCE_DATE_EPOCH is set.
- modules/busybox: Pass SOURCE_DATE_EPOCH=0 in busybox_target.
  Replace 'make install' with direct binary copy + install.sh
  symlink generation.  busybox's install target depends on
  busybox_unstripped which is FORCE, so 'make install' can
  trigger a re-link with no SOURCE_DATE_EPOCH.  Copying the
  already-built binary and running applets/install.sh --symlinks
  avoids this entirely.

Follows the same pattern used by Buildroot's fakedate (2016):
SDE gates nondeterministic behavior off.  No prior art found for
SDE-gated --gc-sections disabling; OpenWrt offers per-package
gc-sections opt-out but for size reduction, not reproducibility.

With the musl-cross-make fix, 67 of 68 tools in tools.cpio
are byte-identical.  This fixes the last one.

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Copilot AI review requested due to automatic review settings August 2, 2026 19:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR targets build reproducibility for the EOL_t480-hotp-maximized board by eliminating known sources of non-determinism in the musl-cross-make toolchain build and the BusyBox link/install flow, aligning local builds with CI artifacts.

Changes:

  • Pin musl-cross-make build settings (BUILD triplet, assembler section padding, debug prefix mapping, debug-section compression) and propagate SOURCE_DATE_EPOCH into the toolchain build.
  • Patch BusyBox scripts/trylink to disable --gc-sections when SOURCE_DATE_EPOCH is set to avoid nondeterministic ld.bfd behavior.
  • Avoid BusyBox re-link during install by copying the built binary and running applets/install.sh directly; set SOURCE_DATE_EPOCH=0 for BusyBox builds.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
patches/busybox-1.36.1/0004-trylink-reproducible.patch Disables BusyBox --gc-sections when SOURCE_DATE_EPOCH is set to avoid nondeterministic linking.
modules/musl-cross-make Adds reproducibility-related config (pinned BUILD, no-pad-sections, debug prefix mapping, disables compressed debug sections) and exports SOURCE_DATE_EPOCH into the build.
modules/busybox Forces SOURCE_DATE_EPOCH=0 and changes install to avoid re-linking by using direct copy + applets/install.sh.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread modules/musl-cross-make Outdated
@tlaurion tlaurion changed the title Fix build reproducibility for EOL_t480-hotp-maximized (issue #2168) Fix build reproducibility regression from PR #2136 (musl-cross-make binutils bump) Aug 2, 2026
@tlaurion

tlaurion commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

Damn we need StageX.

@tlaurion
tlaurion force-pushed the reproducibility-fixes-v2 branch 2 times, most recently from ffb6cf5 to c543c64 Compare August 3, 2026 02:05
doc/reproducible-builds.md (new):
- Reproducible build practices: musl-cross-make flags, heads_cc flags,
  kernel flags, prefix overrides, busybox trylink patch, openssl patch
- hashes.txt structure: SHA256 of every file inside every cpio, cascade
  from innermost file -> cpio -> initrd.cpio.xz -> ROM
- GIT_HASH in /etc/config explains why tools.cpio differs between
  commits (documentation-only changes expected to cascade)
- Verification procedure: ROM -> payload -> cpio -> file top-down check

doc/index.md: regrouped by subject, added missing BOARDS_AND_TESTERS.md,
  circleci.md, and reproducible-builds.md entries

doc/modules.md: added Toolchain Modules section (MUSL_CROSS_ONCE guard,
  early include ordering, cross-ref to doc/circleci.md)

Cross-references added from docker.md, architecture.md, and
build-freshness.md to reproducible-builds.md

Module comments added to musl-cross-make and busybox following the
modules/gpg pattern

Removed file:line references throughout — replaced with descriptive
references that won't decay when source files change

Corrected GIT_HASH documentation (git rev-parse HEAD, not git log --oneline)
and --prefix override (modules use --prefix, not --exec-prefix)

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
@tlaurion
tlaurion force-pushed the reproducibility-fixes-v2 branch from c543c64 to 9648246 Compare August 3, 2026 02:07
Three download improvements:

- GNU_SITE = https://ftp.gnu.org/gnu overrides the unreliable
  ftpmirror.gnu.org (frequently returns 502 Bad Gateway).

- DL_CMD = bin/musl-mirror-fetch.sh provides a multi-mirror download
  wrapper that tries ftp.gnu.org, mirrors.kernel.org, and
  mirror.math.princeton.edu in order for each GNU tarball.

- HEADS_DISABLE_USB=1 + script -qec enables unattended docker builds
  (HEADS_DISABLE_USB=1 skips USB cleanup needing sudo, script -qec
  provides a pseudo-TTY for docker_repro.sh's -ti flag).

No SOURCES = $(packages) override — packages/x86/ does not yet contain
the component tarballs (gcc, binutils, gmp, mpc, mpfr, etc.).
Add back once packages/ is seeded and Purism mirrors are populated.

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
@tlaurion
tlaurion force-pushed the reproducibility-fixes-v2 branch from 9648246 to 1f6688d Compare August 3, 2026 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mismatch between hashes of local build and Circle public build of bin/gpg-agent

2 participants