From 6a83406e88047f19a6e3c95587c1a5b4d34afad2 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 11:32:27 +0800 Subject: [PATCH 01/11] docs: dep build-cache scoping design (global cache is currently a no-op) --- ...26-07-30-dep-build-cache-scoping-design.md | 539 ++++++++++++++++++ 1 file changed, 539 insertions(+) create mode 100644 .agents/docs/2026-07-30-dep-build-cache-scoping-design.md diff --git a/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md new file mode 100644 index 00000000..de2e249a --- /dev/null +++ b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md @@ -0,0 +1,539 @@ +# 依赖构建产物的全局缓存收敛 — 设计 + +日期：2026-07-30 +状态：设计定稿，待实施 +分支/worktree：`worktree-build-cache-design` +关联：本仓库 `.agents/docs/2026-07-30-issue311-bmi-staging-and-cache-root-design.md`（**前置依赖**） +建议目标版本：阶段 1 → issue311（2026.7.30.1）之后的下一版；阶段 2 紧随 + +--- + +## 0. 一句话结论 + +mcpp **已经有**一套全局依赖 BMI/对象缓存（`$MCPP_HOME/bmi//deps/...` + `mcpp cache +list|info|prune|clean`），但它当前**净收益为零**： + +1. 缓存键用的是**全工程指纹**，任何工程只要包名/版本/无关依赖/flag 不同就完全不共享； +2. 即使命中，产物被拷进 build dir 后 **ninja 仍然全部重编**（`command line not found in + log`）——所以 CLI 打印的 `Cached ` 是**假的**； +3. 顺带一个已经存在但被 (2) 掩盖住的**正确性缺陷**：`--profile` 不进缓存键。一旦修好 (2)， + `mcpp build --release` 会直接吃到 `-O0 -g` 的依赖对象。 + +本设计把缓存键从「全工程指纹」收敛为「**每包 Merkle 键**」，把命中路径从「拷贝后祈祷 ninja +不重编」改为「**发 stage 边而不发 compile 边**」，并按硬性顺序先补齐正确性前置。 + +--- + +## 1. 现状机制（逐段核过的链条） + +### 1.1 缓存已经存在，且已接入构建 + +| 环节 | 位置 | +|---|---| +| 缓存条目布局与读写 | `src/bmi_cache.cppm`（`CacheKey` / `is_cached` / `stage_into` / `populate_from`） | +| 命中判定 + staging（prepare 阶段） | `src/build/prepare.cppm:3725-3831` | +| 回填（build 成功之后） | `src/build/execute.cppm:300-308`、`:951` | +| std BMI 缓存 | `src/toolchain/stdmod.cppm:189-303`（`ensure_built`，键 = `/`） | +| 运维命令 | `src/bmi_cache/maintenance.cppm` → `mcpp cache list/info/prune/clean` | +| 全量清理 | `src/build/execute.cppm:1118`（`mcpp clean --bmi-cache`） | + +缓存根目录：`mcpp::toolchain::default_cache_root()` = `$MCPP_HOME/bmi` +（该函数自身的三份拷贝问题由 issue311 的 S2 收敛，本设计直接复用其 `mcpp::home::bmi_root()`）。 + +### 1.2 缓存键 = 全工程指纹 + +```cpp +// src/build/prepare.cppm:3788-3798 +mcpp::bmi_cache::CacheKey key { + .mcppHome = (*cfg2)->mcppHome, + .fingerprint = fp.hex, // ← 全工程指纹 + .indexName = depIdent ? depIdent->indexName : (*cfg2)->defaultIndex, + .packageName = depName, + .version = depVer, + ... +}; +``` + +而 `fp.hex` 的第 7 个字段（`compileFlags`）是 +`canonical_compile_flags(*m) + canonical_package_build_metadata(packages)` +（`prepare.cppm:3587-3588`），后者遍历 **`packages` 全表，含 `packages[0]` = root 工程本身** +（`prepare.cppm:1363` / `2216` 建立，`267-334` 序列化 `name`/`version`/flags/include dirs/ +generated_files）。 + +std BMI 同样按 `fp.hex` 分目录（`stdmod.cppm:204`：`cache_root / fingerprint_hex`）。 + +### 1.3 命中之后 ninja 并不认账 + +`stage_into`（`bmi_cache.cppm:139-181`）把缓存里的 `.gcm/.pcm` 与 `.o` 拷进 +`target///{gcm.cache,obj}/`，并 `touch_now`。但 ninja 判脏不只看 mtime： +`build.ninja` 里这些文件仍然是 **compile 边的输出**，而 ninja 对「输出存在但 `.ninja_log` +里没有该输出的命令行记录」的判定是 **dirty**。新 build dir 天然没有 `.ninja_log`，于是 +全部重编。 + +--- + +## 2. 实测证据 + +全部在本机跑过，命令与输出可复现。 + +### E1 — 全局缓存命中却 100% 重编（决定性） + +``` +$ mcpp build # cachetest: 依赖 compat.zlib@1.3.2 + Compiling cachetest v0.1.0 (.) + Cached compat.zlib v1.3.2 ← mcpp 说命中 +[3/19] ... gcc ... -c .../zlib-1.3.2/gzclose.c -o obj/gzclose.o +[4/19] ... gcc ... -c .../zlib-1.3.2/uncompr.c -o obj/uncompr.o + ...（16 个 .c 全部重编） +``` + +ninja 自己的解释： + +``` +$ ninja -C target/x86_64-linux-gnu/ -d explain -n +ninja explain: command line not found in log for obj/zutil.o +ninja explain: obj/zutil.o is dirty +ninja explain: command line not found in log for obj/uncompr.o +ninja explain: obj/uncompr.o is dirty + ... +``` + +⇒ 缓存只在「同一个 build dir 已经有 `.ninja_log`」时"命中"，而那时本地产物本来就在。 +**全局缓存的净收益 = 0，只剩拷贝开销与磁盘占用，外加一条假的 `Cached` 状态行。** + +### E2 — 改自己的版本号就换指纹（缓存键过宽） + +``` +[package] name = "demoa", version = "0.1.0" → Fingerprint: 3b8a8ae4fc217233 + version = "0.1.1" → Fingerprint: 2138d7ce160e1154 + name = "demob" → Fingerprint: bdd6c2981d862423 +``` + +⇒ `mcpp version bump` / 改项目名 / 加一个无关依赖 ⇒ std BMI 与**全部**依赖缓存整体失效。 +两个工程只要包名不同，即使依赖集与工具链完全一致，也**一条都不共享**。 + +### E3 — 本机 26 GB 缓存的重复度 + +``` +$ du -sh ~/.mcpp/bmi → 26G +指纹目录数 → 1198 +``` + +**std 侧**（按 `std-module.json` 的 15 个身份字段归一后统计）： + +``` +std-module.json 目录数: 1014, distinct std 身份: 15, 合计 16.10 GB + 529 份 7.77 GB gcc@16.1.0 c++23 libstdc++ x86_64-linux-gnu -std=c++23 + 243 份 3.81 GB gcc@16.1.0 c++26 libstdc++ x86_64-linux-gnu -std=c++26 + 103 份 3.69 GB clang@22.1.8 c++26 libc++ x86_64-unknown-linux-gnu -std=c++26 + 71 份 0.06 GB gcc@15.1.0 c++23 ... + （其余 11 个身份合计 < 0.8 GB） +``` + +⇒ **1014 份，只有 15 个真身份。16.10 GB → ~0.5 GB（约 97% 可省）。** + +**依赖侧**： + +``` +dep 条目数 3937, distinct pkg@ver 113, 合计 8.80 GB +按 (工具链身份, pkg@ver) 归一 → 221 条, 0.52 GB （94.1% 可省） + 162 份 compat.zlib@1.3.2 + 107 份 compat.zstd@1.5.7 + 73 份 compat.x11@1.8.13 （单包合计 1.486 GB） + 63 份 mcpplibs/ftxui@6.1.9（单包合计 1.490 GB） +``` + +⇒ 合计 **26 GB → ~1 GB**。 + +### E4 — 重复的产物在语义上就是同一个（且为什么内容不相同） + +同一个 `compat.zlib` 的 `adler32.o` 在两个指纹目录下字节不同（7432 vs 7448）。唯一差异： + +``` +DW_AT_producer : GNU C11 16.1.0 -mtune=generic -march=x86-64 -g -O0 -std=c11 -fPIC （两侧完全一致） +DW_AT_name : /home/speak/.mcpp/registry/data/xpkgs/compat-x-compat.zlib/1.3.2/... （两侧完全一致） +DW_AT_comp_dir : .../scratchpad/be/tests/examples/eui-neo/target/x86_64-linux-gnu/01c3e0484b536e72 +DW_AT_comp_dir : .../scratchpad/w133/tests/examples/eui-gui-vk-glfw/target/x86_64-linux-gnu/0659ceea0b4ded58 +``` + +⇒ 语义完全等价，差异 100% 来自**消费者 build dir 的绝对路径**被烙进了 debug info。 +两个推论：(a) 内容寻址去重在今天做不了；(b) 缓存复用后 debug 信息指向另一个工程的目录。 + +### E5 — profile 不进指纹（正确性） + +``` +$ mcpp build --dev --print-fingerprint → Fingerprint: bdd6c2981d862423 +$ mcpp build --release --print-fingerprint → Fingerprint: bdd6c2981d862423 +$ mcpp build --profile dist --print-fingerprint → Fingerprint: bdd6c2981d862423 +``` + +`prepare.cppm:799-802` 把 profile 落到 `buildConfig.optLevel/debug/lto/strip`，而 +`canonical_compile_flags`（`prepare.cppm:209-265`）**只序列化 cflags/cxxflags/ldflags， +不序列化这四个字段**。三个 profile 因此共用同一个 build dir 与同一条缓存条目。 + +今天没出事，靠的是 ninja 的命令哈希在本地重编。**一旦修好 E1，`--release` 就会直接 stage +进 `-O0 -g` 的依赖对象。** 这决定了实施顺序（§5）。 + +### E6 — 传递 path 依赖被错误地写进全局缓存 + +`skipCache` 谓词（`prepare.cppm:3774-3785`）只在**root manifest** 的 +`dependencies`/`devDependencies` 里查 `isPath()/isGit()`。传递依赖查不到 ⇒ +`specIt == end()` ⇒ `skipCache = false` ⇒ 照样缓存。 + +``` +root(rootp) → A{path} → B{path} +$ mcpp build +$ find ~/.mcpp/bmi//deps -mindepth 2 -maxdepth 2 -type d +/home/speak/.mcpp/bmi/c9a9bded8f95af7e/deps/mcpplibs/B@0.1.0 ← B 被缓存了 +``` + +且 `indexName` 回落到 `defaultIndex`，本地包被**误挂到 `mcpplibs` 索引名下**。 + +接着改 `B/src/B.cppm` 的函数体、不动版本号： + +``` +$ mcpp build --print-fingerprint → Fingerprint: c9a9bded8f95af7e （不变） +$ ls -l ~/.mcpp/bmi/c9a9bded8f95af7e/deps/mcpplibs/B@0.1.0/obj/ +-rw-rw-r-- 1 speak speak 3128 ... B.m.o （旧对象仍在） +``` + +⇒ 同样被 E1 掩盖着。**修好 E1 之后这是静默错误产物。** + +本机真实缓存里已经躺着这类条目：`mcpp cache list` 输出中含 `mcpplibs/B@0.1.0` +（正是上面这个实验造出来的）以及一批 `mcpplibs/@0.1.0` 形态的条目 —— `@0.1.0` + +挂在 `mcpplibs` 名下，与本节的误判路径吻合。 + +### E7 — 用户 flag 到底影响不影响依赖（跨工程共享成立的前提） + +``` +[build] cxxflags = ["-DROOT_ONLY_FLAG=1"] + cflags = ["-DROOT_C_FLAG=1"] +``` + +``` +root TU : g++ ... -std=c++23 -fmodules -O0 -g --sysroot=... -B... -DROOT_ONLY_FLAG=1 -c src/main.cpp +dep TU : gcc ... -std=c11 -O0 -g --sysroot=... -B... -D_GNU_SOURCE -include mcpp_zlib_config.h -c .../adler32.c +``` + +⇒ **root 的 `[build]` cflags/cxxflags 不下发到依赖。** 依赖只吃：自己的 +`buildConfig`（含 feature 折叠出来的 `-DMCPP_FEATURE_*` 与 per-glob flags）、 +工程级 profile（`-O0 -g`）、工具链级 flag（`--sysroot`/`-B`/standard）。 + +这正是「不同工程引用同一个 index 包，产物应当一致」成立的机制依据。**会**影响依赖产物的轴 +只有：工具链身份、target triple、C++/C standard 与方言（含 `c++fly`）、macOS deployment +target、**profile**、该包自身激活的 features。这些全部必须进键，且都是**应该**影响的。 + +### E8 — 顺带发现的两处小问题 + +- `mcpp cache prune --older-than` 用 `last_write_time(entry.dir)`，而 `stage_into` 命中时 + **不更新缓存目录的时间**（只写消费者 build dir）。所以它是 "last **populated**"，不是 LRU： + 一个天天命中的热包会被当成冷条目剪掉。 +- `cache_clean()`（`maintenance.cppm:167-175`）第一行 `remove_all(bmi / "deps")` 指向一个 + 不存在的路径（deps 在 `bmi//deps`），是死代码；真正生效的是后面的循环。 + +--- + +## 3. 缺陷编号汇总 + +| 编号 | 缺陷 | 层级 | 证据 | +|---|---|---|---| +| **C1** | 命中的依赖产物 ninja 一律重编（`command line not found in log`），全局缓存净收益为零，且 UI 谎报 `Cached` | 构建后端 | E1 | +| **C2** | 缓存键 = 全工程指纹（含 root 包名/版本、全图 flags）⇒ 跨工程零共享、改自己版本号即全失效 | 键 | E2 E3 | +| **C3** | profile（`-O`/`-g`/lto/strip）不进指纹 ⇒ dev/release/dist 共用缓存条目 | 正确性 | E5 | +| **C4** | 传递 path/git 依赖被写进全局缓存，且源码变更不改键 | 正确性 | E6 | +| **C5** | 缓存条目无自描述元数据（只有 `manifest.txt` 文件清单），`is_cached` 是"存在即命中"，无法校验、无法审计 | 键 | 代码 | +| **C6** | 依赖对象里烙进消费者 build dir 路径（`DW_AT_comp_dir`）⇒ 无法内容寻址去重、debug 信息指向别的工程 | 可复现性 | E4 | +| **C7** | 整个 `MCPP_VERSION` 进指纹 ⇒ 每次 mcpp 发版把全部缓存作废（含纯 C 目标文件） | 键 | 代码 + E3 的 1198 目录 | +| **C8** | `cache prune` 是 last-populated 而非 LRU；无容量上限；`cache clean` 有一行死代码；无 `cache dir` | 运维 | E8 | + +C3/C4 与 C1 的关系是硬约束：**C1 修好之前它们是无害的浪费，修好之后是静默错误产物。** + +--- + +## 4. 设计 + +### S1 — `PackageBuildKey`：每包 Merkle 键（治 C2 / C3 / C5） + +新建叶模块 `src/build/cache_key.cppm`（`export module mcpp.build.cache_key;`），从 +`BuildPlan` + `PackageRoot` 计算每个包**自己**的键。`CompileUnit` 已经带 +`packageName` 与 per-package 的 `packageCflags/packageCxxflags/localIncludeDirs` +（`plan.cppm:20-36`），**每包粒度的数据已经在 plan 里了**，不需要新的解析。 + +键的输入，按轴分组（序列化格式固定、有序、带字段前缀，与 `canonical_*` 现有风格一致）： + +| 轴 | 字段 | +|---|---| +| A 工具链 | `compiler_id`, `compiler_version`, `driver_identity`, `target_triple`, `stdlib_id`, `stdlib_version` | +| B 语言/方言 | `cpp_standard`, `std_flag`, `dialect_flags`（含 `c++fly` 解析结果）, `c_standard`, `macos_deployment_target` | +| C profile ← **C3 的修复** | `opt_level`, `debug`, `lto`, `strip` | +| D 包身份 | `index_name`, `package_fqn`, `package_version` | +| E 该包自身的有效构建配置 | `sorted(active_features)`, `cflags`, `cxxflags`, `asmflags`, `ldflags`, `globFlags`（有序全量，同 `prepare.cppm:301-307`）, `defines`, `generated_files(path=content)`, `include_dirs`（**相对 xpkgs store 根归一后**）, `sources`（相对包根、有序） | +| F 上游接口闭包 | `sorted( 每个直接依赖 → 它的 PackageBuildKey )`，递归 ⇒ Merkle | +| G BMI 纪元 | `mcpp_bmi_epoch`（见 S5，**不是**完整 `MCPP_VERSION`） | + +**不含**：root 包名/版本、root 的 `[build]` flags（E7 已证不下发）、图里无关的兄弟依赖、 +消费者的 build dir 路径。 + +F 轴用递归键而不是「上游的 public include dirs + defines」，理由：上游 flag 变化会改变它 +导出的接口（宏、生成文件、ABI），逐项枚举必然漏。递归键是保守且完备的上界；代价是上游 +换 profile 会级联下游全部 key —— 这是正确的（profile 本来就是全图轴）。 + +拓扑序已由 `report.topoOrder`（`prepare.cppm:3561`）给出，自底向上一次遍历即可算完全图键。 + +### S2 — 缓存布局与自描述条目（治 C5） + +``` +$MCPP_HOME/cache/v1/ + pkg//@// + entry.json ← 自描述：键的全部输入 + 文件清单 + created/accessed + bmi/.{gcm,pcm} + obj/.o + std// + std-module.json （沿用现有 14+1 字段元数据） + {gcm,pcm}.cache/std*.{gcm,pcm} + std*.o +``` + +`entry.json` 是把 std 侧已经做对的事（`stdmod.cppm:88-145` 的 `metadata_for` + +`metadata_matches`）搬到依赖侧：**命中判定从「manifest.txt 存在且文件都在」升级为 +「键匹配 ∧ `entry.json` 的输入逐字段等于本次算出的输入 ∧ 清单文件齐全」**。 + +`v1/` 前缀 + 新目录名意味着老的 `$MCPP_HOME/bmi//` 一次性弃用。**不自动删**： +`mcpp doctor` 报一行 "legacy BMI cache at ~/.mcpp/bmi (26.0 GiB), safe to delete: +`mcpp cache clean --legacy`"。理由：E1 证明这 26 GB 从来没产生过收益，没有任何保留价值， +但删 26 GB 必须是用户的显式动作。 + +### S3 — std BMI 换键（治 C2 的 std 侧，收益最大、改动最小） + +`stdmod.cppm:204`： + +```cpp +- sm.cacheDir = cache_root / std::string(fingerprint_hex); ++ sm.cacheDir = cache_root / "std" / std_identity_key(metadata); +``` + +`metadata_for(...)` 已经是**正确且完整**的 std 身份（compiler/version/driver_identity/ +triple/stdlib/std_flag/源文件 hash/构建命令），`metadata_matches` 已经在按它校验。今天唯一 +的错误就是**目录名用的不是它**。 + +**实现陷阱（必须处理）**：`metadata["std_build_commands"]` 里含 `sm.cacheDir` 的绝对路径， +而 `cacheDir` 又要由这个 metadata 算出来 ⇒ 自指。算键前必须把 cacheDir 出现处替换成占位符 +``（§2 的 E3 统计脚本正是这么归一才得出「15 个身份」的）。`ensure_built` 的签名 +里 `fingerprint_hex` 参数随之删除 —— 它是唯一的调用点（`prepare.cppm:3612`）。 + +预期效果：1014 目录 / 16.10 GB → 15 条 / ~0.5 GB。`mcpp version bump` 不再触发 std 重编 +（当前约 10–60 s + 31 MB 写盘）。 + +### S4 — 命中的依赖发 stage 边，而不是 compile 边（治 C1） + +`CompileUnit` 加一个字段： + +```cpp +bool servedFromCache = false; // 命中全局缓存，产物由 stage 边提供 +std::filesystem::path cachedObject; // 缓存内绝对路径 +std::filesystem::path cachedBmi; // 缓存内绝对路径（有模块时） +``` + +`ninja_backend` 对 `servedFromCache` 的 CU： + +- **不发** compile 边（`ninja_backend.cppm:913-1010`）； +- **不发** P1689 scan / dyndep 边（`:850-905`）—— 依赖的模块不需要重新扫，BMI 已在缓存里； +- **改发** 每产物一条 `stage_file` 边，`$in` = 缓存内绝对路径，`$out` = build dir 内既有位置： + ``` + build gcm.cache/.gcm : stage_file /bmi/.gcm + build obj/.o : stage_file /obj/.o + ``` + +这样 build dir 的**产物位置、链接行、消费者 TU 的 implicit input 全部不变**，改变的只是 +「这些文件由谁产生」。ninja 拿到 stage 边的命令行记录，判脏语义从此自洽。 + +**直接复用 issue311 的 `mcpp stage` 原语 + `restat = 1` + rule 更名 `cp_bmi → stage_file`**： +它已经定义了「size/hash 相等则一个字节都不写、尽力对齐 mtime、原子 rename、退避重试、 +结构化诊断」。有了 `restat = 1`，第二次起 stage 边是 no-op 且**不级联下游重编**。这就是把 +本设计排在 issue311 之后的原因 —— 两者需要的是同一个原语。 + +`plan.compileUnits` **保留** 这些 CU（只是打了标记），因此 +`compile_commands.json`（`compile_commands.cppm:127-170`）仍然为依赖发条目，clangd/IDE +不退化。CDB 生成器忽略 `servedFromCache`。 + +`ctx.cachedDepLabels` 的 `Cached` 状态行从此变成真话；并且应该改成打印**省下的 TU 数**， +例如 `Cached compat.zlib v1.3.2 (16 units)` —— 假状态行能骗人三个月，带计数就骗不了。 + +### S5 — 收窄失效轴（治 C7） + +`fingerprint.cppm` 的 `fp.parts[7] = MCPP_VERSION` 保持不变（build dir 命名与本地增量归它 +管，宁可保守）。但**缓存键的 G 轴换成 `mcpp_bmi_epoch`**：一个手动递增的整型常量，只在 +BMI 编码/staging 语义/键计算方式发生不兼容变化时 +1，与发版号解耦。 + +```cpp +// src/build/cache_key.cppm +inline constexpr int kCacheEpoch = 1; // bump ONLY on cache-incompatible changes +``` + +同时进 `check_version_pins.sh` 的机器校验清单？**不进** —— 它不是 pin，不存在跨文件不变量， +只需在 `entry.json` 里记下 epoch，读到不同 epoch 的条目当 miss 处理即可（自愈）。 + +### S6 — 三种构建模式（用户面） + +一个正交开关承载三态： + +``` +mcpp build # = --cache=global（默认） +mcpp build --cache=local # 依赖全部在本工程 target/ 内编，不读不写全局缓存 +mcpp build --cache=off # 清空 target/ 全量重编，且不读不写全局缓存 +mcpp build --no-cache # deprecated alias of --cache=off（help 文案修正） +``` + +优先级：CLI `--cache` > 环境变量 `MCPP_BUILD_CACHE` > `[build] cache = "global|local|off"` +> 默认 `global`。`mcpp run` / `mcpp test` 透传同一开关（`run`/`test` 目前连 `--no-cache` +都没有 —— 顺手补齐，`cmd_build.cppm:43` 是唯一读取点）。 + +语义表： + +| 模式 | 读全局缓存 | 写全局缓存 | 先清 `target/` | +|---|---|---|---| +| `global`（默认） | 是 | 是 | 否 | +| `local` | 否 | 否 | 否 | +| `off` | 否 | 否 | **是** | + +`local` 存在的意义：排障时把「是不是缓存的问题」一次性排除掉，以及给 CI 一个可复现的 +无共享状态基线。`off` 保留今天 `--no-cache` 的全部行为。 + +### S7 — 运维命令补齐（治 C8） + +`mcpp cache` 已经是正确的归属（不需要放到 `mcpp self` 下）。新增/修正： + +| 命令 | 说明 | +|---|---| +| `mcpp cache dir` | 打印缓存根绝对路径。今天 `cache *`/`doctor`/`clean --bmi-cache` 用 `default_cache_root()` 而 `config.cppm` 的重置路径用 `cfg.bmiCacheDir`，两者可能不是同一个目录（issue311 D2）；有了这条命令，"到底在哪"永远可验证 | +| `mcpp cache gc [--max-size {MiB,GiB}] [--older-than {s,m,h,d}]` | 真 LRU。前提：**`stage_into` 命中时必须 touch 条目**（写 `entry.json` 的 `accessed` 字段，不动产物 mtime）。默认上限建议 `[cache] max_size`（config），未设则不设限只做 `--older-than` | +| `mcpp cache clean [--deps\|--std\|--all\|--legacy]` | 今天只删 deps 且第一行是死代码。`--legacy` 删旧的 `$MCPP_HOME/bmi/` | +| `mcpp cache list --json` | 让 CI / 工具可消费；顺手把 `list` 的 `key16` 与 `entry.json` 摘要打出来 | +| `mcpp cache verify` | 逐条目校验 `entry.json` ↔ 磁盘清单，报孤儿/残缺条目（承接 `.mcpp_ok` 内容盲区那类问题） | + +`mcpp clean --bmi-cache` 的文案改为指向 `mcpp cache clean --all`，行为保持。 + +### S8 — 依赖产物路径归一（治 C6，可与 S1–S7 并行） + +给**依赖的** compile 边加： + +``` +-ffile-prefix-map==/mcpp/build +-ffile-prefix-map==/mcpp/store +``` + +（GCC ≥ 8 / Clang ≥ 10 支持 `-ffile-prefix-map`；MSVC 无对应项 ⇒ 该轴在 MSVC 上跳过， +不作为硬要求。） + +收益：依赖对象变得与消费者工程无关 ⇒ (a) debug 信息不再指向别人的工程目录；(b) 为后续 +内容寻址去重/硬链接共享打开门。**不作为 C1/C2 的前置**，因为键已经保证语义等价。 + +### S9 — 排除本地来源的包（治 C4） + +`skipCache` 谓词从「查 root manifest 的 dependencies」改为**按 `PackageRoot` 的来源判定**： + +> 包根路径不在 registry/xpkgs store 之下（`cfg.mcppHome/registry/data/xpkgs/...`）的一律 +> 不缓存 —— 不管它是 root 的直接 path 依赖、传递 path 依赖、workspace 成员，还是 git 依赖 +> 的 checkout。 + +判据方向与 `mcpp add` 存在性门的教训相反：那里是「不可证伪就放行」，这里是 +「**不能证明它来自不可变的 store 就不缓存**」。缓存的错误代价是静默错产物，必须取保守侧。 + +顺带修 `indexName` 的回落：`depIdent` 缺失时不要回落到 `defaultIndex`（E6 里把本地包 `B` +误挂到 `mcpplibs` 名下），本地包在 S9 之后根本不会进缓存，回落分支应该改成"进不了缓存"的 +显式路径。 + +--- + +## 5. 实施顺序（硬约束） + +``` +issue311（2026.7.30.1，已设计） + └─ mcpp stage 原语 + restat=1 + rule 更名 + mcpp.home 收敛 + ↓ 复用 + +阶段 1 · 正确性前置 —— 必须先于阶段 2 合入 + P1 profile 进失效轴：本阶段用**最小改法**——在 canonical_compile_flags 里补序列化 + optLevel/debug/lto/strip（4 行），让现有 fp.hex 立刻区分 dev/release/dist。 + 阶段 2 把它作为 S1 的 C 轴迁到每包键上；两处不是重复实现，是同一语义的迁移。 + 副作用（可接受、须写 CHANGELOG）：三个 profile 从此各占一个 target/// + 目录，互不覆盖 —— 这本来就是 cargo 的行为。 + S9 本地来源包（含传递 path/git、workspace 成员）排除出缓存 + indexName 回落修正 + S2 entry.json 自描述 + 命中时逐字段校验（取代"存在即命中"） + e2e 负例：dev 构建后 --release，必须重编而不是复用；改 path dep 源码不改版本号，必须重编 + +阶段 2 · 性能 + S1 缓存键：全工程指纹 → 每包 Merkle 键（新模块 mcpp.build.cache_key） + S3 std BMI 键：fp.hex → std-module.json 身份键（注意 占位化） + S4 命中的依赖发 stage 边、不发 compile/scan 边（复用 mcpp stage + restat） + S5 缓存失效轴：MCPP_VERSION → kCacheEpoch + S6 --cache=global|local|off + [build] cache + MCPP_BUILD_CACHE + e2e 正例：两个不同包名的工程依赖同一个 index 包 ⇒ 第二个工程 0 条依赖编译边 + e2e 正例：mcpp version bump ⇒ std 与依赖均不重编 + +阶段 3 · 后续（不在本设计的实施范围内，只定方向） + S7 cache dir / gc / clean --std|--all|--legacy / list --json / verify + S8 -ffile-prefix-map 归一 + 零拷贝 直接 -fmodule-file= / 绝对路径 .o 进链接行，取消 staging + （与 issue311 §7 合并，GCC 的 gcm.cache 相对查找需要 -fmodule-mapper） +``` + +**为什么阶段 1 不能与阶段 2 并肩或落后**：阶段 2 的 S4 让 ninja 第一次真正接受缓存产物。 +在 C3/C4 未修的前提下，这等于把「白白重编一遍（结果正确）」变成「直接用错对象（结果错误）」。 +E5/E6 两条实测就是这两颗雷的引信。 + +--- + +## 6. 不采纳 / 需要纠正的方案 + +| 方案 | 判定 | 理由 | +|---|---|---| +| 给 staging 输出加 `restat = 1` 就能让 ninja 跳过 | **不成立** | `restat` 只在**有** log entry 时改写 mtime 判定。E1 的 ninja 解释是 `command line not found in log`，即根本没有 entry ⇒ 直接 dirty。必须让缓存产物成为**某条边的输出**（S4） | +| `generator = 1` 标记依赖的 compile 边（ninja 对 generator 边跳过命令哈希检查） | **拒绝** | 会把「命令行变了要重编」这条规则整体关掉，源码/flag 改动也不再重编。用错误换性能 | +| 缓存内容寻址（`obj/.o`），天然去重 | **本批不采纳** | E4 证明今天同语义对象字节不同（`DW_AT_comp_dir`）。要先做 S8 才有意义；且键已经保证语义等价，内容寻址只多省磁盘不多省时间 | +| 把 `dependencyLockHash`（`fingerprint.cppm` 第 9 字段，今天恒为空）填上，让指纹更精确 | **方向相反** | 那会让缓存键更宽（锁文件里任何一个无关依赖变化都失效）。本设计要的是**更窄且更准**的每包键。第 9 字段留给 build dir 命名用 | +| 依赖用硬链接而非拷贝 stage 进 build dir | **不采纳** | build dir 里的文件一旦被判脏重写就会**写穿**到缓存（本仓库已有 `_inherit_toolchain.sh` 写穿符号链接损坏真实工具链的先例）。reflink（`FICLONE`）可以，但只在 btrfs/xfs 且同设备 ⇒ 作为 S4 里的可选优化，默认走 issue311 `mcpp stage` 的拷贝 | +| 把缓存清理放到 `mcpp self` 下 | **拒绝** | `mcpp cache` 已经存在且是正确归属（`cli.cppm:391-410`）。`self` 是 mcpp 自身的安装/升级面 | +| 直接删掉旧的 `$MCPP_HOME/bmi/`（26 GB） | **不自动删** | 内容确实无价值（E1），但删 26 GB 必须是显式动作：doctor 提示 + `mcpp cache clean --legacy` | + +--- + +## 7. 风险与验证 + +| 风险 | 缓解 | +|---|---| +| Merkle 键漏掉某个真实影响产物的轴 ⇒ 复用错对象 | 键的输入全量写进 `entry.json`，命中时**逐字段比对**（不只比 hash）。字段不匹配 = miss + 重建，永不"信任 hash 相等"。这是 std 侧 `metadata_matches` 已验证过的模式 | +| 跳过 scan/dyndep 边后，消费者 TU 的 dyndep 找不到依赖模块的 BMI | BMI 由 stage 边落在原位置（`gcm.cache/.gcm`），消费者的 implicit input 不变。e2e 必须覆盖「消费者 `import` 一个命中缓存的模块依赖」这条正例 —— 这是 S4 最可能出问题的地方 | +| `restat = 1` 引入「永不重建」 | 单测锁 `build.ninja` 文本；e2e 正例：改依赖版本 ⇒ 键变 ⇒ 新条目 ⇒ 重建 | +| 阶段 1 落地后老缓存全 miss，用户体感"变慢了一次" | 新布局 `cache/v1/` 与旧 `bmi//` 无关，本来就是全 miss；CHANGELOG 写明一次性重建，doctor 提示旧目录可删 | +| workspace 场景：成员之间互为 path 依赖 | S9 把它们全部排除出缓存（正确：成员源码随时可改）。`--workspace` / `-p ` 的 e2e 必须断言成员产物**不**进全局缓存 | +| MSVC 无 `-ffile-prefix-map` | S8 在 MSVC 上跳过；不影响 S1–S7 | +| 多进程并发写同一条目 | 沿用 `populate_from` 已有的 `FileLock::try_acquire`（`bmi_cache.cppm:192`，拿不到锁即视为成功让给对方）。新增 `entry.json` 同样最后写、temp+rename 作 sentinel | + +必须新增的验证（e2e）： + +- **跨工程共享（正例，本设计的核心断言）**：两个不同包名/不同版本的工程依赖同一个 index 包， + 第二个工程构建时依赖的 compile 边数 = 0。用结构化载荷断言（读 `build.ninja` 或 + `-d explain`），**不要 grep 全日志**。 +- **版本 bump 不失效（正例）**：`mcpp version bump` 后 std BMI 与依赖条目均命中。 +- **profile 隔离（负例，阶段 1 必须先绿）**：`--dev` 构建后 `--release`，依赖必须重编。 +- **path dep 源码变更（负例，阶段 1 必须先绿）**：改传递 path 依赖的源码不改版本号， + 必须重编，且该包**不出现**在 `mcpp cache list` 里。 +- **假 `Cached` 回归锁**：状态行打印的 unit 数与实际跳过的边数一致。 + +本地复现注意（已知坑，沿用既有教训）：e2e 必须统一 `MCPP_HOME`；测本设计时**不要**用本机 +`~/.mcpp`（26 GB 旧缓存 + 1198 个指纹目录会干扰计时与断言），用 clean-room `MCPP_HOME`； +`! cmd | grep` 在 `errexit` 下被豁免、永不失败，负例断言不要写成这种形状。 + +--- + +## 8. 预期收益 + +| 维度 | 现状 | 本设计后 | +|---|---|---| +| 全局缓存对构建时间的贡献 | **0**（E1：命中也全编） | 依赖 TU 从"每工程编一次"降为"每 (工具链×profile×features) 编一次" | +| 本机缓存体积 | 26 GB / 1198 指纹目录 | ~1 GB（std 16.10→~0.5 GB，deps 8.80→~0.5 GB） | +| `mcpp version bump` 之后 | std 重编（10–60 s）+ 全部依赖重编 | 全部命中 | +| 新工程首次构建（依赖已被别的工程编过） | 全量编依赖 | 只 stage（数十 MB 拷贝，`restat` 后二次为 no-op） | +| `--release` 复用 `--dev` 的依赖对象 | 今天靠 ninja 侥幸不出错；S4 之后会出错 | 键隔离，不可能发生 | +| 缓存可审计性 | 只有文件清单 | `entry.json` 自描述 + `cache verify` + `cache dir` | From 1005c2fa7279a25424d7d12056a47571e8cbc9de Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 11:39:35 +0800 Subject: [PATCH 02/11] docs: add E8 (fast-path delivers wrong profile) + E9 (GCC BMI CRC binding), justify the recursive F axis --- ...26-07-30-dep-build-cache-scoping-design.md | 90 +++++++++++++++++-- 1 file changed, 84 insertions(+), 6 deletions(-) diff --git a/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md index de2e249a..d4a080e9 100644 --- a/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md +++ b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md @@ -223,7 +223,55 @@ dep TU : gcc ... -std=c11 -O0 -g --sysroot=... -B... -D_GNU_SOURCE -inc 只有：工具链身份、target triple、C++/C standard 与方言（含 `c++fly`）、macOS deployment target、**profile**、该包自身激活的 features。这些全部必须进键，且都是**应该**影响的。 -### E8 — 顺带发现的两处小问题 +### E8 — profile 不进指纹的**第二个**受害者：fast path 直接交付错 profile 的产物 + +不只是缓存条目串号。`.build_cache` 的条目**只按 `targetTriple` 去重** +（`execute.cppm:143-145`：`erase_if(e.targetTriple == targetTriple)`），profile 不在键里； +而 fast path 的准入条件里 `ov.profile.empty()`（`cmd_build.cppm:81-83`）只挡显式 +`--profile/--dev/--release`，**挡不住裸 `mcpp build`**。于是： + +``` +$ mcpp build # 默认 dev + obj/main.o → DW_AT_producer: GNU C++23 16.1.0 ... -g -O0 -std=c++23 -fmodules ✔ dev + +$ mcpp build --release # 同一个目录 9900f2252da562b2（fp 不变） + obj/main.o → 无 debug info ✔ release + +$ mcpp build # 应当回到 dev + Finished release [optimized] in 0.00s + obj/main.o → 无 debug info ✘ 仍是 release +$ grep -oE '\-O[0-9s]' build.ninja → -O2 ✘ build.ninja 里就是 -O2 +``` + +⇒ **今天 `mcpp build --release` 之后，裸 `mcpp build` 会 0.00s "成功"并交付 release 产物。** +这是一个独立于缓存的现存 bug，与 C3 同根（profile 不是失效轴），修 C3 时必须一起修，否则 +把 fp 分家只是把"同一个目录里内容是错的"换成"指向另一个 profile 的目录"。 + +### E9 — GCC 把被导入模块的 BMI CRC 烙进导入者的 BMI（决定 §4 S1 的 F 轴形态） + +手工三组对照（`g++ 16.1.0 -std=c++23 -fmodules`，`A` 导入 `B`，只重编 `B`、保留旧 `A.gcm`）： + +| 改动 | 结果 | +|---|---| +| 只改 `bfn()` **函数体**（接口逐字节不变） | 旧 `A.gcm` + 新 `B.gcm` **编译通过** | +| 改 `bfn()` 返回类型 + 新增导出类型（**接口**变） | **硬失败**（下方输出） | +| `-DWIDE` 改变导出结构体布局（**ABI** 变，源码字节不变） | **硬失败**（同下） | + +``` +B: error: module 'B' CRC mismatch +B: error: failed to read compiled module: Bad file data +A: error: failed to read compiled module: Bad import dependency +A: fatal error: returning to the gate for a mechanical issue +``` + +两个推论： + +1. `A` 的 BMI 与它当初读到的那一份 `B` 的 BMI 是**硬绑定**的 —— 所以 `A` 的缓存键必须包含 + 「`B` 的 BMI 身份」，而不是「`B` 的 public 接口的某个枚举摘要」。 +2. **失败模态不对称**：BMI 轴上键取窄了 ⇒ GCC CRC 硬报错（吵、但不出错产物）； + `.o` 轴上键取窄了 ⇒ **静默错对象**（无任何校验）。设计必须按后者取保守侧。 + +### E10 — 顺带发现的两处小问题 - `mcpp cache prune --older-than` 用 `last_write_time(entry.dir)`，而 `stage_into` 命中时 **不更新缓存目录的时间**（只写消费者 build dir）。所以它是 "last **populated**"，不是 LRU： @@ -239,7 +287,8 @@ target、**profile**、该包自身激活的 features。这些全部必须进键 |---|---|---|---| | **C1** | 命中的依赖产物 ninja 一律重编（`command line not found in log`），全局缓存净收益为零，且 UI 谎报 `Cached` | 构建后端 | E1 | | **C2** | 缓存键 = 全工程指纹（含 root 包名/版本、全图 flags）⇒ 跨工程零共享、改自己版本号即全失效 | 键 | E2 E3 | -| **C3** | profile（`-O`/`-g`/lto/strip）不进指纹 ⇒ dev/release/dist 共用缓存条目 | 正确性 | E5 | +| **C3** | profile（`-O`/`-g`/lto/strip）不进指纹 ⇒ dev/release/dist 共用同一个 build dir 与同一条缓存条目 | 正确性 | E5 | +| **C3b** | `.build_cache` 条目只按 `targetTriple` 去重，fast path 的准入只挡显式 `--profile/--dev/--release` ⇒ `--release` 之后裸 `mcpp build` 0.00s "成功"并交付 release 产物。**这是今天就在生效的 bug**，与 C3 同根 | 正确性（独立于缓存） | E8 | | **C4** | 传递 path/git 依赖被写进全局缓存，且源码变更不改键 | 正确性 | E6 | | **C5** | 缓存条目无自描述元数据（只有 `manifest.txt` 文件清单），`is_cached` 是"存在即命中"，无法校验、无法审计 | 键 | 代码 | | **C6** | 依赖对象里烙进消费者 build dir 路径（`DW_AT_comp_dir`）⇒ 无法内容寻址去重、debug 信息指向别的工程 | 可复现性 | E4 | @@ -274,9 +323,30 @@ C3/C4 与 C1 的关系是硬约束：**C1 修好之前它们是无害的浪费 **不含**：root 包名/版本、root 的 `[build]` flags（E7 已证不下发）、图里无关的兄弟依赖、 消费者的 build dir 路径。 -F 轴用递归键而不是「上游的 public include dirs + defines」，理由：上游 flag 变化会改变它 -导出的接口（宏、生成文件、ABI），逐项枚举必然漏。递归键是保守且完备的上界；代价是上游 -换 profile 会级联下游全部 key —— 这是正确的（profile 本来就是全图轴）。 +**F 轴为什么必须是递归键，而不是「枚举上游的 public include dirs + interface defines」：** + +1. **BMI 是硬绑定的，不是"接口等价即可"。** E9 实测：GCC 把被导入模块 BMI 的 CRC 写进导入者 + 的 BMI，接口或 ABI 一变就 `module 'B' CRC mismatch` + `Bad import dependency`。所以 + `A` 的键需要的是「`B` 的 BMI 身份」本身，而 `B` 的 BMI 身份 = `B` 的完整键。 +2. **可枚举清单必然漏项。** 要枚举的是 `B` 的 public include dirs 的**内容**、interface + defines、`generated_files`、以及 `B` **re-export** 出去的传递模块接口 —— 最后这项从 `B` + 的 manifest 里根本看不出来（它是 `R` 的接口，经 `B` 转出）。少一项 = 命中一条 BMI + 加载不了的条目。 +3. **失败模态不对称（E9 的第二个推论）**：BMI 轴取窄 ⇒ CRC 硬报错（吵）；`.o` 轴取窄 ⇒ + 静默错对象（哑，`.o` 没有任何自校验）。设计必须按后者取保守侧。 + +递归的真实代价**不是**「上游换 profile 级联下游」—— profile 是 C 轴，它本来就在**每一个** +包的键里，有没有 F 轴都会让全图 key 变。递归的真实代价是：**上游的私有输入变化会级联下游** +（例如 `B` 给自己加一个 `-DB_INTERNAL_LOG=1`，或改一个不提供模块的私有源文件，`B` 的接口 +逐字节没变，但 `A` 的键跟着变、`A` 白重编一次）。 + +而这个代价在真正吃缓存的人群里 ≈ 0：**index 包的描述符按版本冻结**，`B` 的 buildConfig 不 +可能不 bump 版本就变（仓库里 #253 的注释依赖的正是这条不变量），所以 `B` 的键只能因 +「版本 bump」或「全图轴（工具链/standard/profile）」而变 —— 而版本 bump 本来就该让 `A` +失效（`A` 要链接 `B` 的对象）。唯一能不改版本就动 `B` 私有 flag 的是 path/git 包，而它们 +已被 S9 整体排除出缓存。 + +⇒ 递归是唯一可靠的形态，且在目标人群里几乎不付代价。 拓扑序已由 `report.topoOrder`（`prepare.cppm:3561`）给出，自底向上一次遍历即可算完全图键。 @@ -457,7 +527,15 @@ issue311（2026.7.30.1，已设计） optLevel/debug/lto/strip（4 行），让现有 fp.hex 立刻区分 dev/release/dist。 阶段 2 把它作为 S1 的 C 轴迁到每包键上；两处不是重复实现，是同一语义的迁移。 副作用（可接受、须写 CHANGELOG）：三个 profile 从此各占一个 target/// - 目录，互不覆盖 —— 这本来就是 cargo 的行为。 + 目录，互不覆盖 —— 这本来就是 cargo 的行为。代价是磁盘 ×profile 数（每个 build dir + 带一份 staged std BMI，本机 std.gcm = 31,466,112 字节）。 + P2 C3b：.build_cache 条目按 (targetTriple, **resolved profile**) 去重，profile 不匹配 + 即当 miss 走 prepare_build。**只做 P1 不做 P2 是不够的**：fp 分家只会把"同一个目录 + 里内容是错的"换成"fast path 指向另一个 profile 的目录"，E8 的洞照旧。 + 顺手修 run_build_plan 里硬编码的 ui::finished("release", ...)——它在 dev 构建下 + 也打印 "Finished release [optimized]"，是 E8 里最误导人的一行。 + P3 fingerprint changed (X → Y), full rebuild 警告（execute.cppm:310-324）会在每次切 + profile 时触发。改成能区分"仅 profile 变"的文案，否则它从有用信号退化成噪音。 S9 本地来源包（含传递 path/git、workspace 成员）排除出缓存 + indexName 回落修正 S2 entry.json 自描述 + 命中时逐字段校验（取代"存在即命中"） e2e 负例：dev 构建后 --release，必须重编而不是复用；改 path dep 源码不改版本号，必须重编 From 5daaacb8f9ffec7efab9e56eab1f039749aca296 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 14:55:46 +0800 Subject: [PATCH 03/11] feat(build): scope the dependency cache per package, and make hits actually skip work (2026.7.30.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The global dependency cache existed but its net benefit was zero, for two independent reasons. It never hit across projects. The key was the whole-project fingerprint, whose flags field serializes every package in the graph INCLUDING the root — its name, its version, its [build] flags. So bumping a project's own version invalidated every dependency it had, and two projects with identical dependencies and toolchain shared nothing. Measured on one machine: 26 GB across 1198 fingerprint directories, compat.zlib@1.3.2 stored 162 times, 15 distinct std module identities occupying 1014 directories (16.1 GB where ~0.5 GB was needed). And when it did hit, nothing was saved. Artifacts were copied into the build dir from inside prepare_build while those paths stayed declared as compile edge outputs — and ninja treats an output it has no command line for in .ninja_log as dirty, which a fresh build dir always is. Every "cached" unit was recompiled while the CLI printed "Cached". ninja explain: command line not found in log for obj/zutil.o ninja explain: obj/zutil.o is dirty Keying is now per package (new mcpp.build.cache_key): toolchain identity, language/dialect, profile, package identity, the package's own build config, and — recursively — the keys of its direct dependencies. Nothing about the consumer, which is sound because the root's [build] flags verifiably do not reach dependency translation units. The recursion is not conservatism: GCC embeds a CRC of an imported module's BMI into the importer's BMI, so an importer's artifacts are bound to the exact upstream artifacts they read. Hits now emit stage_file edges instead of compile edges (and no scan/dyndep edges), so ninja has a command-line record for the staged outputs. Artifacts land where a compile edge would have put them, leaving link edges, BMI implicit inputs and runtime deployment unchanged. The status line carries the unit count it saved, because a bare "Cached" was printed for months while every unit was recompiled behind it. Three correctness defects had to land with it, all of them harmless only while the cache was a no-op: - The profile was not an invalidation axis. --dev, --release and --profile dist shared one fingerprint, one build dir and one cache entry, so a release build would have been served -O0 -g objects. - .build_cache keyed fast-path entries by target triple alone, and the fast path only refuses to run for an EXPLICIT profile flag. `mcpp build --release` followed by a bare `mcpp build` reported success in 0.00s and left the release artifacts in place. This one was live regardless of the cache. - Transitively reached path/git dependencies were cached: the exclusion predicate consulted the root manifest's dependency maps, where a transitive package does not appear. Their sources can change without name@version changing. Also: --cache=global|local|off (with --no-cache as a deprecated alias for off, and its inaccurate help text corrected), and mcpp cache grown into something operable — dir / gc with a real LRU / clean --deps|--std|--all|--legacy / list --json / verify, with each entry now describing itself in entry.json so a suspected wrong hit can be audited. The cache root is $MCPP_HOME/build-cache/v1, deliberately not $MCPP_HOME/cache: that name belongs to the index metadata cache, whose reset path removes the whole directory. Design: .agents/docs/2026-07-30-dep-build-cache-scoping-design.md Plan: .agents/docs/2026-07-30-dep-build-cache-implementation-plan.md --- ...-30-dep-build-cache-implementation-plan.md | 396 +++++++++++++ CHANGELOG.md | 66 +++ docs/05-mcpp-toml.md | 55 +- docs/zh/05-mcpp-toml.md | 48 +- mcpp.toml | 2 +- src/bmi_cache.cppm | 329 ++++++----- src/bmi_cache/maintenance.cppm | 537 ++++++++++++++---- src/build/cache_key.cppm | 356 ++++++++++++ src/build/execute.cppm | 182 ++++-- src/build/ninja_backend.cppm | 44 ++ src/build/plan.cppm | 8 + src/build/prepare.cppm | 343 +++++++++-- src/cli.cppm | 54 +- src/cli/cmd_build.cppm | 17 +- src/cli/cmd_cache.cppm | 25 +- src/config.cppm | 6 +- src/doctor.cppm | 55 +- src/home.cppm | 29 +- src/manifest/toml.cppm | 5 +- src/manifest/types.cppm | 6 + src/toolchain/fingerprint.cppm | 2 +- src/toolchain/stdmod.cppm | 116 +++- src/ui.cppm | 18 +- tests/e2e/100_cppfly_reflection.sh | 2 +- tests/e2e/10_env_command.sh | 2 +- tests/e2e/170_bmi_staging_no_cascade.sh | 10 +- tests/e2e/172_build_cache_cross_project.sh | 189 ++++++ tests/e2e/173_build_profile_isolation.sh | 127 +++++ tests/e2e/174_cache_modes_and_commands.sh | 302 ++++++++++ tests/e2e/19_bmi_cache_reuse.sh | 130 +++-- tests/e2e/49_bmi_cache_nested_custom_index.sh | 46 +- tests/e2e/59_cpp_standard_config.sh | 2 +- tests/unit/test_bmi_cache.cpp | 339 +++++++---- tests/unit/test_build_profile.cpp | 134 +++++ tests/unit/test_cache_key.cpp | 265 +++++++++ tests/unit/test_home.cpp | 37 +- tests/unit/test_ninja_backend.cpp | 114 ++++ 37 files changed, 3789 insertions(+), 609 deletions(-) create mode 100644 .agents/docs/2026-07-30-dep-build-cache-implementation-plan.md create mode 100644 src/build/cache_key.cppm create mode 100644 tests/e2e/172_build_cache_cross_project.sh create mode 100644 tests/e2e/173_build_profile_isolation.sh create mode 100644 tests/e2e/174_cache_modes_and_commands.sh create mode 100644 tests/unit/test_build_profile.cpp create mode 100644 tests/unit/test_cache_key.cpp diff --git a/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md b/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md new file mode 100644 index 00000000..388ba6ee --- /dev/null +++ b/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md @@ -0,0 +1,396 @@ +# 依赖构建产物的全局缓存收敛 — 实施计划 + +配套设计：`2026-07-30-dep-build-cache-scoping-design.md` +前置：issue311 / PR#312（`mcpp stage` 原语 + `stage_file` rule + `restat = 1` + `mcpp.home`） +**已合入 main**（`b794856`，2026.7.30.1），故本计划直接复用，不再自建 staging 原语。 +建议目标版本：**2026.7.30.2**（常规迭代，`.0` 保留给稳定版） + +单 PR 交付。阶段有硬依赖顺序：**A（正确性）→ B（换键）→ C（stage 边）→ D（模式与运维）**。 +A 必须先合入的判据见设计 §5：C 让 ninja 第一次真正接受缓存产物，A 未落地时那等于把 +"白重编一遍但结果对" 变成 "直接用错对象"。 + +--- + +## 落地前已核实的现状事实（避免设计→实现的重映射错误） + +| 事实 | 位置 | 对实现的意义 | +|---|---|---| +| `stage_file` rule 已存在，命令 `$mcpp stage $verify --output $out $in`，带 `restat = 1` | `ninja_backend.cppm:456-459` | C 阶段直接复用，`$verify` 用 `size`（依赖缓存条目是 key-scoped，等长即等价，与 std 边同理） | +| `$mcpp` 变量绑定已提到 `if (dyndep)` 之外 | `ninja_backend.cppm:427` | 不需要再动 | +| **restat 与对齐 mtime 互斥**（PR#312 的教训：对齐 mtime 会重新级联） | 同上注释 | `stage_into` 里的 `touch_now` 在 C 阶段要删掉，改由 restat 负责 | +| `DepCacheIdentity` 已有 `{indexName, packageName, version}`，两个 push 点 | `prepare.cppm:1375-1380` / `:2684` / `:2979` | A 阶段加 `sourceKind` 字段即可，**不需要**按路径前缀猜来源 | +| `sourceKind`（`"version"｜"path"｜"git"`）在两个 push 点都在作用域内 | `prepare.cppm:2699` / `:2982` | S9 的判据是现成的 | +| `dependencyEdges`（`consumerPackageIndex → dependencyPackageIndex` + visibility） | `prepare.cppm:2059-2073` | B 阶段 F 轴（Merkle）直接遍历它，不需要新建图 | +| `CompileUnit` 已带 `packageName` + per-package flags/include dirs | `plan.cppm:20-36` | B 阶段按包分组是现成的；C 阶段加标记字段 | +| `report.topoOrder` 已有拓扑序 | `prepare.cppm:3561` | B 阶段自底向上算 Merkle 键一次遍历 | +| `metadata_for()` 已是完整正确的 std 身份（15 字段） | `stdmod.cppm:88-112` | B 阶段只改目录名，不重新定义身份 | +| `mcpp::home::bmi_root()` 已是单一解析器 | `src/home.cppm` | 新缓存根走它，不再碰 `default_cache_root()` 的旧兜底 | +| 版本号两处 | `mcpp.toml:3`、`src/toolchain/fingerprint.cppm:21` | `.github/tools/check_version_pins.sh` 机器校验 | + +--- + +## 阶段 A — 正确性前置 + +### A1 · profile 进失效轴 + +`prepare.cppm:canonical_compile_flags()`（:209-265）末尾追加： + +```cpp +s += " opt=" ; s += m.buildConfig.optLevel; +s += " debug=" ; s += m.buildConfig.debug ? "1" : "0"; +s += " lto=" ; s += m.buildConfig.lto ? "1" : "0"; +s += " strip=" ; s += m.buildConfig.strip ? "1" : "0"; +``` + +调用点在 profile 解析（:776-809）之后，无需移动。 + +**副作用（写 CHANGELOG）**：三个 profile 从此各占一个 `target///`，互不覆盖 +（cargo 行为）。代价是磁盘 × profile 数——每个 build dir 带一份 staged std BMI +（本机 `std.gcm` = 31,466,112 B）。 + +### A2 · `.build_cache` 按 profile 区分（C3b，今天在生效的 bug） + +`BuildCacheEntry` 加 `std::string profile;`（`execute.cppm:35-58`）。 + +- 序列化：P3 格式加一行 `profile=`；**旧条目缺该字段 ⇒ 读成空 ⇒ 与任何 resolved + profile 都不匹配 ⇒ 当 miss**（自愈，不需要迁移代码）。 +- `write_build_cache` 的去重键：`(targetTriple, profile)`（:143-145）。 +- `try_fast_build` / `try_fast_run` 需要知道"本次请求的 resolved profile"。但 fast path 的 + 存在意义就是**不跑 prepare_build**，而 profile 解析在 prepare_build 里。 + ⇒ 解析规则本身是纯的（`overrides.profile > [build].default-profile > "dev"`， + `prepare.cppm:789-791`），把它抽成一个 `mcpp::build::resolve_profile_name( + const Manifest&, std::string_view override)` 供两侧共用。fast path 只需 load manifest + （它已经 load 了，用于 staleness 检查）。 +- 顺手修 `ui::finished("release", ...)`（`execute.cppm:346`）硬编码 → 传 resolved profile。 + 它在 dev 构建下也打 `Finished release [optimized]`，是 E8 里最误导人的一行。 + +### A3 · fingerprint-changed 警告降噪 + +`execute.cppm:310-324`。条目里现在有 profile 了，所以能判断"fp 变化是否伴随 profile 变化"： +profile 也变了 ⇒ 不打这条（这是预期的、用户自己要求的），只在 profile 相同而 fp 变时打。 + +### A4 · 本地来源包排除出缓存（S9） + +1. `DepCacheIdentity` 加 `std::string sourceKind;`，两个 push 点（:2684 用字面 + `"version"`，:2979 用作用域内的 `sourceKind`）填上。 +2. 缓存循环（:3758-3830）的 `skipCache` 谓词整体替换： + +```cpp +// 旧：查 root manifest 的 dependencies/devDependencies（传递依赖查不到 ⇒ 误缓存） +// 新：唯一判据 —— 该包必须来自 index（不可变的 xpkgs store） +if (!depIdent || depIdent->sourceKind != "version") continue; +``` + + 这同时干掉 `indexName` 回落到 `defaultIndex` 的误挂（E6：本地包 `B` 被挂到 + `mcpplibs` 名下）——`depIdent` 为空时直接不缓存，回落分支消失。 +3. 判据方向说明写进代码注释：与 `mcpp add` 存在性门的"不可证伪就放行"**相反**， + 缓存要"不能证明来自不可变 store 就不缓存"。缓存的错误代价是静默错产物。 + +### A5 · 条目自描述 + 命中逐字段校验（S2） + +`bmi_cache.cppm`： + +- `CacheKey` 加 `std::string keyHex;`（B 阶段填 Merkle 键）与 `nlohmann::json inputs;`。 +- 新 `entry.json`（取代 `manifest.txt` 作为 sentinel，`manifest.txt` 保留写出以便人读）： + +```json +{ "schema": 1, "epoch": 1, "key": "", + "inputs": { ...键的全部输入... }, + "bmi": ["A.gcm"], "obj": ["a.o"], + "created": "", "accessed": "" } +``` + +- `is_cached(key)` ⇒ `entry.json` 存在 ∧ `schema/epoch` 匹配 ∧ **`inputs` 逐字段等于本次 + 算出的 inputs** ∧ 清单文件全部存在。字段不等 = miss，**永不"信任 hash 相等"** + （照抄 `stdmod.cppm::metadata_matches` 的模式）。 +- 写序：产物 → `entry.json.tmp` → rename（sentinel 最后）。沿用已有的 + `FileLock::try_acquire`。 +- 命中时更新 `accessed`（只重写 `entry.json`，**不动产物 mtime**）——这是 D 阶段 + `cache gc` 能做真 LRU 的前提。 + +### A6 · 阶段 A 的测试 + +单测（`tests/unit/test_bmi_cache.cpp` 扩写，`tests/unit/test_fingerprint.cpp` 扩写）： + +| 用例 | 断言 | +|---|---| +| profile 进指纹 | 同一 manifest，optLevel/debug 不同 ⇒ `canonical_compile_flags` 串不同 ⇒ fp 不同 | +| `resolve_profile_name` 优先级 | override > default-profile > "dev" | +| `.build_cache` 往返 | 写 (triple="", profile="dev") 与 (triple="", profile="release") ⇒ **两条并存** | +| 旧格式条目 | 无 `profile=` 行 ⇒ 读出空 profile ⇒ 任何 resolved profile 都 miss | +| `entry.json` 校验 | inputs 改一个字段 ⇒ `is_cached` 为 false | +| `accessed` 更新 | `stage_into` 后 `accessed` 变新、产物 mtime 不变 | + +e2e（新文件 `tests/e2e/1NN_build_cache_correctness.sh`）： + +- **负例**：`mcpp build --dev` → `mcpp build --release`，断言 release 构建里依赖**有**编译边。 +- **负例（E8 回归闸）**：`mcpp build --release` → 裸 `mcpp build`，断言产物是 dev + （`readelf --debug-dump=info` 有 `-O0`／`DW_AT_producer` 含 `-g`），且**不是** 0.00s 假成功。 +- **负例（E6 回归闸）**：root → A{path} → B{path}，构建后断言 `mcpp cache list` 里**没有** B。 + +--- + +## 阶段 B — 换键 + +### B1 · 新模块 `src/build/cache_key.cppm` + +```cpp +export module mcpp.build.cache_key; +import std; import mcpp.libs.json; import mcpp.toolchain.detect; +import mcpp.toolchain.fingerprint; // hash_string + +export namespace mcpp::build::cache_key { +inline constexpr int kCacheEpoch = 1; // bump ONLY on cache-incompatible changes + +struct PackageKeyInputs { // 逐轴，A..G（设计 §4 S1） + // A 工具链 / B 语言方言 / C profile / D 包身份 / E 该包自身配置 / F 上游键 / G epoch + ... +}; +nlohmann::json to_json(const PackageKeyInputs&); // 进 entry.json，供逐字段校验 +std::string hex16(const PackageKeyInputs&); // 目录名 +} +``` + +`hex16` 复用 `mcpp::toolchain::hash_string`（fnv1a-64 → 16 hex），与既有指纹同族。 + +**依赖图无环性自查（改前必做）**：`mcpp.build.cache_key` 只 import `std` / +`mcpp.libs.json` / `mcpp.toolchain.{detect,fingerprint}`，均是叶或已在 `mcpp.build.plan` +的闭包内 ⇒ `mcpp.build.prepare → mcpp.build.cache_key` 不成环。 + +### B2 · 在 prepare 里自底向上算全图键 + +位置：缓存循环之前（当前 :3725 之前），拓扑序遍历 `report.topoOrder`／`packages`： + +``` +for pkgIndex in 自底向上顺序: + inputs = 收集 A..E 轴(pkgIndex) + inputs.upstream = sorted( key(dep) for dep in dependencyEdges[pkgIndex] ) // F 轴 + key[pkgIndex] = hex16(inputs) +``` + +- A/B/C 轴对全图相同，一次算出复用。 +- E 轴取 `packages[pkgIndex].manifest.buildConfig` + `privateBuild/publicUsage` 的 + include dirs。**include dirs 必须相对 xpkgs store 根归一**（它们是 + `/registry/data/xpkgs/...` 的绝对路径，不归一则 `MCPP_HOME` 一换全 miss）。 +- `sources` 清单取相对包根、排序后的相对路径（`CompileUnit::source` 按 `packageName` + 分组即可）。 +- F 轴：`dependencyEdges` 里 `consumerPackageIndex == pkgIndex` 的所有边的 + `dependencyPackageIndex` 的键，排序后拼接。**环**：modgraph 已拒环（`validate`）， + 但仍加一个访问集断言，成环时 `return std::unexpected(...)` 而不是无限递归。 + +### B3 · 缓存布局换到 `cache/v1/` + +`bmi_cache.cppm::CacheKey::dir()`： + +```cpp +- return mcppHome / "bmi" / fingerprint / "deps" / indexName / "@"; ++ return mcppHome / "cache" / "v1" / "pkg" / indexName / "@" / keyHex; +``` + +`fingerprint` 字段删除。旧 `$MCPP_HOME/bmi//deps/` **不读不写不删**，由 D 阶段的 +`doctor` 提示 + `cache clean --legacy` 处理。 + +### B4 · std BMI 换键（S3） + +`stdmod.cppm`： + +```cpp +- sm.cacheDir = cache_root / std::string(fingerprint_hex); ++ sm.cacheDir = cache_root / "std" / std_identity_key(tc, cpp_standard, cpp_standard_flag, ++ macos_deployment_target); +``` + +`ensure_built` 的 `fingerprint_hex` 形参删除（唯一调用点 `prepare.cppm:3612`）。 + +⚠️ **自指陷阱（必须处理）**：`metadata["std_build_commands"]` 里含 `sm.cacheDir` 的绝对 +路径，而 cacheDir 又由 metadata 算出。做法：先用占位 `` 生成一份"规范化 +metadata"算键，再用真 cacheDir 生成落盘的 metadata。落盘的 `metadata_matches` 逻辑不变。 + +⚠️ `cache_root` 从 `default_cache_root()`（= `home::bmi_root()` = `/bmi`）改为 +`/cache/v1`，需同步 `maintenance.cppm` / `doctor.cppm` / `clean --bmi-cache` 三处 +读点。 + +### B5 · epoch 取代 MCPP_VERSION（S5） + +`fingerprint.cppm` 的 `fp.parts[7] = MCPP_VERSION` **保持不变**（build dir 命名 + 本地增量 +归它管，保守）。缓存键的 G 轴用 `kCacheEpoch`，写进 `entry.json`；读到不同 epoch 当 miss。 + +### B6 · 阶段 B 的测试 + +单测（新 `tests/unit/test_cache_key.cpp`）： + +| 用例 | 断言 | +|---|---| +| root 身份不影响依赖键 | 只改 root 包名/版本 ⇒ 依赖包的键**不变** | +| root flags 不影响依赖键 | 只改 root `[build] cxxflags` ⇒ 依赖包的键**不变** | +| 无关兄弟依赖不影响 | 图里新增一个与之无边的包 ⇒ 原包的键**不变** | +| profile 影响 | optLevel 变 ⇒ 键变（C 轴） | +| 该包自身 flags 影响 | 该包 cflags 变 ⇒ 键变（E 轴） | +| 上游变化级联 | 上游包的 flags 变 ⇒ 下游包的键**也变**（F 轴 Merkle） | +| include dirs 归一 | 换 `MCPP_HOME` 前缀 ⇒ 键不变 | +| std 身份键 | 同 (compiler, version, triple, stdlib, std_flag, 源 hash) ⇒ 同键；任一变 ⇒ 键变；`` 占位化生效（两个不同 cacheDir 得同键） | + +--- + +## 阶段 C — 命中的依赖发 stage 边 + +### C1 · plan 侧标记 + +`plan.cppm::CompileUnit` 加： + +```cpp +bool servedFromCache = false; +std::filesystem::path cachedObject; // 缓存内绝对路径 +std::filesystem::path cachedBmi; // 缓存内绝对路径（有模块时） +``` + +prepare 的缓存循环命中时，给该包的所有 CU 打上标记并填缓存路径（不再调用 +`stage_into` 做进程内拷贝——staging 交给 ninja 边）。 + +### C2 · ninja 侧分流 + +`ninja_backend.cppm`： + +- P1689 scan／dyndep 的 CU 列表（:850-905 收集 `ddi_paths`）**过滤掉** `servedFromCache`； +- compile 边（:913-1010 两条分支，dyndep / 非 dyndep）**跳过** `servedFromCache`； +- 改发（每产物一条）： + ``` + build gcm.cache/.gcm : stage_file + verify = --verify size + build obj/.o : stage_file + verify = --verify size + ``` + `--verify size` 的判据与 std 边同源：条目目录名就是覆盖了工具链/方言/profile/包配置/ + 上游闭包的键，同键 ⇒ 等长即等价（PR#312 注释里已把这条判据写清）。 +- 消费者 TU 的 implicit input、链接行的对象列表、`bin/` 部署边**都不变**（产物落在原位置）。 +- `plan.compileUnits` 保留这些 CU ⇒ `compile_commands.cppm:127-170` 仍为依赖发条目， + clangd/IDE 不退化。 + +### C3 · `stage_into` 与 `populate_from` 的调整 + +- `stage_into` 只在 `--cache=global` 且 **C 阶段之外的调用者**（无）使用 ⇒ 删除进程内 + staging，改为返回清单供 C1 填路径。函数改名 `resolve_cached(key)`，语义变成"读清单 + + 校验，不落盘"。 +- `populate_from` 保留：未命中的包在 build 成功后回填。删掉 `touch_now` + （restat 与对齐 mtime 互斥，PR#312 的教训）。 + +### C4 · 状态行说真话 + +`cachedDepLabels` 带上省下的 unit 数：`Cached compat.zlib v1.3.2 (16 units)`。 +假状态行能骗人三个月，带计数就骗不了。 + +### C5 · 阶段 C 的测试 + +单测（`tests/unit/test_ninja_backend.cpp` 扩写）： + +| 用例 | 断言 | +|---|---| +| 命中包不发 compile 边 | `servedFromCache` 的 CU ⇒ `build.ninja` 里无其 compile 命令，有 `stage_file` 边 | +| 命中包不发 scan 边 | 无其 `.ddi` | +| `verify = --verify size` | stage 边带该绑定 | +| 未命中包不变 | 无标记 ⇒ 文本与今天逐字节一致（归一化 diff） | + +e2e（新文件 `tests/e2e/1NN_build_cache_crossproject.sh`）——**本设计的核心断言**： + +1. clean-room `MCPP_HOME`；工程 P1（含一个 index 依赖）构建 → 条目落地； +2. 工程 P2（**不同包名、不同版本**、同一依赖）构建 ⇒ 依赖的 compile 边数 **= 0**， + `stage_file` 边数 > 0。断言方式：读 `build.ninja`／`ninja -d explain` + 的结构化输出，**不 grep 全日志**； +3. P1 `mcpp version bump` 后重建 ⇒ std 与依赖**均命中**； +4. 依赖版本改变 ⇒ 键变 ⇒ 重编（正例，防"永不重建"）。 + +--- + +## 阶段 D — 模式与运维 + +### D1 · `--cache=global|local|off`（S6） + +- `cli.cppm`：`build` 加 `--cache `；`--no-cache` 保留为 deprecated 别名并修正 help + 文案（当前写的是 "Force-clear target/ before building"，名不副实）。 +- `run`/`test` 也加（今天它们连 `--no-cache` 都没有）。 +- 优先级：CLI > `MCPP_BUILD_CACHE` > `[build] cache` > `global`。 +- `manifest.cppm`：`[build] cache` 字段 + 未知值的 warning（`--strict` ⇒ error）。 +- `BuildOverrides` 加 `std::string cache_mode;`；`prepare_build` 里 global 之外**不读** + 缓存，`run_build_plan` 里 global 之外**不写**；`off` 额外 `remove_all(outputDir)`。 + +| 模式 | 读 | 写 | 先清 target/ | +|---|---|---|---| +| `global`（默认） | 是 | 是 | 否 | +| `local` | 否 | 否 | 否 | +| `off` | 否 | 否 | 是 | + +### D2 · `mcpp cache` 补齐（S7） + +`maintenance.cppm` 走新布局 `cache/v1/`，并新增： + +| 命令 | 实现要点 | +|---|---| +| `cache dir` | 打印 `home::root() / "cache" / "v1"`。今天 `cache*`/`doctor`/`clean --bmi-cache` 与 config 重置路径可能不是同一个目录，这条让它永远可验证 | +| `cache gc [--max-size N{MiB,GiB}] [--older-than N{s,m,h,d}]` | 按 `entry.json.accessed` 真 LRU（A5 已保证它被更新）。`--max-size` 从最旧开始删到达标 | +| `cache clean [--deps\|--std\|--all\|--legacy]` | 现在只删 deps 且第一行 `remove_all(bmi/"deps")` 是死代码（deps 在 `bmi//deps`）。`--legacy` 删旧 `/bmi/` | +| `cache list --json` | 结构化输出；`list` 增列 `key16` | +| `cache verify` | 逐条目校验 `entry.json` ↔ 磁盘清单，报孤儿/残缺 | + +`clean --bmi-cache` 文案改为指向 `cache clean --all`，行为保持。 +`doctor`：发现旧 `/bmi/` 时报一行大小 + `mcpp cache clean --legacy`（不自动删）。 + +### D3 · 阶段 D 的测试 + +| 用例 | 断言 | +|---|---| +| 模式优先级 | CLI > env > manifest > 默认（单测直接测解析函数） | +| `local` 不写缓存 | clean-room 下 `--cache=local` 构建后 `cache list` 为空 | +| `off` 清 target | 构建产物先存在，`--cache=off` 后 build dir 被重建 | +| `--no-cache` 别名 | 与 `--cache=off` 行为一致 | +| `gc --max-size` | 造 3 条不同 `accessed` 的条目 ⇒ 删到达标，保留最新 | +| `cache verify` | 手动删一个产物 ⇒ 报残缺并非零退出 | + +--- + +## 阶段 E — 收尾 + +1. **版本号**：`mcpp.toml:3` + `src/toolchain/fingerprint.cppm:21` → `2026.7.30.2`； + 跑 `bash .github/tools/check_version_pins.sh`。 + （bootstrap pin `.xlings.json` / `MCPP_PIN` **不动** —— 它是自举起点，不随发布走。） +2. **CHANGELOG.md**：三段——(a) 全局缓存此前净收益为零（附 ninja `-d explain` 证据）； + (b) profile 现在是失效轴，`target/` 下每 profile 一个目录、磁盘上升； + (c) 旧 `/bmi/` 一次性弃用，`mcpp cache clean --legacy` 可回收（本机 26 GB）。 +3. **docs/**：`docs/26-bmi-cache.md` 同步新布局与键定义（`bmi_cache.cppm` 头注释指向它）。 +4. **自举验证**：`mcpp build`（release profile）+ `mcpp test` 全绿； + 用**新构建出的** `mcpp` 二进制跑一遍 e2e，而不是 bootstrap 的那份。 +5. **PR**：单 PR，标题带版本与 issue 号；CI 全绿后 squash 合入。 +6. **xlings 全生态验证**：release → 镜像 xlings-res（gh + gtc 双端）→ xim-pkgindex PR → + clean-room `XLINGS_HOME` 真装 → 跑 e2e → bump bootstrap pin。 + +--- + +## 实施结果与计划的偏差（实现后回填） + +实现于 2026.7.30.2。以下是与本计划不同的地方，以及原因。 + +| 项 | 计划 | 实际 | 原因 | +|---|---|---|---| +| 缓存根 | `$MCPP_HOME/cache/v1` | **`$MCPP_HOME/build-cache/v1`** | `cache` 这个名字已归 `GlobalConfig::metaCacheDir`（索引元数据）所有，而 `config.cppm:222` 的 reset 路径 `remove_all` 整个目录。编译产物塞进别人会清空的目录里，会以「构建缓存有时会自己空掉」的形态浮现 | +| `CacheKey` 字段 | `mcppHome` + 内部拼 `cache/v1` | **`cacheRoot`（调用方传 `mcpp::home::cache_root()`）** | 布局根必须只有一份定义；在 `bmi_cache.cppm` 里再写一遍 `"build-cache"/"v1"` 正是注释管不住的跨文件不变量 | +| `stage_into` | 改名 `resolve_cached`，返回清单 | 同（且 `touch_now` 删除） | 计划一致。restat 与对齐 mtime 互斥（PR#312 的教训） | +| epoch 覆盖 std | G 轴含 `kCacheEpoch` | **std 键不含 epoch** | std 键放在 `stdmod.cppm`（与 `metadata_for` 同处，内聚更好），不引入 `stdmod → build.cache_key` 的跨层依赖。std 的格式兼容由 `build-cache/v1` 路径段承载 —— 一份 std BMI 的有效性取决于工具链，不取决于 mcpp 的缓存簿记 | +| A2 的范围 | `.build_cache` 加 profile | **另加 P2/P3 两项** | 光加字段不够：fast path 的准入只挡显式 `--profile`，挡不住裸 `mcpp build`，所以必须同时让 fast path 自己解析 profile（新 `resolve_profile_name`）。另修 `ui::finished` 的硬编码 `"release" [optimized]` | +| `finished()` 签名 | 不在计划内 | 加可选 `descriptor` 形参 | `[optimized]` 也是硬编码的；描述符由真正解析了 profile 开关的调用方给出，fast path 不给（它没解析过），所以没有调用方需要编造 | +| e2e 文件 | `1NN_*` 两个 | **172 / 173 / 174 三个** | 模式与运维命令的面足够大，值得独立文件；另改写了既有的 19（升级为传递 path 依赖的负例）与 49（改读 `entry.json` + 断言无 compile 边） | +| `--cache=off` 的语义 | 「清 target/」 | **清本次的 `target///`** | 这本来就是 `--no-cache` 的既有行为；旧 help 文案「Force-clear target/」两处不准（不是整个 target/，也与缓存无关），一并改正 | +| S8 `-ffile-prefix-map` | 阶段 3 | **未实施** | 它的收益是为内容寻址去重铺路，而本批不做内容寻址；且改变依赖对象的 debug 路径会让 gdb 需要 `set substitute-path`，是可感知的调试体验回退。留作独立批次 | +| 零拷贝 | 阶段 3 | **未实施** | 与 issue311 §7 合并，需要 GCC 的 `-fmodule-mapper` 与四平台 e2e | + +**未实施项的处置**：S8 与零拷贝均记在设计文档 §7 的后续批次里，不在本批 CHANGELOG 中承诺。 + +## 已知坑（来自本仓库既有教训，实现时逐条对照） + +| 坑 | 规避 | +|---|---| +| 指纹目录随版本变，`ls target// \| head -1` 会自查到旧二进制 | 测试里用 `mcpp build --print-fingerprint` 拿 fp，或对目录按 mtime 排序 | +| `! cmd \| grep` 在 `errexit` 下被豁免、永不失败 | 负例断言不要写成这种形状；用显式 `if grep -q ...; then exit 1; fi` | +| e2e `_inherit_toolchain.sh` 写穿符号链接会损坏真实 payload | 新 e2e 不 `ln -sf` 真实 payload；只用 clean-room `MCPP_HOME` + 工程内 `path` 索引 | +| 本机 `~/.mcpp` 有 26 GB / 1198 个旧指纹目录 | 所有新 e2e 用 clean-room `MCPP_HOME`，否则计时与断言全被污染 | +| 断言读全日志 grep 易假绿 | 读 `build.ninja` / `resolution.json` / `cache list --json` 等结构化载荷 | +| ninja `restat` 与对齐 mtime 互斥 | `populate_from`/`resolve_cached` 都不要 touch 产物 mtime | +| 描述符 Lua 里的跨包字面地址会随布局变化失效 | 本次不动包布局，但 `check_cross_package_refs.lua` 仍要过 | diff --git a/CHANGELOG.md b/CHANGELOG.md index b1335bed..9d1c080a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,72 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.7.30.2] — 2026-07-30 + +### 修复 + +- **依赖的全局缓存此前净收益为零 —— 命中也全量重编。** 命中的依赖产物由 `prepare_build` 直接拷进 build dir,而这些路径在 `build.ninja` 里仍然是 **compile edge 的输出**;ninja 对「输出存在但 `.ninja_log` 里没有该输出的命令行记录」一律判脏,而新 build dir 天然没有 `.ninja_log`。ninja 自己的话: + + ``` + $ ninja -C target/x86_64-linux-gnu/ -d explain -n + ninja explain: command line not found in log for obj/zutil.o + ninja explain: obj/zutil.o is dirty + ``` + + 于是每一个「命中」的 TU 都被重编一遍,而 CLI 照旧打印 `Cached ` —— 缓存只在「同一个 build dir 已经有 `.ninja_log`」时"生效",而那时本地产物本来就在。 + + 修法不是加 `restat`(log entry 根本不存在,restat 只在有 entry 时改写 mtime 判定),也不是给 compile edge 打 `generator = 1`(那会把「命令行变了要重编」整条规则关掉,用错误换性能)。命中的依赖**改发 `stage_file` 边、不发 compile 边、不发 scan/dyndep 边**:产物落在 compile edge 本来会产生的同一个路径上,所以链接行、消费者 TU 的 BMI implicit input、运行期部署边全都不变,变的只是「这些文件由谁产生」—— 而 ninja 从此有了可比对的命令行记录。 + + 状态行同时带上省下的 TU 数(`Cached compat.zlib v1.3.2 (15 units)`):光秃秃一个 `Cached` 能骗人三个月,一个必须与 ninja 实际跳过的边数吻合的数字骗不了。 + +- **`mcpp build --release` 之后裸 `mcpp build` 会 0.00s「成功」并交付 release 产物。** 与缓存无关的独立缺陷,同根于下面的 profile 轴:`.build_cache` 的条目**只按 target triple 去重**,而 fast path 的准入条件只挡显式 `--profile/--dev/--release`,挡不住裸 `mcpp build`(那恰恰是「我要默认 profile」的意思)。于是 fast path 拿着 release 的 `build.ninja` 跑,`build.ninja` 里写着 `-O2`,产物无 debug info。 + + 条目改按 `(target triple, resolved profile)` 去重,fast path 先用同一条纯规则(新的 `resolve_profile_name`,manifest 之外无输入)算出本次请求的 profile 再匹配;不匹配即当 miss 走完整 `prepare_build`。旧条目没有该字段 ⇒ 读成空 ⇒ 任何 profile 都不匹配 ⇒ 自愈,无需迁移。 + +- **`Finished release [optimized]` 是硬编码的。** 唯一调用点传字面量 `"release"`,`ui::finished` 又硬编码 `[optimized]`,所以 `--dev`(`-O0 -g`)也自称优化过的 release 构建。现在 profile 名来自本次真正解析出的 profile,描述符来自本次真正解析出的开关(`unoptimized + debuginfo` 等),没有调用方需要猜。 + +- **传递的 `path` / `git` 依赖被写进全局缓存。** 排除谓词在 **root manifest** 的 `dependencies`/`dev-dependencies` 里查该包并在 spec 为 path/git 时跳过 —— 但传递到达的包两个表里都没有,于是 `specIt == end()` 让谓词返回「不跳过」,本地源码被缓存,且 `indexName` 回落到默认索引(一个 workspace 成员 `B` 于是以 `mcpplibs/B@0.1.0` 的身份落在磁盘上)。它的源码可以在 `name@version` 不变的情况下改变,缓存键看不见这种变化。 + + 唯一可采信的证据改为**解析期记录的来源**(`DepCacheIdentity::sourceKind`,两个 push 点都在作用域内拿得到):非 `"version"` 一律不进缓存。判据方向与 `mcpp add` 的存在性门**相反** —— 那里「不可证伪就放行」,缓存这里「不能证明来自不可变的 xpkgs store 就不缓存」,因为这里的错误代价是静默的错误对象,不是被拒绝的命令。 + + 此前这个缺陷被上面第一条掩盖着(ninja 反正会重编);修好第一条之后它就是静默错误产物,所以两者必须同批。 + +### 变更 + +- **依赖缓存键:全工程指纹 → 每包 Merkle 键。** 旧键是整个工程的 fingerprint,而它的 flags 字段序列化了图里**每一个包,含 root** —— root 的包名、版本、`[build]` flags 都在里面。后果:改自己的版本号(`0.1.0` → `0.1.1`,实测 `3b8a8ae4fc217233` → `2138d7ce160e1154`)就让 std BMI 与**全部**依赖缓存整体失效;两个工程只要包名不同,即使依赖集与工具链完全一致,也一条都不共享。本机实测:26 GB / 1198 个指纹目录,`compat.zlib@1.3.2` 存了 162 份,`compat.x11@1.8.13` 73 份共 1.49 GB。 + + 新模块 `mcpp.build.cache_key` 按七个轴逐包成键:工具链身份 / 语言与方言 / **profile** / 包身份 / 该包自身的构建配置(含 features 折叠后的结果)/ **每个直接依赖的键(递归,Merkle)** / cache epoch。不含 root 的身份与 flags —— 实测 root 的 `[build] cflags/cxxflags` **不会**下发到依赖 TU,这正是跨工程共享成立的机制依据。 + + F 轴之所以递归而不是「枚举上游的 public 接口」:GCC 把被导入模块 BMI 的 CRC 烙进导入者的 BMI,上游接口或 ABI 一变,旧的导入者 BMI 就 `module 'B' CRC mismatch` + `Bad import dependency`(手工三组对照实测:只改函数体通过,改签名/`-D` 改布局都硬失败)。可枚举清单必然漏项(上游 re-export 出去的传递模块接口在上游自己的 manifest 里根本看不出来),而失败模态是不对称的 —— BMI 轴取窄是编译器硬报错,`.o` 轴取窄是静默错对象。 + + 递归的代价是上游的**私有**变化会级联下游;对真正吃这个缓存的人群 ≈ 0:index 包的描述符按版本冻结,`B` 的键只能因版本 bump(那本就该让消费者失效 —— 它链接 `B` 的对象)或全图轴而变,而唯一能不改版本就动私有 flag 的 path/git 包整体不进缓存。 + +- **std 模块缓存键:全工程指纹 → std 自己的身份。** `stdmod.cppm` 写在产物旁边的 15 字段 metadata 一直就是正确完整的 std 身份,`metadata_matches` 一直在按它校验命中 —— 唯一错的是**目录名用的不是它**。本机实测代价:1014 个目录里只有 **15 个真身份**,16.10 GB(需要 ~0.5 GB),且 `mcpp version bump` 会重新 precompile 一次 std(几十 MB、十几秒)。(实现上有个自指要解:build commands 里含 cacheDir,而 cacheDir 由含它的 metadata 算出 —— 先用占位段生成一份规范化 metadata 算键,再用真目录生成落盘的那份。) + +- **缓存布局迁到 `$MCPP_HOME/build-cache/v1/`,条目自描述。** `pkg//@//` 与 `std//`。刻意不放 `$MCPP_HOME/cache` —— 那个名字已经归 `GlobalConfig` 的索引元数据缓存所有,而它的 reset 路径会整目录删除;把编译产物塞进别人会清空的目录里,会以「构建缓存有时会自己空掉」的形态浮现。`v1` 是布局版本段,所以换布局永远不需要迁移或删除旧树。 + + 每条目写 `entry.json`,记下**键的全部输入**;命中判定从「目录存在 + 文件齐」升级为「schema 匹配 ∧ 键匹配 ∧ 记录的输入与本次算出的输入**逐字段相等** ∧ 清单文件齐」。这是把 std 侧一直做对的事搬到依赖侧:此前依赖条目只有一份文件清单,所以怀疑命中错了的时候什么都审计不了。缺字段视为不匹配,**永不**因为 hash 相等就放行。 + + 旧的 `$MCPP_HOME/bmi/`(按全工程指纹分目录)**不读不写不自动删**:`mcpp doctor` 报出它的体积,`mcpp cache clean --legacy` 回收。它从来没产生过收益(见上面第一条),但删几十 GB 必须是用户的显式动作。 + +- **profile 成为失效轴。** `[profile.]` 把开关落在 `buildConfig.optLevel/debug/lto/strip`,由 flags.cppm 变成 `-O`/`-g`/`-flto`,而 `canonical_compile_flags` 只序列化 cflags/cxxflags/ldflags —— 于是 `--dev`、`--release`、`--profile dist` 三者**同一个 fingerprint**、同一个 `target///`、同一条缓存条目。 + + **可感知的行为变化**:`target//` 下从此每个 profile 一个哈希目录。好处是 dev↔release 来回切从「每次全量」变成增量,两个 profile 的二进制可以并存;代价是磁盘随实际使用的 profile 数增长(每个 build dir 带一份 staged std BMI,本机 `std.gcm` = 31,466,112 字节)。 + +- **缓存失效不再挂在 mcpp 版本号上。** 缓存键的兼容轴改用独立的 `kCacheEpoch`,只在缓存布局/键算法/staging 契约真的不兼容时递增。`fingerprint.cppm` 的 `MCPP_VERSION` 保持在 fingerprint 里(build dir 命名与本地增量归它管,宁可保守),但每次发版把整个缓存作废对纯 C 目标文件是过度失效。 + +### 新增 + +- **`--cache `:三种构建模式。** `global`(默认,读+写全局缓存)/ `local`(不读不写,所有依赖在本工程 `target/` 内编 —— 排障时一次性排除「是不是缓存的问题」,也给 CI 一个无共享的可复现基线)/ `off`(不读不写,并先清掉本次的 `target///` 做冷构建)。优先级 `--cache` > `MCPP_BUILD_CACHE` > `[build] cache` > `global`;无法识别的值会被报出来(`--strict` 下为错误),而不是静默回落到 `global`。 + + `--no-cache` 保留为 `off` 的兼容别名。它的旧 help 文案「Force-clear target/ before building」两处不准:它清的是**构建目录**(`target///`)而不是整个 `target/`,而且名字与缓存无关。`mcpp run` / `mcpp test` 一并补上这两个 flag(此前它们连 `--no-cache` 都没有)。 + +- **`mcpp cache` 补齐到可运维。** `cache dir`(缓存到底在哪 —— 此前 `cache *`/`doctor`/`clean --bmi-cache` 各自解析根目录,而 config 的 reset 路径用 `GlobalConfig::bmiCacheDir`,两者可能不是同一个目录)、`cache gc --max-size {MiB,GiB} / --older-than {s,m,h,d}`(**真 LRU**)、`cache clean --deps|--std|--all|--legacy`、`cache list --json`、`cache verify`(逐条目校验清单与磁盘,残缺条目非零退出)。`cache info` 现在打印该条目的键输入 —— 怀疑命中错了时第一件想看的东西。 + + `prune` 此前按**目录 mtime** 排序,而那只记录条目被**写入**的时间:一个每次构建都命中的热包,和一个一个月没人碰过的冷包一样「陈旧」。`entry.json` 的 `accessed` 由每次命中刷新(只重写 `entry.json`,**不动产物 mtime** —— 那些 mtime 参与 ninja 的 restat 判定),`gc` 按它排序。`cache clean` 开头那句 `remove_all(/"deps")` 指向一个从不存在的路径(dep 条目在 `//deps`),是死代码。 + + `gc` 刻意**不动 std 条目**:一份 std BMI 全机共享、重建要几十秒,为了腾一点磁盘赶走它是拿很多时间换很少空间。达不到容量预算时会明确说出来,而不是静默少删。 + ## [2026.7.30.1] — 2026-07-30 ### 修复 diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index d5233cd7..3141a82a 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -146,6 +146,7 @@ static_stdlib = true # Statically link libstdc++ (default true) target = "x86_64-linux-musl" # Default build target when no --target is passed # (≙ cargo build.target; e.g. "ship fully-static") macos_deployment_target = "14.0" # Minimum supported OS version for macOS artifacts (macOS only) +cache = "global" # Global dependency cache: global (default) | local | off (§2.10) ``` `include_dirs_after` (#249) lists header directories that are searched **after** @@ -717,8 +718,56 @@ ldflags = [] - Built-in profiles: `release` (-O2) / `dev`, `debug` (-O0 -g) / `dist` (-O3 + strip; **LTO is not enabled by default**). `[profile.]` can override a built-in definition wholesale. +- **Each profile owns its own build directory.** The resolved profile knobs + participate in the fingerprint, so `target//` holds one hash directory + per profile and switching between them is incremental instead of a full + rebuild. It also means the disk cost scales with the number of profiles you + actually use. -### 2.10 `[runtime]` — Host Runtime Capabilities +### 2.10 `[build] cache` — The Global Dependency Cache + +Compiled artifacts for dependencies fetched from an index are cached across +projects under `$MCPP_HOME/build-cache/v1/`. A dependency's artifacts do not +depend on who consumes them, so two projects with the same toolchain, profile and +dependency versions reuse one entry. + +```toml +[build] +cache = "global" # "global" (default) | "local" | "off" +``` + +| Mode | Reads the cache | Writes the cache | Clears the build dir first | +|---|---|---|---| +| `global` (default) | yes | yes | no | +| `local` | no | no | no | +| `off` | no | no | yes | + +`local` builds every dependency inside this project's `target/` — useful to rule +the cache out while diagnosing something, and to give CI a no-sharing baseline. +`off` additionally clears this build's `target///` for a cold +rebuild; `--no-cache` is a deprecated alias for it. + +Precedence: `--cache ` **>** `MCPP_BUILD_CACHE` **>** `[build] cache` **>** +`global`. An unrecognized value is reported (an error under `--strict`) rather +than silently falling back to `global`. + +What is **not** cached: `path` and `git` dependencies, at any depth, and +workspace members. Their sources can change without their `name@version` +changing, so no key over that identity could notice the change. + +Inspection and reclamation: + +``` +mcpp cache dir # where the cache lives +mcpp cache list [--json] # entries, sizes, last use +mcpp cache info @ # one entry, including the key inputs it was built with +mcpp cache verify # every entry's file list against the disk +mcpp cache gc --max-size 5GiB # LRU-collect package entries to a budget +mcpp cache gc --older-than 30d # ...or by how long since they were last used +mcpp cache clean [--deps|--std|--all|--legacy] +``` + +### 2.11 `[runtime]` — Host Runtime Capabilities ```toml [runtime] @@ -744,7 +793,7 @@ provider = "compat.glx-runtime" `opengl.glx.driver`, `x11.display`) and prefix-style `abi:` (e.g. `abi:glibc`, which participates in toolchain ABI enforcement). -### 2.11 `[package] platforms` — Platform Declaration +### 2.12 `[package] platforms` — Platform Declaration ```toml [package] @@ -756,7 +805,7 @@ The vocabulary is fixed by mcpp (which owns the target/triple system): `linux | macos | windows`; unknown values produce a warning, and an error under `--strict`. -### 2.12 `[xlings]` — Build Environment +### 2.13 `[xlings]` — Build Environment ```toml [xlings] diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 6063a2b0..670703ce 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -139,6 +139,7 @@ ldflags = ["-lfoo"] # 额外链接参数 defines = ["BIZ=1", "QUX"] # 作用于每个 TU 的预处理宏(脱糖为 -D;会进入模块扫描) static_stdlib = true # 静态链接 libstdc++(默认 true) macos_deployment_target = "14.0" # macOS 产物的最低支持系统版本(仅 macOS 生效) +cache = "global" # 依赖的全局构建缓存:global(默认)| local | off(见 §2.10) ``` `include_dirs_after`(#249)列出**排在工具链系统目录之后**搜索的头文件目录 @@ -506,8 +507,49 @@ ldflags = [] (默认 dev 的项目在产出可分发物时应显式 `--release`。) - 内置档案:`release`(-O2)/ `dev`、`debug`(-O0 -g)/ `dist`(-O3 + strip; **不默认开 lto**)。`[profile.<内置名>]` 可整体覆盖内置定义。 +- **每个 profile 各占一个构建目录。** 解析后的 profile 开关参与指纹,所以 + `target//` 下每个 profile 一个哈希目录,来回切换是增量而不是全量重编; + 代价是磁盘占用随实际使用的 profile 数量增长。 -### 2.10 `[runtime]` — 主机运行时能力 +### 2.10 `[build] cache` — 依赖的全局构建缓存 + +从索引获取的依赖,其编译产物按包缓存在 `$MCPP_HOME/build-cache/v1/` 下,跨工程共享。 +依赖的产物与"谁在消费它"无关,所以工具链、profile、依赖版本相同的两个工程复用同一条目。 + +```toml +[build] +cache = "global" # "global"(默认)| "local" | "off" +``` + +| 模式 | 读缓存 | 写缓存 | 先清构建目录 | +|---|---|---|---| +| `global`(默认) | 是 | 是 | 否 | +| `local` | 否 | 否 | 否 | +| `off` | 否 | 否 | 是 | + +`local` 把所有依赖都编在本工程 `target/` 内 —— 排障时一次性排除"是不是缓存的问题", +也给 CI 一个无共享的可复现基线。`off` 额外清掉本次的 `target///` 做冷构建; +`--no-cache` 是它的兼容别名。 + +优先级:`--cache ` **>** `MCPP_BUILD_CACHE` **>** `[build] cache` **>** `global`。 +无法识别的值会被报出来(`--strict` 下为错误),而不是静默回落到 `global`。 + +**不进缓存的**:`path` 与 `git` 依赖(任意深度)以及 workspace 成员。它们的源码可以在 +`name@version` 不变的情况下改变,任何基于该身份的键都看不见这种变化。 + +查看与回收: + +``` +mcpp cache dir # 缓存在哪 +mcpp cache list [--json] # 条目、体积、最后使用时间 +mcpp cache info @ # 单条目详情,含它是用什么键输入编出来的 +mcpp cache verify # 逐条目校验清单与磁盘 +mcpp cache gc --max-size 5GiB # 按 LRU 收到容量预算内 +mcpp cache gc --older-than 30d # 或按"多久没用过"回收 +mcpp cache clean [--deps|--std|--all|--legacy] +``` + +### 2.11 `[runtime]` — 主机运行时能力 ```toml [runtime] @@ -529,7 +571,7 @@ provider = "compat.glx-runtime" - 能力命名约定:分层小写 `domain.sub.role`(如 `opengl.glx.driver`、 `x11.display`)与前缀类 `abi:`(如 `abi:glibc`,参与工具链 ABI 强制)。 -### 2.11 `[package] platforms` — 平台声明 +### 2.12 `[package] platforms` — 平台声明 ```toml [package] @@ -540,7 +582,7 @@ platforms = ["linux", "macos", "windows"] (它拥有 target/triple 体系):`linux | macos | windows`;未知值 warning, `--strict` 下报错。 -### 2.12 `[xlings]` — 构建环境 +### 2.13 `[xlings]` — 构建环境 ```toml [xlings] diff --git a/mcpp.toml b/mcpp.toml index b2ca09a4..b0a3136c 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.7.30.1" +version = "2026.7.30.2" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/bmi_cache.cppm b/src/bmi_cache.cppm index 068d5f61..24e7f63b 100644 --- a/src/bmi_cache.cppm +++ b/src/bmi_cache.cppm @@ -1,86 +1,124 @@ -// mcpp.bmi_cache — cross-project persistent BMI cache (M3.2). +// mcpp.bmi_cache — cross-project persistent cache of dependency build outputs. // -// Layout (per docs/26-bmi-cache.md): -// $MCPP_HOME/bmi//deps//@/ -// {gcm,pcm}.cache/.{gcm,pcm} -// obj/.m.o + .o -// manifest.txt (sentinel + file list) +// Layout: +// /pkg//@// +// (the cache root is $MCPP_HOME/build-cache/v1 — see mcpp.home::cache_root) +// entry.json sentinel + self-description + file list +// bmi/.{gcm,pcm} +// obj/.o // -// fingerprint already covers compiler / flags / stdlib / mcpp version / -// dep-lock hash etc. (docs/06), so different fingerprints get independent -// cache directories — no risk of stale-BMI cross-contamination. +// comes from mcpp.build.cache_key: a per-package Merkle key over the +// toolchain, the language/dialect settings, the resolved profile, the package's +// own identity and build config, and — recursively — the keys of its direct +// dependencies. See that module's header for why each axis is there. // -// M4 #9: populate is wrapped in an advisory exclusive flock(2) on -// /.lock so two concurrent mcpp builds can race to the same -// dep cache without trashing manifest.txt (docs/26 §5.4 V2). +// Two properties this layout has and the previous one did not: +// +// 1. The directory name is derived only from things that actually reach the +// package's compiler command lines. The old key was the whole-project +// fingerprint, so a consumer's own name and version were part of every +// dependency's cache path: `mcpp version bump` invalidated the entire cache +// and no two projects ever shared an entry. +// +// 2. An entry describes itself. `is_cached` compares the recorded inputs +// field by field against the inputs computed for this build, so a hit is +// evidence rather than the mere existence of a directory. The std module +// path has validated hits this way since it was written +// (mcpp.toolchain.stdmod's metadata_matches); the dependency path carried +// only a file list, which is why nothing could be audited when a wrong +// entry was suspected. +// +// populate_from holds an advisory exclusive lock on the entry directory so two +// concurrent builds racing to fill the same entry cannot interleave writes, and +// writes entry.json last so a crash mid-populate leaves a miss, not a +// half-populated hit. module; export module mcpp.bmi_cache; import std; +import mcpp.libs.json; import mcpp.platform; export namespace mcpp::bmi_cache { +// Schema of entry.json. Bumped only if the file's own shape changes; +// cache-content compatibility is carried by cache_key::kCacheEpoch, which +// travels inside `inputs`. +inline constexpr int kEntrySchema = 1; + struct CacheKey { - std::filesystem::path mcppHome; - std::string fingerprint; - std::string indexName; // "mcpplibs" / "xim" / ... - std::string packageName; // "mcpplibs.cmdline" - std::string version; // "0.0.1" - std::string bmiDirName = "gcm.cache"; // "gcm.cache" | "pcm.cache" + // The resolved cache root (mcpp::home::cache_root()). Passed in rather than + // recomputed here: the layout root must have exactly ONE definition, and a + // second copy of "/build-cache/v1" in this file is precisely the kind + // of cross-file invariant a comment cannot enforce. Tests supply a temp + // directory the same way. + std::filesystem::path cacheRoot; + std::string indexName; // "mcpplibs" / "compat" / ... + std::string packageName; // "compat.zlib" + std::string version; // "1.3.2" + std::string keyHex; // cache_key::key_hex(...) + // The full key inputs, recorded in entry.json and compared field by field + // on a hit. Never trust equal hashes alone. + nlohmann::json inputs; + std::string bmiDirName = "gcm.cache"; // consumer-side directory name std::string manifestTag = "gcm"; // "gcm" | "pcm" std::filesystem::path dir() const { - return mcppHome / "bmi" / fingerprint / "deps" - / indexName / std::format("{}@{}", packageName, version); + return cacheRoot / "pkg" / indexName + / std::format("{}@{}", packageName, version) / keyHex; } - std::filesystem::path manifestFile() const { return dir() / "manifest.txt"; } - std::filesystem::path bmiDir() const { return dir() / bmiDirName; } - std::filesystem::path objDir() const { return dir() / "obj"; } + std::filesystem::path entryFile() const { return dir() / "entry.json"; } + std::filesystem::path bmiDir() const { return dir() / "bmi"; } + std::filesystem::path objDir() const { return dir() / "obj"; } }; -// File names (basename only) that belong to one dep package's cache entry. +// File names (basenames for BMIs, output-relative paths for objects) belonging +// to one package's cache entry. struct DepArtifacts { - std::vector bmiFiles; // basenames in bmiDir/ - std::vector objFiles; // basenames in obj/ + std::vector bmiFiles; + std::vector objFiles; }; -// Cache hit if manifest.txt exists AND every listed file is present. +// True when entry.json exists, its schema matches, its recorded inputs equal +// `key.inputs` field for field, and every listed file is present on disk. bool is_cached(const CacheKey& key); -// Read manifest.txt → DepArtifacts. -std::expected -read_manifest(const CacheKey& key); +// The artifact list of a validated entry. Does NOT copy anything: the ninja +// backend stages cached files through its own `stage_file` edges, so that a +// staged file is the output of an edge ninja has a command-line record for. +// Copying them behind ninja's back — which this module used to do — left every +// staged output with no entry in .ninja_log, and ninja treats that as dirty +// ("command line not found in log"), so every cached dependency was recompiled +// anyway while the CLI reported it as cached. +std::expected resolve_cached(const CacheKey& key); -// Copy missing cached files into projectTarget/{bmiDirName,obj}. Existing -// project outputs are left untouched: BMIs may differ byte-for-byte between -// equivalent builds, and overwriting them would dirty downstream modules. -std::expected -stage_into(const CacheKey& key, - const std::filesystem::path& projectTargetDir); +// Refresh the entry's `accessed` stamp. Rewrites entry.json only — never the +// artifacts, whose mtimes must stay put (ninja's restat handling compares them). +// This is what makes `mcpp cache gc` a real LRU: pruning used to read the +// directory's mtime, which only ever recorded when the entry was WRITTEN, so a +// dependency that hit on every build looked stale. +void touch_accessed(const CacheKey& key); -// Copy fresh build outputs from projectTarget/{bmiDirName,obj} → cache dir -// and write manifest.txt last (atomic-ish sentinel). +// Copy fresh build outputs from projectTarget/{bmiDirName,obj} into the entry, +// then write entry.json last as the sentinel. std::expected populate_from(const CacheKey& key, const std::filesystem::path& projectTargetDir, const DepArtifacts& artifacts); +// Absolute paths of an entry's artifacts, for the stage edges. +std::filesystem::path cached_bmi_path(const CacheKey& key, std::string_view basename); +std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel); + } // namespace mcpp::bmi_cache namespace mcpp::bmi_cache { namespace { -void touch_now(const std::filesystem::path& p) { - std::error_code ec; - auto t = std::chrono::file_clock::now() + std::chrono::seconds(1); - std::filesystem::last_write_time(p, t, ec); -} - bool copy_one(const std::filesystem::path& from, const std::filesystem::path& to, std::error_code& ec) @@ -91,113 +129,123 @@ bool copy_one(const std::filesystem::path& from, return !ec; } -std::string serialize_manifest(std::string_view tag, const DepArtifacts& a) { - std::string out = "# Auto-generated by mcpp bmi_cache. Do not edit.\n"; - for (auto& g : a.bmiFiles) out += std::format("{}: {}\n", tag, g); - for (auto& o : a.objFiles) out += std::format("obj: {}\n", o); - return out; +std::optional read_entry(const std::filesystem::path& p) { + std::ifstream is(p); + if (!is) return std::nullopt; + nlohmann::json j; + try { is >> j; } catch (...) { return std::nullopt; } + return j; } -std::expected -parse_manifest(const std::filesystem::path& p) { - std::ifstream is(p); - if (!is) return std::unexpected(std::format("cannot open '{}'", p.string())); +DepArtifacts artifacts_from(const nlohmann::json& j) { DepArtifacts a; - std::string line; - while (std::getline(is, line)) { - while (!line.empty() && (line.back() == '\r' || line.back() == ' ')) line.pop_back(); - if (line.empty() || line[0] == '#') continue; - if (line.starts_with("gcm: ")) a.bmiFiles.push_back(line.substr(5)); - else if (line.starts_with("pcm: ")) a.bmiFiles.push_back(line.substr(5)); - else if (line.starts_with("obj: ")) a.objFiles.push_back(line.substr(5)); - } + if (auto it = j.find("bmi"); it != j.end() && it->is_array()) + for (auto& v : *it) if (v.is_string()) a.bmiFiles.push_back(v.get()); + if (auto it = j.find("obj"); it != j.end() && it->is_array()) + for (auto& v : *it) if (v.is_string()) a.objFiles.push_back(v.get()); return a; } -} // namespace - -bool is_cached(const CacheKey& key) { - auto mf = key.manifestFile(); - if (!std::filesystem::exists(mf)) return false; - auto arts = parse_manifest(mf); - if (!arts) return false; - // Verify every listed file actually exists on disk. - for (auto& g : arts->bmiFiles) { - if (!std::filesystem::exists(key.bmiDir() / g)) return false; - } - for (auto& o : arts->objFiles) { - if (!std::filesystem::exists(key.objDir() / o)) return false; +// Field-by-field, not `==` on the whole object: an entry written by an older +// mcpp may legitimately carry extra keys, but every key the CURRENT build cares +// about has to be present and equal. A missing key is a mismatch, never a pass. +bool inputs_match(const nlohmann::json& recorded, const nlohmann::json& expected) { + if (!recorded.is_object() || !expected.is_object()) return false; + for (auto it = expected.begin(); it != expected.end(); ++it) { + auto found = recorded.find(it.key()); + if (found == recorded.end()) return false; + if (*found != it.value()) return false; } return true; } -std::expected -read_manifest(const CacheKey& key) { - return parse_manifest(key.manifestFile()); +std::string now_iso8601() { + // No zoned formatting: libstdc++'s `import std` support for + // std::format on chrono types is partial (see fingerprint.cppm's + // hand-rolled hex for the same reason). Seconds since epoch is monotonic + // enough for an LRU stamp and needs no formatting support at all. + auto now = std::chrono::system_clock::now(); + auto secs = std::chrono::duration_cast( + now.time_since_epoch()).count(); + return std::to_string(secs); } -std::expected -stage_into(const CacheKey& key, - const std::filesystem::path& projectTargetDir) -{ - auto arts = parse_manifest(key.manifestFile()); - if (!arts) return std::unexpected(arts.error()); - - auto projectBmi = projectTargetDir / key.bmiDirName; - auto projectObj = projectTargetDir / "obj"; - std::error_code ec; - std::filesystem::create_directories(projectBmi, ec); - std::filesystem::create_directories(projectObj, ec); - - for (auto& g : arts->bmiFiles) { - auto from = key.bmiDir() / g; - auto to = projectBmi / g; - if (std::filesystem::exists(to, ec)) { - ec.clear(); - continue; - } - ec.clear(); - if (!copy_one(from, to, ec)) { - return std::unexpected(std::format( - "stage bmi '{}': {}", g, ec.message())); - } - touch_now(to); - } - for (auto& o : arts->objFiles) { - auto from = key.objDir() / o; - auto to = projectObj / o; - if (std::filesystem::exists(to, ec)) { - ec.clear(); - continue; - } - ec.clear(); - if (!copy_one(from, to, ec)) { - return std::unexpected(std::format( - "stage obj '{}': {}", o, ec.message())); - } - touch_now(to); +std::expected +write_entry(const std::filesystem::path& path, const nlohmann::json& j) { + auto tmp = path; + tmp += ".tmp"; + { + std::ofstream os(tmp, std::ios::binary); + if (!os) return std::unexpected(std::format( + "cannot write cache entry '{}'", tmp.string())); + os << j.dump(2) << "\n"; + if (!os) return std::unexpected(std::format( + "failed while writing cache entry '{}'", tmp.string())); } - return arts; + std::error_code ec; + std::filesystem::rename(tmp, path, ec); + if (ec) return std::unexpected(std::format( + "cache entry rename: {}", ec.message())); + return {}; } -namespace { } // namespace +std::filesystem::path cached_bmi_path(const CacheKey& key, std::string_view basename) { + return key.bmiDir() / std::string(basename); +} + +std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel) { + return key.objDir() / std::filesystem::path(std::string(rel)); +} + +bool is_cached(const CacheKey& key) { + auto j = read_entry(key.entryFile()); + if (!j) return false; + if (j->value("schema", 0) != kEntrySchema) return false; + if (j->value("key", std::string{}) != key.keyHex) return false; + auto it = j->find("inputs"); + if (it == j->end() || !inputs_match(*it, key.inputs)) return false; + + auto arts = artifacts_from(*j); + std::error_code ec; + for (auto& g : arts.bmiFiles) + if (!std::filesystem::exists(cached_bmi_path(key, g), ec)) return false; + for (auto& o : arts.objFiles) + if (!std::filesystem::exists(cached_obj_path(key, o), ec)) return false; + return true; +} + +std::expected resolve_cached(const CacheKey& key) { + auto j = read_entry(key.entryFile()); + if (!j) return std::unexpected(std::format( + "cannot read cache entry '{}'", key.entryFile().string())); + return artifacts_from(*j); +} + +void touch_accessed(const CacheKey& key) { + auto j = read_entry(key.entryFile()); + if (!j) return; + (*j)["accessed"] = now_iso8601(); + (void)write_entry(key.entryFile(), *j); +} + std::expected populate_from(const CacheKey& key, const std::filesystem::path& projectTargetDir, const DepArtifacts& arts) { auto cacheDir = key.dir(); + std::error_code ec; + std::filesystem::create_directories(cacheDir, ec); auto lock = mcpp::platform::fs::FileLock::try_acquire(cacheDir); if (!lock) { - // Another writer holds the lock; treat as success (they'll do it). + // Another writer holds the lock; it will finish the entry. return {}; } auto cacheBmi = key.bmiDir(); auto cacheObj = key.objDir(); - std::error_code ec; std::filesystem::create_directories(cacheBmi, ec); std::filesystem::create_directories(cacheObj, ec); @@ -206,43 +254,44 @@ populate_from(const CacheKey& key, for (auto& g : arts.bmiFiles) { auto from = projectBmi / g; - auto to = cacheBmi / g; if (!std::filesystem::exists(from)) { return std::unexpected(std::format( "expected build output missing: {}", from.string())); } - if (!copy_one(from, to, ec)) { + if (!copy_one(from, cacheBmi / g, ec)) { return std::unexpected(std::format( "populate bmi '{}': {}", g, ec.message())); } } for (auto& o : arts.objFiles) { auto from = projectObj / o; - auto to = cacheObj / o; if (!std::filesystem::exists(from)) { return std::unexpected(std::format( "expected build output missing: {}", from.string())); } - if (!copy_one(from, to, ec)) { + if (!copy_one(from, cached_obj_path(key, o), ec)) { return std::unexpected(std::format( "populate obj '{}': {}", o, ec.message())); } } - // Write manifest.txt LAST as the sentinel — atomic via temp + rename. - { - auto tmp = key.manifestFile(); - tmp += ".tmp"; - { - std::ofstream os(tmp); - os << serialize_manifest(key.manifestTag, arts); - } - std::filesystem::rename(tmp, key.manifestFile(), ec); - if (ec) { - return std::unexpected(std::format( - "populate manifest rename: {}", ec.message())); - } - } - return {}; + + // entry.json LAST — it is the sentinel. Preserve `created` when refilling an + // existing entry so gc's age reporting stays meaningful. + nlohmann::json j; + if (auto prev = read_entry(key.entryFile()); prev && prev->contains("created")) + j["created"] = (*prev)["created"]; + else + j["created"] = now_iso8601(); + j["schema"] = kEntrySchema; + j["key"] = key.keyHex; + j["package"] = std::format("{}/{}@{}", key.indexName, key.packageName, key.version); + j["bmi_dir"] = key.bmiDirName; + j["tag"] = key.manifestTag; + j["inputs"] = key.inputs; + j["bmi"] = arts.bmiFiles; + j["obj"] = arts.objFiles; + j["accessed"] = now_iso8601(); + return write_entry(key.entryFile(), j); } } // namespace mcpp::bmi_cache diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index 3d781293..6ddbe742 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -1,6 +1,21 @@ -// mcpp.bmi_cache.maintenance — global BMI cache inspection + pruning, and the -// shared fs-size/byte-formatting helpers they are built on. -// Bodies moved verbatim from the CLI layer. Zero behavior change. +// mcpp.bmi_cache.maintenance — inspection and pruning of the global build +// cache, plus the shared fs-size/byte-formatting helpers they are built on. +// +// Layout walked here (see bmi_cache.cppm): +// $MCPP_HOME/build-cache/v1/pkg//@//entry.json +// $MCPP_HOME/build-cache/v1/std//std-module.json +// +// Two things this file did wrong before the layout change and that the new +// entry.json fixes: +// +// * pruning ranked entries by the DIRECTORY's mtime, which only ever recorded +// when an entry was written. A dependency that hit on every single build +// looked as stale as one nobody had touched in a month, so `prune` was +// "drop what was populated long ago", not an LRU. entry.json carries an +// `accessed` stamp that bmi_cache::touch_accessed refreshes on every hit. +// * `clean` began with remove_all(/"deps"), a path that never existed +// (dep entries lived at //deps) — dead code in front of the loop +// that actually did the work. module; #include @@ -9,7 +24,8 @@ module; export module mcpp.bmi_cache.maintenance; import std; -import mcpp.toolchain.stdmod; +import mcpp.home; +import mcpp.libs.json; import mcpp.ui; namespace mcpp::bmi_cache { @@ -37,124 +53,338 @@ export std::string human_bytes(std::uintmax_t n) { return std::format("{:.1f} {}", v, units[u]); } +// Parse `{s,m,h,d}` into seconds. +export std::optional parse_duration(std::string_view v) { + if (v.size() < 2) return std::nullopt; + char unit = v.back(); + std::int64_t n = 0; + try { n = std::stoll(std::string(v.substr(0, v.size() - 1))); } + catch (...) { return std::nullopt; } + switch (unit) { + case 's': return n; + case 'm': return n * 60; + case 'h': return n * 3600; + case 'd': return n * 86400; + default: return std::nullopt; + } +} + +// Parse `{B,KiB,MiB,GiB,TiB}` (suffix optional, case-insensitive, bare +// number = bytes) into bytes. +export std::optional parse_size(std::string_view v) { + std::size_t digits = 0; + while (digits < v.size() && std::isdigit(static_cast(v[digits]))) + ++digits; + if (digits == 0) return std::nullopt; + std::uintmax_t n = 0; + try { n = std::stoull(std::string(v.substr(0, digits))); } + catch (...) { return std::nullopt; } + std::string suffix; + for (auto c : v.substr(digits)) + if (!std::isspace(static_cast(c))) + suffix.push_back(static_cast(std::tolower( + static_cast(c)))); + if (suffix.empty() || suffix == "b") return n; + if (suffix == "k" || suffix == "kib" || suffix == "kb") return n * 1024ull; + if (suffix == "m" || suffix == "mib" || suffix == "mb") return n * 1024ull * 1024; + if (suffix == "g" || suffix == "gib" || suffix == "gb") return n * 1024ull * 1024 * 1024; + if (suffix == "t" || suffix == "tib" || suffix == "tb") + return n * 1024ull * 1024 * 1024 * 1024; + return std::nullopt; +} + +namespace { -// ─── M4 #4: mcpp cache list / prune / clean / info ────────────────────── -struct CacheEntry { - std::filesystem::path dir; - std::string fingerprint; - std::string pkgAtVer; // "/@" - std::uintmax_t size = 0; - std::filesystem::file_time_type lastWrite{}; - std::size_t fileCount = 0; +std::filesystem::path pkg_root() { return mcpp::home::cache_root() / "pkg"; } +std::filesystem::path std_root() { return mcpp::home::cache_root() / "std"; } + +struct Entry { + std::filesystem::path dir; + std::string kind; // "pkg" | "std" + std::string label; // "/@" | "std" + std::string key; + std::uintmax_t size = 0; + std::int64_t accessed = 0; // seconds since epoch; 0 = unknown + std::size_t fileCount = 0; + bool complete = true; + std::string problem; }; -static std::vector walk_cache_entries() { - std::vector entries; - auto bmi = mcpp::toolchain::default_cache_root(); +std::optional read_json(const std::filesystem::path& p) { + std::ifstream is(p); + if (!is) return std::nullopt; + nlohmann::json j; + try { is >> j; } catch (...) { return std::nullopt; } + return j; +} + +std::int64_t stamp_of(const nlohmann::json& j, const char* field) { + auto it = j.find(field); + if (it == j.end()) return 0; + if (it->is_number_integer()) return it->get(); + if (it->is_string()) { + try { return std::stoll(it->get()); } catch (...) {} + } + return 0; +} + +// mtime fallback for entries whose stamp is missing (written by an mcpp that +// predates `accessed`). Reported as such rather than silently treated as fresh. +std::int64_t dir_mtime_seconds(const std::filesystem::path& p) { + std::error_code ec; + auto t = std::filesystem::last_write_time(p, ec); + if (ec) return 0; + return std::chrono::duration_cast( + t.time_since_epoch()).count(); +} + +void measure(Entry& e) { + e.size = dir_size(e.dir); + std::error_code ec; + for (auto& _ : std::filesystem::recursive_directory_iterator(e.dir, ec)) { + (void)_; + if (ec) break; + ++e.fileCount; + } +} + +// Verify a package entry's file list against the disk. +void check_pkg_files(Entry& e, const nlohmann::json& j) { std::error_code ec; - if (!std::filesystem::exists(bmi, ec)) return entries; - - for (auto& fpEntry : std::filesystem::directory_iterator(bmi, ec)) { - auto fpDir = fpEntry.path(); - auto depsDir = fpDir / "deps"; - if (!std::filesystem::exists(depsDir, ec)) continue; - for (auto& idxEntry : std::filesystem::directory_iterator(depsDir, ec)) { - for (auto& pkgEntry : std::filesystem::directory_iterator(idxEntry.path(), ec)) { - CacheEntry e; - e.dir = pkgEntry.path(); - e.fingerprint = fpDir.filename().string(); - e.pkgAtVer = idxEntry.path().filename().string() - + "/" + pkgEntry.path().filename().string(); - e.size = dir_size(e.dir); - e.lastWrite = std::filesystem::last_write_time(e.dir, ec); - for (auto& _ : std::filesystem::recursive_directory_iterator(e.dir, ec)) { - if (!ec) ++e.fileCount; + auto missing = [&](const std::filesystem::path& p) { + return !std::filesystem::exists(p, ec); + }; + if (auto it = j.find("bmi"); it != j.end() && it->is_array()) { + for (auto& v : *it) { + if (!v.is_string()) continue; + if (missing(e.dir / "bmi" / v.get())) { + e.complete = false; + e.problem = std::format("missing bmi/{}", v.get()); + return; + } + } + } + if (auto it = j.find("obj"); it != j.end() && it->is_array()) { + for (auto& v : *it) { + if (!v.is_string()) continue; + if (missing(e.dir / "obj" / v.get())) { + e.complete = false; + e.problem = std::format("missing obj/{}", v.get()); + return; + } + } + } +} + +std::vector walk_pkg_entries(bool verify = false) { + std::vector out; + std::error_code ec; + auto root = pkg_root(); + if (!std::filesystem::exists(root, ec)) return out; + for (auto& idx : std::filesystem::directory_iterator(root, ec)) { + if (!idx.is_directory()) continue; + for (auto& pkg : std::filesystem::directory_iterator(idx.path(), ec)) { + if (!pkg.is_directory()) continue; + for (auto& key : std::filesystem::directory_iterator(pkg.path(), ec)) { + if (!key.is_directory()) continue; + Entry e; + e.dir = key.path(); + e.kind = "pkg"; + e.key = key.path().filename().string(); + e.label = idx.path().filename().string() + "/" + + pkg.path().filename().string(); + auto j = read_json(e.dir / "entry.json"); + if (!j) { + e.complete = false; + e.problem = "no readable entry.json"; + } else { + e.accessed = stamp_of(*j, "accessed"); + if (verify) check_pkg_files(e, *j); } - entries.push_back(std::move(e)); + if (e.accessed == 0) e.accessed = dir_mtime_seconds(e.dir); + measure(e); + out.push_back(std::move(e)); } } } - return entries; + return out; } -static std::string format_age(std::filesystem::file_time_type t) { - auto now = std::chrono::file_clock::now(); - auto diff = std::chrono::duration_cast(now - t).count(); - if (diff < 60) return std::format("{}s ago", diff); - if (diff < 3600) return std::format("{}m ago", diff / 60); - if (diff < 86400) return std::format("{}h ago", diff / 3600); +std::vector walk_std_entries() { + std::vector out; + std::error_code ec; + auto root = std_root(); + if (!std::filesystem::exists(root, ec)) return out; + for (auto& d : std::filesystem::directory_iterator(root, ec)) { + if (!d.is_directory()) continue; + Entry e; + e.dir = d.path(); + e.kind = "std"; + e.key = d.path().filename().string(); + auto j = read_json(e.dir / "std-module.json"); + if (!j) { + e.complete = false; + e.problem = "no readable std-module.json"; + e.label = "std"; + } else { + e.label = std::format("std {}@{} {} {}", + j->value("compiler", std::string{"?"}), + j->value("compiler_version", std::string{"?"}), + j->value("cpp_standard", std::string{"?"}), + j->value("stdlib", std::string{"?"})); + } + e.accessed = dir_mtime_seconds(e.dir); + measure(e); + out.push_back(std::move(e)); + } + return out; +} + +std::string format_age(std::int64_t stampSeconds) { + if (stampSeconds == 0) return "unknown"; + auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + auto diff = now - stampSeconds; + if (diff < 0) return "just now"; + if (diff < 60) return std::format("{}s ago", diff); + if (diff < 3600) return std::format("{}m ago", diff / 60); + if (diff < 86400) return std::format("{}h ago", diff / 3600); return std::format("{}d ago", diff / 86400); } -// `mcpp cache` is dispatched at the App level — list / info / prune / clean -// each get their own action lambda invoking one of these helpers. +std::uintmax_t remove_entry(const Entry& e) { + std::error_code ec; + std::filesystem::remove_all(e.dir, ec); + if (ec) return 0; + // Drop the now-empty @ and directories so `cache list` + // does not accumulate empty shells. + for (auto p = e.dir.parent_path(); + p != mcpp::home::cache_root() && p.has_parent_path(); + p = p.parent_path()) { + std::error_code rmEc; + if (!std::filesystem::remove(p, rmEc)) break; // non-empty: stop + } + return e.size; +} + +} // namespace -// `mcpp cache list`. -export int cache_list() { - auto entries = walk_cache_entries(); - if (entries.empty()) { - std::println("(BMI cache is empty)"); +// `mcpp cache dir` — where IS the cache? Before this, `cache *`, `doctor` and +// `clean --bmi-cache` each resolved the root themselves while config.cppm's +// reset path used GlobalConfig::bmiCacheDir, and the copies could disagree. +export int cache_dir() { + std::println("{}", mcpp::home::cache_root().string()); + auto legacy = mcpp::home::legacy_bmi_root(); + std::error_code ec; + if (std::filesystem::exists(legacy, ec)) { + std::println("legacy (unused, removable with `mcpp cache clean --legacy`): {}", + legacy.string()); + } + return 0; +} + +// `mcpp cache list [--json]`. +export int cache_list(bool asJson) { + auto entries = walk_pkg_entries(); + auto stds = walk_std_entries(); + + if (asJson) { + nlohmann::json j; + j["root"] = mcpp::home::cache_root().string(); + j["entries"] = nlohmann::json::array(); + for (auto* set : {&entries, &stds}) { + for (auto& e : *set) { + j["entries"].push_back({ + {"kind", e.kind}, + {"label", e.label}, + {"key", e.key}, + {"dir", e.dir.string()}, + {"bytes", e.size}, + {"files", e.fileCount}, + {"accessed", e.accessed}, + {"complete", e.complete}, + }); + } + } + std::println("{}", j.dump(2)); return 0; } - std::println("{:<18} {:>10} {:>14} {}", - "fingerprint", "size", "last accessed", "package"); - for (auto& e : entries) { - auto fp = e.fingerprint.size() > 16 - ? e.fingerprint.substr(0, 16) : e.fingerprint; - std::println("{:<18} {:>10} {:>14} {}", - fp, human_bytes(e.size), format_age(e.lastWrite), e.pkgAtVer); + + if (entries.empty() && stds.empty()) { + std::println("(build cache is empty)"); + return 0; + } + std::println("{:<16} {:<6} {:>10} {:>14} {}", + "key", "kind", "size", "last used", "package"); + std::uintmax_t total = 0; + for (auto* set : {&stds, &entries}) { + for (auto& e : *set) { + total += e.size; + std::println("{:<16} {:<6} {:>10} {:>14} {}{}", + e.key.substr(0, 16), e.kind, human_bytes(e.size), + format_age(e.accessed), e.label, + e.complete ? "" : " (incomplete)"); + } } + std::println(""); + std::println("{} entries, {}", entries.size() + stds.size(), human_bytes(total)); return 0; } // `mcpp cache info @`. export int cache_info(const std::string& needle) { - auto entries = walk_cache_entries(); + auto entries = walk_pkg_entries(/*verify=*/true); + bool found = false; for (auto& e : entries) { - if (e.pkgAtVer.ends_with(needle)) { - std::println("dir = {}", e.dir.string()); - std::println("fingerprint = {}", e.fingerprint); - std::println("package = {}", e.pkgAtVer); - std::println("size = {}", human_bytes(e.size)); - std::println("file count = {}", e.fileCount); - std::println("last write = {}", format_age(e.lastWrite)); - return 0; + if (e.label.find(needle) == std::string::npos) continue; + found = true; + std::println("dir = {}", e.dir.string()); + std::println("key = {}", e.key); + std::println("package = {}", e.label); + std::println("size = {}", human_bytes(e.size)); + std::println("file count = {}", e.fileCount); + std::println("last used = {}", format_age(e.accessed)); + std::println("complete = {}{}", e.complete ? "yes" : "no", + e.complete ? "" : " (" + e.problem + ")"); + if (auto j = read_json(e.dir / "entry.json"); j && j->contains("inputs")) { + std::println("inputs ="); + std::println("{}", (*j)["inputs"].dump(2)); } + std::println(""); } - std::println("no cache entry matching '{}'", needle); - return 1; + if (!found) { + std::println("no cache entry matching '{}'", needle); + return 1; + } + return 0; } -// `mcpp cache prune --older-than {s,m,h,d}` (v = raw option value). +// `mcpp cache prune --older-than {s,m,h,d}` — kept as the age-only form. export int cache_prune(const std::string& v) { if (v.empty()) { mcpp::ui::error("`mcpp cache prune` requires --older-than {s,m,h,d}"); return 2; } - char unit = v.back(); - long long n = 0; - try { n = std::stoll(v.substr(0, v.size() - 1)); } - catch (...) { mcpp::ui::error(std::format("bad --older-than value '{}'", v)); return 2; } - std::chrono::seconds threshold{0}; - if (unit == 's') threshold = std::chrono::seconds(n); - else if (unit == 'm') threshold = std::chrono::seconds(n * 60); - else if (unit == 'h') threshold = std::chrono::seconds(n * 3600); - else if (unit == 'd') threshold = std::chrono::seconds(n * 86400); - else { mcpp::ui::error(std::format("bad time unit '{}': use s/m/h/d", unit)); return 2; } - auto cutoff = std::chrono::file_clock::now() - threshold; - auto entries = walk_cache_entries(); + auto secs = parse_duration(v); + if (!secs) { + mcpp::ui::error(std::format( + "bad --older-than value '{}' (expected {{s,m,h,d}})", v)); + return 2; + } + auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + auto cutoff = now - *secs; + int removed = 0; std::uintmax_t freed = 0; - for (auto& e : entries) { - if (e.lastWrite < cutoff) { - std::error_code ec; - std::filesystem::remove_all(e.dir, ec); - if (!ec) { - ++removed; - freed += e.size; - mcpp::ui::status("Pruned", - std::format("{} ({})", e.pkgAtVer, human_bytes(e.size))); - } + for (auto& e : walk_pkg_entries()) { + if (e.accessed != 0 && e.accessed >= cutoff) continue; + auto n = remove_entry(e); + if (n || !std::filesystem::exists(e.dir)) { + ++removed; + freed += e.size; + mcpp::ui::status("Pruned", + std::format("{} ({})", e.label, human_bytes(e.size))); } } std::println(""); @@ -162,19 +392,132 @@ export int cache_prune(const std::string& v) { return 0; } -// `mcpp cache clean` — drop dep entries, preserve std BMIs. -export int cache_clean() { - auto bmi = mcpp::toolchain::default_cache_root(); +// `mcpp cache gc [--max-size N] [--older-than N]` — real LRU, driven by +// entry.json's `accessed` stamp. +export int cache_gc(const std::string& maxSizeArg, const std::string& olderThanArg) { + if (maxSizeArg.empty() && olderThanArg.empty()) { + mcpp::ui::error("`mcpp cache gc` requires --max-size {MiB,GiB} " + "and/or --older-than {s,m,h,d}"); + return 2; + } + std::optional maxSize; + if (!maxSizeArg.empty()) { + maxSize = parse_size(maxSizeArg); + if (!maxSize) { + mcpp::ui::error(std::format( + "bad --max-size value '{}' (expected {{B,KiB,MiB,GiB,TiB}})", + maxSizeArg)); + return 2; + } + } + std::optional olderThan; + if (!olderThanArg.empty()) { + olderThan = parse_duration(olderThanArg); + if (!olderThan) { + mcpp::ui::error(std::format( + "bad --older-than value '{}' (expected {{s,m,h,d}})", olderThanArg)); + return 2; + } + } + + // Package entries only. A std BMI is shared by every project on the machine + // and costs ~30 s to rebuild; evicting one to hit a size target trades a lot + // of time for a little disk. `cache clean --std` remains the explicit way. + auto entries = walk_pkg_entries(); + std::ranges::sort(entries, [](const Entry& a, const Entry& b) { + return a.accessed < b.accessed; // oldest first + }); + + int removed = 0; + std::uintmax_t freed = 0; + std::uintmax_t live = 0; + for (auto& e : entries) live += e.size; + + auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + + for (auto& e : entries) { + bool tooOld = olderThan && e.accessed != 0 + && (now - e.accessed) > *olderThan; + bool overBudget = maxSize && live > *maxSize; + if (!tooOld && !overBudget) continue; + remove_entry(e); + ++removed; + freed += e.size; + live -= e.size; + mcpp::ui::status("Collected", + std::format("{} ({}, last used {})", + e.label, human_bytes(e.size), format_age(e.accessed))); + } + std::println(""); + std::println("Collected {} entries, freed {} (cache now {})", + removed, human_bytes(freed), human_bytes(live)); + if (maxSize && live > *maxSize) { + // Say it rather than silently under-delivering: a size target that + // cannot be met without evicting std BMIs is a real outcome, and a + // caller that thinks the budget held would be misled. + mcpp::ui::warning(std::format( + "still {} over the {} budget: package entries are exhausted. " + "std BMIs are excluded from gc; use `mcpp cache clean --std`", + human_bytes(live - *maxSize), human_bytes(*maxSize))); + } + return 0; +} + +// `mcpp cache clean [--deps] [--std] [--all] [--legacy]`. +export int cache_clean(bool deps, bool stds, bool all, bool legacy) { + if (all) { deps = true; stds = true; } + if (!deps && !stds && !legacy) deps = true; // bare `clean` = dep entries + std::error_code ec; - std::filesystem::remove_all(bmi / "deps", ec); // deps only; preserve std.gcm - if (std::filesystem::exists(bmi)) { - for (auto& f : std::filesystem::directory_iterator(bmi, ec)) { - auto deps = f.path() / "deps"; - std::filesystem::remove_all(deps, ec); + if (deps) { + auto n = dir_size(pkg_root()); + std::filesystem::remove_all(pkg_root(), ec); + std::println("Cleaned package entries ({})", human_bytes(n)); + } + if (stds) { + auto n = dir_size(std_root()); + std::filesystem::remove_all(std_root(), ec); + std::println("Cleaned std module entries ({})", human_bytes(n)); + } + if (legacy) { + auto legacyRoot = mcpp::home::legacy_bmi_root(); + if (std::filesystem::exists(legacyRoot, ec)) { + auto n = dir_size(legacyRoot); + std::filesystem::remove_all(legacyRoot, ec); + std::println("Removed pre-v1 cache {} ({})", legacyRoot.string(), + human_bytes(n)); + } else { + std::println("No pre-v1 cache at {}", legacyRoot.string()); } } - std::println("Cleaned all dep BMI cache entries (std.gcm preserved)"); return 0; } +// `mcpp cache verify` — entry.json against the disk, for every entry. +export int cache_verify() { + auto entries = walk_pkg_entries(/*verify=*/true); + auto stds = walk_std_entries(); + int bad = 0; + for (auto* set : {&entries, &stds}) { + for (auto& e : *set) { + if (e.complete) continue; + ++bad; + mcpp::ui::error(std::format("{} [{}] {}", + e.label, e.key, e.problem)); + std::println(" {}", e.dir.string()); + } + } + auto total = entries.size() + stds.size(); + if (bad == 0) { + std::println("{} entries verified, all complete", total); + return 0; + } + std::println(""); + std::println("{} of {} entries are incomplete. They are treated as misses " + "and rebuilt; `mcpp cache gc --older-than 0s` or " + "`mcpp cache clean` reclaims them.", bad, total); + return 1; +} + } // namespace mcpp::bmi_cache diff --git a/src/build/cache_key.cppm b/src/build/cache_key.cppm new file mode 100644 index 00000000..f461a7e4 --- /dev/null +++ b/src/build/cache_key.cppm @@ -0,0 +1,356 @@ +// mcpp.build.cache_key — per-package identity for the global build cache. +// +// The global dep cache used to be keyed by the WHOLE-PROJECT fingerprint +// (src/toolchain/fingerprint.cppm), whose flags field folds in every package in +// the graph *including the root* — its name, its version, its [build] flags. +// Consequences, all measured: +// +// * bumping only the root's `version` changed the key ⇒ every dependency +// entry and the std BMI were invalidated by `mcpp version bump`; +// * two projects with identical dependencies and toolchain shared nothing, +// because their package names differ; +// * on one developer machine that produced 26 GB across 1198 fingerprint +// directories: `compat.zlib@1.3.2` stored 162 times, 15 distinct std module +// identities stored 1014 times. +// +// A dependency's artifacts do not depend on the consumer's identity. Verified: +// the root's `[build] cflags/cxxflags` are NOT passed to dependency translation +// units — a dependency compiles with its own buildConfig plus the shared +// toolchain and profile flags. So the key is built per package, from exactly +// the axes that reach that package's compiler command lines: +// +// A toolchain compiler id/version/driver identity, target triple, stdlib +// B language C++ standard + flag, dialect flags, C standard, macOS target +// C profile opt level / debug / lto / strip ← was missing entirely +// D identity index name, package FQN, version +// E own config features, cflags/cxxflags/asmflags/ldflags, per-glob flags, +// defines, generated files, include dirs, source list +// F upstream the KEY of each direct dependency, recursively (Merkle) +// G epoch kCacheEpoch — cache-format compatibility, decoupled from the +// mcpp release number +// +// Why F is recursive rather than "the upstream's public includes + defines": +// GCC embeds a CRC of an imported module's BMI into the importer's BMI. Rebuild +// a dependency with a changed interface (or a changed ABI via a `-D`) and the +// importer's stale BMI hard-fails with `module 'B' CRC mismatch` / +// `Bad import dependency`. The importer's BMI is bound to the exact BMI it read, +// so what its key needs is the upstream's identity itself, not an enumeration +// of interface surface that can silently miss an item (re-exported transitive +// modules are not even visible in the upstream's own manifest). The failure +// modes are asymmetric, which settles the direction: too narrow a key on the +// BMI axis is a loud compiler error, while too narrow a key on the object axis +// is a silently wrong `.o` — objects carry no such self-check. +// +// The recursion's real cost is that an upstream's PRIVATE changes cascade +// downstream. For the population this cache serves that is ~zero: an index +// package's descriptor is frozen per version, so its key can only move via a +// version bump (which must invalidate consumers anyway — they link its objects) +// or via a whole-graph axis. Path and git packages, the only ones that can +// change private flags without a version bump, never enter the cache at all. + +export module mcpp.build.cache_key; + +import std; +import mcpp.libs.json; +import mcpp.manifest; +import mcpp.modgraph.scanner; +import mcpp.toolchain.detect; +import mcpp.toolchain.fingerprint; + +export namespace mcpp::build::cache_key { + +// Bump ONLY when a change makes previously written entries unusable (the +// serialized input shape, the artifact layout, or the staging contract). +// Deliberately NOT the mcpp release number: folding the whole version in +// orphaned every entry on every release, including plain C object files whose +// validity has nothing to do with mcpp's version. +inline constexpr int kCacheEpoch = 1; + +// Axes A/B/C — identical for every package in one build, computed once. +struct BuildAxes { + // A + std::string compilerId; + std::string compilerVersion; + std::string driverIdentity; + std::string targetTriple; + std::string stdlibId; + std::string stdlibVersion; + // B + std::string cppStandard; + std::string cppStandardFlag; + std::vector dialectFlags; + std::string cStandard; + std::string macosDeploymentTarget; + // C + std::string optLevel; + bool debug = false; + bool lto = false; + bool strip = false; +}; + +// Axes D/E/F for one package. +struct PackageAxes { + // D + std::string indexName; + std::string packageName; + std::string version; + // E + // Filled by the caller: the union of features requested of this package + // over every incoming dependency edge, sorted. Their EFFECTS are already + // folded into the vectors below (a feature contributes -DMCPP_FEATURE_*, + // featureDefines, featureFlags and featureSources before the key is + // computed), so this is redundant for correctness — it is here so + // entry.json says which features an entry was built with, which is the + // first thing anyone diagnosing a wrong hit will want. + std::vector features; + std::vector cflags; + std::vector cxxflags; + std::vector ldflags; + std::vector defines; + std::vector globFlags; // pre-serialized, ordered + std::vector generatedFiles;// "path=content", ordered + std::vector includeDirs; // store-relative, ordered + std::vector sourceGlobs; // [build] sources, ordered + std::vector sources; // package-root-relative, sorted + // F — keys of direct dependencies, sorted + std::vector upstreamKeys; +}; + +// The canonical serialization. Also what lands in entry.json, so a cache hit +// can be validated field by field instead of trusting that equal hashes mean +// equal inputs. +nlohmann::json to_json(const BuildAxes& b, const PackageAxes& p); + +// 16 hex chars, from the canonical serialization. +std::string key_hex(const BuildAxes& b, const PackageAxes& p); + +// Axes A/B/C from a resolved toolchain + the root manifest (which is where the +// whole-graph language and profile settings live). +BuildAxes build_axes(const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::Manifest& rootManifest, + std::string_view cppStandardFlag, + const std::vector& dialectFlags, + std::string_view macosDeploymentTarget); + +// Axes E from one PackageRoot. `storeRoot` is stripped off absolute include +// dirs so the key survives a different MCPP_HOME (the payload paths are +// /registry/data/xpkgs/...; leaving them absolute would make every +// entry a miss on another machine, or after a home relocation). +void fill_package_config(PackageAxes& out, + const mcpp::modgraph::PackageRoot& pkg, + const std::filesystem::path& storeRoot); + +} // namespace mcpp::build::cache_key + +namespace mcpp::build::cache_key { + +namespace { + +// Length-prefixed field joining. A plain separator would let ("a", "bc") and +// ("ab", "c") collide, which for a cache key means serving one package's +// objects for another's. +void put(std::string& s, std::string_view label, std::string_view value) { + s += label; + s += '='; + s += std::to_string(value.size()); + s += ':'; + s += value; + s += '\x1f'; +} + +void put_list(std::string& s, std::string_view label, + const std::vector& values) { + s += label; + s += '['; + s += std::to_string(values.size()); + s += ']'; + for (auto& v : values) { + s += '\x1e'; + s += std::to_string(v.size()); + s += ':'; + s += v; + } + s += '\x1f'; +} + +} // namespace + +nlohmann::json to_json(const BuildAxes& b, const PackageAxes& p) { + nlohmann::json j; + j["epoch"] = kCacheEpoch; + j["toolchain"] = { + {"compiler", b.compilerId}, + {"compiler_version", b.compilerVersion}, + {"driver_identity", b.driverIdentity}, + {"target_triple", b.targetTriple}, + {"stdlib", b.stdlibId}, + {"stdlib_version", b.stdlibVersion}, + }; + j["language"] = { + {"cpp_standard", b.cppStandard}, + {"cpp_standard_flag", b.cppStandardFlag}, + {"dialect_flags", b.dialectFlags}, + {"c_standard", b.cStandard}, + {"macos_deployment_target", b.macosDeploymentTarget}, + }; + j["profile"] = { + {"opt_level", b.optLevel}, + {"debug", b.debug}, + {"lto", b.lto}, + {"strip", b.strip}, + }; + j["package"] = { + {"index", p.indexName}, + {"name", p.packageName}, + {"version", p.version}, + }; + j["config"] = { + {"features", p.features}, + {"cflags", p.cflags}, + {"cxxflags", p.cxxflags}, + {"ldflags", p.ldflags}, + {"defines", p.defines}, + {"glob_flags", p.globFlags}, + {"generated_files", p.generatedFiles}, + {"include_dirs", p.includeDirs}, + {"source_globs", p.sourceGlobs}, + {"sources", p.sources}, + }; + j["upstream"] = p.upstreamKeys; + return j; +} + +std::string key_hex(const BuildAxes& b, const PackageAxes& p) { + std::string s = "mcpp-cache-key-v1\x1f"; + put(s, "epoch", std::to_string(kCacheEpoch)); + // A + put(s, "cc", b.compilerId); + put(s, "ccver", b.compilerVersion); + put(s, "driver", b.driverIdentity); + put(s, "triple", b.targetTriple); + put(s, "stdlib", b.stdlibId); + put(s, "stdlibv", b.stdlibVersion); + // B + put(s, "std", b.cppStandard); + put(s, "stdflag", b.cppStandardFlag); + put_list(s, "dialect", b.dialectFlags); + put(s, "cstd", b.cStandard); + put(s, "macos", b.macosDeploymentTarget); + // C + put(s, "opt", b.optLevel); + put(s, "debug", b.debug ? "1" : "0"); + put(s, "lto", b.lto ? "1" : "0"); + put(s, "strip", b.strip ? "1" : "0"); + // D + put(s, "index", p.indexName); + put(s, "pkg", p.packageName); + put(s, "ver", p.version); + // E + put_list(s, "features", p.features); + put_list(s, "cflags", p.cflags); + put_list(s, "cxxflags", p.cxxflags); + put_list(s, "ldflags", p.ldflags); + put_list(s, "defines", p.defines); + put_list(s, "globflags", p.globFlags); + put_list(s, "genfiles", p.generatedFiles); + put_list(s, "includes", p.includeDirs); + put_list(s, "srcglobs", p.sourceGlobs); + put_list(s, "sources", p.sources); + // F + put_list(s, "upstream", p.upstreamKeys); + return mcpp::toolchain::hash_string(s); +} + +BuildAxes build_axes(const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::Manifest& rootManifest, + std::string_view cppStandardFlag, + const std::vector& dialectFlags, + std::string_view macosDeploymentTarget) +{ + BuildAxes b; + b.compilerId = std::string(tc.compiler_name()); + b.compilerVersion = tc.version; + // Same rule the whole-project fingerprint uses: prefer the declared driver + // identity, else hash the driver binary. Two payloads of "gcc 16.1.0" that + // are not the same build must not share a cache entry. + b.driverIdentity = !tc.driverIdent.empty() + ? mcpp::toolchain::hash_string(tc.driverIdent) + : (tc.binaryPath.empty() ? std::string{} + : mcpp::toolchain::hash_file(tc.binaryPath)); + b.targetTriple = tc.targetTriple; + b.stdlibId = tc.stdlibId; + b.stdlibVersion = tc.stdlibVersion; + + b.cppStandard = rootManifest.package.standard; + b.cppStandardFlag = std::string(cppStandardFlag); + b.dialectFlags = dialectFlags; + b.cStandard = rootManifest.buildConfig.cStandard; + b.macosDeploymentTarget = std::string(macosDeploymentTarget); + + b.optLevel = rootManifest.buildConfig.optLevel; + b.debug = rootManifest.buildConfig.debug; + b.lto = rootManifest.buildConfig.lto; + b.strip = rootManifest.buildConfig.strip; + return b; +} + +void fill_package_config(PackageAxes& out, + const mcpp::modgraph::PackageRoot& pkg, + const std::filesystem::path& storeRoot) +{ + const auto& bc = pkg.manifest.buildConfig; + + out.cflags = bc.cflags; + out.cxxflags = bc.cxxflags; + out.ldflags = bc.ldflags; + out.defines = bc.defines; + out.sourceGlobs = bc.sources; + + if (!bc.cStandard.empty()) { + // A package may pin its own C standard; it reaches its own C units. + out.cflags.push_back("__c_standard=" + bc.cStandard); + } + + for (auto const& gf : bc.globFlags) { + std::string one = "glob:" + gf.glob; + for (auto const& f : gf.cflags) one += "\x1egc:" + f; + for (auto const& f : gf.cxxflags) one += "\x1egxx:" + f; + for (auto const& f : gf.asmflags) one += "\x1egas:" + f; + for (auto const& f : gf.defines) one += "\x1egd:" + f; + out.globFlags.push_back(std::move(one)); + } + + for (auto const& [path, content] : bc.generatedFiles) { + out.generatedFiles.push_back( + path.generic_string() + "=" + content); + } + std::ranges::sort(out.generatedFiles); + + // Include dirs are absolute payload paths under the xpkgs store; make them + // store-relative so the key does not encode this machine's MCPP_HOME. + auto storeStr = storeRoot.generic_string(); + auto relativize = [&](const std::filesystem::path& p) { + auto s = p.generic_string(); + if (!storeStr.empty() && s.starts_with(storeStr)) + return "" + s.substr(storeStr.size()); + auto pkgStr = pkg.root.generic_string(); + if (!pkgStr.empty() && s.starts_with(pkgStr)) + return "" + s.substr(pkgStr.size()); + return s; + }; + auto add_dirs = [&](const std::vector& dirs, + std::string_view tag) { + for (auto& d : dirs) + out.includeDirs.push_back(std::string(tag) + ":" + relativize(d)); + }; + if (pkg.usageResolved) { + add_dirs(pkg.privateBuild.includeDirs, "priv"); + add_dirs(pkg.publicUsage.includeDirs, "pub"); + add_dirs(pkg.privateBuild.includeDirsAfter, "priv_after"); + add_dirs(pkg.publicUsage.includeDirsAfter, "pub_after"); + } + add_dirs(bc.includeDirs, "decl"); + add_dirs(bc.includeDirsAfter, "decl_after"); +} + +} // namespace mcpp::build::cache_key diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 06d92e0c..3bab7b45 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -29,7 +29,11 @@ namespace mcpp::build { // ─── P0: build cache for fast-path rebuilds ───────────────────────── constexpr std::string_view kBuildCacheFile = "target/.build_cache"; -constexpr int kBuildCacheMaxEntries = 4; // P3: LRU capacity +// P3: LRU capacity. Entries are keyed by (target triple, profile), so the +// working set is now targets × profiles rather than targets alone — 4 was +// enough for one profile, not for a dev/release/dist rotation across a host +// and a cross target. +constexpr int kBuildCacheMaxEntries = 8; // P3: one entry per (target, fingerprint) pair. struct BuildCacheEntry { @@ -55,6 +59,14 @@ struct BuildCacheEntry { // plan.runtimeLibraryDirs is empty. std::string runEnvKey; std::string runEnvValue; + // The resolved profile this entry was built for. Entries used to be keyed + // by target triple alone, and the fast paths only refuse to run when an + // EXPLICIT --profile/--dev/--release is passed — so a bare `mcpp build` + // after `mcpp build --release` took the fast path against the release + // build.ninja and reported success without ever rebuilding at -O0 -g. + // Empty means "cache predates this field" and is treated as a miss (a + // bare rebuild once, never a wrong artifact). + std::string profile; }; std::vector read_build_cache(const std::filesystem::path& projectRoot) { @@ -120,6 +132,13 @@ std::vector read_build_cache(const std::filesystem::path& proje std::getline(f, e.runEnvValue); haveNextLine = static_cast(std::getline(f, line)); } + // Optional profile line. Same back-compat contract as the two blocks + // above: absent ⇒ e.profile stays empty ⇒ every fast path treats the + // entry as a miss and falls through to prepare_build. + if (haveNextLine && line.starts_with("profile=")) { + e.profile = line.substr(8); + haveNextLine = static_cast(std::getline(f, line)); + } entries.push_back(std::move(e)); if (!haveNextLine || line.empty()) break; } @@ -135,19 +154,23 @@ void write_build_cache(const std::filesystem::path& projectRoot, const std::string& runtimeEnvValue = "", std::vector> runTargets = {}, const std::string& runEnvKey = "", - const std::string& runEnvValue = "") { + const std::string& runEnvValue = "", + const std::string& profile = "") { auto path = projectRoot / kBuildCacheFile; auto entries = read_build_cache(projectRoot); - // Remove existing entry for this target (will be re-added at front). + // Remove the existing entry for this (target, profile) pair. Keying on the + // triple alone made a release build evict the dev entry and vice versa, so + // switching profiles back and forth could never be incremental AND the + // surviving entry pointed at the other profile's build dir. std::erase_if(entries, [&](const BuildCacheEntry& e) { - return e.targetTriple == targetTriple; + return e.targetTriple == targetTriple && e.profile == profile; }); // Insert at front (MRU). BuildCacheEntry newEntry{targetTriple, outputDir.string(), ninjaProgram, fingerprintHex, runtimeEnvKey, runtimeEnvValue, std::move(runTargets), - runEnvKey, runEnvValue}; + runEnvKey, runEnvValue, profile}; entries.insert(entries.begin(), std::move(newEntry)); // Trim to LRU capacity. @@ -175,6 +198,7 @@ void write_build_cache(const std::filesystem::path& projectRoot, for (auto& [name, exe] : e.runTargets) f << name << '\t' << exe << '\n'; f << "runEnv=" << e.runEnvKey << '\n'; f << e.runEnvValue << '\n'; + f << "profile=" << e.profile << '\n'; } } @@ -255,7 +279,10 @@ compute_run_env(const mcpp::build::BuildPlan& plan) { // resolution banner). export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, std::string_view targetOverride = "") { - if (no_cache) { + // `--cache=off` means a cold build: no global cache, and target/ cleared — + // which is exactly what `--no-cache` has always done, hence the alias. + const bool coldBuild = no_cache || ctx.cacheMode == CacheMode::Off; + if (coldBuild) { std::error_code ec; std::filesystem::remove_all(ctx.outputDir, ec); } @@ -267,13 +294,14 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, mcpp::ui::status("Inferred", note); } - // Announce the package being built (and any deps). - // Deps that hit the BMI cache get "Cached" instead of "Compiling". - std::set cachedNames; - for (auto& label : ctx.cachedDepLabels) { - auto sp = label.find(' '); - cachedNames.insert(sp == std::string::npos ? label : label.substr(0, sp)); - } + // Announce the package being built (and any deps). A dep served from the + // global cache says "Cached" and HOW MANY translation units that saved. The + // count is the point: the bare word "Cached" was printed for three months + // while ninja recompiled every one of those units behind it, and no output + // contradicted it. A number that has to match the edges ninja actually skips + // cannot be quietly wrong in the same way. + std::map cachedUnits; + for (auto& dep : ctx.cachedDeps) cachedUnits[dep.name] = dep.units; std::set announced; announced.insert(ctx.manifest.package.name); mcpp::ui::status("Compiling", @@ -283,8 +311,13 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, if (announced.contains(name)) continue; announced.insert(name); std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; - const char* verb = cachedNames.contains(name) ? "Cached" : "Compiling"; - mcpp::ui::status(verb, std::format("{} {}", name, ver)); + auto it = cachedUnits.find(name); + if (it == cachedUnits.end()) { + mcpp::ui::status("Compiling", std::format("{} {}", name, ver)); + } else { + mcpp::ui::status("Cached", std::format("{} {} ({} unit{})", + name, ver, it->second, it->second == 1 ? "" : "s")); + } } mcpp::build::BuildOptions opts; @@ -301,7 +334,11 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, return 1; } - // M3.2: populate BMI cache for deps that did NOT hit cache. + // Populate the global cache for deps that did NOT hit. prepare_build leaves + // depsToPopulate empty under --cache=local|off, so the mode gate is already + // enforced there; asserting it again here keeps the write side legible on + // its own terms rather than as a consequence of something in prepare. + if (ctx.cacheMode != CacheMode::Global) ctx.depsToPopulate.clear(); for (auto& task : ctx.depsToPopulate) { auto pr = mcpp::bmi_cache::populate_from(task.key, ctx.outputDir, task.artifacts); if (!pr) { @@ -312,10 +349,16 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, } // P1.5: warn if fingerprint changed from last build (explains full rebuild). + // Compared against the entry for the SAME profile: the profile is now a + // fingerprint input, so a dev↔release switch always changes the fp. That is + // exactly what the user asked for, and warning about it turns a useful + // signal ("something you didn't expect invalidated your build dir") into + // noise on every profile switch. { auto entries = read_build_cache(ctx.projectRoot); for (auto& e : entries) { - if (e.targetTriple == targetOverride && !e.fingerprint.empty()) { + if (e.targetTriple == targetOverride && e.profile == ctx.profile + && !e.fingerprint.empty()) { auto newFp = ctx.outputDir.filename().string(); if (e.fingerprint != newFp) { mcpp::ui::warning(std::format( @@ -328,7 +371,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, } // P0: save build cache for fast-path on next invocation. - if (!no_cache && !r->ninjaProgram.empty()) { + if (!coldBuild && !r->ninjaProgram.empty()) { auto fpHex = ctx.outputDir.filename().string(); auto runTargets = compute_run_targets(ctx.plan); auto [runEnvKey, runEnvValue] = compute_run_env(ctx.plan); @@ -336,7 +379,8 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, std::string(targetOverride), fpHex, r->runtimeEnvKey.empty() ? "-" : r->runtimeEnvKey, r->runtimeEnvValue, - std::move(runTargets), runEnvKey, runEnvValue); + std::move(runTargets), runEnvKey, runEnvValue, + ctx.profile); } // The one place the --strict policy is settled. Degradations reported by @@ -347,7 +391,17 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, // exact failure mode it exists to prevent. if (!mcpp::diag::flush(ctx.strict)) return 1; - mcpp::ui::finished("release", r->elapsed); + // The descriptor comes from the knobs this build actually resolved, so it + // cannot disagree with the compiler flags the way the old hardcoded + // "release [optimized]" did. + { + const auto& bc = ctx.manifest.buildConfig; + std::string descriptor = + (bc.optLevel.empty() || bc.optLevel == "0") ? "unoptimized" : "optimized"; + if (bc.debug) descriptor += " + debuginfo"; + if (bc.lto) descriptor += " + lto"; + mcpp::ui::finished(ctx.profile, r->elapsed, descriptor); + } return 0; } @@ -461,6 +515,22 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, return 0; } +// Which profile does this invocation mean? The fast paths exist precisely to +// avoid prepare_build, where the profile is normally settled — so they settle +// it here from the same pure rule (resolve_profile_name), which needs nothing +// but the manifest. nullopt = manifest unreadable ⇒ no fast path. +// +// Without this, an entry matched on target triple alone could have been built +// for a different profile, and the fast path would run ninja against that +// profile's build.ninja: `mcpp build --release` then a bare `mcpp build` +// reported success in 0.00s and left -O2 artifacts where -O0 -g was asked for. +std::optional fast_path_profile(const std::filesystem::path& projectRoot, + std::string_view profileOverride = "") { + auto m = mcpp::manifest::load(projectRoot / "mcpp.toml"); + if (!m) return std::nullopt; + return mcpp::build::resolve_profile_name(*m, profileOverride); +} + // Try to fast-path: if build.ninja is newer than all inputs, just run ninja. // Returns exit code on fast-path, or nullopt if full rebuild needed. export std::optional try_fast_build(const std::filesystem::path& projectRoot, @@ -468,11 +538,19 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo std::string_view currentTarget = "") { if (no_cache) return std::nullopt; - // P3: read multi-entry cache and find entry matching currentTarget. + auto wantProfile = fast_path_profile(projectRoot); + if (!wantProfile) return std::nullopt; + + // P3: read multi-entry cache and find the entry matching this + // (target, profile) pair. Matching on the triple alone served the wrong + // profile's artifacts (see fast_path_profile). auto entries = read_build_cache(projectRoot); const BuildCacheEntry* match = nullptr; for (auto& e : entries) { - if (e.targetTriple == currentTarget) { match = &e; break; } + if (e.targetTriple == currentTarget && e.profile == *wantProfile) { + match = &e; + break; + } } if (!match) return std::nullopt; @@ -521,7 +599,7 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo if (!rc) return std::nullopt; if (*rc != 0) return rc; - mcpp::ui::finished("release", elapsed); + mcpp::ui::finished(*wantProfile, elapsed); return 0; } @@ -536,10 +614,16 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo std::optional try_fast_run(const std::filesystem::path& projectRoot, const std::optional& targetName, std::span passthrough) { + auto wantProfile = fast_path_profile(projectRoot); + if (!wantProfile) return std::nullopt; + auto entries = read_build_cache(projectRoot); const BuildCacheEntry* match = nullptr; for (auto& e : entries) { - if (e.targetTriple.empty()) { match = &e; break; } + if (e.targetTriple.empty() && e.profile == *wantProfile) { + match = &e; + break; + } } if (!match || match->runTargets.empty()) return std::nullopt; @@ -615,7 +699,9 @@ std::optional try_fast_run(const std::filesystem::path& projectRoot, // rule mcpp::project::resolve_member_dir documents for build/test). export int build_run_target(const std::optional& targetName, std::span passthrough, - const std::string& package_filter = {}) { + const std::string& package_filter = {}, + const std::string& cache_mode = {}, + bool no_cache = false) { // mcpp#225 (E2): reuse the resolved build cache when it's still fresh, // skipping prepare_build's toolchain resolution + modgraph scan // entirely — mirrors cmd_build's try_fast_build fast path. The cached @@ -623,7 +709,10 @@ export int build_run_target(const std::optional& targetName, // last time; a `-p` filter always needs prepare_build's member switch, // so skip the fast path in that case (mirrors cmd_build's fast-path // bypass whenever ov.package_filter is set). - if (package_filter.empty()) { + // A --cache/--no-cache override also bypasses the fast path, for the same + // reason --profile does: the cached build.ninja was generated under the + // previous mode, so reusing it would silently ignore the flag. + if (package_filter.empty() && cache_mode.empty() && !no_cache) { if (auto root = mcpp::project::find_manifest_root(std::filesystem::current_path())) { if (auto rc = try_fast_run(*root, targetName, passthrough)) { return *rc; @@ -635,10 +724,11 @@ export int build_run_target(const std::optional& targetName, // the binary, so we don't re-resolve the toolchain or re-scan modgraph. mcpp::build::BuildOverrides ov; ov.package_filter = package_filter; + ov.cache_mode = cache_mode; auto ctx = prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, /*extraTargets=*/{}, ov); if (!ctx) { std::println(stderr, "error: {}", ctx.error()); return 2; } - if (auto rc = run_build_plan(*ctx, /*verbose=*/false, /*no_cache=*/false); rc != 0) + if (auto rc = run_build_plan(*ctx, /*verbose=*/false, no_cache); rc != 0) return rc; // Find binary target @@ -843,11 +933,22 @@ export int run_tests(std::span passthrough, } // 4. "Compiling test_X (test)" lines for the test binaries. - std::set cachedNames; - for (auto& label : ctx->cachedDepLabels) { - auto sp = label.find(' '); - cachedNames.insert(sp == std::string::npos ? label : label.substr(0, sp)); - } + std::map cachedUnits; + for (auto& dep : ctx->cachedDeps) cachedUnits[dep.name] = dep.units; + auto announce = [&](const std::string& name, + const mcpp::manifest::DependencySpec& spec, + std::string_view suffix) { + std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; + auto it = cachedUnits.find(name); + if (it == cachedUnits.end()) { + mcpp::ui::status("Compiling", + std::format("{} {}{}", name, ver, suffix)); + } else { + mcpp::ui::status("Cached", + std::format("{} {} ({} unit{}){}", name, ver, it->second, + it->second == 1 ? "" : "s", suffix)); + } + }; std::set announced; announced.insert(ctx->manifest.package.name); mcpp::ui::status("Compiling", @@ -856,17 +957,12 @@ export int run_tests(std::span passthrough, for (auto& [name, spec] : ctx->manifest.dependencies) { if (announced.contains(name)) continue; announced.insert(name); - std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; - const char* verb = cachedNames.contains(name) ? "Cached" : "Compiling"; - mcpp::ui::status(verb, std::format("{} {}", name, ver)); + announce(name, spec, ""); } for (auto& [name, spec] : ctx->manifest.devDependencies) { if (announced.contains(name)) continue; announced.insert(name); - std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; - const char* verb = cachedNames.contains(name) ? "Cached" : "Compiling"; - mcpp::ui::status(verb, - std::format("{} {} (dev)", name, ver)); + announce(name, spec, " (dev)"); } // List test binaries. // (Per-test "Compiling" lines print in Phase B, interleaved with each @@ -1131,9 +1227,11 @@ export int clean_project(bool wipe_bmi) { std::println("Cleaned: {}", (*root / "target").string()); if (wipe_bmi) { - auto bmi = mcpp::toolchain::default_cache_root(); - std::filesystem::remove_all(bmi, ec); - std::println("Cleaned BMI cache: {}", bmi.string()); + auto cache = mcpp::toolchain::default_cache_root(); + std::filesystem::remove_all(cache, ec); + std::println("Cleaned build cache: {}", cache.string()); + std::println(" (`mcpp cache clean --legacy` also removes the unused " + "pre-v1 cache, if any)"); } return 0; } diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index d0b60a9a..76cbb77c 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -896,6 +896,46 @@ std::string emit_ninja_string(const BuildPlan& plan) { return "cxx_object"; }; + // ── Cache-served units: stage edges instead of compile edges ──────── + // + // A unit whose outputs are already in the global cache gets one stage_file + // edge per artifact and is then skipped by every loop below — no scan edge, + // no dyndep edge, no compile edge. Its outputs land at exactly the paths a + // compile edge would have produced, so the link edges, the BMI implicit + // inputs of consuming TUs and the runtime deployment edges are unchanged. + // + // This indirection is the whole point. The cache used to copy artifacts + // into the build dir from inside prepare, leaving them declared as compile + // edge outputs — and ninja treats an output with no command line in + // .ninja_log as dirty ("command line not found in log"), so a fresh build + // dir recompiled every one of them while the CLI printed "Cached". Making + // the staging an edge is what gives ninja a record to compare against. + // + // `--verify size` for the same reason the std artifacts use it: the entry + // directory is named by a key covering the toolchain, dialect, profile, + // this package's own config and its dependencies' keys, so for a given key + // equal size IS equivalence — and size comes from directory metadata, which + // stays readable even when a holder denies opening the file. + { + bool any = false; + for (auto& cu : plan.compileUnits) { + if (!cu.servedFromCache) continue; + if (cu.cachedObject.empty()) continue; + any = true; + append(std::format("build {} : stage_file {}\n", + escape_ninja_path(cu.object), + escape_ninja_path(cu.cachedObject))); + append(" verify = --verify size\n"); + if (cu.providesModule && !cu.cachedBmi.empty()) { + append(std::format("build {} : stage_file {}\n", + bmi_path(*cu.providesModule), + escape_ninja_path(cu.cachedBmi))); + append(" verify = --verify size\n"); + } + } + if (any) append("\n"); + } + if (dyndep) { // ── Phase 1: scan edges (one .ddi per TU). ────────────────────── // .ddi is placed beside the object so multi-version mangling can @@ -910,6 +950,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { std::vector ddi_paths; ddi_paths.reserve(plan.compileUnits.size()); for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; // staged, never scanned if (is_scan_exempt(cu.source)) continue; auto ddi = (cu.object.parent_path() / cu.source.filename()).string() + ".ddi"; @@ -941,6 +982,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { }(); std::map ddi_expect; for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; if (is_scan_exempt(cu.source)) continue; if (!cu.scanOverridden && !verifyAll) continue; auto ddi = (cu.object.parent_path() / cu.source.filename()).string() + ".ddi"; @@ -972,6 +1014,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { // Each compile edge references its OWN .dd file instead of a global one. // P2: module compile edges get a $bmi_out variable for BMI preservation. for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; // a stage_file edge owns these outputs std::string rule = pick_rule(cu.source); std::string out_line = "build " + escape_ninja_path(cu.object); @@ -1014,6 +1057,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { } else { // ── Static-deps mode (M3.2 and earlier). ──────────────────────── for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; // a stage_file edge owns these outputs std::string rule = pick_rule(cu.source); std::string implicit; diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 63f1c884..ff28a4f1 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -33,6 +33,14 @@ struct CompileUnit { // Unit came from a scan_overrides declaration — plan-vs-ddi // verification is mandatory for it (ninja_backend emits --expect-*). bool scanOverridden = false; + // This unit's outputs are already in the global cache: the backend emits + // `stage_file` edges from the cache instead of a compile edge (and skips + // the P1689 scan for it entirely). The unit itself stays in the plan so + // compile_commands.json keeps an entry for it and clangd does not lose the + // dependency's sources. + bool servedFromCache = false; + std::filesystem::path cachedObject; // absolute, inside the cache + std::filesystem::path cachedBmi; // absolute; empty if no module }; struct LinkUnit { diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 9a08d907..83f4a757 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -31,6 +31,7 @@ import mcpp.toolchain.post_install; import mcpp.toolchain.abi; import mcpp.toolchain.triple; import mcpp.build.plan; +import mcpp.build.cache_key; import mcpp.build.build_program; import mcpp.lockfile; import mcpp.config; @@ -207,7 +208,10 @@ export std::filesystem::path target_dir(const mcpp::toolchain::Toolchain& tc, // Compose a stable canonical compile-flags string for fingerprinting. -std::string canonical_compile_flags(const mcpp::manifest::Manifest& m) { +// Exported so the "every build-variant knob is in here" invariant is machine- +// checkable: the profile knobs were absent for a long time precisely because +// nothing could assert on this string. +export std::string canonical_compile_flags(const mcpp::manifest::Manifest& m) { std::string s; s += "-std="; s += m.package.standard; s += " -fmodules"; @@ -262,6 +266,18 @@ std::string canonical_compile_flags(const mcpp::manifest::Manifest& m) { for (auto const& f : gf.asmflags) { s += " gas:"; s += f; } for (auto const& f : gf.defines) { s += " gd:"; s += f; } } + // The resolved [profile] knobs. These are NOT in cflags/cxxflags: the + // profile block (see the profile resolution below) lands them in + // buildConfig.optLevel/debug/lto/strip and flags.cppm turns them into + // -O/-g/-flto at command-construction time. Leaving them out made + // `--dev`, `--release` and `--profile dist` share ONE fingerprint, hence + // one target/// directory AND one global cache entry — so a + // release build could be served -O0 -g dependency objects. They are + // build-variant by definition; they belong here. + s += " opt="; s += m.buildConfig.optLevel; + s += " debug="; s += m.buildConfig.debug ? "1" : "0"; + s += " lto="; s += m.buildConfig.lto ? "1" : "0"; + s += " strip="; s += m.buildConfig.strip ? "1" : "0"; return s; } @@ -570,6 +586,35 @@ bool graph_or_targets_import_std(const mcpp::modgraph::Graph& graph, return false; } +// How this invocation may use the global dependency cache. +// +// Global read + write (default) +// Local neither — every dependency is compiled inside this project's +// target/, which is what every build did before the cache worked +// Off neither, and this build's target/// directory is +// cleared first (a full cold rebuild). Sibling build dirs — other +// profiles, other targets — are left alone. +// +// `--no-cache` used to be the only switch and it meant "clear the build dir", +// which says nothing about a cache (and its help text claimed all of target/); +// it stays as a deprecated alias for Off. +export enum class CacheMode { Global, Local, Off }; + +export std::optional parse_cache_mode(std::string_view v) { + if (v == "global") return CacheMode::Global; + if (v == "local") return CacheMode::Local; + if (v == "off" || v == "none") return CacheMode::Off; + return std::nullopt; +} + +export std::string_view cache_mode_name(CacheMode m) { + switch (m) { + case CacheMode::Local: return "local"; + case CacheMode::Off: return "off"; + default: return "global"; + } +} + export struct BuildContext { // --strict: degradations reported through mcpp::diag become errors. // Carried on the context because the build's degradations are discovered @@ -584,6 +629,14 @@ export struct BuildContext { std::filesystem::path stdBmi; std::filesystem::path stdObject; mcpp::build::BuildPlan plan; + // Resolved profile name (resolve_profile_name). Carried so run_build_plan + // can record it in .build_cache — without it the fast path cannot tell + // whether a cached build.ninja was generated for the profile being asked + // for — and so `Finished ` stops being a hardcoded "release". + std::string profile; + // Resolved global-cache mode. Read side is honored in prepare_build; write + // side in run_build_plan. + CacheMode cacheMode = CacheMode::Global; // M3.2 BMI cache: deps that did NOT hit cache and therefore need // populate_from(...) AFTER backend.build succeeds. @@ -593,10 +646,35 @@ export struct BuildContext { }; std::vector depsToPopulate; - // Names of deps that DID hit cache (for ui status output). - std::vector cachedDepLabels; // "mcpplibs.cmdline v0.0.1" + // Deps that DID hit the global cache, and how many compile units each one + // spared. run_build_plan reports the count so the "Cached" line cannot be + // true-looking and empty at the same time. + struct CachedDep { + std::string name; + std::string version; + std::size_t units = 0; + }; + std::vector cachedDeps; }; +// The ONE profile-name resolver. Shared with execute.cppm's fast paths: +// they deliberately skip prepare_build, so before this existed they had no +// idea which profile the request meant — and `.build_cache` keyed entries by +// target triple alone. Net effect: `mcpp build --release` followed by a bare +// `mcpp build` reported success in 0.00s and left the RELEASE artifacts in +// place. The rule is pure (manifest + one override string), so both sides can +// evaluate it without resolving a toolchain or scanning the module graph. +// +// Precedence: --profile/--release/--dev > [build].default-profile > "dev". +// The global default is "dev" (-O0 -g) per the dominant convention +// (Cargo/Meson/CMake/Zig/Bazel/MSBuild all default to debug). +export std::string resolve_profile_name(const mcpp::manifest::Manifest& m, + std::string_view override_name) { + if (!override_name.empty()) return std::string(override_name); + if (!m.buildConfig.defaultProfile.empty()) return m.buildConfig.defaultProfile; + return "dev"; +} + // Command-level overrides (--target / --static). // Empty defaults preserve pre-existing behaviour exactly. export struct BuildOverrides { @@ -607,6 +685,7 @@ export struct BuildOverrides { std::string features; // --features a,b,c (root package activation) bool strict = false; // --strict: schema warnings become errors std::string capabilities; // --cap blas=openblas,lapack=mkl (provider pins) + std::string cache_mode; // --cache global|local|off ("" = unset) }; // `prepare_build` builds the BuildContext for any verb that compiles. @@ -738,6 +817,31 @@ prepare_build(bool print_fingerprint, mcpp::diag::warning("manifest/schema", w); } + // Global-cache mode: --cache > MCPP_BUILD_CACHE > [build] cache > global. + // An unparseable value is a warning (error under --strict) and falls + // through to the next source rather than silently meaning "global" — a typo + // that quietly re-enabled the cache would be the hardest kind of surprise + // to attribute. + CacheMode cacheMode = CacheMode::Global; + { + std::string cacheModeError; + auto try_source = [&](std::string_view value, std::string_view origin) { + if (value.empty()) return false; + if (auto parsed = parse_cache_mode(value)) { cacheMode = *parsed; return true; } + auto msg = std::format( + "{} has unknown cache mode '{}' (expected: global | local | off)", + origin, value); + if (overrides.strict) { cacheModeError = msg; return true; } + mcpp::diag::warning("build/cache-mode", msg); + return false; + }; + const char* envMode = std::getenv("MCPP_BUILD_CACHE"); + if (!try_source(overrides.cache_mode, "--cache")) + if (!try_source(envMode ? envMode : "", "MCPP_BUILD_CACHE")) + (void)try_source(m->buildConfig.cacheMode, "[build] cache"); + if (!cacheModeError.empty()) return std::unexpected(cacheModeError); + } + // ─── Toolchain resolution (docs/21) ──────────────────────────────── // Priority chain: // 1. mcpp.toml [toolchain]. → resolve_xpkg_path → abs path @@ -780,16 +884,13 @@ prepare_build(bool print_fingerprint, std::string effectiveProfile; { auto& pname = effectiveProfile; - // Precedence: --profile / --release / --dev flag (overrides.profile) > - // [build].default-profile (project default) > "dev" (global default). - // The global default is "dev" (-O0 -g) to follow the dominant convention - // (Cargo/Meson/CMake/Zig/Bazel/MSBuild all default to debug); release is - // opt-in via --release / --profile release. A project that wants its - // plain `mcpp build` optimized sets [build].default-profile = "release" - // (mcpp's own mcpp.toml does this, so the released binary stays -O2). - pname = !overrides.profile.empty() ? overrides.profile - : !m->buildConfig.defaultProfile.empty() ? m->buildConfig.defaultProfile - : "dev"; + // Precedence lives in resolve_profile_name (above) so execute.cppm's + // fast paths settle it identically without running prepare_build. + // Release is opt-in via --release / --profile release; a project that + // wants its plain `mcpp build` optimized sets + // [build].default-profile = "release" (mcpp's own mcpp.toml does this, + // so the released binary stays -O2). + pname = resolve_profile_name(*m, overrides.profile); mcpp::manifest::Profile pr; if (pname == "dev" || pname == "debug") { pr.optLevel = "0"; pr.debug = true; } else if (pname == "dist") { pr.optLevel = "3"; pr.strip = true; } @@ -1376,6 +1477,11 @@ prepare_build(bool print_fingerprint, std::string indexName; std::string packageName; std::string version; + // "version" | "path" | "git". Only "version" is cacheable: an index + // package's payload lives in the immutable xpkgs store under a + // version-keyed directory, so name@version identifies its sources. + // Path and git checkouts can change under an unchanged identity. + std::string sourceKind; }; std::vector dep_cache_identities; struct GitLockIdentity { @@ -2685,6 +2791,7 @@ prepare_build(bool print_fingerprint, .indexName = cache_index_name(key.ns), .packageName = mangled, .version = spec.version, + .sourceKind = "version", }); const auto depPackageIndex = packages.size(); packages.push_back(makePackageRoot(secStage, *dep_manifests.back())); @@ -2982,6 +3089,7 @@ prepare_build(bool print_fingerprint, .version = sourceKind == "version" ? spec.version : dep_manifests.back()->package.version, + .sourceKind = sourceKind, }); const auto depPackageIndex = packages.size(); packages.push_back(makePackageRoot(dep_root, *dep_manifests.back())); @@ -3605,7 +3713,7 @@ prepare_build(bool print_fingerprint, // pieces were already in the fingerprint; this fixes the COMMAND // construction the fingerprint promised (stdFlagAndDialect above). auto sm = mcpp::toolchain::ensure_built( - *tc, fp.hex, m->package.standard, stdFlagAndDialect, + *tc, m->package.standard, stdFlagAndDialect, mcpp::platform::macos::deployment_target( m->buildConfig.macosDeploymentTarget)); if (!sm) return std::unexpected(sm.error().message); @@ -3628,6 +3736,8 @@ prepare_build(bool print_fingerprint, ctx.manifest = *m; ctx.tc = *tc; ctx.fp = fp; + ctx.profile = effectiveProfile; + ctx.cacheMode = cacheMode; ctx.projectRoot= *root; ctx.outputDir = target_dir(*tc, fp, *root); ctx.stdBmi = stdBmiPath; @@ -3717,16 +3827,20 @@ prepare_build(bool print_fingerprint, } } - // ─── M3.2: BMI cache stage / populate-task collection ───────────── - // For each version-based dep package (i.e. fetched from a registry, - // not a path dep), check the global BMI cache. If cached → stage into - // the project's target dir so ninja sees those outputs as up-to-date - // and skips them. If not → record a populate task for AFTER build. + // ─── Global dependency cache: per-package keys, hit → stage edges ── + // + // Every index package gets a key over the axes that actually reach its + // compiler command lines (mcpp.build.cache_key), computed bottom-up so a + // package's key includes its direct dependencies' keys. A hit marks that + // package's compile units `servedFromCache`, and the ninja backend emits + // `stage_file` edges instead of compile edges for them — which is the only + // way ninja will accept a cached artifact. A miss records a populate task + // for after the build. // - // Path deps don't go through the cache: their sources can change at - // any time outside fingerprint awareness. + // `--cache=local|off` skips this block entirely: nothing is read and, in + // run_build_plan, nothing is written. auto cfg2 = get_cfg(); - if (cfg2) { + if (cfg2 && ctx.cacheMode == CacheMode::Global) { std::error_code mkEc; std::filesystem::create_directories(ctx.outputDir, mkEc); auto usable_object_rel = [](const std::filesystem::path& rel) @@ -3750,52 +3864,147 @@ prepare_build(bool print_fingerprint, } return objectPath.filename().generic_string(); }; + + // ── Per-package keys, bottom-up ────────────────────────────────── + // Axes A/B/C are whole-graph, so they are computed once. Axes D/E are + // per package. Axis F is each direct dependency's key, which forces a + // bottom-up order: `dependencyEdges` is a DAG (the modgraph validator + // rejects cycles), so a simple memoized recursion suffices — with an + // explicit in-progress guard so a cycle that slipped past validation + // fails loudly instead of recursing until the stack dies. + namespace ck = mcpp::build::cache_key; + const auto storeRoot = (*cfg2)->xlingsHome() / "data" / "xpkgs"; + auto axes = ck::build_axes( + *tc, *m, stdFlagAndDialect, + mcpp::toolchain::cppfly::effective_dialect_flags( + *tc, m->cppStandard.experimental, + mcpp::manifest::dialect_flags(m->buildConfig)), + mcpp::platform::macos::deployment_target( + m->buildConfig.macosDeploymentTarget)); + + // Sources belonging to each package, package-root-relative and sorted. + std::vector> pkgSources(packages.size()); + for (auto& cu : ctx.plan.compileUnits) { + for (std::size_t p = 0; p < packages.size(); ++p) { + std::error_code ec; + auto rel = std::filesystem::relative(cu.source, packages[p].root, ec); + if (ec || rel.empty()) continue; + auto rels = rel.generic_string(); + if (rels.starts_with("..")) continue; + pkgSources[p].push_back(rels); + break; + } + } + for (auto& v : pkgSources) std::ranges::sort(v); + + std::vector pkgKeys(packages.size()); + std::vector pkgInputs(packages.size(), + nlohmann::json::object()); + std::vector keyState(packages.size(), 0); // 0 new/1 busy/2 done + std::string keyCycleError; + auto compute_key = [&](auto&& self, std::size_t idx) -> const std::string& { + static const std::string kEmpty; + if (keyState[idx] == 2) return pkgKeys[idx]; + if (keyState[idx] == 1) { + if (keyCycleError.empty()) { + keyCycleError = std::format( + "dependency cycle through package '{}' while computing " + "its build-cache key", packages[idx].manifest.package.name); + } + return kEmpty; + } + keyState[idx] = 1; + + ck::PackageAxes pa; + if (idx > 0 && idx - 1 < dep_cache_identities.size()) { + pa.indexName = dep_cache_identities[idx - 1].indexName; + pa.packageName = dep_cache_identities[idx - 1].packageName; + pa.version = dep_cache_identities[idx - 1].version; + } + if (pa.packageName.empty()) { + // The root package, or a package with no resolution identity. + // It is never cached, but its key still has to exist because + // downstream packages fold it in via axis F. + pa.packageName = packages[idx].manifest.package.namespace_.empty() + ? packages[idx].manifest.package.name + : std::format("{}.{}", packages[idx].manifest.package.namespace_, + packages[idx].manifest.package.name); + } + if (pa.version.empty()) pa.version = packages[idx].manifest.package.version; + ck::fill_package_config(pa, packages[idx], storeRoot); + pa.sources = pkgSources[idx]; + for (auto& e : dependencyEdges) { + if (e.consumerPackageIndex != idx) continue; + auto& up = self(self, e.dependencyPackageIndex); + if (!up.empty()) pa.upstreamKeys.push_back(up); + for (auto& f : e.requestedFeatures) pa.features.push_back(f); + } + std::ranges::sort(pa.upstreamKeys); + pa.upstreamKeys.erase(std::unique(pa.upstreamKeys.begin(), + pa.upstreamKeys.end()), + pa.upstreamKeys.end()); + std::ranges::sort(pa.features); + pa.features.erase(std::unique(pa.features.begin(), pa.features.end()), + pa.features.end()); + + pkgKeys[idx] = ck::key_hex(axes, pa); + pkgInputs[idx] = ck::to_json(axes, pa); + keyState[idx] = 2; + return pkgKeys[idx]; + }; + for (std::size_t i = 0; i < packages.size(); ++i) + (void)compute_key(compute_key, i); + if (!keyCycleError.empty()) return std::unexpected(keyCycleError); + for (std::size_t i = 1; i < packages.size(); ++i) { // skip [0] = main const auto& pkgRoot = packages[i]; const auto* depIdent = i - 1 < dep_cache_identities.size() ? &dep_cache_identities[i - 1] : nullptr; - const auto fallbackName = pkgRoot.manifest.package.namespace_.empty() - ? pkgRoot.manifest.package.name - : std::format("{}.{}", pkgRoot.manifest.package.namespace_, - pkgRoot.manifest.package.name); - const auto& depName = depIdent ? depIdent->packageName : fallbackName; - const auto& depVer = depIdent && !depIdent->version.empty() - ? depIdent->version - : pkgRoot.manifest.package.version; - - // Find this dep's spec from the consumer manifest to know - // if it's path-based or version-based. - auto specIt = m->dependencies.find(depName); - // Path AND git deps bypass the BMI cache: their sources can - // change outside the fingerprint's awareness. - bool skipCache = (specIt != m->dependencies.end() && - (specIt->second.isPath() || specIt->second.isGit())); - if (specIt == m->dependencies.end()) { - auto devIt = m->devDependencies.find(depName); - if (devIt != m->devDependencies.end()) { - skipCache = devIt->second.isPath() || devIt->second.isGit(); - } - } - if (skipCache) continue; + // Only index ("version") packages are cacheable, and the identity + // recorded at resolution time is the ONLY admissible evidence. + // + // The predicate this replaces looked the package up in the ROOT + // manifest's dependencies/dev-dependencies and skipped it when the + // spec was path/git. A transitively-reached package is in neither + // map, so `specIt == end()` left skipCache false and local sources + // were cached — with `indexName` falling back to defaultIndex, so a + // workspace member `B` landed on disk as `mcpplibs/B@0.1.0`. Its + // sources can then change without changing name@version, i.e. the + // cache key cannot see the change. + // + // Note the direction of the judgment: `mcpp add`'s existence gate + // admits anything it cannot disprove. A build cache must do the + // opposite — anything it cannot prove came from the immutable + // xpkgs store stays out, because the failure mode here is a + // silently wrong object rather than a rejected command. + if (!depIdent || depIdent->sourceKind != "version") continue; + + const auto& depName = depIdent->packageName; + const auto& depVer = depIdent->version.empty() + ? pkgRoot.manifest.package.version + : depIdent->version; auto bmiT = mcpp::toolchain::bmi_traits(*tc); mcpp::bmi_cache::CacheKey key { - .mcppHome = (*cfg2)->mcppHome, - .fingerprint = fp.hex, - .indexName = depIdent - ? depIdent->indexName - : (*cfg2)->defaultIndex, + .cacheRoot = mcpp::home::cache_root(), + .indexName = depIdent->indexName, .packageName = depName, .version = depVer, + .keyHex = pkgKeys[i], + .inputs = pkgInputs[i], .bmiDirName = std::string(bmiT.bmiDir), .manifestTag = std::string(bmiT.manifestPrefix), }; - // Compute the artifacts list from the build plan: every - // CompileUnit whose source lies under this dep's root contributes. + // The artifacts this package contributes, and the compile units + // that produce them. Collected together so a hit can mark exactly + // those units — the artifact list alone would not say which edges + // must stop being compile edges. mcpp::bmi_cache::DepArtifacts arts; - for (auto& cu : ctx.plan.compileUnits) { + std::vector unitIdx; + for (std::size_t u = 0; u < ctx.plan.compileUnits.size(); ++u) { + auto& cu = ctx.plan.compileUnits[u]; std::error_code ec; auto rel = std::filesystem::relative(cu.source, pkgRoot.root, ec); if (ec || rel.empty()) continue; @@ -3810,16 +4019,32 @@ prepare_build(bool print_fingerprint, arts.bmiFiles.push_back(std::move(bmi)); } arts.objFiles.push_back(object_cache_path(cu.object)); + unitIdx.push_back(u); } if (mcpp::bmi_cache::is_cached(key)) { - auto staged = mcpp::bmi_cache::stage_into(key, ctx.outputDir); - if (staged) { - ctx.cachedDepLabels.push_back( - std::format("{} v{}", depName, depVer)); - continue; // skip populate task; it's already cached + // Mark the units. The backend turns each into a stage_file + // edge; nothing is copied here. Copying behind ninja's back is + // exactly what made the old cache a no-op: the staged file was + // still declared as a compile edge's output, and an output with + // no .ninja_log command-line record is dirty, so every unit was + // recompiled while the CLI printed "Cached". + for (auto u : unitIdx) { + auto& cu = ctx.plan.compileUnits[u]; + cu.servedFromCache = true; + cu.cachedObject = mcpp::bmi_cache::cached_obj_path( + key, object_cache_path(cu.object)); + if (cu.providesModule) { + std::string bmi; + for (char c : *cu.providesModule) + bmi.push_back(c == ':' ? '-' : c); + bmi += std::string(bmiT.bmiExt); + cu.cachedBmi = mcpp::bmi_cache::cached_bmi_path(key, bmi); + } } - // stage failed — fall through to recompile + repopulate + mcpp::bmi_cache::touch_accessed(key); + ctx.cachedDeps.push_back({depName, depVer, unitIdx.size()}); + continue; // no populate task; it is already cached } ctx.depsToPopulate.push_back({ std::move(key), std::move(arts) }); } diff --git a/src/cli.cppm b/src/cli.cppm index 998edcd7..2e25ef35 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -66,7 +66,7 @@ void print_usage() { std::println(""); std::println("Resource management:"); std::println(" mcpp toolchain install|list|default Manage mcpp's private toolchains"); - std::println(" mcpp cache list|prune|clean|info Inspect/manage the global BMI cache"); + std::println(" mcpp cache dir|list|info|gc|... Inspect/manage the global build cache"); std::println(" mcpp index list|add|remove|update Manage package registries"); std::println(""); std::println("About mcpp itself:"); @@ -81,7 +81,8 @@ void print_usage() { std::println(" --verbose, -v Verbose compiler output"); std::println(" --quiet, -q Suppress status output"); std::println(" --print-fingerprint Show toolchain fingerprint and 10 inputs"); - std::println(" --no-cache Force-clear target/ before building"); + std::println(" --cache Dependency cache: global (default) | local | off"); + std::println(" --no-cache Deprecated alias for --cache=off (clears the build dir)"); std::println(" --no-color Disable colored output"); std::println(""); std::println("Docs: https://github.com/mcpp-community/mcpp/tree/main/docs"); @@ -218,8 +219,10 @@ int run(int argc, char** argv) { .description("Build the current package") .option(cl::Option("print-fingerprint") .help("Show toolchain fingerprint and 10 inputs")) + .option(cl::Option("cache").takes_value().value_name("MODE") + .help("Global dependency cache: global (default) | local | off")) .option(cl::Option("no-cache") - .help("Force-clear target/ before building")) + .help("Deprecated alias for --cache=off (also clears the build dir)")) .option(cl::Option("target").takes_value().help( "Build for (e.g. x86_64-linux-musl); looks up [target.] in mcpp.toml")) .option(cl::Option("static").help( @@ -246,6 +249,10 @@ int run(int argc, char** argv) { .arg(cl::Arg("target").help("Binary name (optional)")) .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") .help("Run only the named workspace member (single-member; no --workspace fan-out)")) + .option(cl::Option("cache").takes_value().value_name("MODE") + .help("Global dependency cache: global (default) | local | off")) + .option(cl::Option("no-cache") + .help("Deprecated alias for --cache=off (also clears the build dir)")) .action(wrap_rc([&passthrough](const cl::ParsedArgs& p) { return cmd_run(p, std::span(passthrough)); }))) @@ -269,6 +276,10 @@ int run(int argc, char** argv) { .help("Treat manifest schema warnings (unknown feature/platform) as errors")) .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") .help("Run tests only for the named workspace member")) + .option(cl::Option("cache").takes_value().value_name("MODE") + .help("Global dependency cache: global (default) | local | off")) + .option(cl::Option("no-cache") + .help("Deprecated alias for --cache=off (also clears the build dir)")) .option(cl::Option("workspace") .help("Run tests for all workspace members")) .action(wrap_rc([&passthrough](const cl::ParsedArgs& p) { @@ -389,24 +400,43 @@ int run(int argc, char** argv) { "Remove the payload for instead of the host one"))) .action(wrap_rc(cmd_toolchain))) .subcommand(cl::App("cache") - .description("Inspect and manage the global BMI cache") + .description("Inspect and manage the global build cache") + .subcommand(cl::App("dir") + .description("Print the cache root (and any pre-v1 cache)")) .subcommand(cl::App("list") - .description("List cache entries with size + last-access")) + .description("List cache entries with size + last-use") + .option(cl::Option("json").help("Emit machine-readable JSON"))) .subcommand(cl::App("info") - .description("Show details for a cached package") + .description("Show details (incl. key inputs) for a cached package") .arg(cl::Arg("pkg").help("@").required())) .subcommand(cl::App("prune") - .description("Drop entries older than a threshold") + .description("Drop entries not used within a threshold") .option(cl::Option("older-than").takes_value().value_name("N{s|m|h|d}") .help("Age threshold (e.g. 30d)"))) + .subcommand(cl::App("gc") + .description("LRU-collect package entries to a size and/or age budget") + .option(cl::Option("max-size").takes_value().value_name("N{MiB|GiB}") + .help("Keep the package cache under this size (e.g. 5GiB)")) + .option(cl::Option("older-than").takes_value().value_name("N{s|m|h|d}") + .help("Also drop entries unused for longer than this"))) .subcommand(cl::App("clean") - .description("Drop all dep cache entries (preserves std.gcm)")) + .description("Drop cache entries (default: package entries only)") + .option(cl::Option("deps").help("Drop package entries (default)")) + .option(cl::Option("std").help("Drop std module entries")) + .option(cl::Option("all").help("Drop both")) + .option(cl::Option("legacy") + .help("Remove the unused pre-v1 cache at $MCPP_HOME/bmi"))) + .subcommand(cl::App("verify") + .description("Check every entry's manifest against the files on disk")) .action(wrap_rc([&dispatch_sub](const cl::ParsedArgs& p) { return dispatch_sub("cache", p, { - {"list", cmd_cache_list}, - {"info", cmd_cache_info}, - {"prune", cmd_cache_prune}, - {"clean", cmd_cache_clean}, + {"dir", cmd_cache_dir}, + {"list", cmd_cache_list}, + {"info", cmd_cache_info}, + {"prune", cmd_cache_prune}, + {"gc", cmd_cache_gc}, + {"clean", cmd_cache_clean}, + {"verify", cmd_cache_verify}, }); }))) .subcommand(cl::App("index") diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index f324b5cf..715418b7 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -46,6 +46,11 @@ export int cmd_build(const mcpplibs::cmdline::ParsedArgs& parsed) { mcpp::build::BuildOverrides ov; if (auto t = parsed.value("target")) ov.target_triple = *t; if (auto p = parsed.value("package")) ov.package_filter = *p; + // --cache global|local|off. --no-cache is the deprecated alias for off; the + // old flag only ever cleared target/, which says nothing about a cache, so + // it is expressed in terms of the new one rather than kept as a second axis. + if (auto c = parsed.value("cache")) ov.cache_mode = *c; + else if (no_cache) ov.cache_mode = "off"; // Profile selection precedence: --profile NAME > --release / --dev > the // project default ([build].default-profile) > "release", resolved in // prepare_build. --release/--dev are shorthands only. @@ -81,7 +86,8 @@ export int cmd_build(const mcpplibs::cmdline::ParsedArgs& parsed) { // the fast path would silently ignore the flags. if (!print_fp && ov.target_triple.empty() && !ov.force_static && ov.profile.empty() && ov.features.empty() && !ov.strict - && ov.capabilities.empty() && ov.package_filter.empty()) { + && ov.capabilities.empty() && ov.package_filter.empty() + && ov.cache_mode.empty()) { auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); if (root) { if (auto rc = mcpp::build::try_fast_build(*root, verbose, no_cache)) { @@ -109,7 +115,12 @@ export int cmd_run(const mcpplibs::cmdline::ParsedArgs& parsed, // `mcpp run` is single-member only — no `--workspace` fan-out. std::string package_filter; if (auto p = parsed.value("package")) package_filter = *p; - return mcpp::build::build_run_target(targetName, passthrough, package_filter); + std::string cache_mode; + bool no_cache = parsed.is_flag_set("no-cache"); + if (auto c = parsed.value("cache")) cache_mode = *c; + else if (no_cache) cache_mode = "off"; + return mcpp::build::build_run_target(targetName, passthrough, package_filter, + cache_mode, no_cache); } export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, @@ -125,6 +136,8 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, if (auto cp = parsed.value("cap")) ov.capabilities = *cp; ov.strict = parsed.is_flag_set("strict"); if (auto p = parsed.value("package")) ov.package_filter = *p; + if (auto c = parsed.value("cache")) ov.cache_mode = *c; + else if (parsed.is_flag_set("no-cache")) ov.cache_mode = "off"; mcpp::build::TestOptions to; if (parsed.positional_count() > 0) to.filter = parsed.positional(0); diff --git a/src/cli/cmd_cache.cppm b/src/cli/cmd_cache.cppm index b2fc0e3a..f7a48eff 100644 --- a/src/cli/cmd_cache.cppm +++ b/src/cli/cmd_cache.cppm @@ -14,8 +14,12 @@ import mcpp.ui; namespace mcpp::cli { -export int cmd_cache_list(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { - return mcpp::bmi_cache::cache_list(); +export int cmd_cache_dir(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { + return mcpp::bmi_cache::cache_dir(); +} + +export int cmd_cache_list(const mcpplibs::cmdline::ParsedArgs& parsed) { + return mcpp::bmi_cache::cache_list(parsed.is_flag_set("json")); } export int cmd_cache_info(const mcpplibs::cmdline::ParsedArgs& parsed) { @@ -31,8 +35,21 @@ export int cmd_cache_prune(const mcpplibs::cmdline::ParsedArgs& parsed) { return mcpp::bmi_cache::cache_prune(parsed.option_or_empty("older-than").value()); } -export int cmd_cache_clean(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { - return mcpp::bmi_cache::cache_clean(); +export int cmd_cache_gc(const mcpplibs::cmdline::ParsedArgs& parsed) { + return mcpp::bmi_cache::cache_gc( + parsed.option_or_empty("max-size").value(), + parsed.option_or_empty("older-than").value()); +} + +export int cmd_cache_clean(const mcpplibs::cmdline::ParsedArgs& parsed) { + return mcpp::bmi_cache::cache_clean(parsed.is_flag_set("deps"), + parsed.is_flag_set("std"), + parsed.is_flag_set("all"), + parsed.is_flag_set("legacy")); +} + +export int cmd_cache_verify(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { + return mcpp::bmi_cache::cache_verify(); } } // namespace mcpp::cli diff --git a/src/config.cppm b/src/config.cppm index 68c57631..df174854 100644 --- a/src/config.cppm +++ b/src/config.cppm @@ -59,7 +59,7 @@ struct GlobalConfig { std::filesystem::path binDir; // mcppHome/bin std::filesystem::path xlingsBinary; // mcppHome/registry/bin/xlings std::filesystem::path registryDir; // mcppHome/registry - std::filesystem::path bmiCacheDir; // mcppHome/bmi + std::filesystem::path bmiCacheDir; // mcppHome/build-cache/v1 std::filesystem::path metaCacheDir; // mcppHome/cache std::filesystem::path logDir; // mcppHome/log std::filesystem::path configFile; // mcppHome/config.toml @@ -456,7 +456,7 @@ std::expected load_or_init( // making ensure_sandbox_xlings_binary() a no-op. cfg.xlingsBinary = cfg.registryDir / "bin" / (std::string("xlings") + std::string(mcpp::platform::exe_suffix)); - cfg.bmiCacheDir = mcpp::home::bmi_root(); + cfg.bmiCacheDir = mcpp::home::cache_root(); cfg.metaCacheDir = cfg.mcppHome / "cache"; cfg.logDir = cfg.mcppHome / "log"; cfg.configFile = cfg.mcppHome / "config.toml"; @@ -647,7 +647,7 @@ void print_env(const GlobalConfig& cfg) { std::println("xlings pinned = {}", kXlingsPinnedVersion); std::println("xlings home = {}", cfg.xlingsHome().string()); std::println("config = {}", cfg.configFile.string()); - std::println("BMI cache = {}", cfg.bmiCacheDir.string()); + std::println("build cache = {}", cfg.bmiCacheDir.string()); std::println("meta cache = {}", cfg.metaCacheDir.string()); if (!cfg.defaultToolchain.empty()) std::println("default toolchain = {}", cfg.defaultToolchain); diff --git a/src/doctor.cppm b/src/doctor.cppm index 2e64db30..004c6cb5 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -150,25 +150,29 @@ export int doctor_report() { mcpp::ui::status("Checking", "std module"); if (tc) { - auto cacheRoot = mcpp::toolchain::default_cache_root(); - auto fpDir = cacheRoot; + // Entries live at /std//{gcm,pcm}.cache/std.*; the + // object sits at the entry root. Look for the object rather than one + // compiler's BMI extension so this reports on clang and MSVC too. + auto stdRoot = mcpp::toolchain::default_cache_root() / "std"; std::error_code ec; - if (std::filesystem::exists(cacheRoot, ec)) { + if (std::filesystem::exists(stdRoot, ec)) { bool found = false; - for (auto& e : std::filesystem::directory_iterator(cacheRoot, ec)) { - auto stdgcm = e.path() / "std.gcm"; - if (std::filesystem::exists(stdgcm)) { - ok(std::format("{} ({})", - stdgcm.string(), - mcpp::bmi_cache::human_bytes(std::filesystem::file_size(stdgcm)))); + for (auto& e : std::filesystem::directory_iterator(stdRoot, ec)) { + for (auto name : {"std.o", "std.obj"}) { + auto obj = e.path() / name; + if (!std::filesystem::exists(obj, ec)) continue; + ok(std::format("{} (entry {})", e.path().string(), + mcpp::bmi_cache::human_bytes( + mcpp::bmi_cache::dir_size(e.path())))); found = true; break; } + if (found) break; } - if (!found) warn("no std.gcm found yet (will be built on first `mcpp build`)"); + if (!found) warn("no std module cached yet (built on first `mcpp build`)"); } else { - warn(std::format("cache root '{}' missing (run `mcpp env` to init)", - cacheRoot.string())); + warn(std::format("no std module cache at '{}' yet " + "(built on first `mcpp build`)", stdRoot.string())); } } @@ -189,11 +193,27 @@ export int doctor_report() { mcpp::ui::status("Checking", "cache health"); auto bmiRoot = mcpp::toolchain::default_cache_root(); auto sz = mcpp::bmi_cache::dir_size(bmiRoot); - if (sz > std::uintmax_t(1) * 1024 * 1024 * 1024) { - warn(std::format("BMI cache occupies {} (run `mcpp cache prune` to GC)", + if (sz > std::uintmax_t(4) * 1024 * 1024 * 1024) { + warn(std::format("build cache occupies {} " + "(`mcpp cache gc --max-size 4GiB` to collect)", mcpp::bmi_cache::human_bytes(sz))); } else { - ok(std::format("BMI cache size = {}", mcpp::bmi_cache::human_bytes(sz))); + ok(std::format("build cache size = {}", mcpp::bmi_cache::human_bytes(sz))); + } + // The pre-v1 cache was keyed by whole-project fingerprint, which folded in + // the consumer's own name and version — so it accumulated one copy of every + // dependency per project configuration and never produced a cross-project + // hit. Nothing reads it now. Report the size, never delete it. + { + auto legacyRoot = mcpp::home::legacy_bmi_root(); + std::error_code lec; + if (std::filesystem::is_directory(legacyRoot, lec)) { + auto lsz = mcpp::bmi_cache::dir_size(legacyRoot); + warn(std::format("pre-v1 cache at '{}' occupies {} and is no longer " + "used — `mcpp cache clean --legacy` reclaims it", + legacyRoot.string(), + mcpp::bmi_cache::human_bytes(lsz))); + } } // Pre-#311 builds could park the std BMI cache in the current working // directory (`.mcpp-bmi/`) whenever neither MCPP_HOME nor HOME resolved — @@ -486,8 +506,9 @@ export int self_init(bool force) { if (!home.empty()) { std::error_code ec; std::filesystem::remove_all(home / "registry", ec); - std::filesystem::remove_all(home / "bmi", ec); - std::filesystem::remove_all(home / "cache", ec); + std::filesystem::remove_all(home / "cache", ec); // index metadata + std::filesystem::remove_all(home / "build-cache", ec); // compiled artifacts + std::filesystem::remove_all(home / "bmi", ec); // pre-v1 build cache } } diff --git a/src/home.cppm b/src/home.cppm index 31138431..0f1b5cb0 100644 --- a/src/home.cppm +++ b/src/home.cppm @@ -25,10 +25,25 @@ export namespace mcpp::home { // $MCPP_HOME. See root() below for the resolution order. std::filesystem::path root(); -// The cross-project BMI cache root ($MCPP_HOME/bmi). Both the std module -// cache (toolchain/stdmod.cppm) and the dep BMI cache (bmi_cache.cppm, via -// GlobalConfig::bmiCacheDir) live here — they MUST agree. -std::filesystem::path bmi_root(); +// The cross-project build cache root ($MCPP_HOME/build-cache/v1). Holds both +// the std module cache (toolchain/stdmod.cppm, under std/) and the per-package +// dependency cache (bmi_cache.cppm, under pkg/) — they MUST agree. +// +// NOT `$MCPP_HOME/cache`: that name is already taken by GlobalConfig's index +// metadata cache, which config.cppm's reset path deletes wholesale. Nesting +// compiled artifacts inside a directory something else clears is a trap that +// would surface as "the build cache empties itself sometimes". +// +// The `v1` segment is the layout version. It exists so replacing the layout +// never has to migrate or delete anything: a new layout is a new segment, and +// the old tree becomes inert until `mcpp cache clean --legacy` collects it. +std::filesystem::path cache_root(); + +// The pre-v1 cache root ($MCPP_HOME/bmi), keyed by whole-project fingerprint. +// Nothing reads or writes it any more; `mcpp doctor` reports it and +// `mcpp cache clean --legacy` removes it. Kept resolvable rather than +// hardcoded at the call sites so "where is the old cache" has one answer too. +std::filesystem::path legacy_bmi_root(); } // namespace mcpp::home @@ -90,7 +105,11 @@ std::filesystem::path root() { return default_home(); } -std::filesystem::path bmi_root() { +std::filesystem::path cache_root() { + return root() / "build-cache" / "v1"; +} + +std::filesystem::path legacy_bmi_root() { return root() / "bmi"; } diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index fb9a6119..332c829e 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -877,6 +877,7 @@ std::expected parse_string(std::string_view content, if (auto v = doc->get_string("build.target")) m.buildConfig.target = *v; if (auto v = doc->get_string("build.default-profile")) m.buildConfig.defaultProfile = *v; else if (auto v = doc->get_string("build.profile")) m.buildConfig.defaultProfile = *v; // accepted alias + if (auto v = doc->get_string("build.cache")) m.buildConfig.cacheMode = *v; // [xlings] — build environment (L-1). Subsections mirror .xlings.json 1:1. if (auto v = doc->get_string_array("xlings.deps")) m.xlings.deps = *v; @@ -900,7 +901,7 @@ std::expected parse_string(std::string_view content, // // MUST stay in sync with the `doc->get_*("build.")` reads above. static constexpr std::string_view kKnownBuildKeys[] = { - "allow_host_libs", "c_standard", "cflags", "cxxflags", + "allow_host_libs", "c_standard", "cache", "cflags", "cxxflags", "default-profile", "defines", "dialect_cxxflags", "flags", "include_dirs", "include_dirs_after", "ldflags", "macos_deployment_target", "profile", "sources", "static_stdlib", @@ -915,7 +916,7 @@ std::expected parse_string(std::string_view content, "[build] has unsupported key '{}' (ignored). Supported keys: " "sources, cflags, cxxflags, ldflags, defines, flags, " "include_dirs, include_dirs_after, dialect_cxxflags, " - "c_standard, target, static_stdlib, allow_host_libs, " + "c_standard, target, static_stdlib, allow_host_libs, cache, " "profile, macos_deployment_target.", key)); } } diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 8f931708..c7d56dfd 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -290,6 +290,12 @@ struct BuildConfig : BuildInputs { // footgun): a project that defaults to dev should pass `--profile release` // when producing a distributable (a pack-time release guard is a follow-up). std::string defaultProfile; + // `[build] cache` — "global" (default) | "local" | "off". Project-level + // default for the global dependency cache; --cache and MCPP_BUILD_CACHE + // both override it. Validated in prepare_build (unknown value: warning, or + // error under --strict) rather than here, so parsing a manifest never + // depends on the build-mode vocabulary. + std::string cacheMode; }; // `[runtime]` — requirements needed when launching built binaries. diff --git a/src/toolchain/fingerprint.cppm b/src/toolchain/fingerprint.cppm index ddbb0405..d21a4140 100644 --- a/src/toolchain/fingerprint.cppm +++ b/src/toolchain/fingerprint.cppm @@ -18,7 +18,7 @@ import mcpp.toolchain.detect; export namespace mcpp::toolchain { -inline constexpr std::string_view MCPP_VERSION = "2026.7.30.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.7.30.2"; struct FingerprintInputs { Toolchain toolchain; diff --git a/src/toolchain/stdmod.cppm b/src/toolchain/stdmod.cppm index d32596f4..3dabf39c 100644 --- a/src/toolchain/stdmod.cppm +++ b/src/toolchain/stdmod.cppm @@ -12,13 +12,21 @@ module; // clang++ -std=c++23 pcm.cache/std.pcm -c -o std.o // // We invoke the compiler in a dedicated cache directory so the produced -// BMI is owned by mcpp and reused across all builds with the same fingerprint. +// BMI is owned by mcpp and reused across every build with the same std identity. // // Output layout: -// // +// /std// // gcm.cache/std.gcm ← GCC BMI // pcm.cache/std.pcm ← Clang BMI // std.o ← linked into final binaries +// +// The directory used to be named by the WHOLE-PROJECT fingerprint, which folds +// in the root package's name, version and flags — none of which reach the std +// module's compile commands. The metadata written next to the artifacts +// (metadata_for below) has always been the correct identity and has always been +// what validates a hit; only the directory name was wrong. On one machine that +// cost 1014 directories holding 15 distinct identities: 16.1 GB where ~0.5 GB +// was needed, and a `mcpp version bump` re-precompiled std from scratch. export module mcpp.toolchain.stdmod; @@ -36,7 +44,7 @@ import mcpp.toolchain.msvc; export namespace mcpp::toolchain { struct StdModule { - std::filesystem::path cacheDir; // // + std::filesystem::path cacheDir; // /std// std::filesystem::path bmiPath; // /gcm.cache/std.gcm std::filesystem::path objectPath; // /std.o std::filesystem::path compatBmiPath; // /pcm.cache/std.compat.pcm @@ -54,7 +62,6 @@ std::filesystem::path default_cache_root(); // arm64-apple-macosxNN triple than the code importing it. std::expected ensure_built( const Toolchain& tc, - std::string_view fingerprint_hex, std::string_view cpp_standard, std::string_view cpp_standard_flag, std::string_view macos_deployment_target = {}, @@ -86,6 +93,24 @@ std::filesystem::path metadata_path(const std::filesystem::path& cacheDir) { return cacheDir / "std-module.json"; } +// The directory segment used while deriving the identity, so the build commands +// folded into it do not contain the identity they are about to produce. +constexpr std::string_view kStdKeyPlaceholder = "@std-key@"; + +// The std module's cache directory name. Derived from the metadata that already +// defines std identity (metadata_for below: compiler, version, driver identity, +// target triple, stdlib, standard + flag, source hashes, build commands) — the +// same object metadata_matches validates a hit against. Naming the directory +// after the whole-project fingerprint instead is what let 15 real identities +// occupy 1014 directories. +// +// Format compatibility is carried by the `build-cache/v1` path segment above +// one, not by an epoch field: a std BMI's validity depends on the toolchain, +// never on mcpp's cache bookkeeping. +std::string std_identity_key(const nlohmann::json& normalized) { + return hash_string("mcpp-std-key-v1\x1f" + normalized.dump()); +} + nlohmann::json metadata_for(const Toolchain& tc, std::string_view cppStandard, std::string_view cppStandardFlag, @@ -182,12 +207,11 @@ std::filesystem::path default_cache_root() { // resolution that predated Windows' USERPROFILE branch and self-contained // installs, so the std BMI cache could land in the *current working // directory* (`.mcpp-bmi/`) while dep BMIs went to $MCPP_HOME/bmi. - return mcpp::home::bmi_root(); + return mcpp::home::cache_root(); } std::expected ensure_built( const Toolchain& tc, - std::string_view fingerprint_hex, std::string_view cpp_standard, std::string_view cpp_standard_flag, std::string_view macos_deployment_target, @@ -200,12 +224,6 @@ std::expected ensure_built( const bool isMsvc = tc.compiler == CompilerId::MSVC; StdModule sm; - sm.cacheDir = cache_root / std::string(fingerprint_hex); - sm.bmiPath = isMsvc ? mcpp::toolchain::msvc::std_bmi_path(sm.cacheDir) - : is_clang(tc) - ? mcpp::toolchain::clang::std_bmi_path(sm.cacheDir) - : mcpp::toolchain::gcc::std_bmi_path(sm.cacheDir); - sm.objectPath = sm.cacheDir / (isMsvc ? "std.obj" : "std.o"); // Build sysroot + include flags for std module precompilation, derived // from the shared toolchain link model (same resolver as flags.cppm — @@ -236,26 +254,62 @@ std::expected ensure_built( // All three providers expose the same command-sequence shape (A5 backend // surface normalization) — no per-compiler arity branching here. - std::vector stdCommands = - isMsvc ? mcpp::toolchain::msvc::std_module_build_commands( - tc, sm.cacheDir, cpp_standard_flag) - : is_clang(tc) - ? mcpp::toolchain::clang::std_module_build_commands( - tc, sm.cacheDir, sm.bmiPath, sysroot_flag, cpp_standard_flag) - : mcpp::toolchain::gcc::std_module_build_commands( - tc, sm.cacheDir, sysroot_flag, cpp_standard_flag); - std::vector compatCommands; - if (!tc.stdCompatSource.empty()) { - if (isMsvc) { - compatCommands = mcpp::toolchain::msvc::std_compat_build_commands( - tc, sm.cacheDir, cpp_standard_flag); - } else if (is_clang(tc)) { - auto compatBmi = mcpp::toolchain::clang::std_compat_bmi_path(sm.cacheDir); - compatCommands = mcpp::toolchain::clang::std_compat_build_commands( - tc, sm.cacheDir, compatBmi, sm.bmiPath, sysroot_flag, cpp_standard_flag); + // + // Everything downstream of the cache directory is derived here, because the + // directory NAME is derived from the metadata and the metadata contains the + // build commands, which contain the directory. Resolving that circularity + // by hand: run this once against a placeholder directory to obtain the + // identity (that is what "normalized" means — the commands then name a + // literal placeholder segment instead of a key-specific path), hash it, + // then run it again against the real directory to get what goes on disk. + struct Derived { + std::filesystem::path bmiPath; + std::filesystem::path objectPath; + std::vector stdCommands; + std::vector compatCommands; + nlohmann::json metadata; + }; + auto derive = [&](const std::filesystem::path& cacheDir) { + Derived d; + d.bmiPath = isMsvc ? mcpp::toolchain::msvc::std_bmi_path(cacheDir) + : is_clang(tc) + ? mcpp::toolchain::clang::std_bmi_path(cacheDir) + : mcpp::toolchain::gcc::std_bmi_path(cacheDir); + d.objectPath = cacheDir / (isMsvc ? "std.obj" : "std.o"); + d.stdCommands = + isMsvc ? mcpp::toolchain::msvc::std_module_build_commands( + tc, cacheDir, cpp_standard_flag) + : is_clang(tc) + ? mcpp::toolchain::clang::std_module_build_commands( + tc, cacheDir, d.bmiPath, sysroot_flag, cpp_standard_flag) + : mcpp::toolchain::gcc::std_module_build_commands( + tc, cacheDir, sysroot_flag, cpp_standard_flag); + if (!tc.stdCompatSource.empty()) { + if (isMsvc) { + d.compatCommands = mcpp::toolchain::msvc::std_compat_build_commands( + tc, cacheDir, cpp_standard_flag); + } else if (is_clang(tc)) { + auto compatBmi = mcpp::toolchain::clang::std_compat_bmi_path(cacheDir); + d.compatCommands = mcpp::toolchain::clang::std_compat_build_commands( + tc, cacheDir, compatBmi, d.bmiPath, sysroot_flag, cpp_standard_flag); + } } - } - auto metadata = metadata_for(tc, cpp_standard, cpp_standard_flag, stdCommands, compatCommands); + d.metadata = metadata_for(tc, cpp_standard, cpp_standard_flag, + d.stdCommands, d.compatCommands); + return d; + }; + + const auto stdRoot = cache_root / "std"; + auto normalized = derive(stdRoot / kStdKeyPlaceholder).metadata; + auto identity = std_identity_key(normalized); + sm.cacheDir = stdRoot / identity; + + auto derived = derive(sm.cacheDir); + sm.bmiPath = derived.bmiPath; + sm.objectPath = derived.objectPath; + const auto& stdCommands = derived.stdCommands; + const auto& compatCommands = derived.compatCommands; + const auto& metadata = derived.metadata; auto metaPath = metadata_path(sm.cacheDir); bool std_cached = std::filesystem::exists(sm.bmiPath) && std::filesystem::exists(sm.objectPath) diff --git a/src/ui.cppm b/src/ui.cppm index 6ca9cd61..5229adb8 100644 --- a/src/ui.cppm +++ b/src/ui.cppm @@ -31,7 +31,11 @@ void status(std::string_view verb, std::string_view message); void info(std::string_view verb, std::string_view message); // Bold green Finished line. -void finished(std::string_view profile, std::chrono::milliseconds elapsed); +// `descriptor` annotates the profile's actual effect (e.g. "optimized", +// "unoptimized + debuginfo"). Empty = print the profile name alone; callers +// that never resolved the profile knobs must not invent one. +void finished(std::string_view profile, std::chrono::milliseconds elapsed, + std::string_view descriptor = {}); // "warning:" / "error:" prefix lines (yellow / red). void warning(std::string_view message); @@ -245,12 +249,20 @@ void info(std::string_view verb, std::string_view message) { } } -void finished(std::string_view profile, std::chrono::milliseconds elapsed) { +void finished(std::string_view profile, std::chrono::milliseconds elapsed, + std::string_view descriptor) { if (g_quiet) return; init(); auto v = verb_padded("Finished"); auto secs = static_cast(elapsed.count()) / 1000.0; - auto msg = std::format("{} [optimized] in {:.2f}s", profile, secs); + // `[optimized]` used to be hardcoded here alongside a hardcoded "release" + // at the only call site, so every build — including `--dev` at -O0 -g — + // announced "Finished release [optimized]". The descriptor is now supplied + // by whoever actually resolved the profile knobs, and omitted by callers + // (the fast path) that never resolved them: no caller has to guess. + auto msg = descriptor.empty() + ? std::format("{} in {:.2f}s", profile, secs) + : std::format("{} [{}] in {:.2f}s", profile, descriptor, secs); if (g_color) { std::println("{}{}{}{} {}", kBold, kBrightGreen, v, kReset, msg); diff --git a/tests/e2e/100_cppfly_reflection.sh b/tests/e2e/100_cppfly_reflection.sh index 8177be71..5eb544ee 100755 --- a/tests/e2e/100_cppfly_reflection.sh +++ b/tests/e2e/100_cppfly_reflection.sh @@ -104,7 +104,7 @@ out=$("$binary") } # The std BMI prebuild carries the same dialect (scan/prebuild/compile agree). -metadata="$(find "$MCPP_HOME/bmi" -name std-module.json | head -1)" +metadata="$(find "$MCPP_HOME/build-cache/v1/std" -name std-module.json | head -1)" grep -q '"std_flag": "-std=c++26 -freflection' "$metadata" || { cat "$metadata" echo "FAIL: std-module.json std_flag lacks -std=c++26 -freflection" diff --git a/tests/e2e/10_env_command.sh b/tests/e2e/10_env_command.sh index 5a0eb0ea..77716f71 100755 --- a/tests/e2e/10_env_command.sh +++ b/tests/e2e/10_env_command.sh @@ -14,7 +14,7 @@ out=$("$MCPP" self env 2>&1) # Verify directory tree [[ -d "$MCPP_HOME/bin" ]] || { echo "missing bin/"; exit 1; } [[ -d "$MCPP_HOME/registry" ]] || { echo "missing registry/"; exit 1; } -[[ -d "$MCPP_HOME/bmi" ]] || { echo "missing bmi/"; exit 1; } +[[ -d "$MCPP_HOME/build-cache/v1" ]] || { echo "missing build-cache/v1/"; exit 1; } [[ -d "$MCPP_HOME/cache" ]] || { echo "missing cache/"; exit 1; } [[ -f "$MCPP_HOME/config.toml" ]] || { echo "missing config.toml"; exit 1; } [[ -f "$MCPP_HOME/registry/.xlings.json" ]] || { echo "missing seeded .xlings.json"; exit 1; } diff --git a/tests/e2e/170_bmi_staging_no_cascade.sh b/tests/e2e/170_bmi_staging_no_cascade.sh index 8d365571..32c513ea 100755 --- a/tests/e2e/170_bmi_staging_no_cascade.sh +++ b/tests/e2e/170_bmi_staging_no_cascade.sh @@ -9,7 +9,7 @@ # carried no `restat`, so every importer of `import std` rebuilt whenever # the cache-side BMI got a newer mtime (which happens on any cwd change # before the cache-root fix below). -# 2. The staging SOURCE must live under $MCPP_HOME/bmi. The std module cache +# 2. The staging SOURCE must live under $MCPP_HOME/build-cache/v1/std. That cache # used to resolve its root through a private copy of the home logic that # knew neither %USERPROFILE% nor self-contained installs, so on Windows it # parked the cache in the current working directory as `.mcpp-bmi/`. @@ -56,11 +56,13 @@ DST=$(unescape_ninja "$(echo "$EDGE" | awk '{print $2}')") SRC=$(unescape_ninja "$(echo "$EDGE" | awk '{print $NF}')") echo "staging: $SRC -> $DST" -# ── invariant 2: the cache root is $MCPP_HOME/bmi, never a cwd-local dir ── +# ── invariant 2: the cache root is $MCPP_HOME/build-cache/v1/std, never a cwd-local +# dir (and never the pre-v1 $MCPP_HOME/bmi tree, which nothing reads now) ── HOME_ROOT=$(norm_path "${MCPP_HOME:-$HOME/.mcpp}") case "$(norm_path "$SRC")" in - "$HOME_ROOT"/bmi/*) ;; - *) echo "FAIL: std BMI cache is '$SRC', expected under '$HOME_ROOT/bmi'"; exit 1 ;; + "$HOME_ROOT"/build-cache/v1/std/*) ;; + *) echo "FAIL: std BMI cache is '$SRC', expected under '$HOME_ROOT/build-cache/v1/std'" + exit 1 ;; esac for leftover in "$TMP/.mcpp-bmi" "$TMP/app/.mcpp-bmi"; do [[ -d "$leftover" ]] && { echo "FAIL: legacy cache dir created at $leftover"; exit 1; } diff --git a/tests/e2e/172_build_cache_cross_project.sh b/tests/e2e/172_build_cache_cross_project.sh new file mode 100644 index 00000000..38de385f --- /dev/null +++ b/tests/e2e/172_build_cache_cross_project.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# requires: gcc fresh-sandbox +# 172_build_cache_cross_project.sh — the global build cache must actually save +# work, and it must save it ACROSS projects. +# +# Two defects met here, and this file is the gate for both. +# +# 1. The key was the whole-project fingerprint, which serializes every package +# in the graph INCLUDING the root — its name, its version, its [build] +# flags. So two projects never shared an entry, and bumping a project's own +# version invalidated every dependency it had. (Measured on one machine: +# 26 GB across 1198 fingerprint directories, `compat.zlib@1.3.2` stored 162 +# times.) +# +# 2. Even on a hit, nothing was saved. Artifacts were copied into the build dir +# from inside prepare_build while those paths stayed declared as compile edge +# outputs — and ninja treats an output it has no command line for in +# .ninja_log as dirty ("command line not found in log"), which a fresh build +# dir always is. Every "cached" unit was recompiled while the CLI printed +# "Cached". +# +# So the load-bearing assertion is not "an entry exists" but "the second project +# has ZERO compile edges for the dependency's sources". Read from build.ninja, +# not from the log: a status line is exactly what lied before. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +# ── an offline path index serving one library package ──────────────────────── +INDEX_DIR="$TMP/local-index" +mkdir -p "$INDEX_DIR/pkgs/s" +cat > "$INDEX_DIR/pkgs/s/shared-lib.lua" <<'EOF' +package = { + spec = "1", + name = "shared-lib", + description = "Shared across projects", + licenses = {"MIT"}, + type = "package", + xpm = { + linux = { + ["1.0.0"] = { + url = "https://example.invalid/shared-lib-1.0.0.tar.gz", + sha256 = "0000000000000000000000000000000000000000000000000000000000000000", + }, + }, + }, + mcpp = { + language = "c++23", + import_std = true, + sources = { "src/**/*.cppm" }, + targets = { ["shared-lib"] = { kind = "lib" } }, + deps = {}, + }, +} +EOF + +# `make_project ` — same dependency, and +# deliberately DIFFERENT identity, because the consumer's identity is exactly +# what used to leak into the dependency's cache key. +make_project() { + local dir="$1" name="$2" version="$3" + mkdir -p "$TMP/$dir/src" + mkdir -p "$TMP/$dir/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.0/src" + cat > "$TMP/$dir/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.0/src/lib.cppm" <<'EOF' +export module shared.lib; +export int shared_value() { return 41; } +EOF + cat > "$TMP/$dir/src/main.cpp" <<'EOF' +import std; +import shared.lib; +int main() { std::println("{}", shared_value() + 1); return 0; } +EOF + cat > "$TMP/$dir/mcpp.toml" < build.log 2>&1 || { cat build.log; exit 1; } + +N1="$(find_ninja "$TMP/projone")" +[[ -n "$N1" ]] || { echo "FAIL: projone has no build.ninja"; exit 1; } +[[ "$(dep_compile_edges "$N1")" -gt 0 ]] || { + echo "FAIL: cold build had no compile edges for the dependency" + grep -n 'shared-lib' "$N1" | head + exit 1 +} +"$MCPP" cache list > list1.log 2>&1 +grep -q 'shared-lib' list1.log || { + echo "FAIL: cold build did not populate a cache entry" + cat list1.log; cat build.log + exit 1 +} + +# ── project two: DIFFERENT name and version, same dependency ───────────────── +make_project projtwo projtwo 9.9.9 +cd "$TMP/projtwo" +"$MCPP" build > build.log 2>&1 || { cat build.log; exit 1; } + +N2="$(find_ninja "$TMP/projtwo")" +[[ -n "$N2" ]] || { echo "FAIL: projtwo has no build.ninja"; exit 1; } + +# THE assertion. +edges="$(dep_compile_edges "$N2")" +if [[ "$edges" != "0" ]]; then + echo "FAIL: second project recompiled the dependency ($edges compile edges)" + echo " the cache key is still carrying the consumer's identity" + grep -nE ': (cxx_module|cxx_object|cxx_scan) .*shared-lib' "$N2" | head + exit 1 +fi +staged="$(dep_stage_edges "$N2")" +[[ "$staged" -gt 0 ]] || { + echo "FAIL: dependency has neither compile nor stage edges — it is missing" + grep -n 'shared-lib' "$N2" | head + exit 1 +} + +# The status line must agree, and must carry the unit count. The bare word +# "Cached" was printed for months while every unit was recompiled behind it; a +# number that has to match the skipped edges cannot go quietly wrong that way. +grep -qE 'Cached local-dev\.shared-lib v1\.0\.0 \([0-9]+ unit' build.log || { + echo "FAIL: no 'Cached ... (N units)' line for the reused dependency" + cat build.log + exit 1 +} + +# And it has to WORK: staged artifacts must link and run. +./target/*/*/bin/projtwo > run.log 2>&1 || { cat run.log; exit 1; } +grep -q '^42$' run.log || { echo "FAIL: staged artifacts produced wrong output"; cat run.log; exit 1; } + +# ── the consumer's own version must not invalidate the dependency ──────────── +cd "$TMP/projone" +sed -i.bak 's/version = "0.1.0"/version = "0.2.0"/' mcpp.toml +rm -f mcpp.toml.bak +rm -rf target +"$MCPP" build > bump.log 2>&1 || { cat bump.log; exit 1; } +N3="$(find_ninja "$TMP/projone")" +[[ "$(dep_compile_edges "$N3")" == "0" ]] || { + echo "FAIL: bumping the consumer's own version invalidated the dependency" + grep -nE ': (cxx_module|cxx_object|cxx_scan) .*shared-lib' "$N3" | head + exit 1 +} + +# ── a real change to the dependency MUST invalidate it (no false hits) ─────── +# Bump the dependency's version in the index and in the manifests: a different +# version is a different entry, so it has to be compiled again. +sed -i.bak 's/\["1\.0\.0"\]/["1.0.1"]/' "$INDEX_DIR/pkgs/s/shared-lib.lua" +rm -f "$INDEX_DIR/pkgs/s/shared-lib.lua.bak" +mv "$TMP/projone/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.0" \ + "$TMP/projone/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.1" +sed -i.bak 's/"local-dev.shared-lib" = "1.0.0"/"local-dev.shared-lib" = "1.0.1"/' \ + "$TMP/projone/mcpp.toml" +rm -f "$TMP/projone/mcpp.toml.bak" +rm -rf target +"$MCPP" build > newver.log 2>&1 || { cat newver.log; exit 1; } +N4="$(find_ninja "$TMP/projone")" +[[ "$(dep_compile_edges "$N4")" -gt 0 ]] || { + echo "FAIL: a new dependency version was served from the old entry" + cat newver.log + exit 1 +} + +echo "OK" diff --git a/tests/e2e/173_build_profile_isolation.sh b/tests/e2e/173_build_profile_isolation.sh new file mode 100644 index 00000000..516903cf --- /dev/null +++ b/tests/e2e/173_build_profile_isolation.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# requires: gcc fresh-sandbox +# 173_build_profile_isolation.sh — the build profile must be an invalidation +# axis, and the fast path must not hand back another profile's artifacts. +# +# `[profile.]` lands its knobs in buildConfig.optLevel/debug/lto/strip and +# flags.cppm turns them into -O/-g/-flto, but the fingerprint serialized only +# cflags/cxxflags/ldflags. So `--dev`, `--release` and `--profile dist` produced +# ONE fingerprint, hence one target/// directory and one global cache +# entry — a release build could be served -O0 -g dependency objects. +# +# The second half was worse and user-visible without any cache involved: +# `.build_cache` keyed its fast-path entries by target triple alone, and the fast +# path only refuses to run when an EXPLICIT --profile/--dev/--release is passed. +# A bare `mcpp build` after `mcpp build --release` therefore took the fast path +# against the release build.ninja and reported success in 0.00s, leaving -O2 +# artifacts where -O0 -g was asked for. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +cd "$TMP" +mkdir -p app/src +cd app +cat > src/main.cpp <<'EOF' +#include +int main() { std::printf("hi\n"); return 0; } +EOF +cat > mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" +standard = "c++23" + +[targets.app] +kind = "bin" +main = "src/main.cpp" +EOF + +# The -O level recorded in a build dir's build.ninja. Reading the generated graph +# rather than a status line: a status line is what lied here before. +opt_of() { grep -oE '\-O[0-9s]' "$1/build.ninja" | sort -u | tr -d '\n'; } +dir_for_opt() { + local want="$1" + for d in target/*/*/; do + [[ -f "$d/build.ninja" ]] || continue + [[ "$(opt_of "$d")" == "$want" ]] && { printf '%s' "$d"; return 0; } + done + return 1 +} + +# ── each profile gets its own build dir ───────────────────────────────────── +"$MCPP" build --dev > dev.log 2>&1 || { cat dev.log; exit 1; } +"$MCPP" build --release > release.log 2>&1 || { cat release.log; exit 1; } +"$MCPP" build --profile dist > dist.log 2>&1 || { cat dist.log; exit 1; } + +count=$(find target -name build.ninja | wc -l) +[[ "$count" -eq 3 ]] || { + echo "FAIL: expected one build dir per profile, found $count" + for d in target/*/*/; do echo " $d -> $(opt_of "$d")"; done + exit 1 +} +dir_for_opt "-O0" >/dev/null || { echo "FAIL: no -O0 (dev) build dir"; exit 1; } +dir_for_opt "-O2" >/dev/null || { echo "FAIL: no -O2 (release) build dir"; exit 1; } +dir_for_opt "-O3" >/dev/null || { echo "FAIL: no -O3 (dist) build dir"; exit 1; } + +# The announced profile must be the one that was built. +grep -q 'Finished dev' dev.log || { cat dev.log; echo "FAIL: dev not announced"; exit 1; } +grep -q 'Finished release' release.log || { cat release.log; echo "FAIL: release not announced"; exit 1; } +grep -q 'Finished dist' dist.log || { cat dist.log; echo "FAIL: dist not announced"; exit 1; } +# ...and the descriptor must not contradict the flags. "release [optimized]" was +# hardcoded, so a --dev build used to announce itself as an optimized release. +grep -q 'Finished dev \[unoptimized' dev.log || { + cat dev.log + echo "FAIL: dev build described as optimized" + exit 1 +} + +# ── the regression gate: bare build after --release must be dev ───────────── +DEVDIR="$(dir_for_opt "-O0")" +rm -rf target +"$MCPP" build --release > r2.log 2>&1 || { cat r2.log; exit 1; } +"$MCPP" build > bare.log 2>&1 || { cat bare.log; exit 1; } + +grep -q 'Finished dev' bare.log || { + echo "FAIL: bare build after --release did not resolve to the dev profile" + cat bare.log + exit 1 +} +DEVDIR="$(dir_for_opt "-O0")" || { + echo "FAIL: bare build produced no -O0 build dir (fast path served release)" + for d in target/*/*/; do echo " $d -> $(opt_of "$d")"; done + cat bare.log + exit 1 +} +# The dev objects must actually exist and carry debug info: an existing build dir +# with no objects in it would satisfy a path check but not a build. +[[ -f "$DEVDIR/obj/main.o" ]] || { + echo "FAIL: dev build dir has no object" + find "$DEVDIR" -type f | head + exit 1 +} +if command -v readelf > /dev/null 2>&1; then + readelf --debug-dump=info "$DEVDIR/obj/main.o" 2>/dev/null \ + | grep -q 'DW_AT_producer' || { + echo "FAIL: dev object has no debug info (it is a release object)" + exit 1 + } +fi + +# ── switching back is incremental, not a full rebuild ────────────────────── +# Both entries coexist in .build_cache now; keying on the triple alone made each +# profile evict the other, so a dev/release rotation could never be incremental. +"$MCPP" build --release > r3.log 2>&1 || { cat r3.log; exit 1; } +"$MCPP" build > bare2.log 2>&1 || { cat bare2.log; exit 1; } +if grep -qE 'src/main\.cpp' bare2.log; then + echo "FAIL: returning to the dev profile recompiled from scratch" + cat bare2.log + exit 1 +fi +grep -q 'Finished dev' bare2.log || { cat bare2.log; echo "FAIL: wrong profile"; exit 1; } + +echo "OK" diff --git a/tests/e2e/174_cache_modes_and_commands.sh b/tests/e2e/174_cache_modes_and_commands.sh new file mode 100644 index 00000000..7730739d --- /dev/null +++ b/tests/e2e/174_cache_modes_and_commands.sh @@ -0,0 +1,302 @@ +#!/usr/bin/env bash +# requires: gcc fresh-sandbox +# 174_cache_modes_and_commands.sh — the three build modes, and the cache +# maintenance surface they need. +# +# mcpp build read + write the global cache (default) +# mcpp build --cache=local neither; every dependency compiles in target/ +# mcpp build --cache=off neither, and target/ is cleared first +# mcpp build --no-cache deprecated alias for --cache=off +# +# `--no-cache` used to be the only switch, and it only ever cleared target/ — +# a name that says nothing about a cache. It stays accepted. +# +# The maintenance half exists because a cache nobody can inspect, size or +# reclaim is a cache nobody can trust. `prune` in particular used to rank +# entries by the directory's mtime, which only records when an entry was +# WRITTEN — so a dependency that hit on every build looked as stale as one +# nobody had touched in a month. `gc` reads entry.json's `accessed` stamp. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +INDEX_DIR="$TMP/local-index" +mkdir -p "$INDEX_DIR/pkgs/m" +cat > "$INDEX_DIR/pkgs/m/mode-lib.lua" <<'EOF' +package = { + spec = "1", + name = "mode-lib", + description = "Cache mode fixture", + licenses = {"MIT"}, + type = "package", + xpm = { + linux = { + ["1.0.0"] = { + url = "https://example.invalid/mode-lib-1.0.0.tar.gz", + sha256 = "0000000000000000000000000000000000000000000000000000000000000000", + }, + }, + }, + mcpp = { + language = "c++23", + import_std = true, + sources = { "src/**/*.cppm" }, + targets = { ["mode-lib"] = { kind = "lib" } }, + deps = {}, + }, +} +EOF + +mkdir -p "$TMP/app/src" \ + "$TMP/app/.mcpp/.xlings/data/xpkgs/local-dev.mode-lib/1.0.0/src" +cat > "$TMP/app/.mcpp/.xlings/data/xpkgs/local-dev.mode-lib/1.0.0/src/lib.cppm" <<'EOF' +export module mode.lib; +export int mode_value() { return 5; } +EOF +cat > "$TMP/app/src/main.cpp" <<'EOF' +import std; +import mode.lib; +int main() { std::println("{}", mode_value()); return 0; } +EOF +cat > "$TMP/app/mcpp.toml" </dev/null | wc -l; } + +# ── mcpp cache dir: where IS it? ──────────────────────────────────────────── +out=$("$MCPP" cache dir 2>&1) +[[ "$out" == "$MCPP_HOME/build-cache/v1"* ]] || { + echo "FAIL: cache dir reported '$out', expected '$MCPP_HOME/build-cache/v1'" + exit 1 +} + +# ── local: nothing read, nothing written ─────────────────────────────────── +"$MCPP" build --cache=local > local.log 2>&1 || { cat local.log; exit 1; } +[[ "$(entry_count)" -eq 0 ]] || { + echo "FAIL: --cache=local wrote $(entry_count) cache entries" + find "$MCPP_HOME/build-cache/v1/pkg" -maxdepth 4 2>/dev/null + exit 1 +} +# The dependency still has to be built, in-project. +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: --cache=local did not compile the dependency locally" + grep -n 'mode-lib' "$NINJA" | head + exit 1 +} +./target/*/*/bin/app > run.log 2>&1 || { cat run.log; exit 1; } +grep -q '^5$' run.log || { echo "FAIL: local build produced wrong output"; cat run.log; exit 1; } + +# ── global (default): writes an entry ────────────────────────────────────── +rm -rf target +"$MCPP" build > global.log 2>&1 || { cat global.log; exit 1; } +[[ "$(entry_count)" -eq 1 ]] || { + echo "FAIL: default build wrote $(entry_count) entries, expected 1" + cat global.log + exit 1 +} + +# ── cache list / list --json / info / verify ─────────────────────────────── +"$MCPP" cache list > list.log 2>&1 +grep -q 'mode-lib' list.log || { cat list.log; echo "FAIL: list omits the entry"; exit 1; } + +"$MCPP" cache list --json > list.json 2>&1 +python3 - list.json > jsoncheck.log 2>&1 <<'PYEOF' || { cat jsoncheck.log; exit 1; } +import json, sys +d = json.load(open(sys.argv[1])) +if "root" not in d or "entries" not in d: + sys.exit(f"FAIL: list --json missing keys: {sorted(d)}") +pkgs = [e for e in d["entries"] if e["kind"] == "pkg"] +if not pkgs: + sys.exit("FAIL: list --json has no package entries") +e = pkgs[0] +for k in ("kind", "label", "key", "dir", "bytes", "files", "accessed", "complete"): + if k not in e: + sys.exit(f"FAIL: entry missing '{k}': {sorted(e)}") +if not e["complete"]: + sys.exit(f"FAIL: fresh entry reported incomplete: {e}") +if e["bytes"] <= 0: + sys.exit(f"FAIL: entry reports {e['bytes']} bytes") +PYEOF + +# info must print the recorded key inputs — that is the whole reason a cache +# entry describes itself, and the only way to diagnose a suspected wrong hit. +"$MCPP" cache info mode-lib > info.log 2>&1 || { cat info.log; exit 1; } +grep -q 'inputs' info.log || { cat info.log; echo "FAIL: info omits key inputs"; exit 1; } +grep -q 'opt_level' info.log || { cat info.log; echo "FAIL: info omits the profile axis"; exit 1; } + +"$MCPP" cache verify > verify.log 2>&1 || { cat verify.log; echo "FAIL: verify failed on a healthy cache"; exit 1; } +grep -q 'all complete' verify.log || { cat verify.log; echo "FAIL: verify message"; exit 1; } + +# verify must FAIL (non-zero) when an artifact is gone. Explicit if/exit rather +# than `! cmd`: under `set -e` a negated command is exempted from the errexit +# check, so `! "$MCPP" cache verify` can never fail this test. +victim="$(find "$MCPP_HOME/build-cache/v1/pkg" -path '*/obj/*' -type f | head -1)" +[[ -n "$victim" ]] || { echo "FAIL: no cached object to remove"; exit 1; } +mv "$victim" "$victim.bak" +rc=0 +"$MCPP" cache verify > verify2.log 2>&1 || rc=$? +if [[ "$rc" -eq 0 ]]; then + echo "FAIL: verify passed with a missing artifact" + cat verify2.log + exit 1 +fi +grep -q 'incomplete' verify2.log || { cat verify2.log; echo "FAIL: verify wording"; exit 1; } +mv "$victim.bak" "$victim" + +# ── off: clears this build's build dir and touches no cache ──────────────── +# The build DIRECTORY (target///), not all of target/: sibling dirs +# for other profiles and other targets are not this invocation's business. +builddir="$(dirname "$(find target -name build.ninja | head -1)")" +marker="$builddir/_off_marker" +touch "$marker" +before="$(entry_count)" +"$MCPP" build --cache=off > off.log 2>&1 || { cat off.log; exit 1; } +[[ ! -f "$marker" ]] || { echo "FAIL: --cache=off did not clear the build dir"; exit 1; } +[[ "$(entry_count)" -eq "$before" ]] || { + echo "FAIL: --cache=off changed the cache ($before -> $(entry_count))" + exit 1 +} +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: --cache=off served the dependency from the cache" + exit 1 +} + +# --no-cache must behave identically (deprecated alias). +touch "$marker" +"$MCPP" build --no-cache > nocache.log 2>&1 || { cat nocache.log; exit 1; } +[[ ! -f "$marker" ]] || { echo "FAIL: --no-cache did not clear the build dir"; exit 1; } + +# An unknown mode is refused, not silently taken as "global". +rc=0 +"$MCPP" build --cache=bogus --strict > bogus.log 2>&1 || rc=$? +[[ "$rc" -ne 0 ]] || { echo "FAIL: --cache=bogus accepted under --strict"; cat bogus.log; exit 1; } +grep -qi 'cache mode' bogus.log || { cat bogus.log; echo "FAIL: unhelpful error"; exit 1; } + +# MCPP_BUILD_CACHE is the env-level equivalent. +rm -rf target +MCPP_BUILD_CACHE=local "$MCPP" build > env.log 2>&1 || { cat env.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: MCPP_BUILD_CACHE=local was ignored" + exit 1 +} + +# [build] cache is the project-level equivalent, and --cache beats it. +cat >> mcpp.toml <<'EOF' + +[build] +cache = "local" +EOF +rm -rf target +"$MCPP" build > manifest.log 2>&1 || { cat manifest.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: [build] cache = local was ignored" + exit 1 +} +rm -rf target +"$MCPP" build --cache=global > override.log 2>&1 || { cat override.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +if grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA"; then + echo "FAIL: --cache=global did not override [build] cache = local" + exit 1 +fi + +# Drop the manifest override again — everything below assumes the default mode, +# and a leftover `cache = "local"` would make the gc section measure an empty +# cache and "pass" for the wrong reason. +python3 - mcpp.toml <<'PYEOF' +import sys +p = sys.argv[1] +lines = open(p).read().splitlines(keepends=True) +out, skip = [], False +for ln in lines: + if ln.strip() == "[build]": + skip = True + continue + if skip and ln.strip().startswith("cache"): + skip = False + continue + out.append(ln) +open(p, "w").write("".join(out)) +PYEOF +if grep -q '^cache' mcpp.toml; then + echo "FAIL: could not strip the [build] cache override" + cat mcpp.toml + exit 1 +fi + +# ── gc: LRU by last USE, not by when the entry was written ───────────────── +# Backdate the entry's accessed stamp, then confirm an age-bounded gc collects it +# — and that a build afterwards refreshes the stamp so it would survive next time. +entry="$(find "$MCPP_HOME/build-cache/v1/pkg" -name entry.json | head -1)" +python3 - "$entry" <<'PYEOF' +import json, sys +p = sys.argv[1] +d = json.load(open(p)) +d["accessed"] = "1000" # 1970-ish +json.dump(d, open(p, "w"), indent=2) +PYEOF +"$MCPP" cache gc --older-than 1s > gc.log 2>&1 || { cat gc.log; exit 1; } +[[ "$(entry_count)" -eq 0 ]] || { + echo "FAIL: gc did not collect a long-unused entry" + cat gc.log + exit 1 +} +grep -q 'Collected' gc.log || { cat gc.log; echo "FAIL: gc wording"; exit 1; } + +rm -rf target +"$MCPP" build > refill.log 2>&1 || { cat refill.log; exit 1; } +"$MCPP" build > touch.log 2>&1 || { cat touch.log; exit 1; } # a HIT +"$MCPP" cache gc --older-than 1h > gc2.log 2>&1 || { cat gc2.log; exit 1; } +[[ "$(entry_count)" -eq 1 ]] || { + echo "FAIL: gc collected an entry that was just used" + cat gc2.log + exit 1 +} + +# gc requires a budget; no arguments is a usage error, not a silent no-op. +rc=0 +"$MCPP" cache gc > gcnoargs.log 2>&1 || rc=$? +[[ "$rc" -ne 0 ]] || { echo "FAIL: gc with no budget succeeded"; cat gcnoargs.log; exit 1; } + +# ── clean --std / --all / --legacy ───────────────────────────────────────── +[[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { echo "FAIL: no std cache dir"; exit 1; } +"$MCPP" cache clean --std > cleanstd.log 2>&1 +[[ ! -d "$MCPP_HOME/build-cache/v1/std" ]] || { + echo "FAIL: clean --std left the std entries" + exit 1 +} +[[ "$(entry_count)" -eq 1 ]] || { echo "FAIL: clean --std dropped package entries too"; exit 1; } + +"$MCPP" cache clean --all > cleanall.log 2>&1 +[[ "$(entry_count)" -eq 0 ]] || { echo "FAIL: clean --all left package entries"; exit 1; } + +# --legacy targets the pre-v1 tree, which nothing reads any more. +mkdir -p "$MCPP_HOME/bmi/deadbeef/deps/idx/pkg@1.0.0" +echo x > "$MCPP_HOME/bmi/deadbeef/deps/idx/pkg@1.0.0/old.o" +"$MCPP" cache clean --legacy > cleanlegacy.log 2>&1 +[[ ! -d "$MCPP_HOME/bmi" ]] || { cat cleanlegacy.log; echo "FAIL: clean --legacy left $MCPP_HOME/bmi"; exit 1; } +grep -q 'pre-v1' cleanlegacy.log || { cat cleanlegacy.log; echo "FAIL: legacy wording"; exit 1; } + +echo "OK" diff --git a/tests/e2e/19_bmi_cache_reuse.sh b/tests/e2e/19_bmi_cache_reuse.sh index 2f495724..e83023d9 100755 --- a/tests/e2e/19_bmi_cache_reuse.sh +++ b/tests/e2e/19_bmi_cache_reuse.sh @@ -1,15 +1,21 @@ #!/usr/bin/env bash # requires: -# 19_bmi_cache_reuse.sh — verify M3.2 BMI persistent cache wiring. +# 19_bmi_cache_reuse.sh — local-source packages must never enter the global +# build cache, DIRECTLY or TRANSITIVELY. # -# 1. Path deps don't populate the cache (correctness invariant from docs/26). -# 2. The cache layout matches docs/26 §二 (~/.mcpp/bmi//deps/...). +# 1. A direct path dep is not cached. +# 2. A path dep reached THROUGH another path dep is not cached either. # -# We don't fetch a real registry dep here (would need network, and existing -# E2E tests deliberately avoid that). The cross-project reuse positive case -# is covered by unit tests around the bmi_cache module's stage/populate -# round-trip; this script verifies the *wiring* into mcpp build doesn't -# break path-dep flows and that bmi/ is created lazily. +# (2) is the regression this file exists for. The old predicate looked the +# package up in the ROOT manifest's dependencies/dev-dependencies and skipped it +# when the spec was path/git — but a transitively-reached package is in neither +# map, so it fell through and WAS cached, with its index name defaulting to the +# default index (a workspace member `B` landed on disk as `mcpplibs/B@0.1.0`). +# Its sources can then change without changing name@version, so the cache key +# cannot see the change: a stale object, served silently. +# +# The cross-project positive case needs a registry dep and lives in +# 172_build_cache_cross_project.sh. set -e TMP=$(mktemp -d) @@ -69,11 +75,12 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; exit 1; } -# bmi/ should exist (env init creates it) but no deps/ entry for path deps. -[[ -d "$MCPP_HOME/bmi" ]] || { echo "missing $MCPP_HOME/bmi"; exit 1; } -if find "$MCPP_HOME/bmi" -path "*/deps/*/mylibA*" 2>/dev/null | grep -q .; then - echo "FAIL: path dep mylibA was populated into BMI cache (must be skipped)" - find "$MCPP_HOME/bmi" -maxdepth 5 +# The cache root exists (env init creates it) but holds no package entry for a +# path dep. +[[ -d "$MCPP_HOME/build-cache/v1" ]] || { echo "missing $MCPP_HOME/build-cache/v1"; exit 1; } +if find "$MCPP_HOME/build-cache/v1/pkg" -path "*mylibA*" 2>/dev/null | grep -q .; then + echo "FAIL: path dep mylibA was populated into the build cache (must be skipped)" + find "$MCPP_HOME/build-cache/v1" -maxdepth 5 exit 1 fi @@ -83,36 +90,79 @@ if grep -q 'Cached mylibA' build.log; then cat build.log; exit 1 fi -# --- Part 2: pre-seed a fake cache entry and verify it's staged into target/ --- -# Capture this build's fingerprint dir so we can drop a fake registry-dep -# cache entry beside it. -cd .. -rm -rf myapp/target -fp_dir="$(ls "$MCPP_HOME/bmi" | head -1)" -[[ -n "$fp_dir" ]] || { echo "no fingerprint dir under bmi/"; exit 1; } +# --- Part 2: a path dep reached THROUGH a path dep is also excluded --------- +cd "$TMP" +"$MCPP" new mylibB > /dev/null +cd mylibB +cat > src/inner.cppm <<'EOF' +export module mylibB.inner; +export auto inner() -> int { return 7; } +EOF +rm src/main.cpp +cat > mcpp.toml <<'EOF' +[package] +name = "mylibB" +version = "0.1.0" +[language] +standard = "c++23" +modules = true +import_std = true +[modules] +sources = ["src/**/*.cppm"] +[targets.mylibB] +kind = "lib" +EOF -# Create a synthetic cache entry as if a previous build produced it. -fake_pkg_dir="$MCPP_HOME/bmi/$fp_dir/deps/mcpplibs/fake.pkg@9.9.9" -mkdir -p "$fake_pkg_dir/gcm.cache" "$fake_pkg_dir/obj" -echo SYN > "$fake_pkg_dir/gcm.cache/fake.pkg.gcm" -echo SYN > "$fake_pkg_dir/obj/fake.m.o" -cat > "$fake_pkg_dir/manifest.txt" <<'EOF' -# Auto-generated by mcpp bmi_cache. Do not edit. -gcm: fake.pkg.gcm -obj: fake.m.o +# mylibA now depends on mylibB by path, so myapp reaches mylibB transitively +# and mylibB appears in NEITHER of myapp's dependency maps. +cd "$TMP/mylibA" +cat > src/greet.cppm <<'EOF' +export module mylibA.greet; +import std; +import mylibB.inner; +export auto greet() -> void { std::println("hi {}", inner()); } +EOF +cat >> mcpp.toml <<'EOF' +[dependencies.mylibB] +path = "../mylibB" EOF -[[ -f "$fake_pkg_dir/manifest.txt" ]] || { echo "synth manifest missing"; exit 1; } -# Smoke check: the manifest format documented in docs/26 §5.2 is parseable -# (one gcm: line + one obj: line + leading comment). We don't run mcpp build -# against this synthetic dep because there's no consuming manifest with a -# version-style [dependencies] entry that resolves without a real registry — -# that would require network. The unit tests cover the parse + stage path. +cd "$TMP/myapp" +rm -rf target +"$MCPP" build > build2.log 2>&1 || { cat build2.log; exit 1; } + +for name in mylibA mylibB; do + if find "$MCPP_HOME/build-cache/v1/pkg" -path "*$name*" 2>/dev/null | grep -q .; then + echo "FAIL: local package $name entered the build cache" + find "$MCPP_HOME/build-cache/v1/pkg" -maxdepth 4 2>/dev/null + cat build2.log + exit 1 + fi +done -# Sanity: parse comments + entries the same way bmi_cache does. -gcm_count=$(grep -c '^gcm: ' "$fake_pkg_dir/manifest.txt") -obj_count=$(grep -c '^obj: ' "$fake_pkg_dir/manifest.txt") -[[ "$gcm_count" == "1" ]] || { echo "expected 1 gcm: line, got $gcm_count"; exit 1; } -[[ "$obj_count" == "1" ]] || { echo "expected 1 obj: line, got $obj_count"; exit 1; } +# And `mcpp cache list` — the user-visible surface — must not list them either. +"$MCPP" cache list > list.log 2>&1 +for name in mylibA mylibB; do + if grep -q "$name" list.log; then + echo "FAIL: cache list shows local package $name" + cat list.log + exit 1 + fi +done + +# Editing the transitive path dep must still take effect (it is compiled, not +# served from a cache that cannot see the change). +cd "$TMP/mylibB" +cat > src/inner.cppm <<'EOF' +export module mylibB.inner; +export auto inner() -> int { return 99; } +EOF +cd "$TMP/myapp" +"$MCPP" run > run.log 2>&1 || { cat run.log; exit 1; } +grep -q 'hi 99' run.log || { + echo "FAIL: edit to a transitive path dep did not reach the binary" + cat run.log + exit 1 +} echo "OK" diff --git a/tests/e2e/49_bmi_cache_nested_custom_index.sh b/tests/e2e/49_bmi_cache_nested_custom_index.sh index 7a2c915b..cdc8b633 100644 --- a/tests/e2e/49_bmi_cache_nested_custom_index.sh +++ b/tests/e2e/49_bmi_cache_nested_custom_index.sh @@ -93,27 +93,53 @@ if grep -q "bmi cache populate failed" build.log; then exit 1 fi -manifest="$(find "$MCPP_HOME/bmi" -path "*/deps/local-dev/*collision-lib@1.0.0/manifest.txt" | head -1)" -[[ -n "$manifest" ]] || { - echo "FAIL: missing custom-index BMI manifest" - find "$MCPP_HOME/bmi" -maxdepth 6 -type f | sort +entry="$(find "$MCPP_HOME/build-cache/v1/pkg/local-dev" -name entry.json | head -1)" +[[ -n "$entry" ]] || { + echo "FAIL: missing custom-index cache entry" + find "$MCPP_HOME/build-cache/v1" -maxdepth 6 -type f | sort cat build.log exit 1 } -grep -Eq '^obj: .+/foo\.m\.o$' "$manifest" || { - echo "FAIL: manifest did not preserve nested object path" - cat "$manifest" - exit 1 -} +# Read the structured entry instead of grepping a log: entry.json exists so the +# cache describes itself, and every object it lists must be on disk under obj/. +python3 - "$entry" > entrycheck.log 2>&1 <<'PYEOF' || { cat entrycheck.log; exit 1; } +import json, os, sys +entry = sys.argv[1] +d = json.load(open(entry)) +objs = d.get("obj", []) +if not [o for o in objs if "/" in o and o.endswith("foo.m.o")]: + sys.exit(f"FAIL: entry did not preserve a nested object path: {objs}") +root = os.path.dirname(entry) +for o in objs: + p = os.path.join(root, "obj", o) + if not os.path.exists(p): + sys.exit(f"FAIL: listed object missing on disk: {p}") +PYEOF rm -rf target "$MCPP" build > build2.log 2>&1 || { cat build2.log; exit 1; } grep -q "Cached local-dev.collision-lib v1.0.0" build2.log || { - echo "FAIL: second cold build did not reuse BMI cache" + echo "FAIL: second cold build did not reuse the build cache" cat build2.log exit 1 } +# A reused dependency must not be recompiled — that was the whole defect. Its +# outputs are declared by stage_file edges now, so there is no compile edge for +# its sources at all. +NINJA="$(find target -name build.ninja | head -1)" +[[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja"; exit 1; } +if grep -qE ': (cxx_module|cxx_object) .*collision-lib' "$NINJA"; then + echo "FAIL: cached dependency still has compile edges" + grep -nE ': (cxx_module|cxx_object) .*collision-lib' "$NINJA" + exit 1 +fi +grep -qE '^build .* : stage_file .*collision-lib' "$NINJA" || { + echo "FAIL: cached dependency has no stage_file edge" + grep -n 'stage_file' "$NINJA" | head + exit 1 +} + echo "OK" diff --git a/tests/e2e/59_cpp_standard_config.sh b/tests/e2e/59_cpp_standard_config.sh index 6eacd020..2c31ef91 100644 --- a/tests/e2e/59_cpp_standard_config.sh +++ b/tests/e2e/59_cpp_standard_config.sh @@ -71,7 +71,7 @@ if grep -q -- "-std=c++23" compile_commands.json; then exit 1 fi -metadata="$(find "$MCPP_HOME/bmi" -name std-module.json | head -1)" +metadata="$(find "$MCPP_HOME/build-cache/v1/std" -name std-module.json | head -1)" [[ -n "$metadata" ]] || { echo "FAIL: std module metadata missing"; exit 1; } grep -q '"cpp_standard": "c++26"' "$metadata" || { echo "FAIL: std module metadata missing C++26 standard" diff --git a/tests/unit/test_bmi_cache.cpp b/tests/unit/test_bmi_cache.cpp index 714bc25d..6fe2c517 100644 --- a/tests/unit/test_bmi_cache.cpp +++ b/tests/unit/test_bmi_cache.cpp @@ -7,6 +7,7 @@ import std; import mcpp.bmi_cache; +import mcpp.libs.json; using namespace mcpp::bmi_cache; @@ -26,13 +27,27 @@ struct Tmp { } }; -CacheKey makeKey(const std::filesystem::path& home) { +nlohmann::json makeInputs(std::string_view flavor = "base") { + nlohmann::json j; + j["epoch"] = 1; + j["toolchain"] = {{"compiler", "gcc"}, {"compiler_version", "16.1.0"}}; + j["profile"] = {{"opt_level", "2"}, {"debug", false}}; + j["package"] = {{"index", "mcpplibs"}, {"name", "mcpplibs.cmdline"}, + {"version", "0.0.1"}}; + j["config"] = {{"cxxflags", nlohmann::json::array({std::string(flavor)})}}; + return j; +} + +CacheKey makeKey(const std::filesystem::path& root, + std::string_view key = "deadbeef0123abcd", + std::string_view flavor = "base") { return CacheKey{ - .mcppHome = home, - .fingerprint = "deadbeef0123abcd", + .cacheRoot = root, .indexName = "mcpplibs", .packageName = "mcpplibs.cmdline", .version = "0.0.1", + .keyHex = std::string(key), + .inputs = makeInputs(flavor), }; } @@ -41,153 +56,233 @@ void writeFile(const std::filesystem::path& p, std::string_view body) { std::ofstream(p) << body; } +std::string readFile(const std::filesystem::path& p) { + std::ifstream is(p); + return std::string((std::istreambuf_iterator(is)), {}); +} + +nlohmann::json readJson(const std::filesystem::path& p) { + std::ifstream is(p); + nlohmann::json j; + is >> j; + return j; +} + +DepArtifacts oneOfEach() { + return DepArtifacts{ .bmiFiles = {"lib.gcm"}, .objFiles = {"lib.m.o"} }; +} + +void seedProject(const std::filesystem::path& project) { + writeFile(project / "gcm.cache" / "lib.gcm", "G"); + writeFile(project / "obj" / "lib.m.o", "O"); +} + } // namespace -TEST(BmiCache, KeyDirLayoutMatchesDocs26) { - auto k = makeKey("/home/u/.mcpp"); - auto expected = std::filesystem::path("/home/u/.mcpp/bmi/deadbeef0123abcd/deps/mcpplibs/mcpplibs.cmdline@0.0.1"); +TEST(BmiCache, EntryDirIsKeyedByPackageIdentityAndKeyNotByProject) { + auto k = makeKey("/home/u/.mcpp/build-cache/v1"); + auto expected = std::filesystem::path( + "/home/u/.mcpp/build-cache/v1/pkg/mcpplibs/mcpplibs.cmdline@0.0.1/deadbeef0123abcd"); EXPECT_EQ(k.dir(), expected); - EXPECT_EQ(k.manifestFile().filename().string(), "manifest.txt"); - EXPECT_EQ(k.bmiDir().filename().string(), "gcm.cache"); - EXPECT_EQ(k.objDir().filename().string(), "obj"); + EXPECT_EQ(k.entryFile().filename().string(), "entry.json"); + EXPECT_EQ(k.bmiDir().filename().string(), "bmi"); + EXPECT_EQ(k.objDir().filename().string(), "obj"); +} + +// The layout version must be a path segment: replacing the layout has to be a +// new tree, never a migration of (or a deletion of) the old one. +TEST(BmiCache, EntryDirCarriesTheLayoutVersion) { + auto k = makeKey("/home/u/.mcpp/build-cache/v1"); + auto s = k.dir().generic_string(); + EXPECT_NE(s.find("/build-cache/v1/pkg/"), std::string::npos) << s; + // And nothing may land under the pre-v1 root. + EXPECT_EQ(s.find("/.mcpp/bmi/"), std::string::npos) << s; } -TEST(BmiCache, IsCachedFalseWhenManifestMissing) { +TEST(BmiCache, IsCachedFalseWhenEntryMissing) { Tmp t; - auto k = makeKey(t.path); - EXPECT_FALSE(is_cached(k)); + EXPECT_FALSE(is_cached(makeKey(t.path))); } -TEST(BmiCache, PopulateThenStageRoundTrip) { +TEST(BmiCache, PopulateWritesSelfDescribingEntry) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); - - writeFile(project / "gcm.cache" / "mcpplibs.cmdline.gcm", "GCM-A"); - writeFile(project / "gcm.cache" / "mcpplibs.cmdline-options.gcm", "GCM-B"); - writeFile(project / "obj" / "cmdline.m.o", "OBJ-A"); - - DepArtifacts arts { - .bmiFiles = { "mcpplibs.cmdline.gcm", "mcpplibs.cmdline-options.gcm" }, - .objFiles = { "cmdline.m.o" }, - }; + seedProject(project); auto k = makeKey(home); - auto pop = populate_from(k, project, arts); + auto pop = populate_from(k, project, oneOfEach()); ASSERT_TRUE(pop) << pop.error(); - EXPECT_TRUE(std::filesystem::exists(k.manifestFile())); - EXPECT_TRUE(std::filesystem::exists(k.bmiDir() / "mcpplibs.cmdline.gcm")); - EXPECT_TRUE(std::filesystem::exists(k.bmiDir() / "mcpplibs.cmdline-options.gcm")); - EXPECT_TRUE(std::filesystem::exists(k.objDir() / "cmdline.m.o")); + ASSERT_TRUE(std::filesystem::exists(k.entryFile())); + EXPECT_TRUE(std::filesystem::exists(k.bmiDir() / "lib.gcm")); + EXPECT_TRUE(std::filesystem::exists(k.objDir() / "lib.m.o")); EXPECT_TRUE(is_cached(k)); - // Round-trip: stage into a fresh project dir. - auto project2 = t.path / "proj2" / "target"; - auto staged = stage_into(k, project2); - ASSERT_TRUE(staged) << staged.error(); - EXPECT_EQ(staged->bmiFiles.size(), 2u); - EXPECT_EQ(staged->objFiles.size(), 1u); - EXPECT_TRUE(std::filesystem::exists(project2 / "gcm.cache" / "mcpplibs.cmdline.gcm")); - EXPECT_TRUE(std::filesystem::exists(project2 / "obj" / "cmdline.m.o")); + // The entry has to carry the key AND the full inputs: a hit is validated + // field by field, so a cache whose entries only listed files could never be + // audited when a wrong hit was suspected. + auto j = readJson(k.entryFile()); + EXPECT_EQ(j.value("key", std::string{}), "deadbeef0123abcd"); + EXPECT_EQ(j["inputs"], makeInputs()); + EXPECT_EQ(j["bmi"].size(), 1u); + EXPECT_EQ(j["obj"].size(), 1u); + EXPECT_TRUE(j.contains("created")); + EXPECT_TRUE(j.contains("accessed")); +} - // Staged file content must match original. - std::ifstream is(project2 / "obj" / "cmdline.m.o"); - std::string body((std::istreambuf_iterator(is)), {}); - EXPECT_EQ(body, "OBJ-A"); +// The whole point of recording inputs: equal hashes are never trusted on their +// own. Same key directory, different inputs ⇒ miss. +TEST(BmiCache, IsCachedFalseWhenRecordedInputsDiffer) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto written = makeKey(home, "samekey00000000", "base"); + ASSERT_TRUE(populate_from(written, project, oneOfEach())); + EXPECT_TRUE(is_cached(written)); + + auto probed = makeKey(home, "samekey00000000", "different-flag"); + EXPECT_FALSE(is_cached(probed)); } -TEST(BmiCache, StageIntoDoesNotTouchIdenticalOutputs) { +// An entry written by an older mcpp may carry extra keys, but every field the +// current build cares about must be present. Missing ⇒ mismatch, never a pass. +TEST(BmiCache, IsCachedFalseWhenARequiredInputFieldIsAbsent) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); + seedProject(project); - writeFile(project / "gcm.cache" / "mcpplibs.cmdline.gcm", "GCM-A"); - writeFile(project / "obj" / "cmdline.m.o", "OBJ-A"); + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); - DepArtifacts arts { - .bmiFiles = { "mcpplibs.cmdline.gcm" }, - .objFiles = { "cmdline.m.o" }, - }; + auto j = readJson(k.entryFile()); + j["inputs"].erase("profile"); + std::ofstream(k.entryFile()) << j.dump(2); + + EXPECT_FALSE(is_cached(k)); +} + +TEST(BmiCache, IsCachedFalseWhenSchemaDiffers) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); auto k = makeKey(home); - ASSERT_TRUE(populate_from(k, project, arts)); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); - auto staged = stage_into(k, project); - ASSERT_TRUE(staged) << staged.error(); - auto gcmTime = std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"); - auto objTime = std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"); + auto j = readJson(k.entryFile()); + j["schema"] = kEntrySchema + 1; + std::ofstream(k.entryFile()) << j.dump(2); - auto stagedAgain = stage_into(k, project); - ASSERT_TRUE(stagedAgain) << stagedAgain.error(); - EXPECT_EQ(std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"), gcmTime); - EXPECT_EQ(std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"), objTime); + EXPECT_FALSE(is_cached(k)); } -TEST(BmiCache, StageIntoDoesNotOverwriteExistingOutputs) { +TEST(BmiCache, IsCachedFalseWhenSentinelExistsButFileMissing) { Tmp t; - auto home = t.path / "home"; - auto cacheProject = t.path / "cache-proj" / "target"; - auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(cacheProject / "gcm.cache"); - std::filesystem::create_directories(cacheProject / "obj"); - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); - writeFile(cacheProject / "gcm.cache" / "mcpplibs.cmdline.gcm", "CACHE-GCM"); - writeFile(cacheProject / "obj" / "cmdline.m.o", "CACHE-OBJ"); + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + ASSERT_TRUE(is_cached(k)); - DepArtifacts arts { - .bmiFiles = { "mcpplibs.cmdline.gcm" }, - .objFiles = { "cmdline.m.o" }, - }; + std::filesystem::remove(k.objDir() / "lib.m.o"); + EXPECT_FALSE(is_cached(k)); +} + +TEST(BmiCache, ResolveCachedReportsTheArtifactsWithoutCopying) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); auto k = makeKey(home); - ASSERT_TRUE(populate_from(k, cacheProject, arts)); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); - writeFile(project / "gcm.cache" / "mcpplibs.cmdline.gcm", "PROJECT-GCM"); - writeFile(project / "obj" / "cmdline.m.o", "PROJECT-OBJ"); - auto gcmTime = std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"); - auto objTime = std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"); + auto arts = resolve_cached(k); + ASSERT_TRUE(arts) << arts.error(); + EXPECT_EQ(arts->bmiFiles, std::vector{"lib.gcm"}); + EXPECT_EQ(arts->objFiles, std::vector{"lib.m.o"}); - auto staged = stage_into(k, project); - ASSERT_TRUE(staged) << staged.error(); + // resolve_cached must not write into a project dir. Staging is a ninja edge + // now: copying artifacts in from outside the graph is exactly what made the + // old cache a no-op — ninja rebuilds any output it has no command-line + // record for, so the copies were recompiled over on every fresh build dir. + auto project2 = t.path / "proj2" / "target"; + (void)resolve_cached(k); + EXPECT_FALSE(std::filesystem::exists(project2)); +} - { - std::ifstream is(project / "gcm.cache" / "mcpplibs.cmdline.gcm"); - std::string body((std::istreambuf_iterator(is)), {}); - EXPECT_EQ(body, "PROJECT-GCM"); - } - { - std::ifstream is(project / "obj" / "cmdline.m.o"); - std::string body((std::istreambuf_iterator(is)), {}); - EXPECT_EQ(body, "PROJECT-OBJ"); - } - EXPECT_EQ(std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"), gcmTime); - EXPECT_EQ(std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"), objTime); +TEST(BmiCache, CachedPathsPointIntoTheEntry) { + auto k = makeKey("/home/u/.mcpp/build-cache/v1"); + EXPECT_EQ(cached_bmi_path(k, "lib.gcm"), k.bmiDir() / "lib.gcm"); + EXPECT_EQ(cached_obj_path(k, "sub/lib.m.o"), k.objDir() / "sub" / "lib.m.o"); } -TEST(BmiCache, IsCachedFalseWhenSentinelExistsButFileMissing) { +// Nested object paths (mcpp#233's per-package object prefixes) must round-trip. +TEST(BmiCache, PopulateHandlesNestedObjectPaths) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); - writeFile(project / "gcm.cache" / "lib.gcm", "G"); - writeFile(project / "obj" / "lib.m.o", "O"); + writeFile(project / "obj" / "pkg_zlib" / "zlib-1.3" / "compress.o", "NESTED"); - DepArtifacts arts { .bmiFiles = {"lib.gcm"}, .objFiles = {"lib.m.o"} }; auto k = makeKey(home); + DepArtifacts arts { .objFiles = {"pkg_zlib/zlib-1.3/compress.o"} }; ASSERT_TRUE(populate_from(k, project, arts)); - ASSERT_TRUE(is_cached(k)); + EXPECT_EQ(readFile(k.objDir() / "pkg_zlib" / "zlib-1.3" / "compress.o"), "NESTED"); + EXPECT_TRUE(is_cached(k)); +} - // Delete one cached file → is_cached must return false even though manifest still exists. - std::filesystem::remove(k.objDir() / "lib.m.o"); - EXPECT_FALSE(is_cached(k)); +// touch_accessed is what makes `cache gc` an LRU rather than "drop what was +// populated long ago". It must move the stamp and leave the artifacts alone — +// their mtimes participate in ninja's restat handling. +TEST(BmiCache, TouchAccessedMovesTheStampAndNotTheArtifacts) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + + auto j0 = readJson(k.entryFile()); + auto created0 = j0.value("created", std::string{}); + auto objTime0 = std::filesystem::last_write_time(k.objDir() / "lib.m.o"); + auto bmiTime0 = std::filesystem::last_write_time(k.bmiDir() / "lib.gcm"); + + // Rewrite the stamp to something clearly old, then touch. + j0["accessed"] = "1000"; + std::ofstream(k.entryFile()) << j0.dump(2); + touch_accessed(k); + + auto j1 = readJson(k.entryFile()); + EXPECT_NE(j1.value("accessed", std::string{}), "1000"); + EXPECT_EQ(j1.value("created", std::string{}), created0) + << "touch must not reset the creation stamp"; + EXPECT_EQ(std::filesystem::last_write_time(k.objDir() / "lib.m.o"), objTime0); + EXPECT_EQ(std::filesystem::last_write_time(k.bmiDir() / "lib.gcm"), bmiTime0); + EXPECT_TRUE(is_cached(k)) << "touching must not invalidate the entry"; +} + +TEST(BmiCache, RepopulatePreservesCreatedStamp) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + auto j0 = readJson(k.entryFile()); + j0["created"] = "12345"; + std::ofstream(k.entryFile()) << j0.dump(2); + + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + EXPECT_EQ(readJson(k.entryFile()).value("created", std::string{}), "12345"); } TEST(BmiCache, PopulateFailsIfBuildOutputMissing) { @@ -202,39 +297,49 @@ TEST(BmiCache, PopulateFailsIfBuildOutputMissing) { EXPECT_NE(pop.error().find("expected build output missing"), std::string::npos); } +// Two keys for the same package@version are independent entries — that is what +// lets one machine hold a dev-profile and a release-profile build of the same +// dependency at once. +TEST(BmiCache, DifferentKeysAreIndependentEntries) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto a = makeKey(home, "aaaaaaaaaaaaaaaa", "opt"); + auto b = makeKey(home, "bbbbbbbbbbbbbbbb", "debug"); + ASSERT_TRUE(populate_from(a, project, oneOfEach())); + ASSERT_TRUE(populate_from(b, project, oneOfEach())); + EXPECT_NE(a.dir(), b.dir()); + EXPECT_TRUE(is_cached(a)); + EXPECT_TRUE(is_cached(b)); +} + #if !defined(_WIN32) -// M4 #9: when an external holder takes the .lock, populate_from must skip -// (returns success but does NOT clobber the directory). -// Uses flock() which is POSIX-only. +// When an external holder takes the .lock, populate_from must skip (returns +// success but does NOT clobber the entry). Uses flock(), POSIX-only. TEST(BmiCache, PopulateSkipsWhenLockHeld) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); - writeFile(project / "gcm.cache" / "lib.gcm", "G2"); - writeFile(project / "obj" / "lib.m.o", "O2"); + seedProject(project); auto k = makeKey(home); - // Take the lock manually before populate runs. std::filesystem::create_directories(k.dir()); auto lockPath = k.dir() / ".lock"; int fd = ::open(lockPath.c_str(), O_CREAT | O_RDWR, 0644); ASSERT_GE(fd, 0); ASSERT_EQ(::flock(fd, LOCK_EX | LOCK_NB), 0); - DepArtifacts arts { .bmiFiles = {"lib.gcm"}, .objFiles = {"lib.m.o"} }; - auto pop = populate_from(k, project, arts); + auto pop = populate_from(k, project, oneOfEach()); EXPECT_TRUE(pop) << "should silently skip when lock is held"; - // manifest.txt must NOT have been written by the second writer. - EXPECT_FALSE(std::filesystem::exists(k.manifestFile())); + EXPECT_FALSE(std::filesystem::exists(k.entryFile())); ::flock(fd, LOCK_UN); ::close(fd); - // After lock released, a fresh populate should succeed. - auto pop2 = populate_from(k, project, arts); + auto pop2 = populate_from(k, project, oneOfEach()); ASSERT_TRUE(pop2) << pop2.error(); - EXPECT_TRUE(std::filesystem::exists(k.manifestFile())); + EXPECT_TRUE(std::filesystem::exists(k.entryFile())); } #endif // !defined(_WIN32) diff --git a/tests/unit/test_build_profile.cpp b/tests/unit/test_build_profile.cpp new file mode 100644 index 00000000..5939b07b --- /dev/null +++ b/tests/unit/test_build_profile.cpp @@ -0,0 +1,134 @@ +// The profile axis: profile selection, and the invariant that the resolved +// profile knobs participate in the fingerprint. +// +// Both halves were broken together. `[profile.]` lands its knobs in +// buildConfig.optLevel/debug/lto/strip and flags.cppm turns them into +// -O/-g/-flto, but canonical_compile_flags serialized only cflags/cxxflags/ +// ldflags — so `--dev`, `--release` and `--profile dist` produced ONE +// fingerprint, hence one target/// directory and one global cache +// entry. Meanwhile `.build_cache` keyed its fast-path entries by target triple +// alone and the fast path only refused to run for an EXPLICIT profile flag, so +// `mcpp build --release` followed by a bare `mcpp build` reported success in +// 0.00s and left the release artifacts in place. + +#include + +import std; +import mcpp.build.prepare; +import mcpp.manifest; + +namespace { + +mcpp::manifest::Manifest base() { + mcpp::manifest::Manifest m; + m.package.name = "demo"; + m.package.version = "0.1.0"; + m.package.standard = "c++23"; + m.buildConfig.optLevel = "2"; + return m; +} + +} // namespace + +// ── resolve_profile_name: the ONE rule, shared with the fast paths ─────────── + +TEST(BuildProfile, OverrideBeatsManifestDefault) { + auto m = base(); + m.buildConfig.defaultProfile = "release"; + EXPECT_EQ(mcpp::build::resolve_profile_name(m, "dev"), "dev"); +} + +TEST(BuildProfile, ManifestDefaultBeatsGlobalDefault) { + auto m = base(); + m.buildConfig.defaultProfile = "release"; + EXPECT_EQ(mcpp::build::resolve_profile_name(m, ""), "release"); +} + +TEST(BuildProfile, GlobalDefaultIsDev) { + EXPECT_EQ(mcpp::build::resolve_profile_name(base(), ""), "dev"); +} + +TEST(BuildProfile, CustomProfileNamePassesThrough) { + EXPECT_EQ(mcpp::build::resolve_profile_name(base(), "contracts"), "contracts"); +} + +// ── the fingerprint invariant ──────────────────────────────────────────────── + +TEST(BuildProfile, OptLevelIsInTheCanonicalFlags) { + auto a = base(); + auto b = base(); b.buildConfig.optLevel = "0"; + EXPECT_NE(mcpp::build::canonical_compile_flags(a), + mcpp::build::canonical_compile_flags(b)); +} + +TEST(BuildProfile, DebugLtoStripAreInTheCanonicalFlags) { + auto ref = mcpp::build::canonical_compile_flags(base()); + { auto m = base(); m.buildConfig.debug = true; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.lto = true; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.strip = true; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } +} + +// The exact shape the three built-in profiles resolve to (prepare_build's +// profile block). Asserting on the flag string rather than on a fingerprint hex +// keeps the failure legible when someone adds a knob and forgets to serialize it. +TEST(BuildProfile, BuiltInProfilesProduceDistinctCanonicalFlags) { + auto dev = base(); + dev.buildConfig.optLevel = "0"; + dev.buildConfig.debug = true; + + auto release = base(); + release.buildConfig.optLevel = "2"; + + auto dist = base(); + dist.buildConfig.optLevel = "3"; + dist.buildConfig.strip = true; + + auto fdev = mcpp::build::canonical_compile_flags(dev); + auto frelease = mcpp::build::canonical_compile_flags(release); + auto fdist = mcpp::build::canonical_compile_flags(dist); + + EXPECT_NE(fdev, frelease); + EXPECT_NE(frelease, fdist); + EXPECT_NE(fdev, fdist); +} + +TEST(BuildProfile, CanonicalFlagsStillCoverTheNonProfileKnobs) { + auto ref = mcpp::build::canonical_compile_flags(base()); + { auto m = base(); m.package.standard = "c++26"; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.cxxflags = {"-DFOO"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.cflags = {"-DBAR"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.ldflags = {"-lm"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.dialectCxxflags = {"-freflection"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.cStandard = "c17"; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } +} + +// ── cache-mode parsing ────────────────────────────────────────────────────── + +TEST(BuildProfile, CacheModeParsesTheThreeModes) { + using mcpp::build::CacheMode; + EXPECT_EQ(mcpp::build::parse_cache_mode("global"), CacheMode::Global); + EXPECT_EQ(mcpp::build::parse_cache_mode("local"), CacheMode::Local); + EXPECT_EQ(mcpp::build::parse_cache_mode("off"), CacheMode::Off); + // `none` accepted as a synonym for off; anything else is refused rather + // than silently meaning "global", which would be the worst default for a + // typo to land on. + EXPECT_EQ(mcpp::build::parse_cache_mode("none"), CacheMode::Off); + EXPECT_FALSE(mcpp::build::parse_cache_mode("on").has_value()); + EXPECT_FALSE(mcpp::build::parse_cache_mode("").has_value()); + EXPECT_FALSE(mcpp::build::parse_cache_mode("GLOBAL").has_value()); +} + +TEST(BuildProfile, CacheModeNameRoundTrips) { + using mcpp::build::CacheMode; + for (auto m : {CacheMode::Global, CacheMode::Local, CacheMode::Off}) + EXPECT_EQ(mcpp::build::parse_cache_mode(mcpp::build::cache_mode_name(m)), m); +} diff --git a/tests/unit/test_cache_key.cpp b/tests/unit/test_cache_key.cpp new file mode 100644 index 00000000..529732a1 --- /dev/null +++ b/tests/unit/test_cache_key.cpp @@ -0,0 +1,265 @@ +#include + +import std; +import mcpp.build.cache_key; +import mcpp.libs.json; +import mcpp.manifest; +import mcpp.modgraph.scanner; +import mcpp.toolchain.detect; + +namespace ck = mcpp::build::cache_key; + +namespace { + +ck::BuildAxes axes() { + ck::BuildAxes b; + b.compilerId = "gcc"; + b.compilerVersion = "16.1.0"; + b.driverIdentity = "0123456789abcdef"; + b.targetTriple = "x86_64-linux-gnu"; + b.stdlibId = "libstdc++"; + b.stdlibVersion = "16.1.0"; + b.cppStandard = "c++23"; + b.cppStandardFlag = "-std=c++23"; + b.cStandard = "c11"; + b.optLevel = "2"; + b.debug = false; + return b; +} + +ck::PackageAxes pkg() { + ck::PackageAxes p; + p.indexName = "compat"; + p.packageName = "compat.zlib"; + p.version = "1.3.2"; + p.cflags = {"-D_GNU_SOURCE"}; + p.sources = {"zlib-1.3.2/adler32.c", "zlib-1.3.2/deflate.c"}; + return p; +} + +mcpp::modgraph::PackageRoot rootAt(const std::filesystem::path& at) { + mcpp::modgraph::PackageRoot r; + r.root = at; + r.manifest.package.name = "zlib"; + r.manifest.package.namespace_ = "compat"; + r.manifest.package.version = "1.3.2"; + return r; +} + +} // namespace + +// The defect this key exists to fix: the consumer's identity used to be folded +// into every dependency's cache path (the whole-project fingerprint serializes +// every package in the graph, including the root's name and version), so +// `mcpp version bump` invalidated the whole cache and two projects never shared +// an entry. Axes A–F contain nothing about the consumer, so there is no input +// this test could vary to demonstrate the old behaviour — the closest positive +// statement is that a dependency's key is a pure function of its own axes. +TEST(CacheKey, IsStableAcrossCallsForTheSameInputs) { + EXPECT_EQ(ck::key_hex(axes(), pkg()), ck::key_hex(axes(), pkg())); +} + +TEST(CacheKey, KeyIsSixteenHexChars) { + auto k = ck::key_hex(axes(), pkg()); + EXPECT_EQ(k.size(), 16u); + for (char c : k) + EXPECT_TRUE(std::isxdigit(static_cast(c))) << k; +} + +// ── C axis: the profile. This is the one that was missing entirely, and the +// reason a --release build could be served -O0 -g dependency objects. ──────── +TEST(CacheKey, ProfileOptLevelChangesTheKey) { + auto a = axes(); + auto b = axes(); b.optLevel = "0"; + EXPECT_NE(ck::key_hex(a, pkg()), ck::key_hex(b, pkg())); +} + +TEST(CacheKey, ProfileDebugLtoStripEachChangeTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto b = axes(); b.debug = true; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.lto = true; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.strip = true; EXPECT_NE(ck::key_hex(b, pkg()), base); } +} + +// ── A axis: toolchain identity ─────────────────────────────────────────────── +TEST(CacheKey, ToolchainIdentityChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto b = axes(); b.compilerId = "clang"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.compilerVersion = "15.1.0"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.targetTriple = "x86_64-linux-musl"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.stdlibId = "libc++"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + // Two payloads that both call themselves "gcc 16.1.0" but are not the same + // build must not share an entry. + { auto b = axes(); b.driverIdentity = "ffffffffffffffff"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } +} + +// ── B axis: language and dialect ───────────────────────────────────────────── +TEST(CacheKey, LanguageAndDialectChangeTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto b = axes(); b.cppStandard = "c++26"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.cppStandardFlag = "-std=c++2c"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.dialectFlags = {"-freflection"}; + EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.cStandard = "c17"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.macosDeploymentTarget = "14.0"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } +} + +// ── D axis: package identity ───────────────────────────────────────────────── +TEST(CacheKey, PackageIdentityChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto p = pkg(); p.version = "1.3.1"; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.packageName = "compat.zstd"; + EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.indexName = "other"; EXPECT_NE(ck::key_hex(axes(), p), base); } +} + +// ── E axis: the package's own config ───────────────────────────────────────── +TEST(CacheKey, OwnBuildConfigChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto p = pkg(); p.cflags.push_back("-DEXTRA"); EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.cxxflags = {"-fno-rtti"}; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.defines = {"FOO=1"}; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.globFlags = {"glob:*.c"}; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.generatedFiles = {"cfg.h=1"};EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.includeDirs = {"pub:/x"}; + EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.sources.push_back("zlib-1.3.2/gzread.c"); + EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.features = {"main"}; EXPECT_NE(ck::key_hex(axes(), p), base); } +} + +// ── F axis: upstream keys (Merkle) ─────────────────────────────────────────── +// A dependency's artifacts are bound to the exact upstream artifacts they were +// built against — GCC embeds a CRC of an imported module's BMI into the +// importer's BMI, so a stale importer BMI fails with `module 'X' CRC mismatch`. +// The key therefore folds in the upstream's key, not a summary of its interface. +TEST(CacheKey, UpstreamKeyChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + auto p = pkg(); + p.upstreamKeys = {"1111111111111111"}; + auto withUp = ck::key_hex(axes(), p); + EXPECT_NE(withUp, base); + + p.upstreamKeys = {"2222222222222222"}; + EXPECT_NE(ck::key_hex(axes(), p), withUp); +} + +TEST(CacheKey, UpstreamOrderDoesNotMatterWhenSorted) { + auto p1 = pkg(); p1.upstreamKeys = {"1111111111111111", "2222222222222222"}; + auto p2 = pkg(); p2.upstreamKeys = {"1111111111111111", "2222222222222222"}; + EXPECT_EQ(ck::key_hex(axes(), p1), ck::key_hex(axes(), p2)); +} + +// Length-prefixed field joining: without it, ("a","bc") and ("ab","c") collide, +// and for a cache key a collision means serving one package's objects for +// another's. +TEST(CacheKey, AdjacentFieldsCannotCollide) { + auto p1 = pkg(); p1.packageName = "a"; p1.version = "bc"; + auto p2 = pkg(); p2.packageName = "ab"; p2.version = "c"; + EXPECT_NE(ck::key_hex(axes(), p1), ck::key_hex(axes(), p2)); +} + +TEST(CacheKey, ListBoundariesCannotCollide) { + auto p1 = pkg(); p1.cflags = {"-DA", "-DB"}; + auto p2 = pkg(); p2.cflags = {"-DA-DB"}; + EXPECT_NE(ck::key_hex(axes(), p1), ck::key_hex(axes(), p2)); +} + +// entry.json carries these inputs verbatim so a hit can be validated field by +// field. Anything in the key must therefore also be in the JSON, or validation +// would pass on an input the key distinguishes. +TEST(CacheKey, JsonCarriesEveryAxis) { + auto j = ck::to_json(axes(), pkg()); + EXPECT_EQ(j["epoch"], ck::kCacheEpoch); + for (auto k : {"toolchain", "language", "profile", "package", "config", "upstream"}) + EXPECT_TRUE(j.contains(k)) << k; + EXPECT_EQ(j["toolchain"]["compiler"], "gcc"); + EXPECT_EQ(j["profile"]["opt_level"], "2"); + EXPECT_EQ(j["package"]["version"], "1.3.2"); + for (auto k : {"features", "cflags", "cxxflags", "ldflags", "defines", + "glob_flags", "generated_files", "include_dirs", + "source_globs", "sources"}) + EXPECT_TRUE(j["config"].contains(k)) << k; +} + +TEST(CacheKey, JsonDiffersWheneverTheKeyDiffers) { + auto b = axes(); b.optLevel = "0"; + EXPECT_NE(ck::to_json(axes(), pkg()), ck::to_json(b, pkg())); +} + +// Payload include dirs are absolute paths under /registry/data/xpkgs. +// Left absolute, every entry would miss after a home relocation or on another +// machine — which is most of the point of a cross-project cache. +TEST(CacheKey, IncludeDirsAreRelativizedAgainstTheStoreRoot) { + auto makeKeyFor = [](const std::filesystem::path& store) { + auto pkgRoot = rootAt(store / "compat-x-compat.zlib" / "1.3.2"); + pkgRoot.manifest.buildConfig.includeDirs = { + store / "compat-x-compat.zlib" / "1.3.2" / "include", + }; + ck::PackageAxes p; + p.indexName = "compat"; p.packageName = "compat.zlib"; p.version = "1.3.2"; + ck::fill_package_config(p, pkgRoot, store); + return std::pair{ck::key_hex(axes(), p), p.includeDirs}; + }; + auto [k1, dirs1] = makeKeyFor("/home/alice/.mcpp/registry/data/xpkgs"); + auto [k2, dirs2] = makeKeyFor("/opt/ci/mcpp/registry/data/xpkgs"); + EXPECT_EQ(k1, k2); + ASSERT_FALSE(dirs1.empty()); + EXPECT_NE(dirs1.front().find(""), std::string::npos) << dirs1.front(); + EXPECT_EQ(dirs1, dirs2); +} + +// A path that is under the package root but not under the store (a generated +// directory, say) gets a package-relative placeholder for the same reason. +TEST(CacheKey, PackageRootRelativeDirsAreAlsoRelativized) { + std::filesystem::path store = "/home/u/.mcpp/registry/data/xpkgs"; + auto pkgRoot = rootAt("/somewhere/else/zlib"); + pkgRoot.manifest.buildConfig.includeDirs = {"/somewhere/else/zlib/generated"}; + ck::PackageAxes p; + ck::fill_package_config(p, pkgRoot, store); + ASSERT_FALSE(p.includeDirs.empty()); + EXPECT_NE(p.includeDirs.front().find(""), std::string::npos) + << p.includeDirs.front(); +} + +TEST(CacheKey, FillPackageConfigCarriesFlagsAndGeneratedFiles) { + std::filesystem::path store = "/home/u/.mcpp/registry/data/xpkgs"; + auto pkgRoot = rootAt(store / "compat-x-compat.zlib" / "1.3.2"); + pkgRoot.manifest.buildConfig.cflags = {"-D_GNU_SOURCE"}; + pkgRoot.manifest.buildConfig.cxxflags = {"-fno-exceptions"}; + pkgRoot.manifest.buildConfig.defines = {"ZLIB_CONST"}; + pkgRoot.manifest.buildConfig.cStandard = "c11"; + pkgRoot.manifest.buildConfig.generatedFiles = {{"cfg.h", "#define A 1"}}; + + ck::PackageAxes p; + ck::fill_package_config(p, pkgRoot, store); + EXPECT_NE(std::ranges::find(p.cflags, "-D_GNU_SOURCE"), p.cflags.end()); + EXPECT_NE(std::ranges::find(p.cxxflags, "-fno-exceptions"), p.cxxflags.end()); + EXPECT_NE(std::ranges::find(p.defines, "ZLIB_CONST"), p.defines.end()); + ASSERT_EQ(p.generatedFiles.size(), 1u); + EXPECT_EQ(p.generatedFiles.front(), "cfg.h=#define A 1"); + // A package's own C standard reaches its own C units, so it must be in the + // key even though the whole-graph C standard is on the B axis. + bool sawCStd = false; + for (auto& f : p.cflags) if (f.find("c_standard=c11") != std::string::npos) sawCStd = true; + EXPECT_TRUE(sawCStd); +} + +// Generated files are a map; iteration order must not leak into the key. +TEST(CacheKey, GeneratedFilesAreOrderIndependent) { + std::filesystem::path store = "/home/u/.mcpp/registry/data/xpkgs"; + auto build = [&](std::vector> entries) { + auto pkgRoot = rootAt(store / "p" / "1"); + for (auto& [k, v] : entries) + pkgRoot.manifest.buildConfig.generatedFiles[k] = v; + ck::PackageAxes p; + ck::fill_package_config(p, pkgRoot, store); + return ck::key_hex(axes(), p); + }; + EXPECT_EQ(build({{"a.h", "1"}, {"b.h", "2"}}), + build({{"b.h", "2"}, {"a.h", "1"}})); +} diff --git a/tests/unit/test_home.cpp b/tests/unit/test_home.cpp index 5bab58d5..7a2cd029 100644 --- a/tests/unit/test_home.cpp +++ b/tests/unit/test_home.cpp @@ -43,13 +43,30 @@ class ScopedEnv { TEST(Home, ExplicitMcppHomeWins) { ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test"); EXPECT_EQ(mcpp::home::root(), std::filesystem::path("/tmp/mcpp-home-test")); - EXPECT_EQ(mcpp::home::bmi_root(), - std::filesystem::path("/tmp/mcpp-home-test") / "bmi"); + EXPECT_EQ(mcpp::home::cache_root(), + std::filesystem::path("/tmp/mcpp-home-test") / "build-cache" / "v1"); } -TEST(Home, BmiRootIsAlwaysRootSlashBmi) { +TEST(Home, CacheRootIsAlwaysRootSlashBuildCacheV1) { ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test-2"); - EXPECT_EQ(mcpp::home::bmi_root(), mcpp::home::root() / "bmi"); + EXPECT_EQ(mcpp::home::cache_root(), mcpp::home::root() / "build-cache" / "v1"); +} + +// The layout version has to be a path segment, not an implicit convention: +// replacing the layout must never require migrating or deleting the old tree. +TEST(Home, CacheRootCarriesTheLayoutVersion) { + ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test-2b"); + EXPECT_EQ(mcpp::home::cache_root().filename(), std::filesystem::path("v1")); + EXPECT_EQ(mcpp::home::cache_root().parent_path().filename(), + std::filesystem::path("build-cache")); + // Must NOT nest inside /cache: GlobalConfig's index metadata cache + // owns that name and its reset path removes the whole directory. + EXPECT_NE(mcpp::home::cache_root().parent_path().filename(), + std::filesystem::path("cache")); + // The pre-v1 root must stay resolvable and must be a DIFFERENT tree, so + // `doctor` can report it and `cache clean --legacy` can reclaim it. + EXPECT_EQ(mcpp::home::legacy_bmi_root(), mcpp::home::root() / "bmi"); + EXPECT_NE(mcpp::home::legacy_bmi_root(), mcpp::home::cache_root()); } // The user-home fallback must land on `.mcpp`, never on the legacy flat @@ -63,21 +80,21 @@ TEST(Home, FallbackUsesDotMcppNotLegacyFlatDir) { #endif auto root = mcpp::home::root(); EXPECT_EQ(root.filename(), std::filesystem::path(".mcpp")); - EXPECT_EQ(mcpp::home::bmi_root().filename(), std::filesystem::path("bmi")); - EXPECT_NE(mcpp::home::bmi_root().filename(), std::filesystem::path(".mcpp-bmi")); + EXPECT_NE(mcpp::home::cache_root().filename(), + std::filesystem::path(".mcpp-bmi")); } -// #311's D2 regression gate: the std module cache and the dep BMI cache must +// #311's D2 regression gate: the std module cache and the dep cache must // resolve to ONE root. `default_cache_root()` used to be a private copy of the // home resolution that drifted (no USERPROFILE, no self-contained detection), // so on Windows the std BMI landed in the current working directory. -TEST(Home, StdModuleCacheRootEqualsHomeBmiRoot) { +TEST(Home, StdModuleCacheRootEqualsHomeCacheRoot) { { ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test-3"); - EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::bmi_root()); + EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::cache_root()); } { ScopedEnv mcppHome("MCPP_HOME", nullptr); - EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::bmi_root()); + EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::cache_root()); } } diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 2649cac8..8baf4115 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -926,3 +926,117 @@ TEST(NinjaBackend, FilterKeepsStagingDiagnosticsAndFailedTarget) { EXPECT_EQ(filtered.find("stage --output"), std::string::npos) << filtered; EXPECT_EQ(filtered.find("ninja: Entering directory"), std::string::npos) << filtered; } + +// ── Cache-served units: stage edges, not compile edges ────────────────────── +// +// The global dependency cache used to copy artifacts into the build dir from +// inside prepare_build, while those same paths stayed declared as the outputs of +// compile edges. Ninja treats an output it has no command line for in +// .ninja_log as dirty ("command line not found in log"), so on a fresh build dir +// every "cached" unit was recompiled — the cache saved nothing and the CLI still +// printed "Cached". Staging has to be an edge for ninja to have a record. + +TEST(NinjaBackend, CachedUnitEmitsStageEdgeAndNoCompileEdge) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.cppm", + .object = "obj/dep.m.o", + .packageName = "dep", + .providesModule = "dep", + .servedFromCache = true, + .cachedObject = "/bc/v1/pkg/idx/dep@1.0.0/key/obj/dep.m.o", + .cachedBmi = "/bc/v1/pkg/idx/dep@1.0.0/key/bmi/dep.gcm", + }); + + auto ninja = emit_ninja_string(plan); + + EXPECT_NE(ninja.find("build obj/dep.m.o : stage_file " + "/bc/v1/pkg/idx/dep@1.0.0/key/obj/dep.m.o"), + std::string::npos) << ninja; + // The BMI is staged too, at the path a compile edge would have produced. + EXPECT_NE(ninja.find("stage_file /bc/v1/pkg/idx/dep@1.0.0/key/bmi/dep.gcm"), + std::string::npos) << ninja; + // No compile edge, and no scan edge, for this unit. + EXPECT_EQ(ninja.find("cxx_module /store/dep/src/dep.cppm"), std::string::npos) + << ninja; + EXPECT_EQ(ninja.find("cxx_object /store/dep/src/dep.cppm"), std::string::npos) + << ninja; + EXPECT_EQ(ninja.find("dep.cppm.ddi"), std::string::npos) << ninja; +} + +// `--verify size` for the same reason the std artifacts use it: the entry +// directory is named by a key covering the toolchain, dialect, profile, the +// package's own config and its dependencies' keys, so under one key equal size +// IS equivalence — and size comes from directory metadata, which stays readable +// even when a holder denies opening the file. +TEST(NinjaBackend, CachedStageEdgesUseSizeVerification) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/a.c", + .object = "obj/a.o", + .packageName = "dep", + .servedFromCache = true, + .cachedObject = "/cache/obj/a.o", + }); + + auto ninja = emit_ninja_string(plan); + auto at = ninja.find("build obj/a.o : stage_file /cache/obj/a.o"); + ASSERT_NE(at, std::string::npos) << ninja; + EXPECT_NE(ninja.find("verify = --verify size", at), std::string::npos) << ninja; +} + +// A cached unit sits next to uncached ones; only the cached one changes shape. +TEST(NinjaBackend, UncachedUnitsStillCompileAlongsideCachedOnes) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.c", + .object = "obj/dep.o", + .packageName = "dep", + .servedFromCache = true, + .cachedObject = "/cache/obj/dep.o", + }); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + + auto ninja = emit_ninja_string(plan); + EXPECT_NE(ninja.find("build obj/dep.o : stage_file /cache/obj/dep.o"), + std::string::npos) << ninja; + EXPECT_NE(ninja.find("build obj/main.o : cxx_object src/main.cpp"), + std::string::npos) << ninja; +} + +// The default path must be byte-identical to what it was: `servedFromCache` is +// false everywhere unless a cache hit set it. +TEST(NinjaBackend, NoStageEdgesWithoutCachedUnits) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + + auto ninja = emit_ninja_string(plan); + EXPECT_EQ(ninja.find("build obj/main.o : stage_file"), std::string::npos) << ninja; + EXPECT_NE(ninja.find("build obj/main.o : cxx_object src/main.cpp"), + std::string::npos) << ninja; +} + +// A cached unit whose cachedObject was never filled in is a bug upstream, not a +// reason to emit a stage edge with an empty input — it must fall back to being +// compiled rather than producing an unbuildable graph. +TEST(NinjaBackend, CachedUnitWithoutCachedObjectPathIsStillCompiled) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + .servedFromCache = true, + }); + + auto ninja = emit_ninja_string(plan); + EXPECT_EQ(ninja.find("stage_file \n"), std::string::npos) << ninja; + EXPECT_EQ(ninja.find("build obj/main.o : stage_file"), std::string::npos) << ninja; +} From bbd2eeb314a32eeb6d9a58bc960f723f8e0c9819 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:00:45 +0800 Subject: [PATCH 04/11] fix(build): refuse to cache an index package whose upstream is local MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A key covers an upstream package by folding in that package's KEY, and a local package's key covers its file list but not its file contents — nothing could, without hashing a tree that may change between the hash and the compile. So a cached downstream entry would keep looking valid after a local upstream's source was edited. No index descriptor can declare a path dependency today, which makes the shape unreachable in practice. Enforced structurally anyway: "unreachable today" is exactly how the transitive path-dep leak got in. Also asserts the invariant that cache-served units keep their compile_commands.json entries — they stay in the plan on purpose so clangd does not lose the dependency's sources. --- src/build/prepare.cppm | 20 ++++++++++++++++++++ tests/unit/test_ninja_backend.cpp | 26 ++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 83f4a757..8e0abc9d 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -3902,6 +3902,19 @@ prepare_build(bool print_fingerprint, nlohmann::json::object()); std::vector keyState(packages.size(), 0); // 0 new/1 busy/2 done std::string keyCycleError; + // Does this package's own transitive upstream contain anything that is + // not an immutable index payload? If so it cannot be cached either, even + // when the package itself is an index package. + // + // A key covers an upstream package by folding in that package's KEY, and + // a local package's key covers its file list but not its file CONTENTS — + // nothing could, without hashing a tree that may change between the hash + // and the compile. So editing a local upstream's source would leave a + // downstream entry looking valid. No index descriptor can declare a path + // dependency today, which makes this shape unreachable in practice; it is + // enforced structurally anyway, because "unreachable today" is how the + // transitive path-dep leak got in. + std::vector localTaint(packages.size(), 0); auto compute_key = [&](auto&& self, std::size_t idx) -> const std::string& { static const std::string kEmpty; if (keyState[idx] == 2) return pkgKeys[idx]; @@ -3933,10 +3946,15 @@ prepare_build(bool print_fingerprint, if (pa.version.empty()) pa.version = packages[idx].manifest.package.version; ck::fill_package_config(pa, packages[idx], storeRoot); pa.sources = pkgSources[idx]; + const bool selfIsIndex = idx > 0 + && idx - 1 < dep_cache_identities.size() + && dep_cache_identities[idx - 1].sourceKind == "version"; + if (!selfIsIndex) localTaint[idx] = 1; for (auto& e : dependencyEdges) { if (e.consumerPackageIndex != idx) continue; auto& up = self(self, e.dependencyPackageIndex); if (!up.empty()) pa.upstreamKeys.push_back(up); + if (localTaint[e.dependencyPackageIndex]) localTaint[idx] = 1; for (auto& f : e.requestedFeatures) pa.features.push_back(f); } std::ranges::sort(pa.upstreamKeys); @@ -3979,6 +3997,8 @@ prepare_build(bool print_fingerprint, // xpkgs store stays out, because the failure mode here is a // silently wrong object rather than a rejected command. if (!depIdent || depIdent->sourceKind != "version") continue; + // ...and neither may anything it was built against be local. + if (localTaint[i]) continue; const auto& depName = depIdent->packageName; const auto& depVer = depIdent->version.empty() diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 8baf4115..4d70bf37 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -1040,3 +1040,29 @@ TEST(NinjaBackend, CachedUnitWithoutCachedObjectPathIsStillCompiled) { EXPECT_EQ(ninja.find("stage_file \n"), std::string::npos) << ninja; EXPECT_EQ(ninja.find("build obj/main.o : stage_file"), std::string::npos) << ninja; } + +// A cache-served unit keeps its compile_commands.json entry. The units stay in +// the plan (only their EDGE changes shape) precisely so clangd does not lose the +// dependency's sources — a cache that silently degraded IDE navigation would be +// a bad trade for build time. +TEST(NinjaBackend, CachedUnitsStillAppearInCompileCommands) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.c", + .object = "obj/dep.o", + .packageName = "dep", + .servedFromCache = true, + .cachedObject = "/bc/v1/pkg/idx/dep@1.0.0/key/obj/dep.o", + }); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + + auto flags = compute_flags(plan); + auto cdb = emit_compile_commands(plan, flags); + + EXPECT_NE(cdb.find("/store/dep/src/dep.c"), std::string::npos) << cdb; + EXPECT_NE(cdb.find("src/main.cpp"), std::string::npos) << cdb; +} From 5ac794b598e908fc0add8e34d745d551248c91ae Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:20:53 +0800 Subject: [PATCH 05/11] fix(build): sequence staged artifacts before compilation, and stop hiding std entries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes from the first CI round. Module partitions. Replacing a package's compile edges with stage edges also removes the ordering those compile edges carried. A consumer that imports `pkg` has `pkg`'s BMI in its dyndep and nothing else — the partition BMI `pkg:part` was reached only because `pkg`'s own compile edge depended on it. With independent stage edges ninja may start the consumer while the partition is still unstaged: error: failed to find module file for module 'mcpplibs.cmdline:options' macOS CI hit it; Linux won the race, which is why it is now an invariant rather than a scheduling accident. Every staged artifact becomes an ORDER-ONLY prerequisite of every non-staged edge, aggregated through one phony so no edge repeats the list (mcpp#274: long ninja lines are how a 50781-character command line blew past cmd.exe's 8191 limit). Order-only is the right strength — the real content dependencies are still declared where they always were, so a changed BMI still invalidates its consumers; this adds sequencing, not dirtiness. Verified locally on both GCC and Clang against mcpplibs.cmdline, which has a `:options` partition. Ages were computed against the wrong epoch. file_time_type is std::chrono::file_clock, whose epoch is not the Unix epoch, so `cache list` printed "74509d ago". Converted through clock_cast. Test fallout, all of it real: - 22_doctor_cache_publish asserted `cache list` was empty after `mcpp self doctor`, which precompiles a std module. That assertion only held because the old `cache list` walked dep entries and skipped std ones — and hiding them is how 16 GB of duplicated std BMIs went unnoticed. The empty-cache check moved ahead of doctor; the std entry is now asserted to be visible, with a bound on the age column that would have caught the epoch bug. - 40_llvm_bmi_cache used `--no-cache` to force a cold first build, then expected the second build to reuse it. `--no-cache` is now an alias for `--cache=off`, which means neither read NOR write — so it left nothing to reuse. The test's actual intent (fresh MCPP_HOME is already cold) is now what it says, and it additionally asserts zero compile edges and the partition sequencing. The semantic tightening is called out in the CHANGELOG: a mode named `off` that still writes the cache would not be defensible. - 98_reflection_import_std grepped $MCPP_HOME/bmi, the pre-v1 root. It passed locally purely because this machine still holds 26 GB of legacy entries, one of which happened to record -freflection — a false green of exactly the kind the plan warned about. It and two siblings now select the std entry by CONTENT instead of `find | head -1`, which is no longer well-defined: one MCPP_HOME can hold several std identities now, and that is the point. --- CHANGELOG.md | 2 + src/bmi_cache/maintenance.cppm | 12 +++-- src/build/ninja_backend.cppm | 60 +++++++++++++++++---- src/cli.cppm | 6 +-- src/config.cppm | 3 +- src/doctor.cppm | 9 ++-- tests/e2e/100_cppfly_reflection.sh | 10 ++-- tests/e2e/22_doctor_cache_publish.sh | 27 +++++++++- tests/e2e/40_llvm_bmi_cache.sh | 39 +++++++++++--- tests/e2e/59_cpp_standard_config.sh | 13 +++-- tests/e2e/98_reflection_import_std.sh | 12 ++++- tests/unit/test_ninja_backend.cpp | 77 +++++++++++++++++++++++++++ 12 files changed, 227 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d1c080a..df0beb7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,6 +63,8 @@ `--no-cache` 保留为 `off` 的兼容别名。它的旧 help 文案「Force-clear target/ before building」两处不准:它清的是**构建目录**(`target///`)而不是整个 `target/`,而且名字与缓存无关。`mcpp run` / `mcpp test` 一并补上这两个 flag(此前它们连 `--no-cache` 都没有)。 + ⚠️ **`--no-cache` 的语义有一处收紧**:此前它只清构建目录、**仍然会回填全局缓存**;现在它等于 `off`,即**不读也不写**。想要「从零重编但仍然刷新缓存」的,用 `mcpp clean` 或 `rm -rf target` 后正常构建。这个收紧是为了让三个模式正交:一个叫 `off` 的模式还偷偷写缓存是说不通的。 + - **`mcpp cache` 补齐到可运维。** `cache dir`(缓存到底在哪 —— 此前 `cache *`/`doctor`/`clean --bmi-cache` 各自解析根目录,而 config 的 reset 路径用 `GlobalConfig::bmiCacheDir`,两者可能不是同一个目录)、`cache gc --max-size {MiB,GiB} / --older-than {s,m,h,d}`(**真 LRU**)、`cache clean --deps|--std|--all|--legacy`、`cache list --json`、`cache verify`(逐条目校验清单与磁盘,残缺条目非零退出)。`cache info` 现在打印该条目的键输入 —— 怀疑命中错了时第一件想看的东西。 `prune` 此前按**目录 mtime** 排序,而那只记录条目被**写入**的时间:一个每次构建都命中的热包,和一个一个月没人碰过的冷包一样「陈旧」。`entry.json` 的 `accessed` 由每次命中刷新(只重写 `entry.json`,**不动产物 mtime** —— 那些 mtime 参与 ninja 的 restat 判定),`gc` 按它排序。`cache clean` 开头那句 `remove_all(/"deps")` 指向一个从不存在的路径(dep 条目在 `//deps`),是死代码。 diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index 6ddbe742..71d98b66 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -128,14 +128,20 @@ std::int64_t stamp_of(const nlohmann::json& j, const char* field) { return 0; } -// mtime fallback for entries whose stamp is missing (written by an mcpp that -// predates `accessed`). Reported as such rather than silently treated as fresh. +// mtime fallback for entries with no `accessed` stamp (std entries, and package +// entries written by an mcpp that predates the field). +// +// file_time_type is std::chrono::file_clock, whose epoch is NOT the Unix epoch — +// on libstdc++ it is 1970 shifted, so reading time_since_epoch() and comparing +// it against system_clock produced ages like "74509d ago". Convert through the +// clock rather than assuming a shared epoch. std::int64_t dir_mtime_seconds(const std::filesystem::path& p) { std::error_code ec; auto t = std::filesystem::last_write_time(p, ec); if (ec) return 0; + auto sys = std::chrono::clock_cast(t); return std::chrono::duration_cast( - t.time_since_epoch()).count(); + sys.time_since_epoch()).count(); } void measure(Entry& e) { diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 76cbb77c..fda83ee0 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -872,6 +872,11 @@ std::string emit_ninja_string(const BuildPlan& plan) { append("\n"); } + // Aggregate target for everything staged out of the global cache. Named + // with a leading underscore so it cannot collide with a module or target + // name (both of which are identifiers or paths). + constexpr std::string_view kStagedCachePhony = "_mcpp_staged_cache"; + auto bmi_path = [&traits](std::string_view name) { std::string s(traits.bmiDir); s += '/'; @@ -916,24 +921,55 @@ std::string emit_ninja_string(const BuildPlan& plan) { // this package's own config and its dependencies' keys, so for a given key // equal size IS equivalence — and size comes from directory metadata, which // stays readable even when a holder denies opening the file. + // + // ORDERING. Replacing a package's compile edges with stage edges also + // removes the ordering those compile edges carried. A module partition is + // the case that breaks: a consumer imports `pkg`, so its dyndep declares + // `pkg`'s BMI and nothing else — the partition BMI `pkg:part` was reached + // only because `pkg`'s own compile edge depended on it. With independent + // stage edges, ninja may finish staging `pkg` and start the consumer while + // `pkg:part` is still unstaged, and Clang then fails with + // `failed to find module file for module 'pkg:part'`. (Observed on macOS + // CI; Linux happened to win the race, which is exactly why this is stated + // as an invariant rather than left to scheduling.) + // + // So every staged artifact becomes an ORDER-ONLY prerequisite of every edge + // that is not itself staged. Order-only (`||`) is the right strength: the + // real content dependencies are still declared where they always were (a + // dyndep-supplied implicit input, or the static-mode implicit list), so a + // changed BMI still invalidates its consumers — this adds sequencing, not + // dirtiness. The cost is that a handful of copies finish before compilation + // starts, which is what used to happen anyway when those units were built. + std::string stagedOrderOnly; { - bool any = false; + std::vector staged; for (auto& cu : plan.compileUnits) { if (!cu.servedFromCache) continue; if (cu.cachedObject.empty()) continue; - any = true; - append(std::format("build {} : stage_file {}\n", - escape_ninja_path(cu.object), + auto obj = escape_ninja_path(cu.object); + append(std::format("build {} : stage_file {}\n", obj, escape_ninja_path(cu.cachedObject))); append(" verify = --verify size\n"); + staged.push_back(obj); if (cu.providesModule && !cu.cachedBmi.empty()) { - append(std::format("build {} : stage_file {}\n", - bmi_path(*cu.providesModule), + auto bmi = bmi_path(*cu.providesModule); + append(std::format("build {} : stage_file {}\n", bmi, escape_ninja_path(cu.cachedBmi))); append(" verify = --verify size\n"); + staged.push_back(bmi); } } - if (any) append("\n"); + if (!staged.empty()) { + append("\n"); + // One phony aggregates them so each consuming edge names a single + // prerequisite instead of repeating the whole list (mcpp#274: long + // ninja lines are how a 50781-character command line blew past + // cmd.exe's 8191 limit on Windows). + append("build " + std::string(kStagedCachePhony) + " : phony"); + for (auto& s2 : staged) append(" " + s2); + append("\n\n"); + stagedOrderOnly = " || " + std::string(kStagedCachePhony); + } } if (dyndep) { @@ -955,8 +991,8 @@ std::string emit_ninja_string(const BuildPlan& plan) { continue; auto ddi = (cu.object.parent_path() / cu.source.filename()).string() + ".ddi"; ddi_paths.push_back(ddi); - append(std::format("build {} : cxx_scan {}\n", escape_ninja_path(ddi), - escape_ninja_path(cu.source))); + append(std::format("build {} : cxx_scan {}{}\n", escape_ninja_path(ddi), + escape_ninja_path(cu.source), stagedOrderOnly)); append(std::format(" compile_target = {}\n", escape_ninja_path(cu.object))); if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) append(std::format(" local_includes ={}\n", includes)); @@ -1027,6 +1063,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { auto it = ddi_to_dd.find(ddi); if (it != ddi_to_dd.end()) { out_line += " | " + it->second; + out_line += stagedOrderOnly; out_line += "\n dyndep = " + it->second; // P2: set bmi_out for the copy_if_different logic in cxx_module. if (cu.providesModule) { @@ -1034,10 +1071,10 @@ std::string emit_ninja_string(const BuildPlan& plan) { } out_line += "\n"; } else { - out_line += "\n"; + out_line += stagedOrderOnly + "\n"; } } else { - out_line += "\n"; + out_line += stagedOrderOnly + "\n"; } if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) out_line += " local_includes =" + includes + "\n"; @@ -1089,6 +1126,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { out_line += std::format(" : {} {}", rule, escape_ninja_path(cu.source)); if (!implicit.empty()) out_line += " |" + implicit; + out_line += stagedOrderOnly; out_line += "\n"; if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) out_line += " local_includes =" + includes + "\n"; diff --git a/src/cli.cppm b/src/cli.cppm index 2e25ef35..8f75b35e 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -54,7 +54,7 @@ void print_usage() { std::println(" mcpp build [options] Build the current package"); std::println(" mcpp run [target] [-- args...] Build + run a binary target"); std::println(" mcpp test [pattern] [-- args...] Build + run tests/**/*.cpp (--list, --timeout, --message-format json)"); - std::println(" mcpp clean [--bmi-cache] Remove target/ (and optionally BMI cache)"); + std::println(" mcpp clean [--bmi-cache] Remove target/ (and optionally the build cache)"); std::println(" mcpp add [@] Add a dependency to mcpp.toml"); std::println(" mcpp remove Remove a dependency from mcpp.toml"); std::println(" mcpp update [pkg] Re-resolve deps and rewrite mcpp.lock"); @@ -286,8 +286,8 @@ int run(int argc, char** argv) { return cmd_test(p, std::span(passthrough)); }))) .subcommand(cl::App("clean") - .description("Remove target/ (and optionally the global BMI cache)") - .option(cl::Option("bmi-cache").help("Also wipe the global BMI cache")) + .description("Remove target/ (and optionally the global build cache)") + .option(cl::Option("bmi-cache").help("Also wipe the global build cache (see `mcpp cache clean`)")) .action(wrap_rc(cmd_clean))) .subcommand(cl::App("why") .description("Explain how the toolchain / runtime / deps were resolved") diff --git a/src/config.cppm b/src/config.cppm index df174854..c352f772 100644 --- a/src/config.cppm +++ b/src/config.cppm @@ -6,7 +6,8 @@ // registry/ XLINGS_HOME for mcpp's xlings // bin/xlings vendored xlings binary (= /bin/xlings) // .xlings.json seeded with index_repos = [mcpplibs] -// bmi// BMI cache (existing) +// build-cache/v1/ cross-project build cache (pkg/ + std/) +// bmi/ pre-v1 build cache; unused, `cache clean --legacy` // cache/ metadata caches // config.toml this module's input // diff --git a/src/doctor.cppm b/src/doctor.cppm index 004c6cb5..0105a062 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -461,10 +461,11 @@ export int explain_code(std::string_view code) { "The [toolchain] pin in mcpp.toml does not match the detected toolchain.\n" "Either install the pinned toolchain (xlings install ...) or relax the\n" "pin (e.g. \"gcc@>=15\" instead of \"gcc@15.1.0\")."}, - {"E0005", "BMI cache corruption", - "A cached BMI file referenced by manifest.txt is missing on disk. Run\n" - "`mcpp cache prune --older-than 0d` to drop stale entries; the next build\n" - "will repopulate."}, + {"E0005", "build cache corruption", + "A file listed in a cache entry's entry.json is missing on disk. Such an\n" + "entry is treated as a miss and rebuilt, so this is never wrong output —\n" + "only wasted space. `mcpp cache verify` lists every affected entry and\n" + "`mcpp cache gc --older-than 0s` reclaims them."}, {"E0006", "index requires a newer mcpp", "The package index declares (index.toml [index].min_mcpp) that its\n" "descriptors need a newer mcpp than this binary — parsing them would\n" diff --git a/tests/e2e/100_cppfly_reflection.sh b/tests/e2e/100_cppfly_reflection.sh index 5eb544ee..313edd6d 100755 --- a/tests/e2e/100_cppfly_reflection.sh +++ b/tests/e2e/100_cppfly_reflection.sh @@ -104,10 +104,12 @@ out=$("$binary") } # The std BMI prebuild carries the same dialect (scan/prebuild/compile agree). -metadata="$(find "$MCPP_HOME/build-cache/v1/std" -name std-module.json | head -1)" -grep -q '"std_flag": "-std=c++26 -freflection' "$metadata" || { - cat "$metadata" - echo "FAIL: std-module.json std_flag lacks -std=c++26 -freflection" +# Recursive grep, not `find | head -1`: one MCPP_HOME can hold several std +# identities now, so "the first one" is arbitrary. +grep -rl '"std_flag": "-std=c++26 -freflection' "$MCPP_HOME/build-cache/v1/std" >/dev/null 2>&1 || { + find "$MCPP_HOME/build-cache/v1/std" -name std-module.json \ + -exec grep -H '"std_flag"' {} \; 2>/dev/null + echo "FAIL: no std-module.json records std_flag -std=c++26 -freflection" exit 1 } diff --git a/tests/e2e/22_doctor_cache_publish.sh b/tests/e2e/22_doctor_cache_publish.sh index 6ec661dc..c33960d2 100755 --- a/tests/e2e/22_doctor_cache_publish.sh +++ b/tests/e2e/22_doctor_cache_publish.sh @@ -11,6 +11,14 @@ TMP=$(mktemp -d) trap "rm -rf $TMP" EXIT export MCPP_HOME="$TMP/mcpp-home" +# 0) cache list on a genuinely empty cache: friendly message. +# This has to come BEFORE doctor — doctor resolves a build plan, which +# precompiles the std module, and `cache list` now reports std entries too. +# Hiding them is how 16 GB of duplicated std BMIs went unnoticed, so listing +# them is the fix, not a regression. +out=$("$MCPP" cache list 2>&1) +[[ "$out" == *"empty"* ]] || { echo "cache list on empty cache: '$out'"; exit 1; } + # 1) doctor: should always run, exit 0 or 1 (warn ok), never 2 rc=0 "$MCPP" self doctor > doctor.log 2>&1 || rc=$? @@ -21,9 +29,24 @@ grep -q 'Checking registry' doctor.log || { cat doctor.log; echo "no regis grep -q 'Checking cache health' doctor.log || { cat doctor.log; echo "no cache check"; exit 1; } grep -q 'Doctor result' doctor.log || { cat doctor.log; echo "no result line"; exit 1; } -# 2) cache list (empty): friendly message +# 2) after doctor, the std module it precompiled must be VISIBLE. std entries +# dominate the cache's size, and `cache list` used to walk only dep entries. out=$("$MCPP" cache list 2>&1) -[[ "$out" == *"empty"* ]] || { echo "cache list empty: '$out'"; exit 1; } +[[ "$out" == *"std"* ]] || { echo "cache list omits the std entry: '$out'"; exit 1; } +# The age column must be plausible: file_time_type's epoch is not the Unix +# epoch, and reading it as if it were printed ages like "74509d ago". +if [[ "$out" =~ ([0-9]+)d\ ago ]] && (( BASH_REMATCH[1] > 3650 )); then + echo "cache list reports an implausible age (clock epoch bug): '$out'" + exit 1 +fi + +# 2b) cache dir must point at the cache the rest of the tool uses. +out=$("$MCPP" cache dir 2>&1) +[[ "$out" == "$MCPP_HOME/build-cache/v1"* ]] || { + echo "cache dir '$out' != '$MCPP_HOME/build-cache/v1'"; exit 1; } + +# 2c) verify must pass on a healthy cache. +"$MCPP" cache verify > /tmp/_v.log 2>&1 || { cat /tmp/_v.log; echo "verify failed"; exit 1; } # 3) publish dry-run on a fresh package. Publish uses `git archive` for the # source tarball, so we git-init + commit first. We also need a non-empty diff --git a/tests/e2e/40_llvm_bmi_cache.sh b/tests/e2e/40_llvm_bmi_cache.sh index 41d23eba..698eaa1e 100755 --- a/tests/e2e/40_llvm_bmi_cache.sh +++ b/tests/e2e/40_llvm_bmi_cache.sh @@ -60,17 +60,17 @@ int main() { } EOF -# First build — should compile the dependency -out1=$("$MCPP" build --no-cache 2>&1) +# First build — populates the cache. Deliberately NOT `--no-cache`: that is now +# an alias for `--cache=off`, which means neither read NOR write, so a build that +# used it would leave nothing for the second build to reuse. MCPP_HOME is fresh +# here, so this build is already cold. +out1=$("$MCPP" build 2>&1) echo "$out1" | grep -q "Compiling.*mcpplibs.cmdline" || { - # It's OK if it says "Cached" because global cache may exist - echo "$out1" | grep -q "Cached.*mcpplibs.cmdline" || { - echo "FAIL: mcpplibs.cmdline not mentioned in first build: $out1" - exit 1 - } + echo "FAIL: mcpplibs.cmdline not compiled in the first (cold) build: $out1" + exit 1 } -# Second build (clean target, keep BMI cache) — dependency should be cached +# Second build, clean target dir, cache kept — the dependency must be reused. rm -rf target out2=$("$MCPP" build 2>&1) echo "$out2" | grep -q "Cached.*mcpplibs.cmdline" || { @@ -78,4 +78,27 @@ echo "$out2" | grep -q "Cached.*mcpplibs.cmdline" || { exit 1 } +# ...and reuse must mean "not recompiled". The status line alone used to be +# printed while ninja rebuilt every unit behind it, so assert on the graph. +NINJA="$(find target -name build.ninja | head -1)" +[[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja"; exit 1; } +if grep -qE ': (cxx_module|cxx_object|cxx_scan) .*mcpplibs' "$NINJA"; then + echo "FAIL: cached dependency still has compile edges" + grep -nE ': (cxx_module|cxx_object|cxx_scan) .*mcpplibs' "$NINJA" | head + exit 1 +fi + +# Clang + module partitions: mcpplibs.cmdline has a `:options` partition, which a +# consumer never imports directly. Its stage edge must be sequenced before the +# consumer's compile, or clang fails with `failed to find module file for module +# 'mcpplibs.cmdline:options'` — a race Linux won and macOS lost. +grep -q '_mcpp_staged_cache' "$NINJA" || { + echo "FAIL: staged artifacts are not sequenced before compilation" + exit 1 +} + +# And the whole thing must actually link and run. +out3=$("$MCPP" run 2>&1) || { echo "FAIL: run: $out3"; exit 1; } +echo "$out3" | grep -q 'cache test ok' || { echo "FAIL: run output: $out3"; exit 1; } + echo "OK" diff --git a/tests/e2e/59_cpp_standard_config.sh b/tests/e2e/59_cpp_standard_config.sh index 2c31ef91..8345af61 100644 --- a/tests/e2e/59_cpp_standard_config.sh +++ b/tests/e2e/59_cpp_standard_config.sh @@ -71,11 +71,14 @@ if grep -q -- "-std=c++23" compile_commands.json; then exit 1 fi -metadata="$(find "$MCPP_HOME/build-cache/v1/std" -name std-module.json | head -1)" -[[ -n "$metadata" ]] || { echo "FAIL: std module metadata missing"; exit 1; } -grep -q '"cpp_standard": "c++26"' "$metadata" || { - echo "FAIL: std module metadata missing C++26 standard" - cat "$metadata" +# The entry recording c++26, selected by CONTENT rather than by `find | head -1`: +# one MCPP_HOME can hold several std identities now. +metadata="$(grep -rl '"cpp_standard": "c++26"' \ + "$MCPP_HOME/build-cache/v1/std" 2>/dev/null | head -1)" +[[ -n "$metadata" ]] || { + echo "FAIL: no std module metadata records c++26" + find "$MCPP_HOME/build-cache/v1/std" -name std-module.json \ + -exec grep -H '"cpp_standard"' {} \; 2>/dev/null exit 1 } grep -q '"std_flag": "-std=c++26"' "$metadata" || { diff --git a/tests/e2e/98_reflection_import_std.sh b/tests/e2e/98_reflection_import_std.sh index 86c046a3..4d9430b8 100755 --- a/tests/e2e/98_reflection_import_std.sh +++ b/tests/e2e/98_reflection_import_std.sh @@ -80,7 +80,15 @@ out=$("$MCPP" run 2>&1) || { echo "FAIL: build/run: $out"; exit 1; } [[ "$out" == *"dep-ok"* ]] || { echo "FAIL: dep module output: $out"; exit 1; } # The std BMI build command must carry the dialect flag. -grep -rl '"std_flag": "[^"]*-freflection' "${MCPP_HOME:-$HOME/.mcpp}/bmi/" >/dev/null \ - || { echo "FAIL: std-module.json lacks -freflection in std_flag"; exit 1; } +# Recursive grep across every std entry, not `find | head -1`: one MCPP_HOME can +# now hold several std identities (they are keyed by std identity rather than by +# whole-project fingerprint), so "the first one" is arbitrary. Asserting that SOME +# entry records the flag is the claim that matters. +STD_ROOT="${MCPP_HOME:-$HOME/.mcpp}/build-cache/v1/std" +grep -rl '"std_flag": "[^"]*-freflection' "$STD_ROOT" >/dev/null 2>&1 || { + echo "FAIL: no std-module.json records -freflection in std_flag" + find "$STD_ROOT" -name std-module.json -exec grep -H '"std_flag"' {} \; 2>/dev/null + exit 1 +} echo "PASS: dialect flags reach std BMI + whole module graph (issue #210)" diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 4d70bf37..7afea22d 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -1066,3 +1066,80 @@ TEST(NinjaBackend, CachedUnitsStillAppearInCompileCommands) { EXPECT_NE(cdb.find("/store/dep/src/dep.c"), std::string::npos) << cdb; EXPECT_NE(cdb.find("src/main.cpp"), std::string::npos) << cdb; } + +// Replacing a package's compile edges with stage edges also removes the ordering +// those compile edges carried. A module PARTITION is the case that breaks: a +// consumer imports `pkg`, so its dyndep declares `pkg`'s BMI and nothing else — +// the partition BMI `pkg:part` was reached only because `pkg`'s own compile edge +// depended on it. With independent stage edges ninja may start the consumer while +// the partition is still unstaged, and Clang fails with `failed to find module +// file for module 'pkg:part'`. Observed on macOS CI while Linux won the race, +// which is why it is pinned here rather than left to scheduling. +TEST(NinjaBackend, NonCachedEdgesOrderAfterEveryStagedArtifact) { + auto plan = minimal_plan(); + // A cached package with a primary interface and a partition. + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.cppm", + .object = "obj/dep.m.o", + .packageName = "dep", + .providesModule = "dep", + .servedFromCache = true, + .cachedObject = "/bc/obj/dep.m.o", + .cachedBmi = "/bc/bmi/dep.gcm", + }); + plan.compileUnits.push_back({ + .source = "/store/dep/src/part.cppm", + .object = "obj/part.m.o", + .packageName = "dep", + .providesModule = "dep:part", + .servedFromCache = true, + .cachedObject = "/bc/obj/part.m.o", + .cachedBmi = "/bc/bmi/dep-part.gcm", + }); + // The consumer, which imports only the primary module. + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + .imports = {"dep"}, + }); + + auto ninja = emit_ninja_string(plan); + + // One phony aggregating every staged artifact — a per-edge list would repeat + // it and grow ninja lines without bound (mcpp#274). + auto phony = ninja.find("build _mcpp_staged_cache : phony"); + ASSERT_NE(phony, std::string::npos) << ninja; + auto phonyLine = ninja.substr(phony, ninja.find('\n', phony) - phony); + for (auto* art : {"obj/dep.m.o", "obj/part.m.o", + "gcm.cache/dep.gcm", "gcm.cache/dep-part.gcm"}) { + EXPECT_NE(phonyLine.find(art), std::string::npos) + << art << " missing from: " << phonyLine; + } + + // The consumer's edge orders after it... + auto consumer = ninja.find("build obj/main.o"); + ASSERT_NE(consumer, std::string::npos) << ninja; + auto consumerLine = ninja.substr(consumer, ninja.find('\n', consumer) - consumer); + EXPECT_NE(consumerLine.find("|| _mcpp_staged_cache"), std::string::npos) + << consumerLine; + + // ...as an ORDER-ONLY dependency, not an implicit one: the real content + // dependency is still declared where it always was, so this must add + // sequencing without making a staged BMI dirty its consumers. + auto bar = consumerLine.find("|| _mcpp_staged_cache"); + EXPECT_EQ(consumerLine.find("| _mcpp_staged_cache"), bar + 1) + << "staged phony must appear only after ||, never as an implicit input: " + << consumerLine; +} + +TEST(NinjaBackend, NoStagedPhonyWhenNothingIsCached) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + auto ninja = emit_ninja_string(plan); + EXPECT_EQ(ninja.find("_mcpp_staged_cache"), std::string::npos) << ninja; +} From a5bb4f0130cff31e38d6b861308ba59a9bda0131 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:25:13 +0800 Subject: [PATCH 06/11] fix(cache): compute entry age without assuming file_clock shares an epoch libc++ does not provide std::chrono::clock_cast, so the previous fix broke every Clang build. Measure the mtime as an offset from now IN THE FILE CLOCK and apply that offset to the system clock instead: no shared epoch, no conversion trait, and "how long ago" is all any caller wants. Verified compiling and behaving under llvm@22.1.8 + libc++, which is the configuration that broke. --- src/bmi_cache/maintenance.cppm | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index 71d98b66..d7d2757f 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -132,16 +132,20 @@ std::int64_t stamp_of(const nlohmann::json& j, const char* field) { // entries written by an mcpp that predates the field). // // file_time_type is std::chrono::file_clock, whose epoch is NOT the Unix epoch — -// on libstdc++ it is 1970 shifted, so reading time_since_epoch() and comparing -// it against system_clock produced ages like "74509d ago". Convert through the -// clock rather than assuming a shared epoch. +// reading time_since_epoch() and comparing it against system_clock produced ages +// like "74509d ago". `clock_cast` would be the standard answer but libc++ does +// not provide it, so measure the mtime as an OFFSET FROM NOW in the file clock +// and apply that offset to the system clock. That needs no shared epoch and no +// conversion trait — and "how long ago" is all any caller wants anyway. std::int64_t dir_mtime_seconds(const std::filesystem::path& p) { std::error_code ec; auto t = std::filesystem::last_write_time(p, ec); if (ec) return 0; - auto sys = std::chrono::clock_cast(t); - return std::chrono::duration_cast( - sys.time_since_epoch()).count(); + auto age = std::chrono::duration_cast( + std::chrono::file_clock::now() - t); + auto nowSys = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()); + return (nowSys - age).count(); } void measure(Entry& e) { From 20f5ceb2f542ce37429d004758eaca2b86b50565 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:30:04 +0800 Subject: [PATCH 07/11] fix(build): attribute a source to its most specific package root MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Package roots can nest — a workspace member lives under the workspace root — and taking the first matching root filed the member's sources under the outer package, i.e. into the wrong cache key. Index payloads live in the xpkgs store and cannot be shadowed this way, so no cached entry is affected today; resolving by specificity rather than by iteration order is what keeps that true if roots ever move. --- src/build/prepare.cppm | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 8e0abc9d..339d23a5 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -3885,15 +3885,28 @@ prepare_build(bool print_fingerprint, // Sources belonging to each package, package-root-relative and sorted. std::vector> pkgSources(packages.size()); for (auto& cu : ctx.plan.compileUnits) { + // Longest matching root wins. Package roots can nest — a workspace + // member lives under the workspace root — and taking the first match + // would file the member's sources under the outer package, putting + // them in the wrong key. (Index payloads live in the xpkgs store and + // cannot be shadowed this way, so no cached entry is affected today; + // resolving it by specificity rather than by iteration order is what + // keeps that true if roots ever move.) + std::size_t best = packages.size(); + std::size_t bestLen = 0; + std::string bestRel; for (std::size_t p = 0; p < packages.size(); ++p) { std::error_code ec; auto rel = std::filesystem::relative(cu.source, packages[p].root, ec); if (ec || rel.empty()) continue; auto rels = rel.generic_string(); if (rels.starts_with("..")) continue; - pkgSources[p].push_back(rels); - break; + auto len = packages[p].root.generic_string().size(); + if (best == packages.size() || len > bestLen) { + best = p; bestLen = len; bestRel = std::move(rels); + } } + if (best != packages.size()) pkgSources[best].push_back(std::move(bestRel)); } for (auto& v : pkgSources) std::ranges::sort(v); From 00fcea9858b63bf1c1210d023bce72bc2ebc399c Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:40:03 +0800 Subject: [PATCH 08/11] fix(cache): scope gc's summary figure to package entries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gc deliberately never evicts std entries — one is shared by every project on the machine and costs ~30 s to rebuild, so trading it for a little disk is the wrong trade. But its summary reported the remaining PACKAGE bytes as the cache size, so a run that freed everything in scope printed "cache now 0.0 B" with tens of MB of std BMIs sitting right next to it. e2e now pins both halves: the budget is met, std survives, and the figure says what it measured. --- src/bmi_cache/maintenance.cppm | 6 +++++- tests/e2e/174_cache_modes_and_commands.sh | 24 +++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index d7d2757f..482f02e7 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -460,7 +460,11 @@ export int cache_gc(const std::string& maxSizeArg, const std::string& olderThanA e.label, human_bytes(e.size), format_age(e.accessed))); } std::println(""); - std::println("Collected {} entries, freed {} (cache now {})", + // "package entries", not "cache": `live` only ever counted package entries, + // because std entries are deliberately out of scope here. Reporting it as + // the cache size would read as "the cache is now empty" while tens of MB of + // std BMIs sit right next to it. + std::println("Collected {} entries, freed {} (package entries now {})", removed, human_bytes(freed), human_bytes(live)); if (maxSize && live > *maxSize) { // Say it rather than silently under-delivering: a size target that diff --git a/tests/e2e/174_cache_modes_and_commands.sh b/tests/e2e/174_cache_modes_and_commands.sh index 7730739d..8af47dcc 100644 --- a/tests/e2e/174_cache_modes_and_commands.sh +++ b/tests/e2e/174_cache_modes_and_commands.sh @@ -280,6 +280,30 @@ rc=0 "$MCPP" cache gc > gcnoargs.log 2>&1 || rc=$? [[ "$rc" -ne 0 ]] || { echo "FAIL: gc with no budget succeeded"; cat gcnoargs.log; exit 1; } +# gc --max-size evicts package entries to hit a budget, and leaves std entries +# alone: a std BMI is shared by every project on the machine and costs ~30 s to +# rebuild, so trading it for a little disk is the wrong trade. The summary must +# therefore talk about PACKAGE entries — reporting it as "cache now 0 B" would +# read as an empty cache while tens of MB of std BMIs sit next to it. +[[ "$(entry_count)" -eq 1 ]] || { echo "FAIL: expected one package entry before gc"; exit 1; } +[[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { echo "FAIL: no std entries to protect"; exit 1; } +"$MCPP" cache gc --max-size 1B > gcsize.log 2>&1 || { cat gcsize.log; exit 1; } +[[ "$(entry_count)" -eq 0 ]] || { echo "FAIL: gc --max-size kept package entries"; cat gcsize.log; exit 1; } +[[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { + echo "FAIL: gc --max-size evicted std entries" + cat gcsize.log + exit 1 +} +grep -q 'package entries now' gcsize.log || { + cat gcsize.log + echo "FAIL: gc summary must scope its figure to package entries" + exit 1 +} + +# Refill for the clean tests below. +rm -rf target +"$MCPP" build > refill2.log 2>&1 || { cat refill2.log; exit 1; } + # ── clean --std / --all / --legacy ───────────────────────────────────────── [[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { echo "FAIL: no std cache dir"; exit 1; } "$MCPP" cache clean --std > cleanstd.log 2>&1 From 958bb0010d2dd6e277ca8c7581fcbdcd3e2b86d7 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:41:05 +0800 Subject: [PATCH 09/11] docs: note the cache-root deviation in the design doc itself MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reader hits the design before the plan, and the design's layout section still said $MCPP_HOME/cache/v1 — a path that would nest the build cache inside the index metadata cache, whose reset removes the whole directory. --- .agents/docs/2026-07-30-dep-build-cache-scoping-design.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md index d4a080e9..495d05e4 100644 --- a/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md +++ b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md @@ -352,8 +352,13 @@ C3/C4 与 C1 的关系是硬约束：**C1 修好之前它们是无害的浪费 ### S2 — 缓存布局与自描述条目（治 C5） +> **实施时的修正**：根目录不是 `$MCPP_HOME/cache/v1` 而是 +> **`$MCPP_HOME/build-cache/v1`** —— `cache` 这个名字已归 `GlobalConfig::metaCacheDir` +> （索引元数据）所有，而它的 reset 路径 `remove_all` 整个目录。其余布局如下不变。 +> 完整偏差清单见实施计划的「实施结果与计划的偏差」。 + ``` -$MCPP_HOME/cache/v1/ +$MCPP_HOME/build-cache/v1/ pkg//@// entry.json ← 自描述：键的全部输入 + 文件清单 + created/accessed bmi/.{gcm,pcm} From 9d217737b830436f1cc2d4b9bc6e91240bbd8151 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:55:58 +0800 Subject: [PATCH 10/11] fix(build): make the fast path honour the declared cache mode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same defect shape as the profile one this PR already fixes, in the switch this PR adds. A build.ninja generated under `--cache=global` contains stage_file edges reading the global cache. `.build_cache` did not record the mode, and a bare `mcpp build` (no CLI flag, so nothing bypasses the fast path) replayed that graph even when the manifest said `cache = "local"` — using the cache the project just declared it did not want. Ruling the cache out is `local`'s entire purpose, so this defeated the feature silently. Reproduced with a warm cache and an untouched manifest, which is what makes it reachable: editing mcpp.toml would have invalidated the fast path on mtime, so only the recorded mode can settle it. build --cache=global → 15 stage edges build → 15 stage edges [build] cache = "local" ignored after → 0 stage edges Mode selection moves into resolve_cache_mode, a pure function of (manifest, override, environment), so prepare_build and both fast paths settle it from one rule — the same treatment resolve_profile_name got. prepare_build keeps sole ownership of the diagnostics: an unparseable value must be reported once, by the invocation that actually resolves the build, and must fall through to the next source rather than silently meaning "global". `.build_cache` records the mode alongside the profile; a mismatch is a miss, and entries predating the field have an empty mode that matches nothing. No migration, self-healing on first rebuild. --- src/build/execute.cppm | 56 +++++++++++++++++------ src/build/prepare.cppm | 47 +++++++++++++------ tests/e2e/174_cache_modes_and_commands.sh | 54 ++++++++++++++++++++++ tests/unit/test_build_profile.cpp | 38 +++++++++++++++ 4 files changed, 167 insertions(+), 28 deletions(-) diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 3bab7b45..dacbcc69 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -67,6 +67,12 @@ struct BuildCacheEntry { // Empty means "cache predates this field" and is treated as a miss (a // bare rebuild once, never a wrong artifact). std::string profile; + // The global-cache mode this build.ninja was generated under. A graph built + // under `global` contains stage_file edges reading the cache; replaying it + // for a request that asked for `local` would use the cache the manifest just + // said not to use — and ruling the cache out is `local`'s entire purpose. + // Same back-compat contract as `profile`: empty ⇒ miss. + std::string cacheMode; }; std::vector read_build_cache(const std::filesystem::path& projectRoot) { @@ -139,6 +145,10 @@ std::vector read_build_cache(const std::filesystem::path& proje e.profile = line.substr(8); haveNextLine = static_cast(std::getline(f, line)); } + if (haveNextLine && line.starts_with("cacheMode=")) { + e.cacheMode = line.substr(10); + haveNextLine = static_cast(std::getline(f, line)); + } entries.push_back(std::move(e)); if (!haveNextLine || line.empty()) break; } @@ -155,7 +165,8 @@ void write_build_cache(const std::filesystem::path& projectRoot, std::vector> runTargets = {}, const std::string& runEnvKey = "", const std::string& runEnvValue = "", - const std::string& profile = "") { + const std::string& profile = "", + const std::string& cacheMode = "") { auto path = projectRoot / kBuildCacheFile; auto entries = read_build_cache(projectRoot); @@ -170,7 +181,7 @@ void write_build_cache(const std::filesystem::path& projectRoot, // Insert at front (MRU). BuildCacheEntry newEntry{targetTriple, outputDir.string(), ninjaProgram, fingerprintHex, runtimeEnvKey, runtimeEnvValue, std::move(runTargets), - runEnvKey, runEnvValue, profile}; + runEnvKey, runEnvValue, profile, cacheMode}; entries.insert(entries.begin(), std::move(newEntry)); // Trim to LRU capacity. @@ -199,6 +210,7 @@ void write_build_cache(const std::filesystem::path& projectRoot, f << "runEnv=" << e.runEnvKey << '\n'; f << e.runEnvValue << '\n'; f << "profile=" << e.profile << '\n'; + f << "cacheMode=" << e.cacheMode << '\n'; } } @@ -358,6 +370,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, auto entries = read_build_cache(ctx.projectRoot); for (auto& e : entries) { if (e.targetTriple == targetOverride && e.profile == ctx.profile + && e.cacheMode == cache_mode_name(ctx.cacheMode) && !e.fingerprint.empty()) { auto newFp = ctx.outputDir.filename().string(); if (e.fingerprint != newFp) { @@ -380,7 +393,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, r->runtimeEnvKey.empty() ? "-" : r->runtimeEnvKey, r->runtimeEnvValue, std::move(runTargets), runEnvKey, runEnvValue, - ctx.profile); + ctx.profile, std::string(cache_mode_name(ctx.cacheMode))); } // The one place the --strict policy is settled. Degradations reported by @@ -524,11 +537,21 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, // for a different profile, and the fast path would run ninja against that // profile's build.ninja: `mcpp build --release` then a bare `mcpp build` // reported success in 0.00s and left -O2 artifacts where -O0 -g was asked for. -std::optional fast_path_profile(const std::filesystem::path& projectRoot, - std::string_view profileOverride = "") { +struct FastPathIdentity { + std::string profile; + std::string cacheMode; +}; + +std::optional +fast_path_identity(const std::filesystem::path& projectRoot, + std::string_view profileOverride = "") { auto m = mcpp::manifest::load(projectRoot / "mcpp.toml"); if (!m) return std::nullopt; - return mcpp::build::resolve_profile_name(*m, profileOverride); + return FastPathIdentity{ + mcpp::build::resolve_profile_name(*m, profileOverride), + std::string(mcpp::build::cache_mode_name( + mcpp::build::resolve_cache_mode(*m, ""))), + }; } // Try to fast-path: if build.ninja is newer than all inputs, just run ninja. @@ -538,16 +561,18 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo std::string_view currentTarget = "") { if (no_cache) return std::nullopt; - auto wantProfile = fast_path_profile(projectRoot); - if (!wantProfile) return std::nullopt; + auto want = fast_path_identity(projectRoot); + if (!want) return std::nullopt; // P3: read multi-entry cache and find the entry matching this - // (target, profile) pair. Matching on the triple alone served the wrong - // profile's artifacts (see fast_path_profile). + // (target, profile, cache mode) triple. Matching on the target alone served + // the wrong profile's artifacts, and ignoring the cache mode replayed a + // cache-reading graph for a request that asked not to read the cache. auto entries = read_build_cache(projectRoot); const BuildCacheEntry* match = nullptr; for (auto& e : entries) { - if (e.targetTriple == currentTarget && e.profile == *wantProfile) { + if (e.targetTriple == currentTarget && e.profile == want->profile + && e.cacheMode == want->cacheMode) { match = &e; break; } @@ -599,7 +624,7 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo if (!rc) return std::nullopt; if (*rc != 0) return rc; - mcpp::ui::finished(*wantProfile, elapsed); + mcpp::ui::finished(want->profile, elapsed); return 0; } @@ -614,13 +639,14 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo std::optional try_fast_run(const std::filesystem::path& projectRoot, const std::optional& targetName, std::span passthrough) { - auto wantProfile = fast_path_profile(projectRoot); - if (!wantProfile) return std::nullopt; + auto want = fast_path_identity(projectRoot); + if (!want) return std::nullopt; auto entries = read_build_cache(projectRoot); const BuildCacheEntry* match = nullptr; for (auto& e : entries) { - if (e.targetTriple.empty() && e.profile == *wantProfile) { + if (e.targetTriple.empty() && e.profile == want->profile + && e.cacheMode == want->cacheMode) { match = &e; break; } diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 339d23a5..617ccc0b 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -657,6 +657,26 @@ export struct BuildContext { std::vector cachedDeps; }; +// The ONE cache-mode resolver, for the same reason resolve_profile_name exists: +// execute.cppm's fast paths deliberately skip prepare_build, so they need to +// settle the mode from the same rule. Pure in (manifest, override, environment). +// +// `--cache` on the command line already bypasses the fast path, so the override +// argument is empty there; it is threaded anyway so there is exactly one place +// where precedence is written down. +// +// Precedence: --cache > MCPP_BUILD_CACHE > [build] cache > global. An +// unparseable value falls through to the next source rather than silently +// meaning "global" — see prepare_build, which also reports it. +export CacheMode resolve_cache_mode(const mcpp::manifest::Manifest& m, + std::string_view override_mode) { + if (auto v = parse_cache_mode(override_mode)) return *v; + if (const char* e = std::getenv("MCPP_BUILD_CACHE"); e && *e) + if (auto v = parse_cache_mode(e)) return *v; + if (auto v = parse_cache_mode(m.buildConfig.cacheMode)) return *v; + return CacheMode::Global; +} + // The ONE profile-name resolver. Shared with execute.cppm's fast paths: // they deliberately skip prepare_build, so before this existed they had no // idea which profile the request meant — and `.build_cache` keyed entries by @@ -822,24 +842,25 @@ prepare_build(bool print_fingerprint, // through to the next source rather than silently meaning "global" — a typo // that quietly re-enabled the cache would be the hardest kind of surprise // to attribute. - CacheMode cacheMode = CacheMode::Global; + // Selection lives in resolve_cache_mode (above) so the fast paths settle it + // identically. This block only adds the diagnostics, which the fast paths + // have no business emitting: an unparseable value must be reported once, by + // the invocation that actually resolves the build. + const CacheMode cacheMode = resolve_cache_mode(*m, overrides.cache_mode); { - std::string cacheModeError; - auto try_source = [&](std::string_view value, std::string_view origin) { - if (value.empty()) return false; - if (auto parsed = parse_cache_mode(value)) { cacheMode = *parsed; return true; } + const char* envMode = std::getenv("MCPP_BUILD_CACHE"); + for (auto [value, origin] : std::initializer_list< + std::pair>{ + {overrides.cache_mode, "--cache"}, + {envMode ? envMode : "", "MCPP_BUILD_CACHE"}, + {m->buildConfig.cacheMode, "[build] cache"}}) { + if (value.empty() || parse_cache_mode(value)) continue; auto msg = std::format( "{} has unknown cache mode '{}' (expected: global | local | off)", origin, value); - if (overrides.strict) { cacheModeError = msg; return true; } + if (overrides.strict) return std::unexpected(msg); mcpp::diag::warning("build/cache-mode", msg); - return false; - }; - const char* envMode = std::getenv("MCPP_BUILD_CACHE"); - if (!try_source(overrides.cache_mode, "--cache")) - if (!try_source(envMode ? envMode : "", "MCPP_BUILD_CACHE")) - (void)try_source(m->buildConfig.cacheMode, "[build] cache"); - if (!cacheModeError.empty()) return std::unexpected(cacheModeError); + } } // ─── Toolchain resolution (docs/21) ──────────────────────────────── diff --git a/tests/e2e/174_cache_modes_and_commands.sh b/tests/e2e/174_cache_modes_and_commands.sh index 8af47dcc..dfccd27c 100644 --- a/tests/e2e/174_cache_modes_and_commands.sh +++ b/tests/e2e/174_cache_modes_and_commands.sh @@ -246,6 +246,60 @@ if grep -q '^cache' mcpp.toml; then exit 1 fi +# ── the fast path must honour the declared mode ───────────────────────────── +# A build.ninja generated under `global` contains stage_file edges reading the +# cache. Replaying it for a request that asked for `local` would use the cache the +# manifest just said not to use — and ruling the cache out is `local`'s whole +# purpose. .build_cache therefore records the mode, exactly as it records the +# profile, and a mismatch is a miss. +cat >> mcpp.toml <<'EOF' + +[build] +cache = "local" +EOF +rm -rf target +"$MCPP" build --cache=global > modeglobal.log 2>&1 || { cat modeglobal.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +staged_before=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) +[[ "$staged_before" -gt 0 ]] || { + echo "FAIL: --cache=global did not produce stage edges (cache should be warm here)" + cat modeglobal.log + exit 1 +} +# Bare build: the manifest says local, and the fast path must NOT replay the +# global graph. Note mcpp.toml is untouched between these two builds, so its +# mtime cannot be what saves us — only the recorded mode can. +"$MCPP" build > modebare.log 2>&1 || { cat modebare.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +staged_after=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) +[[ "$staged_after" -eq 0 ]] || { + echo "FAIL: bare build replayed the global-mode graph ($staged_after stage edges)" + echo " [build] cache = local was bypassed by the fast path" + cat modebare.log + exit 1 +} +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: local mode did not compile the dependency" + exit 1 +} +# Strip the override again for the sections below. +python3 - mcpp.toml <<'PYEOF' +import sys +p = sys.argv[1] +out, skip = [], False +for ln in open(p).read().splitlines(keepends=True): + if ln.strip() == "[build]": + skip = True + continue + if skip and ln.strip().startswith("cache"): + skip = False + continue + out.append(ln) +open(p, "w").write("".join(out)) +PYEOF +rm -rf target +"$MCPP" build > moderestore.log 2>&1 || { cat moderestore.log; exit 1; } + # ── gc: LRU by last USE, not by when the entry was written ───────────────── # Backdate the entry's accessed stamp, then confirm an age-bounded gc collects it # — and that a build afterwards refreshes the stamp so it would survive next time. diff --git a/tests/unit/test_build_profile.cpp b/tests/unit/test_build_profile.cpp index 5939b07b..df1cc3b8 100644 --- a/tests/unit/test_build_profile.cpp +++ b/tests/unit/test_build_profile.cpp @@ -132,3 +132,41 @@ TEST(BuildProfile, CacheModeNameRoundTrips) { for (auto m : {CacheMode::Global, CacheMode::Local, CacheMode::Off}) EXPECT_EQ(mcpp::build::parse_cache_mode(mcpp::build::cache_mode_name(m)), m); } + +// ── cache-mode resolution ─────────────────────────────────────────────────── +// Same shape as resolve_profile_name and for the same reason: the fast paths +// skip prepare_build, so they must settle the mode from one shared pure rule. +// Without that, a graph generated under `global` (which contains stage_file +// edges reading the cache) got replayed for a request that asked for `local` — +// and ruling the cache out is `local`'s entire purpose. + +TEST(BuildProfile, CacheModeOverrideBeatsManifest) { + auto m = base(); + m.buildConfig.cacheMode = "local"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, "global"), + mcpp::build::CacheMode::Global); +} + +TEST(BuildProfile, CacheModeManifestBeatsDefault) { + auto m = base(); + m.buildConfig.cacheMode = "off"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, ""), mcpp::build::CacheMode::Off); +} + +TEST(BuildProfile, CacheModeDefaultsToGlobal) { + EXPECT_EQ(mcpp::build::resolve_cache_mode(base(), ""), + mcpp::build::CacheMode::Global); +} + +// An unparseable value must not silently mean "global" at THIS level either: it +// falls through to the next source, and prepare_build reports it separately. +TEST(BuildProfile, UnknownManifestCacheModeFallsThroughToDefault) { + auto m = base(); + m.buildConfig.cacheMode = "bogus"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, ""), + mcpp::build::CacheMode::Global); + // ...and an unparseable override does not shadow a valid manifest value. + m.buildConfig.cacheMode = "local"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, "bogus"), + mcpp::build::CacheMode::Local); +} From 24e2f4e36a469ee22dd2712e526d9f82f51f4a11 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 30 Jul 2026 15:58:31 +0800 Subject: [PATCH 11/11] docs: record the fast-path graph-identity gap as a follow-up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fast path replays build.ninja on the premise that this request would generate the same graph. This PR pins two axes (profile, cache mode) into .build_cache, but MACOSX_DEPLOYMENT_TARGET, MCPP_VERIFY_MODGRAPH and MCPP_SCANNER also change the graph and are recorded nowhere — all three predate this PR, and closing them needs a cheap graph identity rather than re-deriving the fingerprint, which is exactly what the fast path exists to avoid. Written down instead of half-fixed. --- ...-30-dep-build-cache-implementation-plan.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md b/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md index 388ba6ee..a009da9f 100644 --- a/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md +++ b/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md @@ -383,6 +383,29 @@ e2e（新文件 `tests/e2e/1NN_build_cache_crossproject.sh`）——**本设计 **未实施项的处置**：S8 与零拷贝均记在设计文档 §7 的后续批次里，不在本批 CHANGELOG 中承诺。 +### 顺带发现、本批**未**处理的一个同类隐患（记录待办） + +fast path 重放 `build.ninja` 的前提是「本次请求会生成同一张图」。本批把 **profile** 与 +**cache mode** 两个轴记进了 `.build_cache` 并要求匹配，但**图形状还受若干环境变量影响， +它们都不在任何记录里**： + +| 输入 | fast path 是否能发现变化 | +|---|---| +| `--profile` / `--dev` / `--release` | ✅ 记录 + 匹配（本批） | +| `--cache` / `[build] cache` / `MCPP_BUILD_CACHE` | ✅ 记录 + 匹配（本批） | +| `--target` / `-p` / `--features` / `--cap` / `--static` / `--strict` | ✅ 显式 flag 直接绕过 fast path | +| 改 `mcpp.toml` | ✅ mtime 比 `build.ninja` 新 | +| `MACOSX_DEPLOYMENT_TARGET` | ❌ 进 fingerprint，但 fast path 只校验「记录的 fp 与目录名自洽」，不重算 fp | +| `MCPP_VERIFY_MODGRAPH` / `MCPP_SCANNER` | ❌ 改变 scan/dyndep 边的形状 | + +这三个都是**本批之前就存在**的（fast path 从来没有校验过它们），且都需要「重算一次 fp 才能 +比对」——那与 fast path 存在的意义（不跑 prepare_build）直接冲突。正解是给 fast path 一个 +**廉价的图身份**：把这些输入（以及 profile/cache mode）折进一个短哈希写进 `.build_cache`， +匹配即重放。那是一次独立的收敛，不该塞进本批。 + +判据同本批：`.build_cache` 的条目必须自证「我是在什么条件下生成的」，否则「条件变了」与 +「条件没变」在读取端不可区分 —— 这正是 profile 那个 0.00s 假成功的成因。 + ## 已知坑（来自本仓库既有教训，实现时逐条对照） | 坑 | 规避 |