diff --git a/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md b/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md new file mode 100644 index 00000000..a009da9f --- /dev/null +++ b/.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md @@ -0,0 +1,419 @@ +# 依赖构建产物的全局缓存收敛 — 实施计划 + +配套设计：`2026-07-30-dep-build-cache-scoping-design.md` +前置：issue311 / PR#312（`mcpp stage` 原语 + `stage_file` rule + `restat = 1` + `mcpp.home`） +**已合入 main**（`b794856`，2026.7.30.1），故本计划直接复用，不再自建 staging 原语。 +建议目标版本：**2026.7.30.2**（常规迭代，`.0` 保留给稳定版） + +单 PR 交付。阶段有硬依赖顺序：**A（正确性）→ B（换键）→ C（stage 边）→ D（模式与运维）**。 +A 必须先合入的判据见设计 §5：C 让 ninja 第一次真正接受缓存产物，A 未落地时那等于把 +"白重编一遍但结果对" 变成 "直接用错对象"。 + +--- + +## 落地前已核实的现状事实（避免设计→实现的重映射错误） + +| 事实 | 位置 | 对实现的意义 | +|---|---|---| +| `stage_file` rule 已存在，命令 `$mcpp stage $verify --output $out $in`，带 `restat = 1` | `ninja_backend.cppm:456-459` | C 阶段直接复用，`$verify` 用 `size`（依赖缓存条目是 key-scoped，等长即等价，与 std 边同理） | +| `$mcpp` 变量绑定已提到 `if (dyndep)` 之外 | `ninja_backend.cppm:427` | 不需要再动 | +| **restat 与对齐 mtime 互斥**（PR#312 的教训：对齐 mtime 会重新级联） | 同上注释 | `stage_into` 里的 `touch_now` 在 C 阶段要删掉，改由 restat 负责 | +| `DepCacheIdentity` 已有 `{indexName, packageName, version}`，两个 push 点 | `prepare.cppm:1375-1380` / `:2684` / `:2979` | A 阶段加 `sourceKind` 字段即可，**不需要**按路径前缀猜来源 | +| `sourceKind`（`"version"｜"path"｜"git"`）在两个 push 点都在作用域内 | `prepare.cppm:2699` / `:2982` | S9 的判据是现成的 | +| `dependencyEdges`（`consumerPackageIndex → dependencyPackageIndex` + visibility） | `prepare.cppm:2059-2073` | B 阶段 F 轴（Merkle）直接遍历它，不需要新建图 | +| `CompileUnit` 已带 `packageName` + per-package flags/include dirs | `plan.cppm:20-36` | B 阶段按包分组是现成的；C 阶段加标记字段 | +| `report.topoOrder` 已有拓扑序 | `prepare.cppm:3561` | B 阶段自底向上算 Merkle 键一次遍历 | +| `metadata_for()` 已是完整正确的 std 身份（15 字段） | `stdmod.cppm:88-112` | B 阶段只改目录名，不重新定义身份 | +| `mcpp::home::bmi_root()` 已是单一解析器 | `src/home.cppm` | 新缓存根走它，不再碰 `default_cache_root()` 的旧兜底 | +| 版本号两处 | `mcpp.toml:3`、`src/toolchain/fingerprint.cppm:21` | `.github/tools/check_version_pins.sh` 机器校验 | + +--- + +## 阶段 A — 正确性前置 + +### A1 · profile 进失效轴 + +`prepare.cppm:canonical_compile_flags()`（:209-265）末尾追加： + +```cpp +s += " opt=" ; s += m.buildConfig.optLevel; +s += " debug=" ; s += m.buildConfig.debug ? "1" : "0"; +s += " lto=" ; s += m.buildConfig.lto ? "1" : "0"; +s += " strip=" ; s += m.buildConfig.strip ? "1" : "0"; +``` + +调用点在 profile 解析（:776-809）之后，无需移动。 + +**副作用（写 CHANGELOG）**：三个 profile 从此各占一个 `target///`，互不覆盖 +（cargo 行为）。代价是磁盘 × profile 数——每个 build dir 带一份 staged std BMI +（本机 `std.gcm` = 31,466,112 B）。 + +### A2 · `.build_cache` 按 profile 区分（C3b，今天在生效的 bug） + +`BuildCacheEntry` 加 `std::string profile;`（`execute.cppm:35-58`）。 + +- 序列化：P3 格式加一行 `profile=`；**旧条目缺该字段 ⇒ 读成空 ⇒ 与任何 resolved + profile 都不匹配 ⇒ 当 miss**（自愈，不需要迁移代码）。 +- `write_build_cache` 的去重键：`(targetTriple, profile)`（:143-145）。 +- `try_fast_build` / `try_fast_run` 需要知道"本次请求的 resolved profile"。但 fast path 的 + 存在意义就是**不跑 prepare_build**，而 profile 解析在 prepare_build 里。 + ⇒ 解析规则本身是纯的（`overrides.profile > [build].default-profile > "dev"`， + `prepare.cppm:789-791`），把它抽成一个 `mcpp::build::resolve_profile_name( + const Manifest&, std::string_view override)` 供两侧共用。fast path 只需 load manifest + （它已经 load 了，用于 staleness 检查）。 +- 顺手修 `ui::finished("release", ...)`（`execute.cppm:346`）硬编码 → 传 resolved profile。 + 它在 dev 构建下也打 `Finished release [optimized]`，是 E8 里最误导人的一行。 + +### A3 · fingerprint-changed 警告降噪 + +`execute.cppm:310-324`。条目里现在有 profile 了，所以能判断"fp 变化是否伴随 profile 变化"： +profile 也变了 ⇒ 不打这条（这是预期的、用户自己要求的），只在 profile 相同而 fp 变时打。 + +### A4 · 本地来源包排除出缓存（S9） + +1. `DepCacheIdentity` 加 `std::string sourceKind;`，两个 push 点（:2684 用字面 + `"version"`，:2979 用作用域内的 `sourceKind`）填上。 +2. 缓存循环（:3758-3830）的 `skipCache` 谓词整体替换： + +```cpp +// 旧：查 root manifest 的 dependencies/devDependencies（传递依赖查不到 ⇒ 误缓存） +// 新：唯一判据 —— 该包必须来自 index（不可变的 xpkgs store） +if (!depIdent || depIdent->sourceKind != "version") continue; +``` + + 这同时干掉 `indexName` 回落到 `defaultIndex` 的误挂（E6：本地包 `B` 被挂到 + `mcpplibs` 名下）——`depIdent` 为空时直接不缓存，回落分支消失。 +3. 判据方向说明写进代码注释：与 `mcpp add` 存在性门的"不可证伪就放行"**相反**， + 缓存要"不能证明来自不可变 store 就不缓存"。缓存的错误代价是静默错产物。 + +### A5 · 条目自描述 + 命中逐字段校验（S2） + +`bmi_cache.cppm`： + +- `CacheKey` 加 `std::string keyHex;`（B 阶段填 Merkle 键）与 `nlohmann::json inputs;`。 +- 新 `entry.json`（取代 `manifest.txt` 作为 sentinel，`manifest.txt` 保留写出以便人读）： + +```json +{ "schema": 1, "epoch": 1, "key": "", + "inputs": { ...键的全部输入... }, + "bmi": ["A.gcm"], "obj": ["a.o"], + "created": "", "accessed": "" } +``` + +- `is_cached(key)` ⇒ `entry.json` 存在 ∧ `schema/epoch` 匹配 ∧ **`inputs` 逐字段等于本次 + 算出的 inputs** ∧ 清单文件全部存在。字段不等 = miss，**永不"信任 hash 相等"** + （照抄 `stdmod.cppm::metadata_matches` 的模式）。 +- 写序：产物 → `entry.json.tmp` → rename（sentinel 最后）。沿用已有的 + `FileLock::try_acquire`。 +- 命中时更新 `accessed`（只重写 `entry.json`，**不动产物 mtime**）——这是 D 阶段 + `cache gc` 能做真 LRU 的前提。 + +### A6 · 阶段 A 的测试 + +单测（`tests/unit/test_bmi_cache.cpp` 扩写，`tests/unit/test_fingerprint.cpp` 扩写）： + +| 用例 | 断言 | +|---|---| +| profile 进指纹 | 同一 manifest，optLevel/debug 不同 ⇒ `canonical_compile_flags` 串不同 ⇒ fp 不同 | +| `resolve_profile_name` 优先级 | override > default-profile > "dev" | +| `.build_cache` 往返 | 写 (triple="", profile="dev") 与 (triple="", profile="release") ⇒ **两条并存** | +| 旧格式条目 | 无 `profile=` 行 ⇒ 读出空 profile ⇒ 任何 resolved profile 都 miss | +| `entry.json` 校验 | inputs 改一个字段 ⇒ `is_cached` 为 false | +| `accessed` 更新 | `stage_into` 后 `accessed` 变新、产物 mtime 不变 | + +e2e（新文件 `tests/e2e/1NN_build_cache_correctness.sh`）： + +- **负例**：`mcpp build --dev` → `mcpp build --release`，断言 release 构建里依赖**有**编译边。 +- **负例（E8 回归闸）**：`mcpp build --release` → 裸 `mcpp build`，断言产物是 dev + （`readelf --debug-dump=info` 有 `-O0`／`DW_AT_producer` 含 `-g`），且**不是** 0.00s 假成功。 +- **负例（E6 回归闸）**：root → A{path} → B{path}，构建后断言 `mcpp cache list` 里**没有** B。 + +--- + +## 阶段 B — 换键 + +### B1 · 新模块 `src/build/cache_key.cppm` + +```cpp +export module mcpp.build.cache_key; +import std; import mcpp.libs.json; import mcpp.toolchain.detect; +import mcpp.toolchain.fingerprint; // hash_string + +export namespace mcpp::build::cache_key { +inline constexpr int kCacheEpoch = 1; // bump ONLY on cache-incompatible changes + +struct PackageKeyInputs { // 逐轴，A..G（设计 §4 S1） + // A 工具链 / B 语言方言 / C profile / D 包身份 / E 该包自身配置 / F 上游键 / G epoch + ... +}; +nlohmann::json to_json(const PackageKeyInputs&); // 进 entry.json，供逐字段校验 +std::string hex16(const PackageKeyInputs&); // 目录名 +} +``` + +`hex16` 复用 `mcpp::toolchain::hash_string`（fnv1a-64 → 16 hex），与既有指纹同族。 + +**依赖图无环性自查（改前必做）**：`mcpp.build.cache_key` 只 import `std` / +`mcpp.libs.json` / `mcpp.toolchain.{detect,fingerprint}`，均是叶或已在 `mcpp.build.plan` +的闭包内 ⇒ `mcpp.build.prepare → mcpp.build.cache_key` 不成环。 + +### B2 · 在 prepare 里自底向上算全图键 + +位置：缓存循环之前（当前 :3725 之前），拓扑序遍历 `report.topoOrder`／`packages`： + +``` +for pkgIndex in 自底向上顺序: + inputs = 收集 A..E 轴(pkgIndex) + inputs.upstream = sorted( key(dep) for dep in dependencyEdges[pkgIndex] ) // F 轴 + key[pkgIndex] = hex16(inputs) +``` + +- A/B/C 轴对全图相同，一次算出复用。 +- E 轴取 `packages[pkgIndex].manifest.buildConfig` + `privateBuild/publicUsage` 的 + include dirs。**include dirs 必须相对 xpkgs store 根归一**（它们是 + `/registry/data/xpkgs/...` 的绝对路径，不归一则 `MCPP_HOME` 一换全 miss）。 +- `sources` 清单取相对包根、排序后的相对路径（`CompileUnit::source` 按 `packageName` + 分组即可）。 +- F 轴：`dependencyEdges` 里 `consumerPackageIndex == pkgIndex` 的所有边的 + `dependencyPackageIndex` 的键，排序后拼接。**环**：modgraph 已拒环（`validate`）， + 但仍加一个访问集断言，成环时 `return std::unexpected(...)` 而不是无限递归。 + +### B3 · 缓存布局换到 `cache/v1/` + +`bmi_cache.cppm::CacheKey::dir()`： + +```cpp +- return mcppHome / "bmi" / fingerprint / "deps" / indexName / "@"; ++ return mcppHome / "cache" / "v1" / "pkg" / indexName / "@" / keyHex; +``` + +`fingerprint` 字段删除。旧 `$MCPP_HOME/bmi//deps/` **不读不写不删**，由 D 阶段的 +`doctor` 提示 + `cache clean --legacy` 处理。 + +### B4 · std BMI 换键（S3） + +`stdmod.cppm`： + +```cpp +- sm.cacheDir = cache_root / std::string(fingerprint_hex); ++ sm.cacheDir = cache_root / "std" / std_identity_key(tc, cpp_standard, cpp_standard_flag, ++ macos_deployment_target); +``` + +`ensure_built` 的 `fingerprint_hex` 形参删除（唯一调用点 `prepare.cppm:3612`）。 + +⚠️ **自指陷阱（必须处理）**：`metadata["std_build_commands"]` 里含 `sm.cacheDir` 的绝对 +路径，而 cacheDir 又由 metadata 算出。做法：先用占位 `` 生成一份"规范化 +metadata"算键，再用真 cacheDir 生成落盘的 metadata。落盘的 `metadata_matches` 逻辑不变。 + +⚠️ `cache_root` 从 `default_cache_root()`（= `home::bmi_root()` = `/bmi`）改为 +`/cache/v1`，需同步 `maintenance.cppm` / `doctor.cppm` / `clean --bmi-cache` 三处 +读点。 + +### B5 · epoch 取代 MCPP_VERSION（S5） + +`fingerprint.cppm` 的 `fp.parts[7] = MCPP_VERSION` **保持不变**（build dir 命名 + 本地增量 +归它管，保守）。缓存键的 G 轴用 `kCacheEpoch`，写进 `entry.json`；读到不同 epoch 当 miss。 + +### B6 · 阶段 B 的测试 + +单测（新 `tests/unit/test_cache_key.cpp`）： + +| 用例 | 断言 | +|---|---| +| root 身份不影响依赖键 | 只改 root 包名/版本 ⇒ 依赖包的键**不变** | +| root flags 不影响依赖键 | 只改 root `[build] cxxflags` ⇒ 依赖包的键**不变** | +| 无关兄弟依赖不影响 | 图里新增一个与之无边的包 ⇒ 原包的键**不变** | +| profile 影响 | optLevel 变 ⇒ 键变（C 轴） | +| 该包自身 flags 影响 | 该包 cflags 变 ⇒ 键变（E 轴） | +| 上游变化级联 | 上游包的 flags 变 ⇒ 下游包的键**也变**（F 轴 Merkle） | +| include dirs 归一 | 换 `MCPP_HOME` 前缀 ⇒ 键不变 | +| std 身份键 | 同 (compiler, version, triple, stdlib, std_flag, 源 hash) ⇒ 同键；任一变 ⇒ 键变；`` 占位化生效（两个不同 cacheDir 得同键） | + +--- + +## 阶段 C — 命中的依赖发 stage 边 + +### C1 · plan 侧标记 + +`plan.cppm::CompileUnit` 加： + +```cpp +bool servedFromCache = false; +std::filesystem::path cachedObject; // 缓存内绝对路径 +std::filesystem::path cachedBmi; // 缓存内绝对路径（有模块时） +``` + +prepare 的缓存循环命中时，给该包的所有 CU 打上标记并填缓存路径（不再调用 +`stage_into` 做进程内拷贝——staging 交给 ninja 边）。 + +### C2 · ninja 侧分流 + +`ninja_backend.cppm`： + +- P1689 scan／dyndep 的 CU 列表（:850-905 收集 `ddi_paths`）**过滤掉** `servedFromCache`； +- compile 边（:913-1010 两条分支，dyndep / 非 dyndep）**跳过** `servedFromCache`； +- 改发（每产物一条）： + ``` + build gcm.cache/.gcm : stage_file + verify = --verify size + build obj/.o : stage_file + verify = --verify size + ``` + `--verify size` 的判据与 std 边同源：条目目录名就是覆盖了工具链/方言/profile/包配置/ + 上游闭包的键，同键 ⇒ 等长即等价（PR#312 注释里已把这条判据写清）。 +- 消费者 TU 的 implicit input、链接行的对象列表、`bin/` 部署边**都不变**（产物落在原位置）。 +- `plan.compileUnits` 保留这些 CU ⇒ `compile_commands.cppm:127-170` 仍为依赖发条目， + clangd/IDE 不退化。 + +### C3 · `stage_into` 与 `populate_from` 的调整 + +- `stage_into` 只在 `--cache=global` 且 **C 阶段之外的调用者**（无）使用 ⇒ 删除进程内 + staging，改为返回清单供 C1 填路径。函数改名 `resolve_cached(key)`，语义变成"读清单 + + 校验，不落盘"。 +- `populate_from` 保留：未命中的包在 build 成功后回填。删掉 `touch_now` + （restat 与对齐 mtime 互斥，PR#312 的教训）。 + +### C4 · 状态行说真话 + +`cachedDepLabels` 带上省下的 unit 数：`Cached compat.zlib v1.3.2 (16 units)`。 +假状态行能骗人三个月，带计数就骗不了。 + +### C5 · 阶段 C 的测试 + +单测（`tests/unit/test_ninja_backend.cpp` 扩写）： + +| 用例 | 断言 | +|---|---| +| 命中包不发 compile 边 | `servedFromCache` 的 CU ⇒ `build.ninja` 里无其 compile 命令，有 `stage_file` 边 | +| 命中包不发 scan 边 | 无其 `.ddi` | +| `verify = --verify size` | stage 边带该绑定 | +| 未命中包不变 | 无标记 ⇒ 文本与今天逐字节一致（归一化 diff） | + +e2e（新文件 `tests/e2e/1NN_build_cache_crossproject.sh`）——**本设计的核心断言**： + +1. clean-room `MCPP_HOME`；工程 P1（含一个 index 依赖）构建 → 条目落地； +2. 工程 P2（**不同包名、不同版本**、同一依赖）构建 ⇒ 依赖的 compile 边数 **= 0**， + `stage_file` 边数 > 0。断言方式：读 `build.ninja`／`ninja -d explain` + 的结构化输出，**不 grep 全日志**； +3. P1 `mcpp version bump` 后重建 ⇒ std 与依赖**均命中**； +4. 依赖版本改变 ⇒ 键变 ⇒ 重编（正例，防"永不重建"）。 + +--- + +## 阶段 D — 模式与运维 + +### D1 · `--cache=global|local|off`（S6） + +- `cli.cppm`：`build` 加 `--cache `；`--no-cache` 保留为 deprecated 别名并修正 help + 文案（当前写的是 "Force-clear target/ before building"，名不副实）。 +- `run`/`test` 也加（今天它们连 `--no-cache` 都没有）。 +- 优先级：CLI > `MCPP_BUILD_CACHE` > `[build] cache` > `global`。 +- `manifest.cppm`：`[build] cache` 字段 + 未知值的 warning（`--strict` ⇒ error）。 +- `BuildOverrides` 加 `std::string cache_mode;`；`prepare_build` 里 global 之外**不读** + 缓存，`run_build_plan` 里 global 之外**不写**；`off` 额外 `remove_all(outputDir)`。 + +| 模式 | 读 | 写 | 先清 target/ | +|---|---|---|---| +| `global`（默认） | 是 | 是 | 否 | +| `local` | 否 | 否 | 否 | +| `off` | 否 | 否 | 是 | + +### D2 · `mcpp cache` 补齐（S7） + +`maintenance.cppm` 走新布局 `cache/v1/`，并新增： + +| 命令 | 实现要点 | +|---|---| +| `cache dir` | 打印 `home::root() / "cache" / "v1"`。今天 `cache*`/`doctor`/`clean --bmi-cache` 与 config 重置路径可能不是同一个目录，这条让它永远可验证 | +| `cache gc [--max-size N{MiB,GiB}] [--older-than N{s,m,h,d}]` | 按 `entry.json.accessed` 真 LRU（A5 已保证它被更新）。`--max-size` 从最旧开始删到达标 | +| `cache clean [--deps\|--std\|--all\|--legacy]` | 现在只删 deps 且第一行 `remove_all(bmi/"deps")` 是死代码（deps 在 `bmi//deps`）。`--legacy` 删旧 `/bmi/` | +| `cache list --json` | 结构化输出；`list` 增列 `key16` | +| `cache verify` | 逐条目校验 `entry.json` ↔ 磁盘清单，报孤儿/残缺 | + +`clean --bmi-cache` 文案改为指向 `cache clean --all`，行为保持。 +`doctor`：发现旧 `/bmi/` 时报一行大小 + `mcpp cache clean --legacy`（不自动删）。 + +### D3 · 阶段 D 的测试 + +| 用例 | 断言 | +|---|---| +| 模式优先级 | CLI > env > manifest > 默认（单测直接测解析函数） | +| `local` 不写缓存 | clean-room 下 `--cache=local` 构建后 `cache list` 为空 | +| `off` 清 target | 构建产物先存在，`--cache=off` 后 build dir 被重建 | +| `--no-cache` 别名 | 与 `--cache=off` 行为一致 | +| `gc --max-size` | 造 3 条不同 `accessed` 的条目 ⇒ 删到达标，保留最新 | +| `cache verify` | 手动删一个产物 ⇒ 报残缺并非零退出 | + +--- + +## 阶段 E — 收尾 + +1. **版本号**：`mcpp.toml:3` + `src/toolchain/fingerprint.cppm:21` → `2026.7.30.2`； + 跑 `bash .github/tools/check_version_pins.sh`。 + （bootstrap pin `.xlings.json` / `MCPP_PIN` **不动** —— 它是自举起点，不随发布走。） +2. **CHANGELOG.md**：三段——(a) 全局缓存此前净收益为零（附 ninja `-d explain` 证据）； + (b) profile 现在是失效轴，`target/` 下每 profile 一个目录、磁盘上升； + (c) 旧 `/bmi/` 一次性弃用，`mcpp cache clean --legacy` 可回收（本机 26 GB）。 +3. **docs/**：`docs/26-bmi-cache.md` 同步新布局与键定义（`bmi_cache.cppm` 头注释指向它）。 +4. **自举验证**：`mcpp build`（release profile）+ `mcpp test` 全绿； + 用**新构建出的** `mcpp` 二进制跑一遍 e2e，而不是 bootstrap 的那份。 +5. **PR**：单 PR，标题带版本与 issue 号；CI 全绿后 squash 合入。 +6. **xlings 全生态验证**：release → 镜像 xlings-res（gh + gtc 双端）→ xim-pkgindex PR → + clean-room `XLINGS_HOME` 真装 → 跑 e2e → bump bootstrap pin。 + +--- + +## 实施结果与计划的偏差（实现后回填） + +实现于 2026.7.30.2。以下是与本计划不同的地方，以及原因。 + +| 项 | 计划 | 实际 | 原因 | +|---|---|---|---| +| 缓存根 | `$MCPP_HOME/cache/v1` | **`$MCPP_HOME/build-cache/v1`** | `cache` 这个名字已归 `GlobalConfig::metaCacheDir`（索引元数据）所有，而 `config.cppm:222` 的 reset 路径 `remove_all` 整个目录。编译产物塞进别人会清空的目录里，会以「构建缓存有时会自己空掉」的形态浮现 | +| `CacheKey` 字段 | `mcppHome` + 内部拼 `cache/v1` | **`cacheRoot`（调用方传 `mcpp::home::cache_root()`）** | 布局根必须只有一份定义；在 `bmi_cache.cppm` 里再写一遍 `"build-cache"/"v1"` 正是注释管不住的跨文件不变量 | +| `stage_into` | 改名 `resolve_cached`，返回清单 | 同（且 `touch_now` 删除） | 计划一致。restat 与对齐 mtime 互斥（PR#312 的教训） | +| epoch 覆盖 std | G 轴含 `kCacheEpoch` | **std 键不含 epoch** | std 键放在 `stdmod.cppm`（与 `metadata_for` 同处，内聚更好），不引入 `stdmod → build.cache_key` 的跨层依赖。std 的格式兼容由 `build-cache/v1` 路径段承载 —— 一份 std BMI 的有效性取决于工具链，不取决于 mcpp 的缓存簿记 | +| A2 的范围 | `.build_cache` 加 profile | **另加 P2/P3 两项** | 光加字段不够：fast path 的准入只挡显式 `--profile`，挡不住裸 `mcpp build`，所以必须同时让 fast path 自己解析 profile（新 `resolve_profile_name`）。另修 `ui::finished` 的硬编码 `"release" [optimized]` | +| `finished()` 签名 | 不在计划内 | 加可选 `descriptor` 形参 | `[optimized]` 也是硬编码的；描述符由真正解析了 profile 开关的调用方给出，fast path 不给（它没解析过），所以没有调用方需要编造 | +| e2e 文件 | `1NN_*` 两个 | **172 / 173 / 174 三个** | 模式与运维命令的面足够大，值得独立文件；另改写了既有的 19（升级为传递 path 依赖的负例）与 49（改读 `entry.json` + 断言无 compile 边） | +| `--cache=off` 的语义 | 「清 target/」 | **清本次的 `target///`** | 这本来就是 `--no-cache` 的既有行为；旧 help 文案「Force-clear target/」两处不准（不是整个 target/，也与缓存无关），一并改正 | +| S8 `-ffile-prefix-map` | 阶段 3 | **未实施** | 它的收益是为内容寻址去重铺路，而本批不做内容寻址；且改变依赖对象的 debug 路径会让 gdb 需要 `set substitute-path`，是可感知的调试体验回退。留作独立批次 | +| 零拷贝 | 阶段 3 | **未实施** | 与 issue311 §7 合并，需要 GCC 的 `-fmodule-mapper` 与四平台 e2e | + +**未实施项的处置**：S8 与零拷贝均记在设计文档 §7 的后续批次里，不在本批 CHANGELOG 中承诺。 + +### 顺带发现、本批**未**处理的一个同类隐患（记录待办） + +fast path 重放 `build.ninja` 的前提是「本次请求会生成同一张图」。本批把 **profile** 与 +**cache mode** 两个轴记进了 `.build_cache` 并要求匹配，但**图形状还受若干环境变量影响， +它们都不在任何记录里**： + +| 输入 | fast path 是否能发现变化 | +|---|---| +| `--profile` / `--dev` / `--release` | ✅ 记录 + 匹配（本批） | +| `--cache` / `[build] cache` / `MCPP_BUILD_CACHE` | ✅ 记录 + 匹配（本批） | +| `--target` / `-p` / `--features` / `--cap` / `--static` / `--strict` | ✅ 显式 flag 直接绕过 fast path | +| 改 `mcpp.toml` | ✅ mtime 比 `build.ninja` 新 | +| `MACOSX_DEPLOYMENT_TARGET` | ❌ 进 fingerprint，但 fast path 只校验「记录的 fp 与目录名自洽」，不重算 fp | +| `MCPP_VERIFY_MODGRAPH` / `MCPP_SCANNER` | ❌ 改变 scan/dyndep 边的形状 | + +这三个都是**本批之前就存在**的（fast path 从来没有校验过它们），且都需要「重算一次 fp 才能 +比对」——那与 fast path 存在的意义（不跑 prepare_build）直接冲突。正解是给 fast path 一个 +**廉价的图身份**：把这些输入（以及 profile/cache mode）折进一个短哈希写进 `.build_cache`， +匹配即重放。那是一次独立的收敛，不该塞进本批。 + +判据同本批：`.build_cache` 的条目必须自证「我是在什么条件下生成的」，否则「条件变了」与 +「条件没变」在读取端不可区分 —— 这正是 profile 那个 0.00s 假成功的成因。 + +## 已知坑（来自本仓库既有教训，实现时逐条对照） + +| 坑 | 规避 | +|---|---| +| 指纹目录随版本变，`ls target// \| head -1` 会自查到旧二进制 | 测试里用 `mcpp build --print-fingerprint` 拿 fp，或对目录按 mtime 排序 | +| `! cmd \| grep` 在 `errexit` 下被豁免、永不失败 | 负例断言不要写成这种形状；用显式 `if grep -q ...; then exit 1; fi` | +| e2e `_inherit_toolchain.sh` 写穿符号链接会损坏真实 payload | 新 e2e 不 `ln -sf` 真实 payload；只用 clean-room `MCPP_HOME` + 工程内 `path` 索引 | +| 本机 `~/.mcpp` 有 26 GB / 1198 个旧指纹目录 | 所有新 e2e 用 clean-room `MCPP_HOME`，否则计时与断言全被污染 | +| 断言读全日志 grep 易假绿 | 读 `build.ninja` / `resolution.json` / `cache list --json` 等结构化载荷 | +| ninja `restat` 与对齐 mtime 互斥 | `populate_from`/`resolve_cached` 都不要 touch 产物 mtime | +| 描述符 Lua 里的跨包字面地址会随布局变化失效 | 本次不动包布局，但 `check_cross_package_refs.lua` 仍要过 | diff --git a/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md new file mode 100644 index 00000000..495d05e4 --- /dev/null +++ b/.agents/docs/2026-07-30-dep-build-cache-scoping-design.md @@ -0,0 +1,622 @@ +# 依赖构建产物的全局缓存收敛 — 设计 + +日期：2026-07-30 +状态：设计定稿，待实施 +分支/worktree：`worktree-build-cache-design` +关联：本仓库 `.agents/docs/2026-07-30-issue311-bmi-staging-and-cache-root-design.md`（**前置依赖**） +建议目标版本：阶段 1 → issue311（2026.7.30.1）之后的下一版；阶段 2 紧随 + +--- + +## 0. 一句话结论 + +mcpp **已经有**一套全局依赖 BMI/对象缓存（`$MCPP_HOME/bmi//deps/...` + `mcpp cache +list|info|prune|clean`），但它当前**净收益为零**： + +1. 缓存键用的是**全工程指纹**，任何工程只要包名/版本/无关依赖/flag 不同就完全不共享； +2. 即使命中，产物被拷进 build dir 后 **ninja 仍然全部重编**（`command line not found in + log`）——所以 CLI 打印的 `Cached ` 是**假的**； +3. 顺带一个已经存在但被 (2) 掩盖住的**正确性缺陷**：`--profile` 不进缓存键。一旦修好 (2)， + `mcpp build --release` 会直接吃到 `-O0 -g` 的依赖对象。 + +本设计把缓存键从「全工程指纹」收敛为「**每包 Merkle 键**」，把命中路径从「拷贝后祈祷 ninja +不重编」改为「**发 stage 边而不发 compile 边**」，并按硬性顺序先补齐正确性前置。 + +--- + +## 1. 现状机制（逐段核过的链条） + +### 1.1 缓存已经存在，且已接入构建 + +| 环节 | 位置 | +|---|---| +| 缓存条目布局与读写 | `src/bmi_cache.cppm`（`CacheKey` / `is_cached` / `stage_into` / `populate_from`） | +| 命中判定 + staging（prepare 阶段） | `src/build/prepare.cppm:3725-3831` | +| 回填（build 成功之后） | `src/build/execute.cppm:300-308`、`:951` | +| std BMI 缓存 | `src/toolchain/stdmod.cppm:189-303`（`ensure_built`，键 = `/`） | +| 运维命令 | `src/bmi_cache/maintenance.cppm` → `mcpp cache list/info/prune/clean` | +| 全量清理 | `src/build/execute.cppm:1118`（`mcpp clean --bmi-cache`） | + +缓存根目录：`mcpp::toolchain::default_cache_root()` = `$MCPP_HOME/bmi` +（该函数自身的三份拷贝问题由 issue311 的 S2 收敛，本设计直接复用其 `mcpp::home::bmi_root()`）。 + +### 1.2 缓存键 = 全工程指纹 + +```cpp +// src/build/prepare.cppm:3788-3798 +mcpp::bmi_cache::CacheKey key { + .mcppHome = (*cfg2)->mcppHome, + .fingerprint = fp.hex, // ← 全工程指纹 + .indexName = depIdent ? depIdent->indexName : (*cfg2)->defaultIndex, + .packageName = depName, + .version = depVer, + ... +}; +``` + +而 `fp.hex` 的第 7 个字段（`compileFlags`）是 +`canonical_compile_flags(*m) + canonical_package_build_metadata(packages)` +（`prepare.cppm:3587-3588`），后者遍历 **`packages` 全表，含 `packages[0]` = root 工程本身** +（`prepare.cppm:1363` / `2216` 建立，`267-334` 序列化 `name`/`version`/flags/include dirs/ +generated_files）。 + +std BMI 同样按 `fp.hex` 分目录（`stdmod.cppm:204`：`cache_root / fingerprint_hex`）。 + +### 1.3 命中之后 ninja 并不认账 + +`stage_into`（`bmi_cache.cppm:139-181`）把缓存里的 `.gcm/.pcm` 与 `.o` 拷进 +`target///{gcm.cache,obj}/`，并 `touch_now`。但 ninja 判脏不只看 mtime： +`build.ninja` 里这些文件仍然是 **compile 边的输出**，而 ninja 对「输出存在但 `.ninja_log` +里没有该输出的命令行记录」的判定是 **dirty**。新 build dir 天然没有 `.ninja_log`，于是 +全部重编。 + +--- + +## 2. 实测证据 + +全部在本机跑过，命令与输出可复现。 + +### E1 — 全局缓存命中却 100% 重编（决定性） + +``` +$ mcpp build # cachetest: 依赖 compat.zlib@1.3.2 + Compiling cachetest v0.1.0 (.) + Cached compat.zlib v1.3.2 ← mcpp 说命中 +[3/19] ... gcc ... -c .../zlib-1.3.2/gzclose.c -o obj/gzclose.o +[4/19] ... gcc ... -c .../zlib-1.3.2/uncompr.c -o obj/uncompr.o + ...（16 个 .c 全部重编） +``` + +ninja 自己的解释： + +``` +$ ninja -C target/x86_64-linux-gnu/ -d explain -n +ninja explain: command line not found in log for obj/zutil.o +ninja explain: obj/zutil.o is dirty +ninja explain: command line not found in log for obj/uncompr.o +ninja explain: obj/uncompr.o is dirty + ... +``` + +⇒ 缓存只在「同一个 build dir 已经有 `.ninja_log`」时"命中"，而那时本地产物本来就在。 +**全局缓存的净收益 = 0，只剩拷贝开销与磁盘占用，外加一条假的 `Cached` 状态行。** + +### E2 — 改自己的版本号就换指纹（缓存键过宽） + +``` +[package] name = "demoa", version = "0.1.0" → Fingerprint: 3b8a8ae4fc217233 + version = "0.1.1" → Fingerprint: 2138d7ce160e1154 + name = "demob" → Fingerprint: bdd6c2981d862423 +``` + +⇒ `mcpp version bump` / 改项目名 / 加一个无关依赖 ⇒ std BMI 与**全部**依赖缓存整体失效。 +两个工程只要包名不同，即使依赖集与工具链完全一致，也**一条都不共享**。 + +### E3 — 本机 26 GB 缓存的重复度 + +``` +$ du -sh ~/.mcpp/bmi → 26G +指纹目录数 → 1198 +``` + +**std 侧**（按 `std-module.json` 的 15 个身份字段归一后统计）： + +``` +std-module.json 目录数: 1014, distinct std 身份: 15, 合计 16.10 GB + 529 份 7.77 GB gcc@16.1.0 c++23 libstdc++ x86_64-linux-gnu -std=c++23 + 243 份 3.81 GB gcc@16.1.0 c++26 libstdc++ x86_64-linux-gnu -std=c++26 + 103 份 3.69 GB clang@22.1.8 c++26 libc++ x86_64-unknown-linux-gnu -std=c++26 + 71 份 0.06 GB gcc@15.1.0 c++23 ... + （其余 11 个身份合计 < 0.8 GB） +``` + +⇒ **1014 份，只有 15 个真身份。16.10 GB → ~0.5 GB（约 97% 可省）。** + +**依赖侧**： + +``` +dep 条目数 3937, distinct pkg@ver 113, 合计 8.80 GB +按 (工具链身份, pkg@ver) 归一 → 221 条, 0.52 GB （94.1% 可省） + 162 份 compat.zlib@1.3.2 + 107 份 compat.zstd@1.5.7 + 73 份 compat.x11@1.8.13 （单包合计 1.486 GB） + 63 份 mcpplibs/ftxui@6.1.9（单包合计 1.490 GB） +``` + +⇒ 合计 **26 GB → ~1 GB**。 + +### E4 — 重复的产物在语义上就是同一个（且为什么内容不相同） + +同一个 `compat.zlib` 的 `adler32.o` 在两个指纹目录下字节不同（7432 vs 7448）。唯一差异： + +``` +DW_AT_producer : GNU C11 16.1.0 -mtune=generic -march=x86-64 -g -O0 -std=c11 -fPIC （两侧完全一致） +DW_AT_name : /home/speak/.mcpp/registry/data/xpkgs/compat-x-compat.zlib/1.3.2/... （两侧完全一致） +DW_AT_comp_dir : .../scratchpad/be/tests/examples/eui-neo/target/x86_64-linux-gnu/01c3e0484b536e72 +DW_AT_comp_dir : .../scratchpad/w133/tests/examples/eui-gui-vk-glfw/target/x86_64-linux-gnu/0659ceea0b4ded58 +``` + +⇒ 语义完全等价，差异 100% 来自**消费者 build dir 的绝对路径**被烙进了 debug info。 +两个推论：(a) 内容寻址去重在今天做不了；(b) 缓存复用后 debug 信息指向另一个工程的目录。 + +### E5 — profile 不进指纹（正确性） + +``` +$ mcpp build --dev --print-fingerprint → Fingerprint: bdd6c2981d862423 +$ mcpp build --release --print-fingerprint → Fingerprint: bdd6c2981d862423 +$ mcpp build --profile dist --print-fingerprint → Fingerprint: bdd6c2981d862423 +``` + +`prepare.cppm:799-802` 把 profile 落到 `buildConfig.optLevel/debug/lto/strip`，而 +`canonical_compile_flags`（`prepare.cppm:209-265`）**只序列化 cflags/cxxflags/ldflags， +不序列化这四个字段**。三个 profile 因此共用同一个 build dir 与同一条缓存条目。 + +今天没出事，靠的是 ninja 的命令哈希在本地重编。**一旦修好 E1，`--release` 就会直接 stage +进 `-O0 -g` 的依赖对象。** 这决定了实施顺序（§5）。 + +### E6 — 传递 path 依赖被错误地写进全局缓存 + +`skipCache` 谓词（`prepare.cppm:3774-3785`）只在**root manifest** 的 +`dependencies`/`devDependencies` 里查 `isPath()/isGit()`。传递依赖查不到 ⇒ +`specIt == end()` ⇒ `skipCache = false` ⇒ 照样缓存。 + +``` +root(rootp) → A{path} → B{path} +$ mcpp build +$ find ~/.mcpp/bmi//deps -mindepth 2 -maxdepth 2 -type d +/home/speak/.mcpp/bmi/c9a9bded8f95af7e/deps/mcpplibs/B@0.1.0 ← B 被缓存了 +``` + +且 `indexName` 回落到 `defaultIndex`，本地包被**误挂到 `mcpplibs` 索引名下**。 + +接着改 `B/src/B.cppm` 的函数体、不动版本号： + +``` +$ mcpp build --print-fingerprint → Fingerprint: c9a9bded8f95af7e （不变） +$ ls -l ~/.mcpp/bmi/c9a9bded8f95af7e/deps/mcpplibs/B@0.1.0/obj/ +-rw-rw-r-- 1 speak speak 3128 ... B.m.o （旧对象仍在） +``` + +⇒ 同样被 E1 掩盖着。**修好 E1 之后这是静默错误产物。** + +本机真实缓存里已经躺着这类条目：`mcpp cache list` 输出中含 `mcpplibs/B@0.1.0` +（正是上面这个实验造出来的）以及一批 `mcpplibs/@0.1.0` 形态的条目 —— `@0.1.0` + +挂在 `mcpplibs` 名下，与本节的误判路径吻合。 + +### E7 — 用户 flag 到底影响不影响依赖（跨工程共享成立的前提） + +``` +[build] cxxflags = ["-DROOT_ONLY_FLAG=1"] + cflags = ["-DROOT_C_FLAG=1"] +``` + +``` +root TU : g++ ... -std=c++23 -fmodules -O0 -g --sysroot=... -B... -DROOT_ONLY_FLAG=1 -c src/main.cpp +dep TU : gcc ... -std=c11 -O0 -g --sysroot=... -B... -D_GNU_SOURCE -include mcpp_zlib_config.h -c .../adler32.c +``` + +⇒ **root 的 `[build]` cflags/cxxflags 不下发到依赖。** 依赖只吃：自己的 +`buildConfig`（含 feature 折叠出来的 `-DMCPP_FEATURE_*` 与 per-glob flags）、 +工程级 profile（`-O0 -g`）、工具链级 flag（`--sysroot`/`-B`/standard）。 + +这正是「不同工程引用同一个 index 包，产物应当一致」成立的机制依据。**会**影响依赖产物的轴 +只有：工具链身份、target triple、C++/C standard 与方言（含 `c++fly`）、macOS deployment +target、**profile**、该包自身激活的 features。这些全部必须进键，且都是**应该**影响的。 + +### E8 — profile 不进指纹的**第二个**受害者：fast path 直接交付错 profile 的产物 + +不只是缓存条目串号。`.build_cache` 的条目**只按 `targetTriple` 去重** +（`execute.cppm:143-145`：`erase_if(e.targetTriple == targetTriple)`），profile 不在键里； +而 fast path 的准入条件里 `ov.profile.empty()`（`cmd_build.cppm:81-83`）只挡显式 +`--profile/--dev/--release`，**挡不住裸 `mcpp build`**。于是： + +``` +$ mcpp build # 默认 dev + obj/main.o → DW_AT_producer: GNU C++23 16.1.0 ... -g -O0 -std=c++23 -fmodules ✔ dev + +$ mcpp build --release # 同一个目录 9900f2252da562b2（fp 不变） + obj/main.o → 无 debug info ✔ release + +$ mcpp build # 应当回到 dev + Finished release [optimized] in 0.00s + obj/main.o → 无 debug info ✘ 仍是 release +$ grep -oE '\-O[0-9s]' build.ninja → -O2 ✘ build.ninja 里就是 -O2 +``` + +⇒ **今天 `mcpp build --release` 之后，裸 `mcpp build` 会 0.00s "成功"并交付 release 产物。** +这是一个独立于缓存的现存 bug，与 C3 同根（profile 不是失效轴），修 C3 时必须一起修，否则 +把 fp 分家只是把"同一个目录里内容是错的"换成"指向另一个 profile 的目录"。 + +### E9 — GCC 把被导入模块的 BMI CRC 烙进导入者的 BMI（决定 §4 S1 的 F 轴形态） + +手工三组对照（`g++ 16.1.0 -std=c++23 -fmodules`，`A` 导入 `B`，只重编 `B`、保留旧 `A.gcm`）： + +| 改动 | 结果 | +|---|---| +| 只改 `bfn()` **函数体**（接口逐字节不变） | 旧 `A.gcm` + 新 `B.gcm` **编译通过** | +| 改 `bfn()` 返回类型 + 新增导出类型（**接口**变） | **硬失败**（下方输出） | +| `-DWIDE` 改变导出结构体布局（**ABI** 变，源码字节不变） | **硬失败**（同下） | + +``` +B: error: module 'B' CRC mismatch +B: error: failed to read compiled module: Bad file data +A: error: failed to read compiled module: Bad import dependency +A: fatal error: returning to the gate for a mechanical issue +``` + +两个推论： + +1. `A` 的 BMI 与它当初读到的那一份 `B` 的 BMI 是**硬绑定**的 —— 所以 `A` 的缓存键必须包含 + 「`B` 的 BMI 身份」，而不是「`B` 的 public 接口的某个枚举摘要」。 +2. **失败模态不对称**：BMI 轴上键取窄了 ⇒ GCC CRC 硬报错（吵、但不出错产物）； + `.o` 轴上键取窄了 ⇒ **静默错对象**（无任何校验）。设计必须按后者取保守侧。 + +### E10 — 顺带发现的两处小问题 + +- `mcpp cache prune --older-than` 用 `last_write_time(entry.dir)`，而 `stage_into` 命中时 + **不更新缓存目录的时间**（只写消费者 build dir）。所以它是 "last **populated**"，不是 LRU： + 一个天天命中的热包会被当成冷条目剪掉。 +- `cache_clean()`（`maintenance.cppm:167-175`）第一行 `remove_all(bmi / "deps")` 指向一个 + 不存在的路径（deps 在 `bmi//deps`），是死代码；真正生效的是后面的循环。 + +--- + +## 3. 缺陷编号汇总 + +| 编号 | 缺陷 | 层级 | 证据 | +|---|---|---|---| +| **C1** | 命中的依赖产物 ninja 一律重编（`command line not found in log`），全局缓存净收益为零，且 UI 谎报 `Cached` | 构建后端 | E1 | +| **C2** | 缓存键 = 全工程指纹（含 root 包名/版本、全图 flags）⇒ 跨工程零共享、改自己版本号即全失效 | 键 | E2 E3 | +| **C3** | profile（`-O`/`-g`/lto/strip）不进指纹 ⇒ dev/release/dist 共用同一个 build dir 与同一条缓存条目 | 正确性 | E5 | +| **C3b** | `.build_cache` 条目只按 `targetTriple` 去重，fast path 的准入只挡显式 `--profile/--dev/--release` ⇒ `--release` 之后裸 `mcpp build` 0.00s "成功"并交付 release 产物。**这是今天就在生效的 bug**，与 C3 同根 | 正确性（独立于缓存） | E8 | +| **C4** | 传递 path/git 依赖被写进全局缓存，且源码变更不改键 | 正确性 | E6 | +| **C5** | 缓存条目无自描述元数据（只有 `manifest.txt` 文件清单），`is_cached` 是"存在即命中"，无法校验、无法审计 | 键 | 代码 | +| **C6** | 依赖对象里烙进消费者 build dir 路径（`DW_AT_comp_dir`）⇒ 无法内容寻址去重、debug 信息指向别的工程 | 可复现性 | E4 | +| **C7** | 整个 `MCPP_VERSION` 进指纹 ⇒ 每次 mcpp 发版把全部缓存作废（含纯 C 目标文件） | 键 | 代码 + E3 的 1198 目录 | +| **C8** | `cache prune` 是 last-populated 而非 LRU；无容量上限；`cache clean` 有一行死代码；无 `cache dir` | 运维 | E8 | + +C3/C4 与 C1 的关系是硬约束：**C1 修好之前它们是无害的浪费，修好之后是静默错误产物。** + +--- + +## 4. 设计 + +### S1 — `PackageBuildKey`：每包 Merkle 键（治 C2 / C3 / C5） + +新建叶模块 `src/build/cache_key.cppm`（`export module mcpp.build.cache_key;`），从 +`BuildPlan` + `PackageRoot` 计算每个包**自己**的键。`CompileUnit` 已经带 +`packageName` 与 per-package 的 `packageCflags/packageCxxflags/localIncludeDirs` +（`plan.cppm:20-36`），**每包粒度的数据已经在 plan 里了**，不需要新的解析。 + +键的输入，按轴分组（序列化格式固定、有序、带字段前缀，与 `canonical_*` 现有风格一致）： + +| 轴 | 字段 | +|---|---| +| A 工具链 | `compiler_id`, `compiler_version`, `driver_identity`, `target_triple`, `stdlib_id`, `stdlib_version` | +| B 语言/方言 | `cpp_standard`, `std_flag`, `dialect_flags`（含 `c++fly` 解析结果）, `c_standard`, `macos_deployment_target` | +| C profile ← **C3 的修复** | `opt_level`, `debug`, `lto`, `strip` | +| D 包身份 | `index_name`, `package_fqn`, `package_version` | +| E 该包自身的有效构建配置 | `sorted(active_features)`, `cflags`, `cxxflags`, `asmflags`, `ldflags`, `globFlags`（有序全量，同 `prepare.cppm:301-307`）, `defines`, `generated_files(path=content)`, `include_dirs`（**相对 xpkgs store 根归一后**）, `sources`（相对包根、有序） | +| F 上游接口闭包 | `sorted( 每个直接依赖 → 它的 PackageBuildKey )`，递归 ⇒ Merkle | +| G BMI 纪元 | `mcpp_bmi_epoch`（见 S5，**不是**完整 `MCPP_VERSION`） | + +**不含**：root 包名/版本、root 的 `[build]` flags（E7 已证不下发）、图里无关的兄弟依赖、 +消费者的 build dir 路径。 + +**F 轴为什么必须是递归键，而不是「枚举上游的 public include dirs + interface defines」：** + +1. **BMI 是硬绑定的，不是"接口等价即可"。** E9 实测：GCC 把被导入模块 BMI 的 CRC 写进导入者 + 的 BMI，接口或 ABI 一变就 `module 'B' CRC mismatch` + `Bad import dependency`。所以 + `A` 的键需要的是「`B` 的 BMI 身份」本身，而 `B` 的 BMI 身份 = `B` 的完整键。 +2. **可枚举清单必然漏项。** 要枚举的是 `B` 的 public include dirs 的**内容**、interface + defines、`generated_files`、以及 `B` **re-export** 出去的传递模块接口 —— 最后这项从 `B` + 的 manifest 里根本看不出来（它是 `R` 的接口，经 `B` 转出）。少一项 = 命中一条 BMI + 加载不了的条目。 +3. **失败模态不对称（E9 的第二个推论）**：BMI 轴取窄 ⇒ CRC 硬报错（吵）；`.o` 轴取窄 ⇒ + 静默错对象（哑，`.o` 没有任何自校验）。设计必须按后者取保守侧。 + +递归的真实代价**不是**「上游换 profile 级联下游」—— profile 是 C 轴，它本来就在**每一个** +包的键里，有没有 F 轴都会让全图 key 变。递归的真实代价是：**上游的私有输入变化会级联下游** +（例如 `B` 给自己加一个 `-DB_INTERNAL_LOG=1`，或改一个不提供模块的私有源文件，`B` 的接口 +逐字节没变，但 `A` 的键跟着变、`A` 白重编一次）。 + +而这个代价在真正吃缓存的人群里 ≈ 0：**index 包的描述符按版本冻结**，`B` 的 buildConfig 不 +可能不 bump 版本就变（仓库里 #253 的注释依赖的正是这条不变量），所以 `B` 的键只能因 +「版本 bump」或「全图轴（工具链/standard/profile）」而变 —— 而版本 bump 本来就该让 `A` +失效（`A` 要链接 `B` 的对象）。唯一能不改版本就动 `B` 私有 flag 的是 path/git 包，而它们 +已被 S9 整体排除出缓存。 + +⇒ 递归是唯一可靠的形态，且在目标人群里几乎不付代价。 + +拓扑序已由 `report.topoOrder`（`prepare.cppm:3561`）给出，自底向上一次遍历即可算完全图键。 + +### S2 — 缓存布局与自描述条目（治 C5） + +> **实施时的修正**：根目录不是 `$MCPP_HOME/cache/v1` 而是 +> **`$MCPP_HOME/build-cache/v1`** —— `cache` 这个名字已归 `GlobalConfig::metaCacheDir` +> （索引元数据）所有，而它的 reset 路径 `remove_all` 整个目录。其余布局如下不变。 +> 完整偏差清单见实施计划的「实施结果与计划的偏差」。 + +``` +$MCPP_HOME/build-cache/v1/ + pkg//@// + entry.json ← 自描述：键的全部输入 + 文件清单 + created/accessed + bmi/.{gcm,pcm} + obj/.o + std// + std-module.json （沿用现有 14+1 字段元数据） + {gcm,pcm}.cache/std*.{gcm,pcm} + std*.o +``` + +`entry.json` 是把 std 侧已经做对的事（`stdmod.cppm:88-145` 的 `metadata_for` + +`metadata_matches`）搬到依赖侧：**命中判定从「manifest.txt 存在且文件都在」升级为 +「键匹配 ∧ `entry.json` 的输入逐字段等于本次算出的输入 ∧ 清单文件齐全」**。 + +`v1/` 前缀 + 新目录名意味着老的 `$MCPP_HOME/bmi//` 一次性弃用。**不自动删**： +`mcpp doctor` 报一行 "legacy BMI cache at ~/.mcpp/bmi (26.0 GiB), safe to delete: +`mcpp cache clean --legacy`"。理由：E1 证明这 26 GB 从来没产生过收益，没有任何保留价值， +但删 26 GB 必须是用户的显式动作。 + +### S3 — std BMI 换键（治 C2 的 std 侧，收益最大、改动最小） + +`stdmod.cppm:204`： + +```cpp +- sm.cacheDir = cache_root / std::string(fingerprint_hex); ++ sm.cacheDir = cache_root / "std" / std_identity_key(metadata); +``` + +`metadata_for(...)` 已经是**正确且完整**的 std 身份（compiler/version/driver_identity/ +triple/stdlib/std_flag/源文件 hash/构建命令），`metadata_matches` 已经在按它校验。今天唯一 +的错误就是**目录名用的不是它**。 + +**实现陷阱（必须处理）**：`metadata["std_build_commands"]` 里含 `sm.cacheDir` 的绝对路径， +而 `cacheDir` 又要由这个 metadata 算出来 ⇒ 自指。算键前必须把 cacheDir 出现处替换成占位符 +``（§2 的 E3 统计脚本正是这么归一才得出「15 个身份」的）。`ensure_built` 的签名 +里 `fingerprint_hex` 参数随之删除 —— 它是唯一的调用点（`prepare.cppm:3612`）。 + +预期效果：1014 目录 / 16.10 GB → 15 条 / ~0.5 GB。`mcpp version bump` 不再触发 std 重编 +（当前约 10–60 s + 31 MB 写盘）。 + +### S4 — 命中的依赖发 stage 边，而不是 compile 边（治 C1） + +`CompileUnit` 加一个字段： + +```cpp +bool servedFromCache = false; // 命中全局缓存，产物由 stage 边提供 +std::filesystem::path cachedObject; // 缓存内绝对路径 +std::filesystem::path cachedBmi; // 缓存内绝对路径（有模块时） +``` + +`ninja_backend` 对 `servedFromCache` 的 CU： + +- **不发** compile 边（`ninja_backend.cppm:913-1010`）； +- **不发** P1689 scan / dyndep 边（`:850-905`）—— 依赖的模块不需要重新扫，BMI 已在缓存里； +- **改发** 每产物一条 `stage_file` 边，`$in` = 缓存内绝对路径，`$out` = build dir 内既有位置： + ``` + build gcm.cache/.gcm : stage_file /bmi/.gcm + build obj/.o : stage_file /obj/.o + ``` + +这样 build dir 的**产物位置、链接行、消费者 TU 的 implicit input 全部不变**，改变的只是 +「这些文件由谁产生」。ninja 拿到 stage 边的命令行记录，判脏语义从此自洽。 + +**直接复用 issue311 的 `mcpp stage` 原语 + `restat = 1` + rule 更名 `cp_bmi → stage_file`**： +它已经定义了「size/hash 相等则一个字节都不写、尽力对齐 mtime、原子 rename、退避重试、 +结构化诊断」。有了 `restat = 1`，第二次起 stage 边是 no-op 且**不级联下游重编**。这就是把 +本设计排在 issue311 之后的原因 —— 两者需要的是同一个原语。 + +`plan.compileUnits` **保留** 这些 CU（只是打了标记），因此 +`compile_commands.json`（`compile_commands.cppm:127-170`）仍然为依赖发条目，clangd/IDE +不退化。CDB 生成器忽略 `servedFromCache`。 + +`ctx.cachedDepLabels` 的 `Cached` 状态行从此变成真话；并且应该改成打印**省下的 TU 数**， +例如 `Cached compat.zlib v1.3.2 (16 units)` —— 假状态行能骗人三个月，带计数就骗不了。 + +### S5 — 收窄失效轴（治 C7） + +`fingerprint.cppm` 的 `fp.parts[7] = MCPP_VERSION` 保持不变（build dir 命名与本地增量归它 +管，宁可保守）。但**缓存键的 G 轴换成 `mcpp_bmi_epoch`**：一个手动递增的整型常量，只在 +BMI 编码/staging 语义/键计算方式发生不兼容变化时 +1，与发版号解耦。 + +```cpp +// src/build/cache_key.cppm +inline constexpr int kCacheEpoch = 1; // bump ONLY on cache-incompatible changes +``` + +同时进 `check_version_pins.sh` 的机器校验清单？**不进** —— 它不是 pin，不存在跨文件不变量， +只需在 `entry.json` 里记下 epoch，读到不同 epoch 的条目当 miss 处理即可（自愈）。 + +### S6 — 三种构建模式（用户面） + +一个正交开关承载三态： + +``` +mcpp build # = --cache=global（默认） +mcpp build --cache=local # 依赖全部在本工程 target/ 内编，不读不写全局缓存 +mcpp build --cache=off # 清空 target/ 全量重编，且不读不写全局缓存 +mcpp build --no-cache # deprecated alias of --cache=off（help 文案修正） +``` + +优先级：CLI `--cache` > 环境变量 `MCPP_BUILD_CACHE` > `[build] cache = "global|local|off"` +> 默认 `global`。`mcpp run` / `mcpp test` 透传同一开关（`run`/`test` 目前连 `--no-cache` +都没有 —— 顺手补齐，`cmd_build.cppm:43` 是唯一读取点）。 + +语义表： + +| 模式 | 读全局缓存 | 写全局缓存 | 先清 `target/` | +|---|---|---|---| +| `global`（默认） | 是 | 是 | 否 | +| `local` | 否 | 否 | 否 | +| `off` | 否 | 否 | **是** | + +`local` 存在的意义：排障时把「是不是缓存的问题」一次性排除掉，以及给 CI 一个可复现的 +无共享状态基线。`off` 保留今天 `--no-cache` 的全部行为。 + +### S7 — 运维命令补齐（治 C8） + +`mcpp cache` 已经是正确的归属（不需要放到 `mcpp self` 下）。新增/修正： + +| 命令 | 说明 | +|---|---| +| `mcpp cache dir` | 打印缓存根绝对路径。今天 `cache *`/`doctor`/`clean --bmi-cache` 用 `default_cache_root()` 而 `config.cppm` 的重置路径用 `cfg.bmiCacheDir`，两者可能不是同一个目录（issue311 D2）；有了这条命令，"到底在哪"永远可验证 | +| `mcpp cache gc [--max-size {MiB,GiB}] [--older-than {s,m,h,d}]` | 真 LRU。前提：**`stage_into` 命中时必须 touch 条目**（写 `entry.json` 的 `accessed` 字段，不动产物 mtime）。默认上限建议 `[cache] max_size`（config），未设则不设限只做 `--older-than` | +| `mcpp cache clean [--deps\|--std\|--all\|--legacy]` | 今天只删 deps 且第一行是死代码。`--legacy` 删旧的 `$MCPP_HOME/bmi/` | +| `mcpp cache list --json` | 让 CI / 工具可消费；顺手把 `list` 的 `key16` 与 `entry.json` 摘要打出来 | +| `mcpp cache verify` | 逐条目校验 `entry.json` ↔ 磁盘清单，报孤儿/残缺条目（承接 `.mcpp_ok` 内容盲区那类问题） | + +`mcpp clean --bmi-cache` 的文案改为指向 `mcpp cache clean --all`，行为保持。 + +### S8 — 依赖产物路径归一（治 C6，可与 S1–S7 并行） + +给**依赖的** compile 边加： + +``` +-ffile-prefix-map==/mcpp/build +-ffile-prefix-map==/mcpp/store +``` + +（GCC ≥ 8 / Clang ≥ 10 支持 `-ffile-prefix-map`；MSVC 无对应项 ⇒ 该轴在 MSVC 上跳过， +不作为硬要求。） + +收益：依赖对象变得与消费者工程无关 ⇒ (a) debug 信息不再指向别人的工程目录；(b) 为后续 +内容寻址去重/硬链接共享打开门。**不作为 C1/C2 的前置**，因为键已经保证语义等价。 + +### S9 — 排除本地来源的包（治 C4） + +`skipCache` 谓词从「查 root manifest 的 dependencies」改为**按 `PackageRoot` 的来源判定**： + +> 包根路径不在 registry/xpkgs store 之下（`cfg.mcppHome/registry/data/xpkgs/...`）的一律 +> 不缓存 —— 不管它是 root 的直接 path 依赖、传递 path 依赖、workspace 成员，还是 git 依赖 +> 的 checkout。 + +判据方向与 `mcpp add` 存在性门的教训相反：那里是「不可证伪就放行」，这里是 +「**不能证明它来自不可变的 store 就不缓存**」。缓存的错误代价是静默错产物，必须取保守侧。 + +顺带修 `indexName` 的回落：`depIdent` 缺失时不要回落到 `defaultIndex`（E6 里把本地包 `B` +误挂到 `mcpplibs` 名下），本地包在 S9 之后根本不会进缓存，回落分支应该改成"进不了缓存"的 +显式路径。 + +--- + +## 5. 实施顺序（硬约束） + +``` +issue311（2026.7.30.1，已设计） + └─ mcpp stage 原语 + restat=1 + rule 更名 + mcpp.home 收敛 + ↓ 复用 + +阶段 1 · 正确性前置 —— 必须先于阶段 2 合入 + P1 profile 进失效轴：本阶段用**最小改法**——在 canonical_compile_flags 里补序列化 + optLevel/debug/lto/strip（4 行），让现有 fp.hex 立刻区分 dev/release/dist。 + 阶段 2 把它作为 S1 的 C 轴迁到每包键上；两处不是重复实现，是同一语义的迁移。 + 副作用（可接受、须写 CHANGELOG）：三个 profile 从此各占一个 target/// + 目录，互不覆盖 —— 这本来就是 cargo 的行为。代价是磁盘 ×profile 数（每个 build dir + 带一份 staged std BMI，本机 std.gcm = 31,466,112 字节）。 + P2 C3b：.build_cache 条目按 (targetTriple, **resolved profile**) 去重，profile 不匹配 + 即当 miss 走 prepare_build。**只做 P1 不做 P2 是不够的**：fp 分家只会把"同一个目录 + 里内容是错的"换成"fast path 指向另一个 profile 的目录"，E8 的洞照旧。 + 顺手修 run_build_plan 里硬编码的 ui::finished("release", ...)——它在 dev 构建下 + 也打印 "Finished release [optimized]"，是 E8 里最误导人的一行。 + P3 fingerprint changed (X → Y), full rebuild 警告（execute.cppm:310-324）会在每次切 + profile 时触发。改成能区分"仅 profile 变"的文案，否则它从有用信号退化成噪音。 + S9 本地来源包（含传递 path/git、workspace 成员）排除出缓存 + indexName 回落修正 + S2 entry.json 自描述 + 命中时逐字段校验（取代"存在即命中"） + e2e 负例：dev 构建后 --release，必须重编而不是复用；改 path dep 源码不改版本号，必须重编 + +阶段 2 · 性能 + S1 缓存键：全工程指纹 → 每包 Merkle 键（新模块 mcpp.build.cache_key） + S3 std BMI 键：fp.hex → std-module.json 身份键（注意 占位化） + S4 命中的依赖发 stage 边、不发 compile/scan 边（复用 mcpp stage + restat） + S5 缓存失效轴：MCPP_VERSION → kCacheEpoch + S6 --cache=global|local|off + [build] cache + MCPP_BUILD_CACHE + e2e 正例：两个不同包名的工程依赖同一个 index 包 ⇒ 第二个工程 0 条依赖编译边 + e2e 正例：mcpp version bump ⇒ std 与依赖均不重编 + +阶段 3 · 后续（不在本设计的实施范围内，只定方向） + S7 cache dir / gc / clean --std|--all|--legacy / list --json / verify + S8 -ffile-prefix-map 归一 + 零拷贝 直接 -fmodule-file= / 绝对路径 .o 进链接行，取消 staging + （与 issue311 §7 合并，GCC 的 gcm.cache 相对查找需要 -fmodule-mapper） +``` + +**为什么阶段 1 不能与阶段 2 并肩或落后**：阶段 2 的 S4 让 ninja 第一次真正接受缓存产物。 +在 C3/C4 未修的前提下，这等于把「白白重编一遍（结果正确）」变成「直接用错对象（结果错误）」。 +E5/E6 两条实测就是这两颗雷的引信。 + +--- + +## 6. 不采纳 / 需要纠正的方案 + +| 方案 | 判定 | 理由 | +|---|---|---| +| 给 staging 输出加 `restat = 1` 就能让 ninja 跳过 | **不成立** | `restat` 只在**有** log entry 时改写 mtime 判定。E1 的 ninja 解释是 `command line not found in log`，即根本没有 entry ⇒ 直接 dirty。必须让缓存产物成为**某条边的输出**（S4） | +| `generator = 1` 标记依赖的 compile 边（ninja 对 generator 边跳过命令哈希检查） | **拒绝** | 会把「命令行变了要重编」这条规则整体关掉，源码/flag 改动也不再重编。用错误换性能 | +| 缓存内容寻址（`obj/.o`），天然去重 | **本批不采纳** | E4 证明今天同语义对象字节不同（`DW_AT_comp_dir`）。要先做 S8 才有意义；且键已经保证语义等价，内容寻址只多省磁盘不多省时间 | +| 把 `dependencyLockHash`（`fingerprint.cppm` 第 9 字段，今天恒为空）填上，让指纹更精确 | **方向相反** | 那会让缓存键更宽（锁文件里任何一个无关依赖变化都失效）。本设计要的是**更窄且更准**的每包键。第 9 字段留给 build dir 命名用 | +| 依赖用硬链接而非拷贝 stage 进 build dir | **不采纳** | build dir 里的文件一旦被判脏重写就会**写穿**到缓存（本仓库已有 `_inherit_toolchain.sh` 写穿符号链接损坏真实工具链的先例）。reflink（`FICLONE`）可以，但只在 btrfs/xfs 且同设备 ⇒ 作为 S4 里的可选优化，默认走 issue311 `mcpp stage` 的拷贝 | +| 把缓存清理放到 `mcpp self` 下 | **拒绝** | `mcpp cache` 已经存在且是正确归属（`cli.cppm:391-410`）。`self` 是 mcpp 自身的安装/升级面 | +| 直接删掉旧的 `$MCPP_HOME/bmi/`（26 GB） | **不自动删** | 内容确实无价值（E1），但删 26 GB 必须是显式动作：doctor 提示 + `mcpp cache clean --legacy` | + +--- + +## 7. 风险与验证 + +| 风险 | 缓解 | +|---|---| +| Merkle 键漏掉某个真实影响产物的轴 ⇒ 复用错对象 | 键的输入全量写进 `entry.json`，命中时**逐字段比对**（不只比 hash）。字段不匹配 = miss + 重建，永不"信任 hash 相等"。这是 std 侧 `metadata_matches` 已验证过的模式 | +| 跳过 scan/dyndep 边后，消费者 TU 的 dyndep 找不到依赖模块的 BMI | BMI 由 stage 边落在原位置（`gcm.cache/.gcm`），消费者的 implicit input 不变。e2e 必须覆盖「消费者 `import` 一个命中缓存的模块依赖」这条正例 —— 这是 S4 最可能出问题的地方 | +| `restat = 1` 引入「永不重建」 | 单测锁 `build.ninja` 文本；e2e 正例：改依赖版本 ⇒ 键变 ⇒ 新条目 ⇒ 重建 | +| 阶段 1 落地后老缓存全 miss，用户体感"变慢了一次" | 新布局 `cache/v1/` 与旧 `bmi//` 无关，本来就是全 miss；CHANGELOG 写明一次性重建，doctor 提示旧目录可删 | +| workspace 场景：成员之间互为 path 依赖 | S9 把它们全部排除出缓存（正确：成员源码随时可改）。`--workspace` / `-p ` 的 e2e 必须断言成员产物**不**进全局缓存 | +| MSVC 无 `-ffile-prefix-map` | S8 在 MSVC 上跳过；不影响 S1–S7 | +| 多进程并发写同一条目 | 沿用 `populate_from` 已有的 `FileLock::try_acquire`（`bmi_cache.cppm:192`，拿不到锁即视为成功让给对方）。新增 `entry.json` 同样最后写、temp+rename 作 sentinel | + +必须新增的验证（e2e）： + +- **跨工程共享（正例，本设计的核心断言）**：两个不同包名/不同版本的工程依赖同一个 index 包， + 第二个工程构建时依赖的 compile 边数 = 0。用结构化载荷断言（读 `build.ninja` 或 + `-d explain`），**不要 grep 全日志**。 +- **版本 bump 不失效（正例）**：`mcpp version bump` 后 std BMI 与依赖条目均命中。 +- **profile 隔离（负例，阶段 1 必须先绿）**：`--dev` 构建后 `--release`，依赖必须重编。 +- **path dep 源码变更（负例，阶段 1 必须先绿）**：改传递 path 依赖的源码不改版本号， + 必须重编，且该包**不出现**在 `mcpp cache list` 里。 +- **假 `Cached` 回归锁**：状态行打印的 unit 数与实际跳过的边数一致。 + +本地复现注意（已知坑，沿用既有教训）：e2e 必须统一 `MCPP_HOME`；测本设计时**不要**用本机 +`~/.mcpp`（26 GB 旧缓存 + 1198 个指纹目录会干扰计时与断言），用 clean-room `MCPP_HOME`； +`! cmd | grep` 在 `errexit` 下被豁免、永不失败，负例断言不要写成这种形状。 + +--- + +## 8. 预期收益 + +| 维度 | 现状 | 本设计后 | +|---|---|---| +| 全局缓存对构建时间的贡献 | **0**（E1：命中也全编） | 依赖 TU 从"每工程编一次"降为"每 (工具链×profile×features) 编一次" | +| 本机缓存体积 | 26 GB / 1198 指纹目录 | ~1 GB（std 16.10→~0.5 GB，deps 8.80→~0.5 GB） | +| `mcpp version bump` 之后 | std 重编（10–60 s）+ 全部依赖重编 | 全部命中 | +| 新工程首次构建（依赖已被别的工程编过） | 全量编依赖 | 只 stage（数十 MB 拷贝，`restat` 后二次为 no-op） | +| `--release` 复用 `--dev` 的依赖对象 | 今天靠 ninja 侥幸不出错；S4 之后会出错 | 键隔离，不可能发生 | +| 缓存可审计性 | 只有文件清单 | `entry.json` 自描述 + `cache verify` + `cache dir` | diff --git a/CHANGELOG.md b/CHANGELOG.md index b1335bed..df0beb7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,74 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.7.30.2] — 2026-07-30 + +### 修复 + +- **依赖的全局缓存此前净收益为零 —— 命中也全量重编。** 命中的依赖产物由 `prepare_build` 直接拷进 build dir,而这些路径在 `build.ninja` 里仍然是 **compile edge 的输出**;ninja 对「输出存在但 `.ninja_log` 里没有该输出的命令行记录」一律判脏,而新 build dir 天然没有 `.ninja_log`。ninja 自己的话: + + ``` + $ ninja -C target/x86_64-linux-gnu/ -d explain -n + ninja explain: command line not found in log for obj/zutil.o + ninja explain: obj/zutil.o is dirty + ``` + + 于是每一个「命中」的 TU 都被重编一遍,而 CLI 照旧打印 `Cached ` —— 缓存只在「同一个 build dir 已经有 `.ninja_log`」时"生效",而那时本地产物本来就在。 + + 修法不是加 `restat`(log entry 根本不存在,restat 只在有 entry 时改写 mtime 判定),也不是给 compile edge 打 `generator = 1`(那会把「命令行变了要重编」整条规则关掉,用错误换性能)。命中的依赖**改发 `stage_file` 边、不发 compile 边、不发 scan/dyndep 边**:产物落在 compile edge 本来会产生的同一个路径上,所以链接行、消费者 TU 的 BMI implicit input、运行期部署边全都不变,变的只是「这些文件由谁产生」—— 而 ninja 从此有了可比对的命令行记录。 + + 状态行同时带上省下的 TU 数(`Cached compat.zlib v1.3.2 (15 units)`):光秃秃一个 `Cached` 能骗人三个月,一个必须与 ninja 实际跳过的边数吻合的数字骗不了。 + +- **`mcpp build --release` 之后裸 `mcpp build` 会 0.00s「成功」并交付 release 产物。** 与缓存无关的独立缺陷,同根于下面的 profile 轴:`.build_cache` 的条目**只按 target triple 去重**,而 fast path 的准入条件只挡显式 `--profile/--dev/--release`,挡不住裸 `mcpp build`(那恰恰是「我要默认 profile」的意思)。于是 fast path 拿着 release 的 `build.ninja` 跑,`build.ninja` 里写着 `-O2`,产物无 debug info。 + + 条目改按 `(target triple, resolved profile)` 去重,fast path 先用同一条纯规则(新的 `resolve_profile_name`,manifest 之外无输入)算出本次请求的 profile 再匹配;不匹配即当 miss 走完整 `prepare_build`。旧条目没有该字段 ⇒ 读成空 ⇒ 任何 profile 都不匹配 ⇒ 自愈,无需迁移。 + +- **`Finished release [optimized]` 是硬编码的。** 唯一调用点传字面量 `"release"`,`ui::finished` 又硬编码 `[optimized]`,所以 `--dev`(`-O0 -g`)也自称优化过的 release 构建。现在 profile 名来自本次真正解析出的 profile,描述符来自本次真正解析出的开关(`unoptimized + debuginfo` 等),没有调用方需要猜。 + +- **传递的 `path` / `git` 依赖被写进全局缓存。** 排除谓词在 **root manifest** 的 `dependencies`/`dev-dependencies` 里查该包并在 spec 为 path/git 时跳过 —— 但传递到达的包两个表里都没有,于是 `specIt == end()` 让谓词返回「不跳过」,本地源码被缓存,且 `indexName` 回落到默认索引(一个 workspace 成员 `B` 于是以 `mcpplibs/B@0.1.0` 的身份落在磁盘上)。它的源码可以在 `name@version` 不变的情况下改变,缓存键看不见这种变化。 + + 唯一可采信的证据改为**解析期记录的来源**(`DepCacheIdentity::sourceKind`,两个 push 点都在作用域内拿得到):非 `"version"` 一律不进缓存。判据方向与 `mcpp add` 的存在性门**相反** —— 那里「不可证伪就放行」,缓存这里「不能证明来自不可变的 xpkgs store 就不缓存」,因为这里的错误代价是静默的错误对象,不是被拒绝的命令。 + + 此前这个缺陷被上面第一条掩盖着(ninja 反正会重编);修好第一条之后它就是静默错误产物,所以两者必须同批。 + +### 变更 + +- **依赖缓存键:全工程指纹 → 每包 Merkle 键。** 旧键是整个工程的 fingerprint,而它的 flags 字段序列化了图里**每一个包,含 root** —— root 的包名、版本、`[build]` flags 都在里面。后果:改自己的版本号(`0.1.0` → `0.1.1`,实测 `3b8a8ae4fc217233` → `2138d7ce160e1154`)就让 std BMI 与**全部**依赖缓存整体失效;两个工程只要包名不同,即使依赖集与工具链完全一致,也一条都不共享。本机实测:26 GB / 1198 个指纹目录,`compat.zlib@1.3.2` 存了 162 份,`compat.x11@1.8.13` 73 份共 1.49 GB。 + + 新模块 `mcpp.build.cache_key` 按七个轴逐包成键:工具链身份 / 语言与方言 / **profile** / 包身份 / 该包自身的构建配置(含 features 折叠后的结果)/ **每个直接依赖的键(递归,Merkle)** / cache epoch。不含 root 的身份与 flags —— 实测 root 的 `[build] cflags/cxxflags` **不会**下发到依赖 TU,这正是跨工程共享成立的机制依据。 + + F 轴之所以递归而不是「枚举上游的 public 接口」:GCC 把被导入模块 BMI 的 CRC 烙进导入者的 BMI,上游接口或 ABI 一变,旧的导入者 BMI 就 `module 'B' CRC mismatch` + `Bad import dependency`(手工三组对照实测:只改函数体通过,改签名/`-D` 改布局都硬失败)。可枚举清单必然漏项(上游 re-export 出去的传递模块接口在上游自己的 manifest 里根本看不出来),而失败模态是不对称的 —— BMI 轴取窄是编译器硬报错,`.o` 轴取窄是静默错对象。 + + 递归的代价是上游的**私有**变化会级联下游;对真正吃这个缓存的人群 ≈ 0:index 包的描述符按版本冻结,`B` 的键只能因版本 bump(那本就该让消费者失效 —— 它链接 `B` 的对象)或全图轴而变,而唯一能不改版本就动私有 flag 的 path/git 包整体不进缓存。 + +- **std 模块缓存键:全工程指纹 → std 自己的身份。** `stdmod.cppm` 写在产物旁边的 15 字段 metadata 一直就是正确完整的 std 身份,`metadata_matches` 一直在按它校验命中 —— 唯一错的是**目录名用的不是它**。本机实测代价:1014 个目录里只有 **15 个真身份**,16.10 GB(需要 ~0.5 GB),且 `mcpp version bump` 会重新 precompile 一次 std(几十 MB、十几秒)。(实现上有个自指要解:build commands 里含 cacheDir,而 cacheDir 由含它的 metadata 算出 —— 先用占位段生成一份规范化 metadata 算键,再用真目录生成落盘的那份。) + +- **缓存布局迁到 `$MCPP_HOME/build-cache/v1/`,条目自描述。** `pkg//@//` 与 `std//`。刻意不放 `$MCPP_HOME/cache` —— 那个名字已经归 `GlobalConfig` 的索引元数据缓存所有,而它的 reset 路径会整目录删除;把编译产物塞进别人会清空的目录里,会以「构建缓存有时会自己空掉」的形态浮现。`v1` 是布局版本段,所以换布局永远不需要迁移或删除旧树。 + + 每条目写 `entry.json`,记下**键的全部输入**;命中判定从「目录存在 + 文件齐」升级为「schema 匹配 ∧ 键匹配 ∧ 记录的输入与本次算出的输入**逐字段相等** ∧ 清单文件齐」。这是把 std 侧一直做对的事搬到依赖侧:此前依赖条目只有一份文件清单,所以怀疑命中错了的时候什么都审计不了。缺字段视为不匹配,**永不**因为 hash 相等就放行。 + + 旧的 `$MCPP_HOME/bmi/`(按全工程指纹分目录)**不读不写不自动删**:`mcpp doctor` 报出它的体积,`mcpp cache clean --legacy` 回收。它从来没产生过收益(见上面第一条),但删几十 GB 必须是用户的显式动作。 + +- **profile 成为失效轴。** `[profile.]` 把开关落在 `buildConfig.optLevel/debug/lto/strip`,由 flags.cppm 变成 `-O`/`-g`/`-flto`,而 `canonical_compile_flags` 只序列化 cflags/cxxflags/ldflags —— 于是 `--dev`、`--release`、`--profile dist` 三者**同一个 fingerprint**、同一个 `target///`、同一条缓存条目。 + + **可感知的行为变化**:`target//` 下从此每个 profile 一个哈希目录。好处是 dev↔release 来回切从「每次全量」变成增量,两个 profile 的二进制可以并存;代价是磁盘随实际使用的 profile 数增长(每个 build dir 带一份 staged std BMI,本机 `std.gcm` = 31,466,112 字节)。 + +- **缓存失效不再挂在 mcpp 版本号上。** 缓存键的兼容轴改用独立的 `kCacheEpoch`,只在缓存布局/键算法/staging 契约真的不兼容时递增。`fingerprint.cppm` 的 `MCPP_VERSION` 保持在 fingerprint 里(build dir 命名与本地增量归它管,宁可保守),但每次发版把整个缓存作废对纯 C 目标文件是过度失效。 + +### 新增 + +- **`--cache `:三种构建模式。** `global`(默认,读+写全局缓存)/ `local`(不读不写,所有依赖在本工程 `target/` 内编 —— 排障时一次性排除「是不是缓存的问题」,也给 CI 一个无共享的可复现基线)/ `off`(不读不写,并先清掉本次的 `target///` 做冷构建)。优先级 `--cache` > `MCPP_BUILD_CACHE` > `[build] cache` > `global`;无法识别的值会被报出来(`--strict` 下为错误),而不是静默回落到 `global`。 + + `--no-cache` 保留为 `off` 的兼容别名。它的旧 help 文案「Force-clear target/ before building」两处不准:它清的是**构建目录**(`target///`)而不是整个 `target/`,而且名字与缓存无关。`mcpp run` / `mcpp test` 一并补上这两个 flag(此前它们连 `--no-cache` 都没有)。 + + ⚠️ **`--no-cache` 的语义有一处收紧**:此前它只清构建目录、**仍然会回填全局缓存**;现在它等于 `off`,即**不读也不写**。想要「从零重编但仍然刷新缓存」的,用 `mcpp clean` 或 `rm -rf target` 后正常构建。这个收紧是为了让三个模式正交:一个叫 `off` 的模式还偷偷写缓存是说不通的。 + +- **`mcpp cache` 补齐到可运维。** `cache dir`(缓存到底在哪 —— 此前 `cache *`/`doctor`/`clean --bmi-cache` 各自解析根目录,而 config 的 reset 路径用 `GlobalConfig::bmiCacheDir`,两者可能不是同一个目录)、`cache gc --max-size {MiB,GiB} / --older-than {s,m,h,d}`(**真 LRU**)、`cache clean --deps|--std|--all|--legacy`、`cache list --json`、`cache verify`(逐条目校验清单与磁盘,残缺条目非零退出)。`cache info` 现在打印该条目的键输入 —— 怀疑命中错了时第一件想看的东西。 + + `prune` 此前按**目录 mtime** 排序,而那只记录条目被**写入**的时间:一个每次构建都命中的热包,和一个一个月没人碰过的冷包一样「陈旧」。`entry.json` 的 `accessed` 由每次命中刷新(只重写 `entry.json`,**不动产物 mtime** —— 那些 mtime 参与 ninja 的 restat 判定),`gc` 按它排序。`cache clean` 开头那句 `remove_all(/"deps")` 指向一个从不存在的路径(dep 条目在 `//deps`),是死代码。 + + `gc` 刻意**不动 std 条目**:一份 std BMI 全机共享、重建要几十秒,为了腾一点磁盘赶走它是拿很多时间换很少空间。达不到容量预算时会明确说出来,而不是静默少删。 + ## [2026.7.30.1] — 2026-07-30 ### 修复 diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index d5233cd7..3141a82a 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -146,6 +146,7 @@ static_stdlib = true # Statically link libstdc++ (default true) target = "x86_64-linux-musl" # Default build target when no --target is passed # (≙ cargo build.target; e.g. "ship fully-static") macos_deployment_target = "14.0" # Minimum supported OS version for macOS artifacts (macOS only) +cache = "global" # Global dependency cache: global (default) | local | off (§2.10) ``` `include_dirs_after` (#249) lists header directories that are searched **after** @@ -717,8 +718,56 @@ ldflags = [] - Built-in profiles: `release` (-O2) / `dev`, `debug` (-O0 -g) / `dist` (-O3 + strip; **LTO is not enabled by default**). `[profile.]` can override a built-in definition wholesale. +- **Each profile owns its own build directory.** The resolved profile knobs + participate in the fingerprint, so `target//` holds one hash directory + per profile and switching between them is incremental instead of a full + rebuild. It also means the disk cost scales with the number of profiles you + actually use. -### 2.10 `[runtime]` — Host Runtime Capabilities +### 2.10 `[build] cache` — The Global Dependency Cache + +Compiled artifacts for dependencies fetched from an index are cached across +projects under `$MCPP_HOME/build-cache/v1/`. A dependency's artifacts do not +depend on who consumes them, so two projects with the same toolchain, profile and +dependency versions reuse one entry. + +```toml +[build] +cache = "global" # "global" (default) | "local" | "off" +``` + +| Mode | Reads the cache | Writes the cache | Clears the build dir first | +|---|---|---|---| +| `global` (default) | yes | yes | no | +| `local` | no | no | no | +| `off` | no | no | yes | + +`local` builds every dependency inside this project's `target/` — useful to rule +the cache out while diagnosing something, and to give CI a no-sharing baseline. +`off` additionally clears this build's `target///` for a cold +rebuild; `--no-cache` is a deprecated alias for it. + +Precedence: `--cache ` **>** `MCPP_BUILD_CACHE` **>** `[build] cache` **>** +`global`. An unrecognized value is reported (an error under `--strict`) rather +than silently falling back to `global`. + +What is **not** cached: `path` and `git` dependencies, at any depth, and +workspace members. Their sources can change without their `name@version` +changing, so no key over that identity could notice the change. + +Inspection and reclamation: + +``` +mcpp cache dir # where the cache lives +mcpp cache list [--json] # entries, sizes, last use +mcpp cache info @ # one entry, including the key inputs it was built with +mcpp cache verify # every entry's file list against the disk +mcpp cache gc --max-size 5GiB # LRU-collect package entries to a budget +mcpp cache gc --older-than 30d # ...or by how long since they were last used +mcpp cache clean [--deps|--std|--all|--legacy] +``` + +### 2.11 `[runtime]` — Host Runtime Capabilities ```toml [runtime] @@ -744,7 +793,7 @@ provider = "compat.glx-runtime" `opengl.glx.driver`, `x11.display`) and prefix-style `abi:` (e.g. `abi:glibc`, which participates in toolchain ABI enforcement). -### 2.11 `[package] platforms` — Platform Declaration +### 2.12 `[package] platforms` — Platform Declaration ```toml [package] @@ -756,7 +805,7 @@ The vocabulary is fixed by mcpp (which owns the target/triple system): `linux | macos | windows`; unknown values produce a warning, and an error under `--strict`. -### 2.12 `[xlings]` — Build Environment +### 2.13 `[xlings]` — Build Environment ```toml [xlings] diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 6063a2b0..670703ce 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -139,6 +139,7 @@ ldflags = ["-lfoo"] # 额外链接参数 defines = ["BIZ=1", "QUX"] # 作用于每个 TU 的预处理宏(脱糖为 -D;会进入模块扫描) static_stdlib = true # 静态链接 libstdc++(默认 true) macos_deployment_target = "14.0" # macOS 产物的最低支持系统版本(仅 macOS 生效) +cache = "global" # 依赖的全局构建缓存:global(默认)| local | off(见 §2.10) ``` `include_dirs_after`(#249)列出**排在工具链系统目录之后**搜索的头文件目录 @@ -506,8 +507,49 @@ ldflags = [] (默认 dev 的项目在产出可分发物时应显式 `--release`。) - 内置档案:`release`(-O2)/ `dev`、`debug`(-O0 -g)/ `dist`(-O3 + strip; **不默认开 lto**)。`[profile.<内置名>]` 可整体覆盖内置定义。 +- **每个 profile 各占一个构建目录。** 解析后的 profile 开关参与指纹,所以 + `target//` 下每个 profile 一个哈希目录,来回切换是增量而不是全量重编; + 代价是磁盘占用随实际使用的 profile 数量增长。 -### 2.10 `[runtime]` — 主机运行时能力 +### 2.10 `[build] cache` — 依赖的全局构建缓存 + +从索引获取的依赖,其编译产物按包缓存在 `$MCPP_HOME/build-cache/v1/` 下,跨工程共享。 +依赖的产物与"谁在消费它"无关,所以工具链、profile、依赖版本相同的两个工程复用同一条目。 + +```toml +[build] +cache = "global" # "global"(默认)| "local" | "off" +``` + +| 模式 | 读缓存 | 写缓存 | 先清构建目录 | +|---|---|---|---| +| `global`(默认) | 是 | 是 | 否 | +| `local` | 否 | 否 | 否 | +| `off` | 否 | 否 | 是 | + +`local` 把所有依赖都编在本工程 `target/` 内 —— 排障时一次性排除"是不是缓存的问题", +也给 CI 一个无共享的可复现基线。`off` 额外清掉本次的 `target///` 做冷构建; +`--no-cache` 是它的兼容别名。 + +优先级:`--cache ` **>** `MCPP_BUILD_CACHE` **>** `[build] cache` **>** `global`。 +无法识别的值会被报出来(`--strict` 下为错误),而不是静默回落到 `global`。 + +**不进缓存的**:`path` 与 `git` 依赖(任意深度)以及 workspace 成员。它们的源码可以在 +`name@version` 不变的情况下改变,任何基于该身份的键都看不见这种变化。 + +查看与回收: + +``` +mcpp cache dir # 缓存在哪 +mcpp cache list [--json] # 条目、体积、最后使用时间 +mcpp cache info @ # 单条目详情,含它是用什么键输入编出来的 +mcpp cache verify # 逐条目校验清单与磁盘 +mcpp cache gc --max-size 5GiB # 按 LRU 收到容量预算内 +mcpp cache gc --older-than 30d # 或按"多久没用过"回收 +mcpp cache clean [--deps|--std|--all|--legacy] +``` + +### 2.11 `[runtime]` — 主机运行时能力 ```toml [runtime] @@ -529,7 +571,7 @@ provider = "compat.glx-runtime" - 能力命名约定:分层小写 `domain.sub.role`(如 `opengl.glx.driver`、 `x11.display`)与前缀类 `abi:`(如 `abi:glibc`,参与工具链 ABI 强制)。 -### 2.11 `[package] platforms` — 平台声明 +### 2.12 `[package] platforms` — 平台声明 ```toml [package] @@ -540,7 +582,7 @@ platforms = ["linux", "macos", "windows"] (它拥有 target/triple 体系):`linux | macos | windows`;未知值 warning, `--strict` 下报错。 -### 2.12 `[xlings]` — 构建环境 +### 2.13 `[xlings]` — 构建环境 ```toml [xlings] diff --git a/mcpp.toml b/mcpp.toml index b2ca09a4..b0a3136c 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.7.30.1" +version = "2026.7.30.2" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/bmi_cache.cppm b/src/bmi_cache.cppm index 068d5f61..24e7f63b 100644 --- a/src/bmi_cache.cppm +++ b/src/bmi_cache.cppm @@ -1,86 +1,124 @@ -// mcpp.bmi_cache — cross-project persistent BMI cache (M3.2). +// mcpp.bmi_cache — cross-project persistent cache of dependency build outputs. // -// Layout (per docs/26-bmi-cache.md): -// $MCPP_HOME/bmi//deps//@/ -// {gcm,pcm}.cache/.{gcm,pcm} -// obj/.m.o + .o -// manifest.txt (sentinel + file list) +// Layout: +// /pkg//@// +// (the cache root is $MCPP_HOME/build-cache/v1 — see mcpp.home::cache_root) +// entry.json sentinel + self-description + file list +// bmi/.{gcm,pcm} +// obj/.o // -// fingerprint already covers compiler / flags / stdlib / mcpp version / -// dep-lock hash etc. (docs/06), so different fingerprints get independent -// cache directories — no risk of stale-BMI cross-contamination. +// comes from mcpp.build.cache_key: a per-package Merkle key over the +// toolchain, the language/dialect settings, the resolved profile, the package's +// own identity and build config, and — recursively — the keys of its direct +// dependencies. See that module's header for why each axis is there. // -// M4 #9: populate is wrapped in an advisory exclusive flock(2) on -// /.lock so two concurrent mcpp builds can race to the same -// dep cache without trashing manifest.txt (docs/26 §5.4 V2). +// Two properties this layout has and the previous one did not: +// +// 1. The directory name is derived only from things that actually reach the +// package's compiler command lines. The old key was the whole-project +// fingerprint, so a consumer's own name and version were part of every +// dependency's cache path: `mcpp version bump` invalidated the entire cache +// and no two projects ever shared an entry. +// +// 2. An entry describes itself. `is_cached` compares the recorded inputs +// field by field against the inputs computed for this build, so a hit is +// evidence rather than the mere existence of a directory. The std module +// path has validated hits this way since it was written +// (mcpp.toolchain.stdmod's metadata_matches); the dependency path carried +// only a file list, which is why nothing could be audited when a wrong +// entry was suspected. +// +// populate_from holds an advisory exclusive lock on the entry directory so two +// concurrent builds racing to fill the same entry cannot interleave writes, and +// writes entry.json last so a crash mid-populate leaves a miss, not a +// half-populated hit. module; export module mcpp.bmi_cache; import std; +import mcpp.libs.json; import mcpp.platform; export namespace mcpp::bmi_cache { +// Schema of entry.json. Bumped only if the file's own shape changes; +// cache-content compatibility is carried by cache_key::kCacheEpoch, which +// travels inside `inputs`. +inline constexpr int kEntrySchema = 1; + struct CacheKey { - std::filesystem::path mcppHome; - std::string fingerprint; - std::string indexName; // "mcpplibs" / "xim" / ... - std::string packageName; // "mcpplibs.cmdline" - std::string version; // "0.0.1" - std::string bmiDirName = "gcm.cache"; // "gcm.cache" | "pcm.cache" + // The resolved cache root (mcpp::home::cache_root()). Passed in rather than + // recomputed here: the layout root must have exactly ONE definition, and a + // second copy of "/build-cache/v1" in this file is precisely the kind + // of cross-file invariant a comment cannot enforce. Tests supply a temp + // directory the same way. + std::filesystem::path cacheRoot; + std::string indexName; // "mcpplibs" / "compat" / ... + std::string packageName; // "compat.zlib" + std::string version; // "1.3.2" + std::string keyHex; // cache_key::key_hex(...) + // The full key inputs, recorded in entry.json and compared field by field + // on a hit. Never trust equal hashes alone. + nlohmann::json inputs; + std::string bmiDirName = "gcm.cache"; // consumer-side directory name std::string manifestTag = "gcm"; // "gcm" | "pcm" std::filesystem::path dir() const { - return mcppHome / "bmi" / fingerprint / "deps" - / indexName / std::format("{}@{}", packageName, version); + return cacheRoot / "pkg" / indexName + / std::format("{}@{}", packageName, version) / keyHex; } - std::filesystem::path manifestFile() const { return dir() / "manifest.txt"; } - std::filesystem::path bmiDir() const { return dir() / bmiDirName; } - std::filesystem::path objDir() const { return dir() / "obj"; } + std::filesystem::path entryFile() const { return dir() / "entry.json"; } + std::filesystem::path bmiDir() const { return dir() / "bmi"; } + std::filesystem::path objDir() const { return dir() / "obj"; } }; -// File names (basename only) that belong to one dep package's cache entry. +// File names (basenames for BMIs, output-relative paths for objects) belonging +// to one package's cache entry. struct DepArtifacts { - std::vector bmiFiles; // basenames in bmiDir/ - std::vector objFiles; // basenames in obj/ + std::vector bmiFiles; + std::vector objFiles; }; -// Cache hit if manifest.txt exists AND every listed file is present. +// True when entry.json exists, its schema matches, its recorded inputs equal +// `key.inputs` field for field, and every listed file is present on disk. bool is_cached(const CacheKey& key); -// Read manifest.txt → DepArtifacts. -std::expected -read_manifest(const CacheKey& key); +// The artifact list of a validated entry. Does NOT copy anything: the ninja +// backend stages cached files through its own `stage_file` edges, so that a +// staged file is the output of an edge ninja has a command-line record for. +// Copying them behind ninja's back — which this module used to do — left every +// staged output with no entry in .ninja_log, and ninja treats that as dirty +// ("command line not found in log"), so every cached dependency was recompiled +// anyway while the CLI reported it as cached. +std::expected resolve_cached(const CacheKey& key); -// Copy missing cached files into projectTarget/{bmiDirName,obj}. Existing -// project outputs are left untouched: BMIs may differ byte-for-byte between -// equivalent builds, and overwriting them would dirty downstream modules. -std::expected -stage_into(const CacheKey& key, - const std::filesystem::path& projectTargetDir); +// Refresh the entry's `accessed` stamp. Rewrites entry.json only — never the +// artifacts, whose mtimes must stay put (ninja's restat handling compares them). +// This is what makes `mcpp cache gc` a real LRU: pruning used to read the +// directory's mtime, which only ever recorded when the entry was WRITTEN, so a +// dependency that hit on every build looked stale. +void touch_accessed(const CacheKey& key); -// Copy fresh build outputs from projectTarget/{bmiDirName,obj} → cache dir -// and write manifest.txt last (atomic-ish sentinel). +// Copy fresh build outputs from projectTarget/{bmiDirName,obj} into the entry, +// then write entry.json last as the sentinel. std::expected populate_from(const CacheKey& key, const std::filesystem::path& projectTargetDir, const DepArtifacts& artifacts); +// Absolute paths of an entry's artifacts, for the stage edges. +std::filesystem::path cached_bmi_path(const CacheKey& key, std::string_view basename); +std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel); + } // namespace mcpp::bmi_cache namespace mcpp::bmi_cache { namespace { -void touch_now(const std::filesystem::path& p) { - std::error_code ec; - auto t = std::chrono::file_clock::now() + std::chrono::seconds(1); - std::filesystem::last_write_time(p, t, ec); -} - bool copy_one(const std::filesystem::path& from, const std::filesystem::path& to, std::error_code& ec) @@ -91,113 +129,123 @@ bool copy_one(const std::filesystem::path& from, return !ec; } -std::string serialize_manifest(std::string_view tag, const DepArtifacts& a) { - std::string out = "# Auto-generated by mcpp bmi_cache. Do not edit.\n"; - for (auto& g : a.bmiFiles) out += std::format("{}: {}\n", tag, g); - for (auto& o : a.objFiles) out += std::format("obj: {}\n", o); - return out; +std::optional read_entry(const std::filesystem::path& p) { + std::ifstream is(p); + if (!is) return std::nullopt; + nlohmann::json j; + try { is >> j; } catch (...) { return std::nullopt; } + return j; } -std::expected -parse_manifest(const std::filesystem::path& p) { - std::ifstream is(p); - if (!is) return std::unexpected(std::format("cannot open '{}'", p.string())); +DepArtifacts artifacts_from(const nlohmann::json& j) { DepArtifacts a; - std::string line; - while (std::getline(is, line)) { - while (!line.empty() && (line.back() == '\r' || line.back() == ' ')) line.pop_back(); - if (line.empty() || line[0] == '#') continue; - if (line.starts_with("gcm: ")) a.bmiFiles.push_back(line.substr(5)); - else if (line.starts_with("pcm: ")) a.bmiFiles.push_back(line.substr(5)); - else if (line.starts_with("obj: ")) a.objFiles.push_back(line.substr(5)); - } + if (auto it = j.find("bmi"); it != j.end() && it->is_array()) + for (auto& v : *it) if (v.is_string()) a.bmiFiles.push_back(v.get()); + if (auto it = j.find("obj"); it != j.end() && it->is_array()) + for (auto& v : *it) if (v.is_string()) a.objFiles.push_back(v.get()); return a; } -} // namespace - -bool is_cached(const CacheKey& key) { - auto mf = key.manifestFile(); - if (!std::filesystem::exists(mf)) return false; - auto arts = parse_manifest(mf); - if (!arts) return false; - // Verify every listed file actually exists on disk. - for (auto& g : arts->bmiFiles) { - if (!std::filesystem::exists(key.bmiDir() / g)) return false; - } - for (auto& o : arts->objFiles) { - if (!std::filesystem::exists(key.objDir() / o)) return false; +// Field-by-field, not `==` on the whole object: an entry written by an older +// mcpp may legitimately carry extra keys, but every key the CURRENT build cares +// about has to be present and equal. A missing key is a mismatch, never a pass. +bool inputs_match(const nlohmann::json& recorded, const nlohmann::json& expected) { + if (!recorded.is_object() || !expected.is_object()) return false; + for (auto it = expected.begin(); it != expected.end(); ++it) { + auto found = recorded.find(it.key()); + if (found == recorded.end()) return false; + if (*found != it.value()) return false; } return true; } -std::expected -read_manifest(const CacheKey& key) { - return parse_manifest(key.manifestFile()); +std::string now_iso8601() { + // No zoned formatting: libstdc++'s `import std` support for + // std::format on chrono types is partial (see fingerprint.cppm's + // hand-rolled hex for the same reason). Seconds since epoch is monotonic + // enough for an LRU stamp and needs no formatting support at all. + auto now = std::chrono::system_clock::now(); + auto secs = std::chrono::duration_cast( + now.time_since_epoch()).count(); + return std::to_string(secs); } -std::expected -stage_into(const CacheKey& key, - const std::filesystem::path& projectTargetDir) -{ - auto arts = parse_manifest(key.manifestFile()); - if (!arts) return std::unexpected(arts.error()); - - auto projectBmi = projectTargetDir / key.bmiDirName; - auto projectObj = projectTargetDir / "obj"; - std::error_code ec; - std::filesystem::create_directories(projectBmi, ec); - std::filesystem::create_directories(projectObj, ec); - - for (auto& g : arts->bmiFiles) { - auto from = key.bmiDir() / g; - auto to = projectBmi / g; - if (std::filesystem::exists(to, ec)) { - ec.clear(); - continue; - } - ec.clear(); - if (!copy_one(from, to, ec)) { - return std::unexpected(std::format( - "stage bmi '{}': {}", g, ec.message())); - } - touch_now(to); - } - for (auto& o : arts->objFiles) { - auto from = key.objDir() / o; - auto to = projectObj / o; - if (std::filesystem::exists(to, ec)) { - ec.clear(); - continue; - } - ec.clear(); - if (!copy_one(from, to, ec)) { - return std::unexpected(std::format( - "stage obj '{}': {}", o, ec.message())); - } - touch_now(to); +std::expected +write_entry(const std::filesystem::path& path, const nlohmann::json& j) { + auto tmp = path; + tmp += ".tmp"; + { + std::ofstream os(tmp, std::ios::binary); + if (!os) return std::unexpected(std::format( + "cannot write cache entry '{}'", tmp.string())); + os << j.dump(2) << "\n"; + if (!os) return std::unexpected(std::format( + "failed while writing cache entry '{}'", tmp.string())); } - return arts; + std::error_code ec; + std::filesystem::rename(tmp, path, ec); + if (ec) return std::unexpected(std::format( + "cache entry rename: {}", ec.message())); + return {}; } -namespace { } // namespace +std::filesystem::path cached_bmi_path(const CacheKey& key, std::string_view basename) { + return key.bmiDir() / std::string(basename); +} + +std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel) { + return key.objDir() / std::filesystem::path(std::string(rel)); +} + +bool is_cached(const CacheKey& key) { + auto j = read_entry(key.entryFile()); + if (!j) return false; + if (j->value("schema", 0) != kEntrySchema) return false; + if (j->value("key", std::string{}) != key.keyHex) return false; + auto it = j->find("inputs"); + if (it == j->end() || !inputs_match(*it, key.inputs)) return false; + + auto arts = artifacts_from(*j); + std::error_code ec; + for (auto& g : arts.bmiFiles) + if (!std::filesystem::exists(cached_bmi_path(key, g), ec)) return false; + for (auto& o : arts.objFiles) + if (!std::filesystem::exists(cached_obj_path(key, o), ec)) return false; + return true; +} + +std::expected resolve_cached(const CacheKey& key) { + auto j = read_entry(key.entryFile()); + if (!j) return std::unexpected(std::format( + "cannot read cache entry '{}'", key.entryFile().string())); + return artifacts_from(*j); +} + +void touch_accessed(const CacheKey& key) { + auto j = read_entry(key.entryFile()); + if (!j) return; + (*j)["accessed"] = now_iso8601(); + (void)write_entry(key.entryFile(), *j); +} + std::expected populate_from(const CacheKey& key, const std::filesystem::path& projectTargetDir, const DepArtifacts& arts) { auto cacheDir = key.dir(); + std::error_code ec; + std::filesystem::create_directories(cacheDir, ec); auto lock = mcpp::platform::fs::FileLock::try_acquire(cacheDir); if (!lock) { - // Another writer holds the lock; treat as success (they'll do it). + // Another writer holds the lock; it will finish the entry. return {}; } auto cacheBmi = key.bmiDir(); auto cacheObj = key.objDir(); - std::error_code ec; std::filesystem::create_directories(cacheBmi, ec); std::filesystem::create_directories(cacheObj, ec); @@ -206,43 +254,44 @@ populate_from(const CacheKey& key, for (auto& g : arts.bmiFiles) { auto from = projectBmi / g; - auto to = cacheBmi / g; if (!std::filesystem::exists(from)) { return std::unexpected(std::format( "expected build output missing: {}", from.string())); } - if (!copy_one(from, to, ec)) { + if (!copy_one(from, cacheBmi / g, ec)) { return std::unexpected(std::format( "populate bmi '{}': {}", g, ec.message())); } } for (auto& o : arts.objFiles) { auto from = projectObj / o; - auto to = cacheObj / o; if (!std::filesystem::exists(from)) { return std::unexpected(std::format( "expected build output missing: {}", from.string())); } - if (!copy_one(from, to, ec)) { + if (!copy_one(from, cached_obj_path(key, o), ec)) { return std::unexpected(std::format( "populate obj '{}': {}", o, ec.message())); } } - // Write manifest.txt LAST as the sentinel — atomic via temp + rename. - { - auto tmp = key.manifestFile(); - tmp += ".tmp"; - { - std::ofstream os(tmp); - os << serialize_manifest(key.manifestTag, arts); - } - std::filesystem::rename(tmp, key.manifestFile(), ec); - if (ec) { - return std::unexpected(std::format( - "populate manifest rename: {}", ec.message())); - } - } - return {}; + + // entry.json LAST — it is the sentinel. Preserve `created` when refilling an + // existing entry so gc's age reporting stays meaningful. + nlohmann::json j; + if (auto prev = read_entry(key.entryFile()); prev && prev->contains("created")) + j["created"] = (*prev)["created"]; + else + j["created"] = now_iso8601(); + j["schema"] = kEntrySchema; + j["key"] = key.keyHex; + j["package"] = std::format("{}/{}@{}", key.indexName, key.packageName, key.version); + j["bmi_dir"] = key.bmiDirName; + j["tag"] = key.manifestTag; + j["inputs"] = key.inputs; + j["bmi"] = arts.bmiFiles; + j["obj"] = arts.objFiles; + j["accessed"] = now_iso8601(); + return write_entry(key.entryFile(), j); } } // namespace mcpp::bmi_cache diff --git a/src/bmi_cache/maintenance.cppm b/src/bmi_cache/maintenance.cppm index 3d781293..482f02e7 100644 --- a/src/bmi_cache/maintenance.cppm +++ b/src/bmi_cache/maintenance.cppm @@ -1,6 +1,21 @@ -// mcpp.bmi_cache.maintenance — global BMI cache inspection + pruning, and the -// shared fs-size/byte-formatting helpers they are built on. -// Bodies moved verbatim from the CLI layer. Zero behavior change. +// mcpp.bmi_cache.maintenance — inspection and pruning of the global build +// cache, plus the shared fs-size/byte-formatting helpers they are built on. +// +// Layout walked here (see bmi_cache.cppm): +// $MCPP_HOME/build-cache/v1/pkg//@//entry.json +// $MCPP_HOME/build-cache/v1/std//std-module.json +// +// Two things this file did wrong before the layout change and that the new +// entry.json fixes: +// +// * pruning ranked entries by the DIRECTORY's mtime, which only ever recorded +// when an entry was written. A dependency that hit on every single build +// looked as stale as one nobody had touched in a month, so `prune` was +// "drop what was populated long ago", not an LRU. entry.json carries an +// `accessed` stamp that bmi_cache::touch_accessed refreshes on every hit. +// * `clean` began with remove_all(/"deps"), a path that never existed +// (dep entries lived at //deps) — dead code in front of the loop +// that actually did the work. module; #include @@ -9,7 +24,8 @@ module; export module mcpp.bmi_cache.maintenance; import std; -import mcpp.toolchain.stdmod; +import mcpp.home; +import mcpp.libs.json; import mcpp.ui; namespace mcpp::bmi_cache { @@ -37,124 +53,348 @@ export std::string human_bytes(std::uintmax_t n) { return std::format("{:.1f} {}", v, units[u]); } +// Parse `{s,m,h,d}` into seconds. +export std::optional parse_duration(std::string_view v) { + if (v.size() < 2) return std::nullopt; + char unit = v.back(); + std::int64_t n = 0; + try { n = std::stoll(std::string(v.substr(0, v.size() - 1))); } + catch (...) { return std::nullopt; } + switch (unit) { + case 's': return n; + case 'm': return n * 60; + case 'h': return n * 3600; + case 'd': return n * 86400; + default: return std::nullopt; + } +} + +// Parse `{B,KiB,MiB,GiB,TiB}` (suffix optional, case-insensitive, bare +// number = bytes) into bytes. +export std::optional parse_size(std::string_view v) { + std::size_t digits = 0; + while (digits < v.size() && std::isdigit(static_cast(v[digits]))) + ++digits; + if (digits == 0) return std::nullopt; + std::uintmax_t n = 0; + try { n = std::stoull(std::string(v.substr(0, digits))); } + catch (...) { return std::nullopt; } + std::string suffix; + for (auto c : v.substr(digits)) + if (!std::isspace(static_cast(c))) + suffix.push_back(static_cast(std::tolower( + static_cast(c)))); + if (suffix.empty() || suffix == "b") return n; + if (suffix == "k" || suffix == "kib" || suffix == "kb") return n * 1024ull; + if (suffix == "m" || suffix == "mib" || suffix == "mb") return n * 1024ull * 1024; + if (suffix == "g" || suffix == "gib" || suffix == "gb") return n * 1024ull * 1024 * 1024; + if (suffix == "t" || suffix == "tib" || suffix == "tb") + return n * 1024ull * 1024 * 1024 * 1024; + return std::nullopt; +} + +namespace { -// ─── M4 #4: mcpp cache list / prune / clean / info ────────────────────── -struct CacheEntry { - std::filesystem::path dir; - std::string fingerprint; - std::string pkgAtVer; // "/@" - std::uintmax_t size = 0; - std::filesystem::file_time_type lastWrite{}; - std::size_t fileCount = 0; +std::filesystem::path pkg_root() { return mcpp::home::cache_root() / "pkg"; } +std::filesystem::path std_root() { return mcpp::home::cache_root() / "std"; } + +struct Entry { + std::filesystem::path dir; + std::string kind; // "pkg" | "std" + std::string label; // "/@" | "std" + std::string key; + std::uintmax_t size = 0; + std::int64_t accessed = 0; // seconds since epoch; 0 = unknown + std::size_t fileCount = 0; + bool complete = true; + std::string problem; }; -static std::vector walk_cache_entries() { - std::vector entries; - auto bmi = mcpp::toolchain::default_cache_root(); +std::optional read_json(const std::filesystem::path& p) { + std::ifstream is(p); + if (!is) return std::nullopt; + nlohmann::json j; + try { is >> j; } catch (...) { return std::nullopt; } + return j; +} + +std::int64_t stamp_of(const nlohmann::json& j, const char* field) { + auto it = j.find(field); + if (it == j.end()) return 0; + if (it->is_number_integer()) return it->get(); + if (it->is_string()) { + try { return std::stoll(it->get()); } catch (...) {} + } + return 0; +} + +// mtime fallback for entries with no `accessed` stamp (std entries, and package +// entries written by an mcpp that predates the field). +// +// file_time_type is std::chrono::file_clock, whose epoch is NOT the Unix epoch — +// reading time_since_epoch() and comparing it against system_clock produced ages +// like "74509d ago". `clock_cast` would be the standard answer but libc++ does +// not provide it, so measure the mtime as an OFFSET FROM NOW in the file clock +// and apply that offset to the system clock. That needs no shared epoch and no +// conversion trait — and "how long ago" is all any caller wants anyway. +std::int64_t dir_mtime_seconds(const std::filesystem::path& p) { + std::error_code ec; + auto t = std::filesystem::last_write_time(p, ec); + if (ec) return 0; + auto age = std::chrono::duration_cast( + std::chrono::file_clock::now() - t); + auto nowSys = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()); + return (nowSys - age).count(); +} + +void measure(Entry& e) { + e.size = dir_size(e.dir); + std::error_code ec; + for (auto& _ : std::filesystem::recursive_directory_iterator(e.dir, ec)) { + (void)_; + if (ec) break; + ++e.fileCount; + } +} + +// Verify a package entry's file list against the disk. +void check_pkg_files(Entry& e, const nlohmann::json& j) { std::error_code ec; - if (!std::filesystem::exists(bmi, ec)) return entries; - - for (auto& fpEntry : std::filesystem::directory_iterator(bmi, ec)) { - auto fpDir = fpEntry.path(); - auto depsDir = fpDir / "deps"; - if (!std::filesystem::exists(depsDir, ec)) continue; - for (auto& idxEntry : std::filesystem::directory_iterator(depsDir, ec)) { - for (auto& pkgEntry : std::filesystem::directory_iterator(idxEntry.path(), ec)) { - CacheEntry e; - e.dir = pkgEntry.path(); - e.fingerprint = fpDir.filename().string(); - e.pkgAtVer = idxEntry.path().filename().string() - + "/" + pkgEntry.path().filename().string(); - e.size = dir_size(e.dir); - e.lastWrite = std::filesystem::last_write_time(e.dir, ec); - for (auto& _ : std::filesystem::recursive_directory_iterator(e.dir, ec)) { - if (!ec) ++e.fileCount; + auto missing = [&](const std::filesystem::path& p) { + return !std::filesystem::exists(p, ec); + }; + if (auto it = j.find("bmi"); it != j.end() && it->is_array()) { + for (auto& v : *it) { + if (!v.is_string()) continue; + if (missing(e.dir / "bmi" / v.get())) { + e.complete = false; + e.problem = std::format("missing bmi/{}", v.get()); + return; + } + } + } + if (auto it = j.find("obj"); it != j.end() && it->is_array()) { + for (auto& v : *it) { + if (!v.is_string()) continue; + if (missing(e.dir / "obj" / v.get())) { + e.complete = false; + e.problem = std::format("missing obj/{}", v.get()); + return; + } + } + } +} + +std::vector walk_pkg_entries(bool verify = false) { + std::vector out; + std::error_code ec; + auto root = pkg_root(); + if (!std::filesystem::exists(root, ec)) return out; + for (auto& idx : std::filesystem::directory_iterator(root, ec)) { + if (!idx.is_directory()) continue; + for (auto& pkg : std::filesystem::directory_iterator(idx.path(), ec)) { + if (!pkg.is_directory()) continue; + for (auto& key : std::filesystem::directory_iterator(pkg.path(), ec)) { + if (!key.is_directory()) continue; + Entry e; + e.dir = key.path(); + e.kind = "pkg"; + e.key = key.path().filename().string(); + e.label = idx.path().filename().string() + "/" + + pkg.path().filename().string(); + auto j = read_json(e.dir / "entry.json"); + if (!j) { + e.complete = false; + e.problem = "no readable entry.json"; + } else { + e.accessed = stamp_of(*j, "accessed"); + if (verify) check_pkg_files(e, *j); } - entries.push_back(std::move(e)); + if (e.accessed == 0) e.accessed = dir_mtime_seconds(e.dir); + measure(e); + out.push_back(std::move(e)); } } } - return entries; + return out; } -static std::string format_age(std::filesystem::file_time_type t) { - auto now = std::chrono::file_clock::now(); - auto diff = std::chrono::duration_cast(now - t).count(); - if (diff < 60) return std::format("{}s ago", diff); - if (diff < 3600) return std::format("{}m ago", diff / 60); - if (diff < 86400) return std::format("{}h ago", diff / 3600); +std::vector walk_std_entries() { + std::vector out; + std::error_code ec; + auto root = std_root(); + if (!std::filesystem::exists(root, ec)) return out; + for (auto& d : std::filesystem::directory_iterator(root, ec)) { + if (!d.is_directory()) continue; + Entry e; + e.dir = d.path(); + e.kind = "std"; + e.key = d.path().filename().string(); + auto j = read_json(e.dir / "std-module.json"); + if (!j) { + e.complete = false; + e.problem = "no readable std-module.json"; + e.label = "std"; + } else { + e.label = std::format("std {}@{} {} {}", + j->value("compiler", std::string{"?"}), + j->value("compiler_version", std::string{"?"}), + j->value("cpp_standard", std::string{"?"}), + j->value("stdlib", std::string{"?"})); + } + e.accessed = dir_mtime_seconds(e.dir); + measure(e); + out.push_back(std::move(e)); + } + return out; +} + +std::string format_age(std::int64_t stampSeconds) { + if (stampSeconds == 0) return "unknown"; + auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + auto diff = now - stampSeconds; + if (diff < 0) return "just now"; + if (diff < 60) return std::format("{}s ago", diff); + if (diff < 3600) return std::format("{}m ago", diff / 60); + if (diff < 86400) return std::format("{}h ago", diff / 3600); return std::format("{}d ago", diff / 86400); } -// `mcpp cache` is dispatched at the App level — list / info / prune / clean -// each get their own action lambda invoking one of these helpers. +std::uintmax_t remove_entry(const Entry& e) { + std::error_code ec; + std::filesystem::remove_all(e.dir, ec); + if (ec) return 0; + // Drop the now-empty @ and directories so `cache list` + // does not accumulate empty shells. + for (auto p = e.dir.parent_path(); + p != mcpp::home::cache_root() && p.has_parent_path(); + p = p.parent_path()) { + std::error_code rmEc; + if (!std::filesystem::remove(p, rmEc)) break; // non-empty: stop + } + return e.size; +} + +} // namespace -// `mcpp cache list`. -export int cache_list() { - auto entries = walk_cache_entries(); - if (entries.empty()) { - std::println("(BMI cache is empty)"); +// `mcpp cache dir` — where IS the cache? Before this, `cache *`, `doctor` and +// `clean --bmi-cache` each resolved the root themselves while config.cppm's +// reset path used GlobalConfig::bmiCacheDir, and the copies could disagree. +export int cache_dir() { + std::println("{}", mcpp::home::cache_root().string()); + auto legacy = mcpp::home::legacy_bmi_root(); + std::error_code ec; + if (std::filesystem::exists(legacy, ec)) { + std::println("legacy (unused, removable with `mcpp cache clean --legacy`): {}", + legacy.string()); + } + return 0; +} + +// `mcpp cache list [--json]`. +export int cache_list(bool asJson) { + auto entries = walk_pkg_entries(); + auto stds = walk_std_entries(); + + if (asJson) { + nlohmann::json j; + j["root"] = mcpp::home::cache_root().string(); + j["entries"] = nlohmann::json::array(); + for (auto* set : {&entries, &stds}) { + for (auto& e : *set) { + j["entries"].push_back({ + {"kind", e.kind}, + {"label", e.label}, + {"key", e.key}, + {"dir", e.dir.string()}, + {"bytes", e.size}, + {"files", e.fileCount}, + {"accessed", e.accessed}, + {"complete", e.complete}, + }); + } + } + std::println("{}", j.dump(2)); return 0; } - std::println("{:<18} {:>10} {:>14} {}", - "fingerprint", "size", "last accessed", "package"); - for (auto& e : entries) { - auto fp = e.fingerprint.size() > 16 - ? e.fingerprint.substr(0, 16) : e.fingerprint; - std::println("{:<18} {:>10} {:>14} {}", - fp, human_bytes(e.size), format_age(e.lastWrite), e.pkgAtVer); + + if (entries.empty() && stds.empty()) { + std::println("(build cache is empty)"); + return 0; + } + std::println("{:<16} {:<6} {:>10} {:>14} {}", + "key", "kind", "size", "last used", "package"); + std::uintmax_t total = 0; + for (auto* set : {&stds, &entries}) { + for (auto& e : *set) { + total += e.size; + std::println("{:<16} {:<6} {:>10} {:>14} {}{}", + e.key.substr(0, 16), e.kind, human_bytes(e.size), + format_age(e.accessed), e.label, + e.complete ? "" : " (incomplete)"); + } } + std::println(""); + std::println("{} entries, {}", entries.size() + stds.size(), human_bytes(total)); return 0; } // `mcpp cache info @`. export int cache_info(const std::string& needle) { - auto entries = walk_cache_entries(); + auto entries = walk_pkg_entries(/*verify=*/true); + bool found = false; for (auto& e : entries) { - if (e.pkgAtVer.ends_with(needle)) { - std::println("dir = {}", e.dir.string()); - std::println("fingerprint = {}", e.fingerprint); - std::println("package = {}", e.pkgAtVer); - std::println("size = {}", human_bytes(e.size)); - std::println("file count = {}", e.fileCount); - std::println("last write = {}", format_age(e.lastWrite)); - return 0; + if (e.label.find(needle) == std::string::npos) continue; + found = true; + std::println("dir = {}", e.dir.string()); + std::println("key = {}", e.key); + std::println("package = {}", e.label); + std::println("size = {}", human_bytes(e.size)); + std::println("file count = {}", e.fileCount); + std::println("last used = {}", format_age(e.accessed)); + std::println("complete = {}{}", e.complete ? "yes" : "no", + e.complete ? "" : " (" + e.problem + ")"); + if (auto j = read_json(e.dir / "entry.json"); j && j->contains("inputs")) { + std::println("inputs ="); + std::println("{}", (*j)["inputs"].dump(2)); } + std::println(""); } - std::println("no cache entry matching '{}'", needle); - return 1; + if (!found) { + std::println("no cache entry matching '{}'", needle); + return 1; + } + return 0; } -// `mcpp cache prune --older-than {s,m,h,d}` (v = raw option value). +// `mcpp cache prune --older-than {s,m,h,d}` — kept as the age-only form. export int cache_prune(const std::string& v) { if (v.empty()) { mcpp::ui::error("`mcpp cache prune` requires --older-than {s,m,h,d}"); return 2; } - char unit = v.back(); - long long n = 0; - try { n = std::stoll(v.substr(0, v.size() - 1)); } - catch (...) { mcpp::ui::error(std::format("bad --older-than value '{}'", v)); return 2; } - std::chrono::seconds threshold{0}; - if (unit == 's') threshold = std::chrono::seconds(n); - else if (unit == 'm') threshold = std::chrono::seconds(n * 60); - else if (unit == 'h') threshold = std::chrono::seconds(n * 3600); - else if (unit == 'd') threshold = std::chrono::seconds(n * 86400); - else { mcpp::ui::error(std::format("bad time unit '{}': use s/m/h/d", unit)); return 2; } - auto cutoff = std::chrono::file_clock::now() - threshold; - auto entries = walk_cache_entries(); + auto secs = parse_duration(v); + if (!secs) { + mcpp::ui::error(std::format( + "bad --older-than value '{}' (expected {{s,m,h,d}})", v)); + return 2; + } + auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + auto cutoff = now - *secs; + int removed = 0; std::uintmax_t freed = 0; - for (auto& e : entries) { - if (e.lastWrite < cutoff) { - std::error_code ec; - std::filesystem::remove_all(e.dir, ec); - if (!ec) { - ++removed; - freed += e.size; - mcpp::ui::status("Pruned", - std::format("{} ({})", e.pkgAtVer, human_bytes(e.size))); - } + for (auto& e : walk_pkg_entries()) { + if (e.accessed != 0 && e.accessed >= cutoff) continue; + auto n = remove_entry(e); + if (n || !std::filesystem::exists(e.dir)) { + ++removed; + freed += e.size; + mcpp::ui::status("Pruned", + std::format("{} ({})", e.label, human_bytes(e.size))); } } std::println(""); @@ -162,19 +402,136 @@ export int cache_prune(const std::string& v) { return 0; } -// `mcpp cache clean` — drop dep entries, preserve std BMIs. -export int cache_clean() { - auto bmi = mcpp::toolchain::default_cache_root(); +// `mcpp cache gc [--max-size N] [--older-than N]` — real LRU, driven by +// entry.json's `accessed` stamp. +export int cache_gc(const std::string& maxSizeArg, const std::string& olderThanArg) { + if (maxSizeArg.empty() && olderThanArg.empty()) { + mcpp::ui::error("`mcpp cache gc` requires --max-size {MiB,GiB} " + "and/or --older-than {s,m,h,d}"); + return 2; + } + std::optional maxSize; + if (!maxSizeArg.empty()) { + maxSize = parse_size(maxSizeArg); + if (!maxSize) { + mcpp::ui::error(std::format( + "bad --max-size value '{}' (expected {{B,KiB,MiB,GiB,TiB}})", + maxSizeArg)); + return 2; + } + } + std::optional olderThan; + if (!olderThanArg.empty()) { + olderThan = parse_duration(olderThanArg); + if (!olderThan) { + mcpp::ui::error(std::format( + "bad --older-than value '{}' (expected {{s,m,h,d}})", olderThanArg)); + return 2; + } + } + + // Package entries only. A std BMI is shared by every project on the machine + // and costs ~30 s to rebuild; evicting one to hit a size target trades a lot + // of time for a little disk. `cache clean --std` remains the explicit way. + auto entries = walk_pkg_entries(); + std::ranges::sort(entries, [](const Entry& a, const Entry& b) { + return a.accessed < b.accessed; // oldest first + }); + + int removed = 0; + std::uintmax_t freed = 0; + std::uintmax_t live = 0; + for (auto& e : entries) live += e.size; + + auto now = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + + for (auto& e : entries) { + bool tooOld = olderThan && e.accessed != 0 + && (now - e.accessed) > *olderThan; + bool overBudget = maxSize && live > *maxSize; + if (!tooOld && !overBudget) continue; + remove_entry(e); + ++removed; + freed += e.size; + live -= e.size; + mcpp::ui::status("Collected", + std::format("{} ({}, last used {})", + e.label, human_bytes(e.size), format_age(e.accessed))); + } + std::println(""); + // "package entries", not "cache": `live` only ever counted package entries, + // because std entries are deliberately out of scope here. Reporting it as + // the cache size would read as "the cache is now empty" while tens of MB of + // std BMIs sit right next to it. + std::println("Collected {} entries, freed {} (package entries now {})", + removed, human_bytes(freed), human_bytes(live)); + if (maxSize && live > *maxSize) { + // Say it rather than silently under-delivering: a size target that + // cannot be met without evicting std BMIs is a real outcome, and a + // caller that thinks the budget held would be misled. + mcpp::ui::warning(std::format( + "still {} over the {} budget: package entries are exhausted. " + "std BMIs are excluded from gc; use `mcpp cache clean --std`", + human_bytes(live - *maxSize), human_bytes(*maxSize))); + } + return 0; +} + +// `mcpp cache clean [--deps] [--std] [--all] [--legacy]`. +export int cache_clean(bool deps, bool stds, bool all, bool legacy) { + if (all) { deps = true; stds = true; } + if (!deps && !stds && !legacy) deps = true; // bare `clean` = dep entries + std::error_code ec; - std::filesystem::remove_all(bmi / "deps", ec); // deps only; preserve std.gcm - if (std::filesystem::exists(bmi)) { - for (auto& f : std::filesystem::directory_iterator(bmi, ec)) { - auto deps = f.path() / "deps"; - std::filesystem::remove_all(deps, ec); + if (deps) { + auto n = dir_size(pkg_root()); + std::filesystem::remove_all(pkg_root(), ec); + std::println("Cleaned package entries ({})", human_bytes(n)); + } + if (stds) { + auto n = dir_size(std_root()); + std::filesystem::remove_all(std_root(), ec); + std::println("Cleaned std module entries ({})", human_bytes(n)); + } + if (legacy) { + auto legacyRoot = mcpp::home::legacy_bmi_root(); + if (std::filesystem::exists(legacyRoot, ec)) { + auto n = dir_size(legacyRoot); + std::filesystem::remove_all(legacyRoot, ec); + std::println("Removed pre-v1 cache {} ({})", legacyRoot.string(), + human_bytes(n)); + } else { + std::println("No pre-v1 cache at {}", legacyRoot.string()); } } - std::println("Cleaned all dep BMI cache entries (std.gcm preserved)"); return 0; } +// `mcpp cache verify` — entry.json against the disk, for every entry. +export int cache_verify() { + auto entries = walk_pkg_entries(/*verify=*/true); + auto stds = walk_std_entries(); + int bad = 0; + for (auto* set : {&entries, &stds}) { + for (auto& e : *set) { + if (e.complete) continue; + ++bad; + mcpp::ui::error(std::format("{} [{}] {}", + e.label, e.key, e.problem)); + std::println(" {}", e.dir.string()); + } + } + auto total = entries.size() + stds.size(); + if (bad == 0) { + std::println("{} entries verified, all complete", total); + return 0; + } + std::println(""); + std::println("{} of {} entries are incomplete. They are treated as misses " + "and rebuilt; `mcpp cache gc --older-than 0s` or " + "`mcpp cache clean` reclaims them.", bad, total); + return 1; +} + } // namespace mcpp::bmi_cache diff --git a/src/build/cache_key.cppm b/src/build/cache_key.cppm new file mode 100644 index 00000000..f461a7e4 --- /dev/null +++ b/src/build/cache_key.cppm @@ -0,0 +1,356 @@ +// mcpp.build.cache_key — per-package identity for the global build cache. +// +// The global dep cache used to be keyed by the WHOLE-PROJECT fingerprint +// (src/toolchain/fingerprint.cppm), whose flags field folds in every package in +// the graph *including the root* — its name, its version, its [build] flags. +// Consequences, all measured: +// +// * bumping only the root's `version` changed the key ⇒ every dependency +// entry and the std BMI were invalidated by `mcpp version bump`; +// * two projects with identical dependencies and toolchain shared nothing, +// because their package names differ; +// * on one developer machine that produced 26 GB across 1198 fingerprint +// directories: `compat.zlib@1.3.2` stored 162 times, 15 distinct std module +// identities stored 1014 times. +// +// A dependency's artifacts do not depend on the consumer's identity. Verified: +// the root's `[build] cflags/cxxflags` are NOT passed to dependency translation +// units — a dependency compiles with its own buildConfig plus the shared +// toolchain and profile flags. So the key is built per package, from exactly +// the axes that reach that package's compiler command lines: +// +// A toolchain compiler id/version/driver identity, target triple, stdlib +// B language C++ standard + flag, dialect flags, C standard, macOS target +// C profile opt level / debug / lto / strip ← was missing entirely +// D identity index name, package FQN, version +// E own config features, cflags/cxxflags/asmflags/ldflags, per-glob flags, +// defines, generated files, include dirs, source list +// F upstream the KEY of each direct dependency, recursively (Merkle) +// G epoch kCacheEpoch — cache-format compatibility, decoupled from the +// mcpp release number +// +// Why F is recursive rather than "the upstream's public includes + defines": +// GCC embeds a CRC of an imported module's BMI into the importer's BMI. Rebuild +// a dependency with a changed interface (or a changed ABI via a `-D`) and the +// importer's stale BMI hard-fails with `module 'B' CRC mismatch` / +// `Bad import dependency`. The importer's BMI is bound to the exact BMI it read, +// so what its key needs is the upstream's identity itself, not an enumeration +// of interface surface that can silently miss an item (re-exported transitive +// modules are not even visible in the upstream's own manifest). The failure +// modes are asymmetric, which settles the direction: too narrow a key on the +// BMI axis is a loud compiler error, while too narrow a key on the object axis +// is a silently wrong `.o` — objects carry no such self-check. +// +// The recursion's real cost is that an upstream's PRIVATE changes cascade +// downstream. For the population this cache serves that is ~zero: an index +// package's descriptor is frozen per version, so its key can only move via a +// version bump (which must invalidate consumers anyway — they link its objects) +// or via a whole-graph axis. Path and git packages, the only ones that can +// change private flags without a version bump, never enter the cache at all. + +export module mcpp.build.cache_key; + +import std; +import mcpp.libs.json; +import mcpp.manifest; +import mcpp.modgraph.scanner; +import mcpp.toolchain.detect; +import mcpp.toolchain.fingerprint; + +export namespace mcpp::build::cache_key { + +// Bump ONLY when a change makes previously written entries unusable (the +// serialized input shape, the artifact layout, or the staging contract). +// Deliberately NOT the mcpp release number: folding the whole version in +// orphaned every entry on every release, including plain C object files whose +// validity has nothing to do with mcpp's version. +inline constexpr int kCacheEpoch = 1; + +// Axes A/B/C — identical for every package in one build, computed once. +struct BuildAxes { + // A + std::string compilerId; + std::string compilerVersion; + std::string driverIdentity; + std::string targetTriple; + std::string stdlibId; + std::string stdlibVersion; + // B + std::string cppStandard; + std::string cppStandardFlag; + std::vector dialectFlags; + std::string cStandard; + std::string macosDeploymentTarget; + // C + std::string optLevel; + bool debug = false; + bool lto = false; + bool strip = false; +}; + +// Axes D/E/F for one package. +struct PackageAxes { + // D + std::string indexName; + std::string packageName; + std::string version; + // E + // Filled by the caller: the union of features requested of this package + // over every incoming dependency edge, sorted. Their EFFECTS are already + // folded into the vectors below (a feature contributes -DMCPP_FEATURE_*, + // featureDefines, featureFlags and featureSources before the key is + // computed), so this is redundant for correctness — it is here so + // entry.json says which features an entry was built with, which is the + // first thing anyone diagnosing a wrong hit will want. + std::vector features; + std::vector cflags; + std::vector cxxflags; + std::vector ldflags; + std::vector defines; + std::vector globFlags; // pre-serialized, ordered + std::vector generatedFiles;// "path=content", ordered + std::vector includeDirs; // store-relative, ordered + std::vector sourceGlobs; // [build] sources, ordered + std::vector sources; // package-root-relative, sorted + // F — keys of direct dependencies, sorted + std::vector upstreamKeys; +}; + +// The canonical serialization. Also what lands in entry.json, so a cache hit +// can be validated field by field instead of trusting that equal hashes mean +// equal inputs. +nlohmann::json to_json(const BuildAxes& b, const PackageAxes& p); + +// 16 hex chars, from the canonical serialization. +std::string key_hex(const BuildAxes& b, const PackageAxes& p); + +// Axes A/B/C from a resolved toolchain + the root manifest (which is where the +// whole-graph language and profile settings live). +BuildAxes build_axes(const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::Manifest& rootManifest, + std::string_view cppStandardFlag, + const std::vector& dialectFlags, + std::string_view macosDeploymentTarget); + +// Axes E from one PackageRoot. `storeRoot` is stripped off absolute include +// dirs so the key survives a different MCPP_HOME (the payload paths are +// /registry/data/xpkgs/...; leaving them absolute would make every +// entry a miss on another machine, or after a home relocation). +void fill_package_config(PackageAxes& out, + const mcpp::modgraph::PackageRoot& pkg, + const std::filesystem::path& storeRoot); + +} // namespace mcpp::build::cache_key + +namespace mcpp::build::cache_key { + +namespace { + +// Length-prefixed field joining. A plain separator would let ("a", "bc") and +// ("ab", "c") collide, which for a cache key means serving one package's +// objects for another's. +void put(std::string& s, std::string_view label, std::string_view value) { + s += label; + s += '='; + s += std::to_string(value.size()); + s += ':'; + s += value; + s += '\x1f'; +} + +void put_list(std::string& s, std::string_view label, + const std::vector& values) { + s += label; + s += '['; + s += std::to_string(values.size()); + s += ']'; + for (auto& v : values) { + s += '\x1e'; + s += std::to_string(v.size()); + s += ':'; + s += v; + } + s += '\x1f'; +} + +} // namespace + +nlohmann::json to_json(const BuildAxes& b, const PackageAxes& p) { + nlohmann::json j; + j["epoch"] = kCacheEpoch; + j["toolchain"] = { + {"compiler", b.compilerId}, + {"compiler_version", b.compilerVersion}, + {"driver_identity", b.driverIdentity}, + {"target_triple", b.targetTriple}, + {"stdlib", b.stdlibId}, + {"stdlib_version", b.stdlibVersion}, + }; + j["language"] = { + {"cpp_standard", b.cppStandard}, + {"cpp_standard_flag", b.cppStandardFlag}, + {"dialect_flags", b.dialectFlags}, + {"c_standard", b.cStandard}, + {"macos_deployment_target", b.macosDeploymentTarget}, + }; + j["profile"] = { + {"opt_level", b.optLevel}, + {"debug", b.debug}, + {"lto", b.lto}, + {"strip", b.strip}, + }; + j["package"] = { + {"index", p.indexName}, + {"name", p.packageName}, + {"version", p.version}, + }; + j["config"] = { + {"features", p.features}, + {"cflags", p.cflags}, + {"cxxflags", p.cxxflags}, + {"ldflags", p.ldflags}, + {"defines", p.defines}, + {"glob_flags", p.globFlags}, + {"generated_files", p.generatedFiles}, + {"include_dirs", p.includeDirs}, + {"source_globs", p.sourceGlobs}, + {"sources", p.sources}, + }; + j["upstream"] = p.upstreamKeys; + return j; +} + +std::string key_hex(const BuildAxes& b, const PackageAxes& p) { + std::string s = "mcpp-cache-key-v1\x1f"; + put(s, "epoch", std::to_string(kCacheEpoch)); + // A + put(s, "cc", b.compilerId); + put(s, "ccver", b.compilerVersion); + put(s, "driver", b.driverIdentity); + put(s, "triple", b.targetTriple); + put(s, "stdlib", b.stdlibId); + put(s, "stdlibv", b.stdlibVersion); + // B + put(s, "std", b.cppStandard); + put(s, "stdflag", b.cppStandardFlag); + put_list(s, "dialect", b.dialectFlags); + put(s, "cstd", b.cStandard); + put(s, "macos", b.macosDeploymentTarget); + // C + put(s, "opt", b.optLevel); + put(s, "debug", b.debug ? "1" : "0"); + put(s, "lto", b.lto ? "1" : "0"); + put(s, "strip", b.strip ? "1" : "0"); + // D + put(s, "index", p.indexName); + put(s, "pkg", p.packageName); + put(s, "ver", p.version); + // E + put_list(s, "features", p.features); + put_list(s, "cflags", p.cflags); + put_list(s, "cxxflags", p.cxxflags); + put_list(s, "ldflags", p.ldflags); + put_list(s, "defines", p.defines); + put_list(s, "globflags", p.globFlags); + put_list(s, "genfiles", p.generatedFiles); + put_list(s, "includes", p.includeDirs); + put_list(s, "srcglobs", p.sourceGlobs); + put_list(s, "sources", p.sources); + // F + put_list(s, "upstream", p.upstreamKeys); + return mcpp::toolchain::hash_string(s); +} + +BuildAxes build_axes(const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::Manifest& rootManifest, + std::string_view cppStandardFlag, + const std::vector& dialectFlags, + std::string_view macosDeploymentTarget) +{ + BuildAxes b; + b.compilerId = std::string(tc.compiler_name()); + b.compilerVersion = tc.version; + // Same rule the whole-project fingerprint uses: prefer the declared driver + // identity, else hash the driver binary. Two payloads of "gcc 16.1.0" that + // are not the same build must not share a cache entry. + b.driverIdentity = !tc.driverIdent.empty() + ? mcpp::toolchain::hash_string(tc.driverIdent) + : (tc.binaryPath.empty() ? std::string{} + : mcpp::toolchain::hash_file(tc.binaryPath)); + b.targetTriple = tc.targetTriple; + b.stdlibId = tc.stdlibId; + b.stdlibVersion = tc.stdlibVersion; + + b.cppStandard = rootManifest.package.standard; + b.cppStandardFlag = std::string(cppStandardFlag); + b.dialectFlags = dialectFlags; + b.cStandard = rootManifest.buildConfig.cStandard; + b.macosDeploymentTarget = std::string(macosDeploymentTarget); + + b.optLevel = rootManifest.buildConfig.optLevel; + b.debug = rootManifest.buildConfig.debug; + b.lto = rootManifest.buildConfig.lto; + b.strip = rootManifest.buildConfig.strip; + return b; +} + +void fill_package_config(PackageAxes& out, + const mcpp::modgraph::PackageRoot& pkg, + const std::filesystem::path& storeRoot) +{ + const auto& bc = pkg.manifest.buildConfig; + + out.cflags = bc.cflags; + out.cxxflags = bc.cxxflags; + out.ldflags = bc.ldflags; + out.defines = bc.defines; + out.sourceGlobs = bc.sources; + + if (!bc.cStandard.empty()) { + // A package may pin its own C standard; it reaches its own C units. + out.cflags.push_back("__c_standard=" + bc.cStandard); + } + + for (auto const& gf : bc.globFlags) { + std::string one = "glob:" + gf.glob; + for (auto const& f : gf.cflags) one += "\x1egc:" + f; + for (auto const& f : gf.cxxflags) one += "\x1egxx:" + f; + for (auto const& f : gf.asmflags) one += "\x1egas:" + f; + for (auto const& f : gf.defines) one += "\x1egd:" + f; + out.globFlags.push_back(std::move(one)); + } + + for (auto const& [path, content] : bc.generatedFiles) { + out.generatedFiles.push_back( + path.generic_string() + "=" + content); + } + std::ranges::sort(out.generatedFiles); + + // Include dirs are absolute payload paths under the xpkgs store; make them + // store-relative so the key does not encode this machine's MCPP_HOME. + auto storeStr = storeRoot.generic_string(); + auto relativize = [&](const std::filesystem::path& p) { + auto s = p.generic_string(); + if (!storeStr.empty() && s.starts_with(storeStr)) + return "" + s.substr(storeStr.size()); + auto pkgStr = pkg.root.generic_string(); + if (!pkgStr.empty() && s.starts_with(pkgStr)) + return "" + s.substr(pkgStr.size()); + return s; + }; + auto add_dirs = [&](const std::vector& dirs, + std::string_view tag) { + for (auto& d : dirs) + out.includeDirs.push_back(std::string(tag) + ":" + relativize(d)); + }; + if (pkg.usageResolved) { + add_dirs(pkg.privateBuild.includeDirs, "priv"); + add_dirs(pkg.publicUsage.includeDirs, "pub"); + add_dirs(pkg.privateBuild.includeDirsAfter, "priv_after"); + add_dirs(pkg.publicUsage.includeDirsAfter, "pub_after"); + } + add_dirs(bc.includeDirs, "decl"); + add_dirs(bc.includeDirsAfter, "decl_after"); +} + +} // namespace mcpp::build::cache_key diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 06d92e0c..dacbcc69 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -29,7 +29,11 @@ namespace mcpp::build { // ─── P0: build cache for fast-path rebuilds ───────────────────────── constexpr std::string_view kBuildCacheFile = "target/.build_cache"; -constexpr int kBuildCacheMaxEntries = 4; // P3: LRU capacity +// P3: LRU capacity. Entries are keyed by (target triple, profile), so the +// working set is now targets × profiles rather than targets alone — 4 was +// enough for one profile, not for a dev/release/dist rotation across a host +// and a cross target. +constexpr int kBuildCacheMaxEntries = 8; // P3: one entry per (target, fingerprint) pair. struct BuildCacheEntry { @@ -55,6 +59,20 @@ struct BuildCacheEntry { // plan.runtimeLibraryDirs is empty. std::string runEnvKey; std::string runEnvValue; + // The resolved profile this entry was built for. Entries used to be keyed + // by target triple alone, and the fast paths only refuse to run when an + // EXPLICIT --profile/--dev/--release is passed — so a bare `mcpp build` + // after `mcpp build --release` took the fast path against the release + // build.ninja and reported success without ever rebuilding at -O0 -g. + // Empty means "cache predates this field" and is treated as a miss (a + // bare rebuild once, never a wrong artifact). + std::string profile; + // The global-cache mode this build.ninja was generated under. A graph built + // under `global` contains stage_file edges reading the cache; replaying it + // for a request that asked for `local` would use the cache the manifest just + // said not to use — and ruling the cache out is `local`'s entire purpose. + // Same back-compat contract as `profile`: empty ⇒ miss. + std::string cacheMode; }; std::vector read_build_cache(const std::filesystem::path& projectRoot) { @@ -120,6 +138,17 @@ std::vector read_build_cache(const std::filesystem::path& proje std::getline(f, e.runEnvValue); haveNextLine = static_cast(std::getline(f, line)); } + // Optional profile line. Same back-compat contract as the two blocks + // above: absent ⇒ e.profile stays empty ⇒ every fast path treats the + // entry as a miss and falls through to prepare_build. + if (haveNextLine && line.starts_with("profile=")) { + e.profile = line.substr(8); + haveNextLine = static_cast(std::getline(f, line)); + } + if (haveNextLine && line.starts_with("cacheMode=")) { + e.cacheMode = line.substr(10); + haveNextLine = static_cast(std::getline(f, line)); + } entries.push_back(std::move(e)); if (!haveNextLine || line.empty()) break; } @@ -135,19 +164,24 @@ void write_build_cache(const std::filesystem::path& projectRoot, const std::string& runtimeEnvValue = "", std::vector> runTargets = {}, const std::string& runEnvKey = "", - const std::string& runEnvValue = "") { + const std::string& runEnvValue = "", + const std::string& profile = "", + const std::string& cacheMode = "") { auto path = projectRoot / kBuildCacheFile; auto entries = read_build_cache(projectRoot); - // Remove existing entry for this target (will be re-added at front). + // Remove the existing entry for this (target, profile) pair. Keying on the + // triple alone made a release build evict the dev entry and vice versa, so + // switching profiles back and forth could never be incremental AND the + // surviving entry pointed at the other profile's build dir. std::erase_if(entries, [&](const BuildCacheEntry& e) { - return e.targetTriple == targetTriple; + return e.targetTriple == targetTriple && e.profile == profile; }); // Insert at front (MRU). BuildCacheEntry newEntry{targetTriple, outputDir.string(), ninjaProgram, fingerprintHex, runtimeEnvKey, runtimeEnvValue, std::move(runTargets), - runEnvKey, runEnvValue}; + runEnvKey, runEnvValue, profile, cacheMode}; entries.insert(entries.begin(), std::move(newEntry)); // Trim to LRU capacity. @@ -175,6 +209,8 @@ void write_build_cache(const std::filesystem::path& projectRoot, for (auto& [name, exe] : e.runTargets) f << name << '\t' << exe << '\n'; f << "runEnv=" << e.runEnvKey << '\n'; f << e.runEnvValue << '\n'; + f << "profile=" << e.profile << '\n'; + f << "cacheMode=" << e.cacheMode << '\n'; } } @@ -255,7 +291,10 @@ compute_run_env(const mcpp::build::BuildPlan& plan) { // resolution banner). export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, std::string_view targetOverride = "") { - if (no_cache) { + // `--cache=off` means a cold build: no global cache, and target/ cleared — + // which is exactly what `--no-cache` has always done, hence the alias. + const bool coldBuild = no_cache || ctx.cacheMode == CacheMode::Off; + if (coldBuild) { std::error_code ec; std::filesystem::remove_all(ctx.outputDir, ec); } @@ -267,13 +306,14 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, mcpp::ui::status("Inferred", note); } - // Announce the package being built (and any deps). - // Deps that hit the BMI cache get "Cached" instead of "Compiling". - std::set cachedNames; - for (auto& label : ctx.cachedDepLabels) { - auto sp = label.find(' '); - cachedNames.insert(sp == std::string::npos ? label : label.substr(0, sp)); - } + // Announce the package being built (and any deps). A dep served from the + // global cache says "Cached" and HOW MANY translation units that saved. The + // count is the point: the bare word "Cached" was printed for three months + // while ninja recompiled every one of those units behind it, and no output + // contradicted it. A number that has to match the edges ninja actually skips + // cannot be quietly wrong in the same way. + std::map cachedUnits; + for (auto& dep : ctx.cachedDeps) cachedUnits[dep.name] = dep.units; std::set announced; announced.insert(ctx.manifest.package.name); mcpp::ui::status("Compiling", @@ -283,8 +323,13 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, if (announced.contains(name)) continue; announced.insert(name); std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; - const char* verb = cachedNames.contains(name) ? "Cached" : "Compiling"; - mcpp::ui::status(verb, std::format("{} {}", name, ver)); + auto it = cachedUnits.find(name); + if (it == cachedUnits.end()) { + mcpp::ui::status("Compiling", std::format("{} {}", name, ver)); + } else { + mcpp::ui::status("Cached", std::format("{} {} ({} unit{})", + name, ver, it->second, it->second == 1 ? "" : "s")); + } } mcpp::build::BuildOptions opts; @@ -301,7 +346,11 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, return 1; } - // M3.2: populate BMI cache for deps that did NOT hit cache. + // Populate the global cache for deps that did NOT hit. prepare_build leaves + // depsToPopulate empty under --cache=local|off, so the mode gate is already + // enforced there; asserting it again here keeps the write side legible on + // its own terms rather than as a consequence of something in prepare. + if (ctx.cacheMode != CacheMode::Global) ctx.depsToPopulate.clear(); for (auto& task : ctx.depsToPopulate) { auto pr = mcpp::bmi_cache::populate_from(task.key, ctx.outputDir, task.artifacts); if (!pr) { @@ -312,10 +361,17 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, } // P1.5: warn if fingerprint changed from last build (explains full rebuild). + // Compared against the entry for the SAME profile: the profile is now a + // fingerprint input, so a dev↔release switch always changes the fp. That is + // exactly what the user asked for, and warning about it turns a useful + // signal ("something you didn't expect invalidated your build dir") into + // noise on every profile switch. { auto entries = read_build_cache(ctx.projectRoot); for (auto& e : entries) { - if (e.targetTriple == targetOverride && !e.fingerprint.empty()) { + if (e.targetTriple == targetOverride && e.profile == ctx.profile + && e.cacheMode == cache_mode_name(ctx.cacheMode) + && !e.fingerprint.empty()) { auto newFp = ctx.outputDir.filename().string(); if (e.fingerprint != newFp) { mcpp::ui::warning(std::format( @@ -328,7 +384,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, } // P0: save build cache for fast-path on next invocation. - if (!no_cache && !r->ninjaProgram.empty()) { + if (!coldBuild && !r->ninjaProgram.empty()) { auto fpHex = ctx.outputDir.filename().string(); auto runTargets = compute_run_targets(ctx.plan); auto [runEnvKey, runEnvValue] = compute_run_env(ctx.plan); @@ -336,7 +392,8 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, std::string(targetOverride), fpHex, r->runtimeEnvKey.empty() ? "-" : r->runtimeEnvKey, r->runtimeEnvValue, - std::move(runTargets), runEnvKey, runEnvValue); + std::move(runTargets), runEnvKey, runEnvValue, + ctx.profile, std::string(cache_mode_name(ctx.cacheMode))); } // The one place the --strict policy is settled. Degradations reported by @@ -347,7 +404,17 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, // exact failure mode it exists to prevent. if (!mcpp::diag::flush(ctx.strict)) return 1; - mcpp::ui::finished("release", r->elapsed); + // The descriptor comes from the knobs this build actually resolved, so it + // cannot disagree with the compiler flags the way the old hardcoded + // "release [optimized]" did. + { + const auto& bc = ctx.manifest.buildConfig; + std::string descriptor = + (bc.optLevel.empty() || bc.optLevel == "0") ? "unoptimized" : "optimized"; + if (bc.debug) descriptor += " + debuginfo"; + if (bc.lto) descriptor += " + lto"; + mcpp::ui::finished(ctx.profile, r->elapsed, descriptor); + } return 0; } @@ -461,6 +528,32 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, return 0; } +// Which profile does this invocation mean? The fast paths exist precisely to +// avoid prepare_build, where the profile is normally settled — so they settle +// it here from the same pure rule (resolve_profile_name), which needs nothing +// but the manifest. nullopt = manifest unreadable ⇒ no fast path. +// +// Without this, an entry matched on target triple alone could have been built +// for a different profile, and the fast path would run ninja against that +// profile's build.ninja: `mcpp build --release` then a bare `mcpp build` +// reported success in 0.00s and left -O2 artifacts where -O0 -g was asked for. +struct FastPathIdentity { + std::string profile; + std::string cacheMode; +}; + +std::optional +fast_path_identity(const std::filesystem::path& projectRoot, + std::string_view profileOverride = "") { + auto m = mcpp::manifest::load(projectRoot / "mcpp.toml"); + if (!m) return std::nullopt; + return FastPathIdentity{ + mcpp::build::resolve_profile_name(*m, profileOverride), + std::string(mcpp::build::cache_mode_name( + mcpp::build::resolve_cache_mode(*m, ""))), + }; +} + // Try to fast-path: if build.ninja is newer than all inputs, just run ninja. // Returns exit code on fast-path, or nullopt if full rebuild needed. export std::optional try_fast_build(const std::filesystem::path& projectRoot, @@ -468,11 +561,21 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo std::string_view currentTarget = "") { if (no_cache) return std::nullopt; - // P3: read multi-entry cache and find entry matching currentTarget. + auto want = fast_path_identity(projectRoot); + if (!want) return std::nullopt; + + // P3: read multi-entry cache and find the entry matching this + // (target, profile, cache mode) triple. Matching on the target alone served + // the wrong profile's artifacts, and ignoring the cache mode replayed a + // cache-reading graph for a request that asked not to read the cache. auto entries = read_build_cache(projectRoot); const BuildCacheEntry* match = nullptr; for (auto& e : entries) { - if (e.targetTriple == currentTarget) { match = &e; break; } + if (e.targetTriple == currentTarget && e.profile == want->profile + && e.cacheMode == want->cacheMode) { + match = &e; + break; + } } if (!match) return std::nullopt; @@ -521,7 +624,7 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo if (!rc) return std::nullopt; if (*rc != 0) return rc; - mcpp::ui::finished("release", elapsed); + mcpp::ui::finished(want->profile, elapsed); return 0; } @@ -536,10 +639,17 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo std::optional try_fast_run(const std::filesystem::path& projectRoot, const std::optional& targetName, std::span passthrough) { + auto want = fast_path_identity(projectRoot); + if (!want) return std::nullopt; + auto entries = read_build_cache(projectRoot); const BuildCacheEntry* match = nullptr; for (auto& e : entries) { - if (e.targetTriple.empty()) { match = &e; break; } + if (e.targetTriple.empty() && e.profile == want->profile + && e.cacheMode == want->cacheMode) { + match = &e; + break; + } } if (!match || match->runTargets.empty()) return std::nullopt; @@ -615,7 +725,9 @@ std::optional try_fast_run(const std::filesystem::path& projectRoot, // rule mcpp::project::resolve_member_dir documents for build/test). export int build_run_target(const std::optional& targetName, std::span passthrough, - const std::string& package_filter = {}) { + const std::string& package_filter = {}, + const std::string& cache_mode = {}, + bool no_cache = false) { // mcpp#225 (E2): reuse the resolved build cache when it's still fresh, // skipping prepare_build's toolchain resolution + modgraph scan // entirely — mirrors cmd_build's try_fast_build fast path. The cached @@ -623,7 +735,10 @@ export int build_run_target(const std::optional& targetName, // last time; a `-p` filter always needs prepare_build's member switch, // so skip the fast path in that case (mirrors cmd_build's fast-path // bypass whenever ov.package_filter is set). - if (package_filter.empty()) { + // A --cache/--no-cache override also bypasses the fast path, for the same + // reason --profile does: the cached build.ninja was generated under the + // previous mode, so reusing it would silently ignore the flag. + if (package_filter.empty() && cache_mode.empty() && !no_cache) { if (auto root = mcpp::project::find_manifest_root(std::filesystem::current_path())) { if (auto rc = try_fast_run(*root, targetName, passthrough)) { return *rc; @@ -635,10 +750,11 @@ export int build_run_target(const std::optional& targetName, // the binary, so we don't re-resolve the toolchain or re-scan modgraph. mcpp::build::BuildOverrides ov; ov.package_filter = package_filter; + ov.cache_mode = cache_mode; auto ctx = prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, /*extraTargets=*/{}, ov); if (!ctx) { std::println(stderr, "error: {}", ctx.error()); return 2; } - if (auto rc = run_build_plan(*ctx, /*verbose=*/false, /*no_cache=*/false); rc != 0) + if (auto rc = run_build_plan(*ctx, /*verbose=*/false, no_cache); rc != 0) return rc; // Find binary target @@ -843,11 +959,22 @@ export int run_tests(std::span passthrough, } // 4. "Compiling test_X (test)" lines for the test binaries. - std::set cachedNames; - for (auto& label : ctx->cachedDepLabels) { - auto sp = label.find(' '); - cachedNames.insert(sp == std::string::npos ? label : label.substr(0, sp)); - } + std::map cachedUnits; + for (auto& dep : ctx->cachedDeps) cachedUnits[dep.name] = dep.units; + auto announce = [&](const std::string& name, + const mcpp::manifest::DependencySpec& spec, + std::string_view suffix) { + std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; + auto it = cachedUnits.find(name); + if (it == cachedUnits.end()) { + mcpp::ui::status("Compiling", + std::format("{} {}{}", name, ver, suffix)); + } else { + mcpp::ui::status("Cached", + std::format("{} {} ({} unit{}){}", name, ver, it->second, + it->second == 1 ? "" : "s", suffix)); + } + }; std::set announced; announced.insert(ctx->manifest.package.name); mcpp::ui::status("Compiling", @@ -856,17 +983,12 @@ export int run_tests(std::span passthrough, for (auto& [name, spec] : ctx->manifest.dependencies) { if (announced.contains(name)) continue; announced.insert(name); - std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; - const char* verb = cachedNames.contains(name) ? "Cached" : "Compiling"; - mcpp::ui::status(verb, std::format("{} {}", name, ver)); + announce(name, spec, ""); } for (auto& [name, spec] : ctx->manifest.devDependencies) { if (announced.contains(name)) continue; announced.insert(name); - std::string ver = spec.isPath() ? "(path)" : std::string("v") + spec.version; - const char* verb = cachedNames.contains(name) ? "Cached" : "Compiling"; - mcpp::ui::status(verb, - std::format("{} {} (dev)", name, ver)); + announce(name, spec, " (dev)"); } // List test binaries. // (Per-test "Compiling" lines print in Phase B, interleaved with each @@ -1131,9 +1253,11 @@ export int clean_project(bool wipe_bmi) { std::println("Cleaned: {}", (*root / "target").string()); if (wipe_bmi) { - auto bmi = mcpp::toolchain::default_cache_root(); - std::filesystem::remove_all(bmi, ec); - std::println("Cleaned BMI cache: {}", bmi.string()); + auto cache = mcpp::toolchain::default_cache_root(); + std::filesystem::remove_all(cache, ec); + std::println("Cleaned build cache: {}", cache.string()); + std::println(" (`mcpp cache clean --legacy` also removes the unused " + "pre-v1 cache, if any)"); } return 0; } diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index d0b60a9a..fda83ee0 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -872,6 +872,11 @@ std::string emit_ninja_string(const BuildPlan& plan) { append("\n"); } + // Aggregate target for everything staged out of the global cache. Named + // with a leading underscore so it cannot collide with a module or target + // name (both of which are identifiers or paths). + constexpr std::string_view kStagedCachePhony = "_mcpp_staged_cache"; + auto bmi_path = [&traits](std::string_view name) { std::string s(traits.bmiDir); s += '/'; @@ -896,6 +901,77 @@ std::string emit_ninja_string(const BuildPlan& plan) { return "cxx_object"; }; + // ── Cache-served units: stage edges instead of compile edges ──────── + // + // A unit whose outputs are already in the global cache gets one stage_file + // edge per artifact and is then skipped by every loop below — no scan edge, + // no dyndep edge, no compile edge. Its outputs land at exactly the paths a + // compile edge would have produced, so the link edges, the BMI implicit + // inputs of consuming TUs and the runtime deployment edges are unchanged. + // + // This indirection is the whole point. The cache used to copy artifacts + // into the build dir from inside prepare, leaving them declared as compile + // edge outputs — and ninja treats an output with no command line in + // .ninja_log as dirty ("command line not found in log"), so a fresh build + // dir recompiled every one of them while the CLI printed "Cached". Making + // the staging an edge is what gives ninja a record to compare against. + // + // `--verify size` for the same reason the std artifacts use it: the entry + // directory is named by a key covering the toolchain, dialect, profile, + // this package's own config and its dependencies' keys, so for a given key + // equal size IS equivalence — and size comes from directory metadata, which + // stays readable even when a holder denies opening the file. + // + // ORDERING. Replacing a package's compile edges with stage edges also + // removes the ordering those compile edges carried. A module partition is + // the case that breaks: a consumer imports `pkg`, so its dyndep declares + // `pkg`'s BMI and nothing else — the partition BMI `pkg:part` was reached + // only because `pkg`'s own compile edge depended on it. With independent + // stage edges, ninja may finish staging `pkg` and start the consumer while + // `pkg:part` is still unstaged, and Clang then fails with + // `failed to find module file for module 'pkg:part'`. (Observed on macOS + // CI; Linux happened to win the race, which is exactly why this is stated + // as an invariant rather than left to scheduling.) + // + // So every staged artifact becomes an ORDER-ONLY prerequisite of every edge + // that is not itself staged. Order-only (`||`) is the right strength: the + // real content dependencies are still declared where they always were (a + // dyndep-supplied implicit input, or the static-mode implicit list), so a + // changed BMI still invalidates its consumers — this adds sequencing, not + // dirtiness. The cost is that a handful of copies finish before compilation + // starts, which is what used to happen anyway when those units were built. + std::string stagedOrderOnly; + { + std::vector staged; + for (auto& cu : plan.compileUnits) { + if (!cu.servedFromCache) continue; + if (cu.cachedObject.empty()) continue; + auto obj = escape_ninja_path(cu.object); + append(std::format("build {} : stage_file {}\n", obj, + escape_ninja_path(cu.cachedObject))); + append(" verify = --verify size\n"); + staged.push_back(obj); + if (cu.providesModule && !cu.cachedBmi.empty()) { + auto bmi = bmi_path(*cu.providesModule); + append(std::format("build {} : stage_file {}\n", bmi, + escape_ninja_path(cu.cachedBmi))); + append(" verify = --verify size\n"); + staged.push_back(bmi); + } + } + if (!staged.empty()) { + append("\n"); + // One phony aggregates them so each consuming edge names a single + // prerequisite instead of repeating the whole list (mcpp#274: long + // ninja lines are how a 50781-character command line blew past + // cmd.exe's 8191 limit on Windows). + append("build " + std::string(kStagedCachePhony) + " : phony"); + for (auto& s2 : staged) append(" " + s2); + append("\n\n"); + stagedOrderOnly = " || " + std::string(kStagedCachePhony); + } + } + if (dyndep) { // ── Phase 1: scan edges (one .ddi per TU). ────────────────────── // .ddi is placed beside the object so multi-version mangling can @@ -910,12 +986,13 @@ std::string emit_ninja_string(const BuildPlan& plan) { std::vector ddi_paths; ddi_paths.reserve(plan.compileUnits.size()); for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; // staged, never scanned if (is_scan_exempt(cu.source)) continue; auto ddi = (cu.object.parent_path() / cu.source.filename()).string() + ".ddi"; ddi_paths.push_back(ddi); - append(std::format("build {} : cxx_scan {}\n", escape_ninja_path(ddi), - escape_ninja_path(cu.source))); + append(std::format("build {} : cxx_scan {}{}\n", escape_ninja_path(ddi), + escape_ninja_path(cu.source), stagedOrderOnly)); append(std::format(" compile_target = {}\n", escape_ninja_path(cu.object))); if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) append(std::format(" local_includes ={}\n", includes)); @@ -941,6 +1018,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { }(); std::map ddi_expect; for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; if (is_scan_exempt(cu.source)) continue; if (!cu.scanOverridden && !verifyAll) continue; auto ddi = (cu.object.parent_path() / cu.source.filename()).string() + ".ddi"; @@ -972,6 +1050,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { // Each compile edge references its OWN .dd file instead of a global one. // P2: module compile edges get a $bmi_out variable for BMI preservation. for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; // a stage_file edge owns these outputs std::string rule = pick_rule(cu.source); std::string out_line = "build " + escape_ninja_path(cu.object); @@ -984,6 +1063,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { auto it = ddi_to_dd.find(ddi); if (it != ddi_to_dd.end()) { out_line += " | " + it->second; + out_line += stagedOrderOnly; out_line += "\n dyndep = " + it->second; // P2: set bmi_out for the copy_if_different logic in cxx_module. if (cu.providesModule) { @@ -991,10 +1071,10 @@ std::string emit_ninja_string(const BuildPlan& plan) { } out_line += "\n"; } else { - out_line += "\n"; + out_line += stagedOrderOnly + "\n"; } } else { - out_line += "\n"; + out_line += stagedOrderOnly + "\n"; } if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) out_line += " local_includes =" + includes + "\n"; @@ -1014,6 +1094,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { } else { // ── Static-deps mode (M3.2 and earlier). ──────────────────────── for (auto& cu : plan.compileUnits) { + if (cu.servedFromCache) continue; // a stage_file edge owns these outputs std::string rule = pick_rule(cu.source); std::string implicit; @@ -1045,6 +1126,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { out_line += std::format(" : {} {}", rule, escape_ninja_path(cu.source)); if (!implicit.empty()) out_line += " |" + implicit; + out_line += stagedOrderOnly; out_line += "\n"; if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) out_line += " local_includes =" + includes + "\n"; diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 63f1c884..ff28a4f1 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -33,6 +33,14 @@ struct CompileUnit { // Unit came from a scan_overrides declaration — plan-vs-ddi // verification is mandatory for it (ninja_backend emits --expect-*). bool scanOverridden = false; + // This unit's outputs are already in the global cache: the backend emits + // `stage_file` edges from the cache instead of a compile edge (and skips + // the P1689 scan for it entirely). The unit itself stays in the plan so + // compile_commands.json keeps an entry for it and clangd does not lose the + // dependency's sources. + bool servedFromCache = false; + std::filesystem::path cachedObject; // absolute, inside the cache + std::filesystem::path cachedBmi; // absolute; empty if no module }; struct LinkUnit { diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 9a08d907..617ccc0b 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -31,6 +31,7 @@ import mcpp.toolchain.post_install; import mcpp.toolchain.abi; import mcpp.toolchain.triple; import mcpp.build.plan; +import mcpp.build.cache_key; import mcpp.build.build_program; import mcpp.lockfile; import mcpp.config; @@ -207,7 +208,10 @@ export std::filesystem::path target_dir(const mcpp::toolchain::Toolchain& tc, // Compose a stable canonical compile-flags string for fingerprinting. -std::string canonical_compile_flags(const mcpp::manifest::Manifest& m) { +// Exported so the "every build-variant knob is in here" invariant is machine- +// checkable: the profile knobs were absent for a long time precisely because +// nothing could assert on this string. +export std::string canonical_compile_flags(const mcpp::manifest::Manifest& m) { std::string s; s += "-std="; s += m.package.standard; s += " -fmodules"; @@ -262,6 +266,18 @@ std::string canonical_compile_flags(const mcpp::manifest::Manifest& m) { for (auto const& f : gf.asmflags) { s += " gas:"; s += f; } for (auto const& f : gf.defines) { s += " gd:"; s += f; } } + // The resolved [profile] knobs. These are NOT in cflags/cxxflags: the + // profile block (see the profile resolution below) lands them in + // buildConfig.optLevel/debug/lto/strip and flags.cppm turns them into + // -O/-g/-flto at command-construction time. Leaving them out made + // `--dev`, `--release` and `--profile dist` share ONE fingerprint, hence + // one target/// directory AND one global cache entry — so a + // release build could be served -O0 -g dependency objects. They are + // build-variant by definition; they belong here. + s += " opt="; s += m.buildConfig.optLevel; + s += " debug="; s += m.buildConfig.debug ? "1" : "0"; + s += " lto="; s += m.buildConfig.lto ? "1" : "0"; + s += " strip="; s += m.buildConfig.strip ? "1" : "0"; return s; } @@ -570,6 +586,35 @@ bool graph_or_targets_import_std(const mcpp::modgraph::Graph& graph, return false; } +// How this invocation may use the global dependency cache. +// +// Global read + write (default) +// Local neither — every dependency is compiled inside this project's +// target/, which is what every build did before the cache worked +// Off neither, and this build's target/// directory is +// cleared first (a full cold rebuild). Sibling build dirs — other +// profiles, other targets — are left alone. +// +// `--no-cache` used to be the only switch and it meant "clear the build dir", +// which says nothing about a cache (and its help text claimed all of target/); +// it stays as a deprecated alias for Off. +export enum class CacheMode { Global, Local, Off }; + +export std::optional parse_cache_mode(std::string_view v) { + if (v == "global") return CacheMode::Global; + if (v == "local") return CacheMode::Local; + if (v == "off" || v == "none") return CacheMode::Off; + return std::nullopt; +} + +export std::string_view cache_mode_name(CacheMode m) { + switch (m) { + case CacheMode::Local: return "local"; + case CacheMode::Off: return "off"; + default: return "global"; + } +} + export struct BuildContext { // --strict: degradations reported through mcpp::diag become errors. // Carried on the context because the build's degradations are discovered @@ -584,6 +629,14 @@ export struct BuildContext { std::filesystem::path stdBmi; std::filesystem::path stdObject; mcpp::build::BuildPlan plan; + // Resolved profile name (resolve_profile_name). Carried so run_build_plan + // can record it in .build_cache — without it the fast path cannot tell + // whether a cached build.ninja was generated for the profile being asked + // for — and so `Finished ` stops being a hardcoded "release". + std::string profile; + // Resolved global-cache mode. Read side is honored in prepare_build; write + // side in run_build_plan. + CacheMode cacheMode = CacheMode::Global; // M3.2 BMI cache: deps that did NOT hit cache and therefore need // populate_from(...) AFTER backend.build succeeds. @@ -593,10 +646,55 @@ export struct BuildContext { }; std::vector depsToPopulate; - // Names of deps that DID hit cache (for ui status output). - std::vector cachedDepLabels; // "mcpplibs.cmdline v0.0.1" + // Deps that DID hit the global cache, and how many compile units each one + // spared. run_build_plan reports the count so the "Cached" line cannot be + // true-looking and empty at the same time. + struct CachedDep { + std::string name; + std::string version; + std::size_t units = 0; + }; + std::vector cachedDeps; }; +// The ONE cache-mode resolver, for the same reason resolve_profile_name exists: +// execute.cppm's fast paths deliberately skip prepare_build, so they need to +// settle the mode from the same rule. Pure in (manifest, override, environment). +// +// `--cache` on the command line already bypasses the fast path, so the override +// argument is empty there; it is threaded anyway so there is exactly one place +// where precedence is written down. +// +// Precedence: --cache > MCPP_BUILD_CACHE > [build] cache > global. An +// unparseable value falls through to the next source rather than silently +// meaning "global" — see prepare_build, which also reports it. +export CacheMode resolve_cache_mode(const mcpp::manifest::Manifest& m, + std::string_view override_mode) { + if (auto v = parse_cache_mode(override_mode)) return *v; + if (const char* e = std::getenv("MCPP_BUILD_CACHE"); e && *e) + if (auto v = parse_cache_mode(e)) return *v; + if (auto v = parse_cache_mode(m.buildConfig.cacheMode)) return *v; + return CacheMode::Global; +} + +// The ONE profile-name resolver. Shared with execute.cppm's fast paths: +// they deliberately skip prepare_build, so before this existed they had no +// idea which profile the request meant — and `.build_cache` keyed entries by +// target triple alone. Net effect: `mcpp build --release` followed by a bare +// `mcpp build` reported success in 0.00s and left the RELEASE artifacts in +// place. The rule is pure (manifest + one override string), so both sides can +// evaluate it without resolving a toolchain or scanning the module graph. +// +// Precedence: --profile/--release/--dev > [build].default-profile > "dev". +// The global default is "dev" (-O0 -g) per the dominant convention +// (Cargo/Meson/CMake/Zig/Bazel/MSBuild all default to debug). +export std::string resolve_profile_name(const mcpp::manifest::Manifest& m, + std::string_view override_name) { + if (!override_name.empty()) return std::string(override_name); + if (!m.buildConfig.defaultProfile.empty()) return m.buildConfig.defaultProfile; + return "dev"; +} + // Command-level overrides (--target / --static). // Empty defaults preserve pre-existing behaviour exactly. export struct BuildOverrides { @@ -607,6 +705,7 @@ export struct BuildOverrides { std::string features; // --features a,b,c (root package activation) bool strict = false; // --strict: schema warnings become errors std::string capabilities; // --cap blas=openblas,lapack=mkl (provider pins) + std::string cache_mode; // --cache global|local|off ("" = unset) }; // `prepare_build` builds the BuildContext for any verb that compiles. @@ -738,6 +837,32 @@ prepare_build(bool print_fingerprint, mcpp::diag::warning("manifest/schema", w); } + // Global-cache mode: --cache > MCPP_BUILD_CACHE > [build] cache > global. + // An unparseable value is a warning (error under --strict) and falls + // through to the next source rather than silently meaning "global" — a typo + // that quietly re-enabled the cache would be the hardest kind of surprise + // to attribute. + // Selection lives in resolve_cache_mode (above) so the fast paths settle it + // identically. This block only adds the diagnostics, which the fast paths + // have no business emitting: an unparseable value must be reported once, by + // the invocation that actually resolves the build. + const CacheMode cacheMode = resolve_cache_mode(*m, overrides.cache_mode); + { + const char* envMode = std::getenv("MCPP_BUILD_CACHE"); + for (auto [value, origin] : std::initializer_list< + std::pair>{ + {overrides.cache_mode, "--cache"}, + {envMode ? envMode : "", "MCPP_BUILD_CACHE"}, + {m->buildConfig.cacheMode, "[build] cache"}}) { + if (value.empty() || parse_cache_mode(value)) continue; + auto msg = std::format( + "{} has unknown cache mode '{}' (expected: global | local | off)", + origin, value); + if (overrides.strict) return std::unexpected(msg); + mcpp::diag::warning("build/cache-mode", msg); + } + } + // ─── Toolchain resolution (docs/21) ──────────────────────────────── // Priority chain: // 1. mcpp.toml [toolchain]. → resolve_xpkg_path → abs path @@ -780,16 +905,13 @@ prepare_build(bool print_fingerprint, std::string effectiveProfile; { auto& pname = effectiveProfile; - // Precedence: --profile / --release / --dev flag (overrides.profile) > - // [build].default-profile (project default) > "dev" (global default). - // The global default is "dev" (-O0 -g) to follow the dominant convention - // (Cargo/Meson/CMake/Zig/Bazel/MSBuild all default to debug); release is - // opt-in via --release / --profile release. A project that wants its - // plain `mcpp build` optimized sets [build].default-profile = "release" - // (mcpp's own mcpp.toml does this, so the released binary stays -O2). - pname = !overrides.profile.empty() ? overrides.profile - : !m->buildConfig.defaultProfile.empty() ? m->buildConfig.defaultProfile - : "dev"; + // Precedence lives in resolve_profile_name (above) so execute.cppm's + // fast paths settle it identically without running prepare_build. + // Release is opt-in via --release / --profile release; a project that + // wants its plain `mcpp build` optimized sets + // [build].default-profile = "release" (mcpp's own mcpp.toml does this, + // so the released binary stays -O2). + pname = resolve_profile_name(*m, overrides.profile); mcpp::manifest::Profile pr; if (pname == "dev" || pname == "debug") { pr.optLevel = "0"; pr.debug = true; } else if (pname == "dist") { pr.optLevel = "3"; pr.strip = true; } @@ -1376,6 +1498,11 @@ prepare_build(bool print_fingerprint, std::string indexName; std::string packageName; std::string version; + // "version" | "path" | "git". Only "version" is cacheable: an index + // package's payload lives in the immutable xpkgs store under a + // version-keyed directory, so name@version identifies its sources. + // Path and git checkouts can change under an unchanged identity. + std::string sourceKind; }; std::vector dep_cache_identities; struct GitLockIdentity { @@ -2685,6 +2812,7 @@ prepare_build(bool print_fingerprint, .indexName = cache_index_name(key.ns), .packageName = mangled, .version = spec.version, + .sourceKind = "version", }); const auto depPackageIndex = packages.size(); packages.push_back(makePackageRoot(secStage, *dep_manifests.back())); @@ -2982,6 +3110,7 @@ prepare_build(bool print_fingerprint, .version = sourceKind == "version" ? spec.version : dep_manifests.back()->package.version, + .sourceKind = sourceKind, }); const auto depPackageIndex = packages.size(); packages.push_back(makePackageRoot(dep_root, *dep_manifests.back())); @@ -3605,7 +3734,7 @@ prepare_build(bool print_fingerprint, // pieces were already in the fingerprint; this fixes the COMMAND // construction the fingerprint promised (stdFlagAndDialect above). auto sm = mcpp::toolchain::ensure_built( - *tc, fp.hex, m->package.standard, stdFlagAndDialect, + *tc, m->package.standard, stdFlagAndDialect, mcpp::platform::macos::deployment_target( m->buildConfig.macosDeploymentTarget)); if (!sm) return std::unexpected(sm.error().message); @@ -3628,6 +3757,8 @@ prepare_build(bool print_fingerprint, ctx.manifest = *m; ctx.tc = *tc; ctx.fp = fp; + ctx.profile = effectiveProfile; + ctx.cacheMode = cacheMode; ctx.projectRoot= *root; ctx.outputDir = target_dir(*tc, fp, *root); ctx.stdBmi = stdBmiPath; @@ -3717,16 +3848,20 @@ prepare_build(bool print_fingerprint, } } - // ─── M3.2: BMI cache stage / populate-task collection ───────────── - // For each version-based dep package (i.e. fetched from a registry, - // not a path dep), check the global BMI cache. If cached → stage into - // the project's target dir so ninja sees those outputs as up-to-date - // and skips them. If not → record a populate task for AFTER build. + // ─── Global dependency cache: per-package keys, hit → stage edges ── + // + // Every index package gets a key over the axes that actually reach its + // compiler command lines (mcpp.build.cache_key), computed bottom-up so a + // package's key includes its direct dependencies' keys. A hit marks that + // package's compile units `servedFromCache`, and the ninja backend emits + // `stage_file` edges instead of compile edges for them — which is the only + // way ninja will accept a cached artifact. A miss records a populate task + // for after the build. // - // Path deps don't go through the cache: their sources can change at - // any time outside fingerprint awareness. + // `--cache=local|off` skips this block entirely: nothing is read and, in + // run_build_plan, nothing is written. auto cfg2 = get_cfg(); - if (cfg2) { + if (cfg2 && ctx.cacheMode == CacheMode::Global) { std::error_code mkEc; std::filesystem::create_directories(ctx.outputDir, mkEc); auto usable_object_rel = [](const std::filesystem::path& rel) @@ -3750,52 +3885,180 @@ prepare_build(bool print_fingerprint, } return objectPath.filename().generic_string(); }; + + // ── Per-package keys, bottom-up ────────────────────────────────── + // Axes A/B/C are whole-graph, so they are computed once. Axes D/E are + // per package. Axis F is each direct dependency's key, which forces a + // bottom-up order: `dependencyEdges` is a DAG (the modgraph validator + // rejects cycles), so a simple memoized recursion suffices — with an + // explicit in-progress guard so a cycle that slipped past validation + // fails loudly instead of recursing until the stack dies. + namespace ck = mcpp::build::cache_key; + const auto storeRoot = (*cfg2)->xlingsHome() / "data" / "xpkgs"; + auto axes = ck::build_axes( + *tc, *m, stdFlagAndDialect, + mcpp::toolchain::cppfly::effective_dialect_flags( + *tc, m->cppStandard.experimental, + mcpp::manifest::dialect_flags(m->buildConfig)), + mcpp::platform::macos::deployment_target( + m->buildConfig.macosDeploymentTarget)); + + // Sources belonging to each package, package-root-relative and sorted. + std::vector> pkgSources(packages.size()); + for (auto& cu : ctx.plan.compileUnits) { + // Longest matching root wins. Package roots can nest — a workspace + // member lives under the workspace root — and taking the first match + // would file the member's sources under the outer package, putting + // them in the wrong key. (Index payloads live in the xpkgs store and + // cannot be shadowed this way, so no cached entry is affected today; + // resolving it by specificity rather than by iteration order is what + // keeps that true if roots ever move.) + std::size_t best = packages.size(); + std::size_t bestLen = 0; + std::string bestRel; + for (std::size_t p = 0; p < packages.size(); ++p) { + std::error_code ec; + auto rel = std::filesystem::relative(cu.source, packages[p].root, ec); + if (ec || rel.empty()) continue; + auto rels = rel.generic_string(); + if (rels.starts_with("..")) continue; + auto len = packages[p].root.generic_string().size(); + if (best == packages.size() || len > bestLen) { + best = p; bestLen = len; bestRel = std::move(rels); + } + } + if (best != packages.size()) pkgSources[best].push_back(std::move(bestRel)); + } + for (auto& v : pkgSources) std::ranges::sort(v); + + std::vector pkgKeys(packages.size()); + std::vector pkgInputs(packages.size(), + nlohmann::json::object()); + std::vector keyState(packages.size(), 0); // 0 new/1 busy/2 done + std::string keyCycleError; + // Does this package's own transitive upstream contain anything that is + // not an immutable index payload? If so it cannot be cached either, even + // when the package itself is an index package. + // + // A key covers an upstream package by folding in that package's KEY, and + // a local package's key covers its file list but not its file CONTENTS — + // nothing could, without hashing a tree that may change between the hash + // and the compile. So editing a local upstream's source would leave a + // downstream entry looking valid. No index descriptor can declare a path + // dependency today, which makes this shape unreachable in practice; it is + // enforced structurally anyway, because "unreachable today" is how the + // transitive path-dep leak got in. + std::vector localTaint(packages.size(), 0); + auto compute_key = [&](auto&& self, std::size_t idx) -> const std::string& { + static const std::string kEmpty; + if (keyState[idx] == 2) return pkgKeys[idx]; + if (keyState[idx] == 1) { + if (keyCycleError.empty()) { + keyCycleError = std::format( + "dependency cycle through package '{}' while computing " + "its build-cache key", packages[idx].manifest.package.name); + } + return kEmpty; + } + keyState[idx] = 1; + + ck::PackageAxes pa; + if (idx > 0 && idx - 1 < dep_cache_identities.size()) { + pa.indexName = dep_cache_identities[idx - 1].indexName; + pa.packageName = dep_cache_identities[idx - 1].packageName; + pa.version = dep_cache_identities[idx - 1].version; + } + if (pa.packageName.empty()) { + // The root package, or a package with no resolution identity. + // It is never cached, but its key still has to exist because + // downstream packages fold it in via axis F. + pa.packageName = packages[idx].manifest.package.namespace_.empty() + ? packages[idx].manifest.package.name + : std::format("{}.{}", packages[idx].manifest.package.namespace_, + packages[idx].manifest.package.name); + } + if (pa.version.empty()) pa.version = packages[idx].manifest.package.version; + ck::fill_package_config(pa, packages[idx], storeRoot); + pa.sources = pkgSources[idx]; + const bool selfIsIndex = idx > 0 + && idx - 1 < dep_cache_identities.size() + && dep_cache_identities[idx - 1].sourceKind == "version"; + if (!selfIsIndex) localTaint[idx] = 1; + for (auto& e : dependencyEdges) { + if (e.consumerPackageIndex != idx) continue; + auto& up = self(self, e.dependencyPackageIndex); + if (!up.empty()) pa.upstreamKeys.push_back(up); + if (localTaint[e.dependencyPackageIndex]) localTaint[idx] = 1; + for (auto& f : e.requestedFeatures) pa.features.push_back(f); + } + std::ranges::sort(pa.upstreamKeys); + pa.upstreamKeys.erase(std::unique(pa.upstreamKeys.begin(), + pa.upstreamKeys.end()), + pa.upstreamKeys.end()); + std::ranges::sort(pa.features); + pa.features.erase(std::unique(pa.features.begin(), pa.features.end()), + pa.features.end()); + + pkgKeys[idx] = ck::key_hex(axes, pa); + pkgInputs[idx] = ck::to_json(axes, pa); + keyState[idx] = 2; + return pkgKeys[idx]; + }; + for (std::size_t i = 0; i < packages.size(); ++i) + (void)compute_key(compute_key, i); + if (!keyCycleError.empty()) return std::unexpected(keyCycleError); + for (std::size_t i = 1; i < packages.size(); ++i) { // skip [0] = main const auto& pkgRoot = packages[i]; const auto* depIdent = i - 1 < dep_cache_identities.size() ? &dep_cache_identities[i - 1] : nullptr; - const auto fallbackName = pkgRoot.manifest.package.namespace_.empty() - ? pkgRoot.manifest.package.name - : std::format("{}.{}", pkgRoot.manifest.package.namespace_, - pkgRoot.manifest.package.name); - const auto& depName = depIdent ? depIdent->packageName : fallbackName; - const auto& depVer = depIdent && !depIdent->version.empty() - ? depIdent->version - : pkgRoot.manifest.package.version; - - // Find this dep's spec from the consumer manifest to know - // if it's path-based or version-based. - auto specIt = m->dependencies.find(depName); - // Path AND git deps bypass the BMI cache: their sources can - // change outside the fingerprint's awareness. - bool skipCache = (specIt != m->dependencies.end() && - (specIt->second.isPath() || specIt->second.isGit())); - if (specIt == m->dependencies.end()) { - auto devIt = m->devDependencies.find(depName); - if (devIt != m->devDependencies.end()) { - skipCache = devIt->second.isPath() || devIt->second.isGit(); - } - } - if (skipCache) continue; + // Only index ("version") packages are cacheable, and the identity + // recorded at resolution time is the ONLY admissible evidence. + // + // The predicate this replaces looked the package up in the ROOT + // manifest's dependencies/dev-dependencies and skipped it when the + // spec was path/git. A transitively-reached package is in neither + // map, so `specIt == end()` left skipCache false and local sources + // were cached — with `indexName` falling back to defaultIndex, so a + // workspace member `B` landed on disk as `mcpplibs/B@0.1.0`. Its + // sources can then change without changing name@version, i.e. the + // cache key cannot see the change. + // + // Note the direction of the judgment: `mcpp add`'s existence gate + // admits anything it cannot disprove. A build cache must do the + // opposite — anything it cannot prove came from the immutable + // xpkgs store stays out, because the failure mode here is a + // silently wrong object rather than a rejected command. + if (!depIdent || depIdent->sourceKind != "version") continue; + // ...and neither may anything it was built against be local. + if (localTaint[i]) continue; + + const auto& depName = depIdent->packageName; + const auto& depVer = depIdent->version.empty() + ? pkgRoot.manifest.package.version + : depIdent->version; auto bmiT = mcpp::toolchain::bmi_traits(*tc); mcpp::bmi_cache::CacheKey key { - .mcppHome = (*cfg2)->mcppHome, - .fingerprint = fp.hex, - .indexName = depIdent - ? depIdent->indexName - : (*cfg2)->defaultIndex, + .cacheRoot = mcpp::home::cache_root(), + .indexName = depIdent->indexName, .packageName = depName, .version = depVer, + .keyHex = pkgKeys[i], + .inputs = pkgInputs[i], .bmiDirName = std::string(bmiT.bmiDir), .manifestTag = std::string(bmiT.manifestPrefix), }; - // Compute the artifacts list from the build plan: every - // CompileUnit whose source lies under this dep's root contributes. + // The artifacts this package contributes, and the compile units + // that produce them. Collected together so a hit can mark exactly + // those units — the artifact list alone would not say which edges + // must stop being compile edges. mcpp::bmi_cache::DepArtifacts arts; - for (auto& cu : ctx.plan.compileUnits) { + std::vector unitIdx; + for (std::size_t u = 0; u < ctx.plan.compileUnits.size(); ++u) { + auto& cu = ctx.plan.compileUnits[u]; std::error_code ec; auto rel = std::filesystem::relative(cu.source, pkgRoot.root, ec); if (ec || rel.empty()) continue; @@ -3810,16 +4073,32 @@ prepare_build(bool print_fingerprint, arts.bmiFiles.push_back(std::move(bmi)); } arts.objFiles.push_back(object_cache_path(cu.object)); + unitIdx.push_back(u); } if (mcpp::bmi_cache::is_cached(key)) { - auto staged = mcpp::bmi_cache::stage_into(key, ctx.outputDir); - if (staged) { - ctx.cachedDepLabels.push_back( - std::format("{} v{}", depName, depVer)); - continue; // skip populate task; it's already cached + // Mark the units. The backend turns each into a stage_file + // edge; nothing is copied here. Copying behind ninja's back is + // exactly what made the old cache a no-op: the staged file was + // still declared as a compile edge's output, and an output with + // no .ninja_log command-line record is dirty, so every unit was + // recompiled while the CLI printed "Cached". + for (auto u : unitIdx) { + auto& cu = ctx.plan.compileUnits[u]; + cu.servedFromCache = true; + cu.cachedObject = mcpp::bmi_cache::cached_obj_path( + key, object_cache_path(cu.object)); + if (cu.providesModule) { + std::string bmi; + for (char c : *cu.providesModule) + bmi.push_back(c == ':' ? '-' : c); + bmi += std::string(bmiT.bmiExt); + cu.cachedBmi = mcpp::bmi_cache::cached_bmi_path(key, bmi); + } } - // stage failed — fall through to recompile + repopulate + mcpp::bmi_cache::touch_accessed(key); + ctx.cachedDeps.push_back({depName, depVer, unitIdx.size()}); + continue; // no populate task; it is already cached } ctx.depsToPopulate.push_back({ std::move(key), std::move(arts) }); } diff --git a/src/cli.cppm b/src/cli.cppm index 998edcd7..8f75b35e 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -54,7 +54,7 @@ void print_usage() { std::println(" mcpp build [options] Build the current package"); std::println(" mcpp run [target] [-- args...] Build + run a binary target"); std::println(" mcpp test [pattern] [-- args...] Build + run tests/**/*.cpp (--list, --timeout, --message-format json)"); - std::println(" mcpp clean [--bmi-cache] Remove target/ (and optionally BMI cache)"); + std::println(" mcpp clean [--bmi-cache] Remove target/ (and optionally the build cache)"); std::println(" mcpp add [@] Add a dependency to mcpp.toml"); std::println(" mcpp remove Remove a dependency from mcpp.toml"); std::println(" mcpp update [pkg] Re-resolve deps and rewrite mcpp.lock"); @@ -66,7 +66,7 @@ void print_usage() { std::println(""); std::println("Resource management:"); std::println(" mcpp toolchain install|list|default Manage mcpp's private toolchains"); - std::println(" mcpp cache list|prune|clean|info Inspect/manage the global BMI cache"); + std::println(" mcpp cache dir|list|info|gc|... Inspect/manage the global build cache"); std::println(" mcpp index list|add|remove|update Manage package registries"); std::println(""); std::println("About mcpp itself:"); @@ -81,7 +81,8 @@ void print_usage() { std::println(" --verbose, -v Verbose compiler output"); std::println(" --quiet, -q Suppress status output"); std::println(" --print-fingerprint Show toolchain fingerprint and 10 inputs"); - std::println(" --no-cache Force-clear target/ before building"); + std::println(" --cache Dependency cache: global (default) | local | off"); + std::println(" --no-cache Deprecated alias for --cache=off (clears the build dir)"); std::println(" --no-color Disable colored output"); std::println(""); std::println("Docs: https://github.com/mcpp-community/mcpp/tree/main/docs"); @@ -218,8 +219,10 @@ int run(int argc, char** argv) { .description("Build the current package") .option(cl::Option("print-fingerprint") .help("Show toolchain fingerprint and 10 inputs")) + .option(cl::Option("cache").takes_value().value_name("MODE") + .help("Global dependency cache: global (default) | local | off")) .option(cl::Option("no-cache") - .help("Force-clear target/ before building")) + .help("Deprecated alias for --cache=off (also clears the build dir)")) .option(cl::Option("target").takes_value().help( "Build for (e.g. x86_64-linux-musl); looks up [target.] in mcpp.toml")) .option(cl::Option("static").help( @@ -246,6 +249,10 @@ int run(int argc, char** argv) { .arg(cl::Arg("target").help("Binary name (optional)")) .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") .help("Run only the named workspace member (single-member; no --workspace fan-out)")) + .option(cl::Option("cache").takes_value().value_name("MODE") + .help("Global dependency cache: global (default) | local | off")) + .option(cl::Option("no-cache") + .help("Deprecated alias for --cache=off (also clears the build dir)")) .action(wrap_rc([&passthrough](const cl::ParsedArgs& p) { return cmd_run(p, std::span(passthrough)); }))) @@ -269,14 +276,18 @@ int run(int argc, char** argv) { .help("Treat manifest schema warnings (unknown feature/platform) as errors")) .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") .help("Run tests only for the named workspace member")) + .option(cl::Option("cache").takes_value().value_name("MODE") + .help("Global dependency cache: global (default) | local | off")) + .option(cl::Option("no-cache") + .help("Deprecated alias for --cache=off (also clears the build dir)")) .option(cl::Option("workspace") .help("Run tests for all workspace members")) .action(wrap_rc([&passthrough](const cl::ParsedArgs& p) { return cmd_test(p, std::span(passthrough)); }))) .subcommand(cl::App("clean") - .description("Remove target/ (and optionally the global BMI cache)") - .option(cl::Option("bmi-cache").help("Also wipe the global BMI cache")) + .description("Remove target/ (and optionally the global build cache)") + .option(cl::Option("bmi-cache").help("Also wipe the global build cache (see `mcpp cache clean`)")) .action(wrap_rc(cmd_clean))) .subcommand(cl::App("why") .description("Explain how the toolchain / runtime / deps were resolved") @@ -389,24 +400,43 @@ int run(int argc, char** argv) { "Remove the payload for instead of the host one"))) .action(wrap_rc(cmd_toolchain))) .subcommand(cl::App("cache") - .description("Inspect and manage the global BMI cache") + .description("Inspect and manage the global build cache") + .subcommand(cl::App("dir") + .description("Print the cache root (and any pre-v1 cache)")) .subcommand(cl::App("list") - .description("List cache entries with size + last-access")) + .description("List cache entries with size + last-use") + .option(cl::Option("json").help("Emit machine-readable JSON"))) .subcommand(cl::App("info") - .description("Show details for a cached package") + .description("Show details (incl. key inputs) for a cached package") .arg(cl::Arg("pkg").help("@").required())) .subcommand(cl::App("prune") - .description("Drop entries older than a threshold") + .description("Drop entries not used within a threshold") .option(cl::Option("older-than").takes_value().value_name("N{s|m|h|d}") .help("Age threshold (e.g. 30d)"))) + .subcommand(cl::App("gc") + .description("LRU-collect package entries to a size and/or age budget") + .option(cl::Option("max-size").takes_value().value_name("N{MiB|GiB}") + .help("Keep the package cache under this size (e.g. 5GiB)")) + .option(cl::Option("older-than").takes_value().value_name("N{s|m|h|d}") + .help("Also drop entries unused for longer than this"))) .subcommand(cl::App("clean") - .description("Drop all dep cache entries (preserves std.gcm)")) + .description("Drop cache entries (default: package entries only)") + .option(cl::Option("deps").help("Drop package entries (default)")) + .option(cl::Option("std").help("Drop std module entries")) + .option(cl::Option("all").help("Drop both")) + .option(cl::Option("legacy") + .help("Remove the unused pre-v1 cache at $MCPP_HOME/bmi"))) + .subcommand(cl::App("verify") + .description("Check every entry's manifest against the files on disk")) .action(wrap_rc([&dispatch_sub](const cl::ParsedArgs& p) { return dispatch_sub("cache", p, { - {"list", cmd_cache_list}, - {"info", cmd_cache_info}, - {"prune", cmd_cache_prune}, - {"clean", cmd_cache_clean}, + {"dir", cmd_cache_dir}, + {"list", cmd_cache_list}, + {"info", cmd_cache_info}, + {"prune", cmd_cache_prune}, + {"gc", cmd_cache_gc}, + {"clean", cmd_cache_clean}, + {"verify", cmd_cache_verify}, }); }))) .subcommand(cl::App("index") diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index f324b5cf..715418b7 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -46,6 +46,11 @@ export int cmd_build(const mcpplibs::cmdline::ParsedArgs& parsed) { mcpp::build::BuildOverrides ov; if (auto t = parsed.value("target")) ov.target_triple = *t; if (auto p = parsed.value("package")) ov.package_filter = *p; + // --cache global|local|off. --no-cache is the deprecated alias for off; the + // old flag only ever cleared target/, which says nothing about a cache, so + // it is expressed in terms of the new one rather than kept as a second axis. + if (auto c = parsed.value("cache")) ov.cache_mode = *c; + else if (no_cache) ov.cache_mode = "off"; // Profile selection precedence: --profile NAME > --release / --dev > the // project default ([build].default-profile) > "release", resolved in // prepare_build. --release/--dev are shorthands only. @@ -81,7 +86,8 @@ export int cmd_build(const mcpplibs::cmdline::ParsedArgs& parsed) { // the fast path would silently ignore the flags. if (!print_fp && ov.target_triple.empty() && !ov.force_static && ov.profile.empty() && ov.features.empty() && !ov.strict - && ov.capabilities.empty() && ov.package_filter.empty()) { + && ov.capabilities.empty() && ov.package_filter.empty() + && ov.cache_mode.empty()) { auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); if (root) { if (auto rc = mcpp::build::try_fast_build(*root, verbose, no_cache)) { @@ -109,7 +115,12 @@ export int cmd_run(const mcpplibs::cmdline::ParsedArgs& parsed, // `mcpp run` is single-member only — no `--workspace` fan-out. std::string package_filter; if (auto p = parsed.value("package")) package_filter = *p; - return mcpp::build::build_run_target(targetName, passthrough, package_filter); + std::string cache_mode; + bool no_cache = parsed.is_flag_set("no-cache"); + if (auto c = parsed.value("cache")) cache_mode = *c; + else if (no_cache) cache_mode = "off"; + return mcpp::build::build_run_target(targetName, passthrough, package_filter, + cache_mode, no_cache); } export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, @@ -125,6 +136,8 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, if (auto cp = parsed.value("cap")) ov.capabilities = *cp; ov.strict = parsed.is_flag_set("strict"); if (auto p = parsed.value("package")) ov.package_filter = *p; + if (auto c = parsed.value("cache")) ov.cache_mode = *c; + else if (parsed.is_flag_set("no-cache")) ov.cache_mode = "off"; mcpp::build::TestOptions to; if (parsed.positional_count() > 0) to.filter = parsed.positional(0); diff --git a/src/cli/cmd_cache.cppm b/src/cli/cmd_cache.cppm index b2fc0e3a..f7a48eff 100644 --- a/src/cli/cmd_cache.cppm +++ b/src/cli/cmd_cache.cppm @@ -14,8 +14,12 @@ import mcpp.ui; namespace mcpp::cli { -export int cmd_cache_list(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { - return mcpp::bmi_cache::cache_list(); +export int cmd_cache_dir(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { + return mcpp::bmi_cache::cache_dir(); +} + +export int cmd_cache_list(const mcpplibs::cmdline::ParsedArgs& parsed) { + return mcpp::bmi_cache::cache_list(parsed.is_flag_set("json")); } export int cmd_cache_info(const mcpplibs::cmdline::ParsedArgs& parsed) { @@ -31,8 +35,21 @@ export int cmd_cache_prune(const mcpplibs::cmdline::ParsedArgs& parsed) { return mcpp::bmi_cache::cache_prune(parsed.option_or_empty("older-than").value()); } -export int cmd_cache_clean(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { - return mcpp::bmi_cache::cache_clean(); +export int cmd_cache_gc(const mcpplibs::cmdline::ParsedArgs& parsed) { + return mcpp::bmi_cache::cache_gc( + parsed.option_or_empty("max-size").value(), + parsed.option_or_empty("older-than").value()); +} + +export int cmd_cache_clean(const mcpplibs::cmdline::ParsedArgs& parsed) { + return mcpp::bmi_cache::cache_clean(parsed.is_flag_set("deps"), + parsed.is_flag_set("std"), + parsed.is_flag_set("all"), + parsed.is_flag_set("legacy")); +} + +export int cmd_cache_verify(const mcpplibs::cmdline::ParsedArgs& /*parsed*/) { + return mcpp::bmi_cache::cache_verify(); } } // namespace mcpp::cli diff --git a/src/config.cppm b/src/config.cppm index 68c57631..c352f772 100644 --- a/src/config.cppm +++ b/src/config.cppm @@ -6,7 +6,8 @@ // registry/ XLINGS_HOME for mcpp's xlings // bin/xlings vendored xlings binary (= /bin/xlings) // .xlings.json seeded with index_repos = [mcpplibs] -// bmi// BMI cache (existing) +// build-cache/v1/ cross-project build cache (pkg/ + std/) +// bmi/ pre-v1 build cache; unused, `cache clean --legacy` // cache/ metadata caches // config.toml this module's input // @@ -59,7 +60,7 @@ struct GlobalConfig { std::filesystem::path binDir; // mcppHome/bin std::filesystem::path xlingsBinary; // mcppHome/registry/bin/xlings std::filesystem::path registryDir; // mcppHome/registry - std::filesystem::path bmiCacheDir; // mcppHome/bmi + std::filesystem::path bmiCacheDir; // mcppHome/build-cache/v1 std::filesystem::path metaCacheDir; // mcppHome/cache std::filesystem::path logDir; // mcppHome/log std::filesystem::path configFile; // mcppHome/config.toml @@ -456,7 +457,7 @@ std::expected load_or_init( // making ensure_sandbox_xlings_binary() a no-op. cfg.xlingsBinary = cfg.registryDir / "bin" / (std::string("xlings") + std::string(mcpp::platform::exe_suffix)); - cfg.bmiCacheDir = mcpp::home::bmi_root(); + cfg.bmiCacheDir = mcpp::home::cache_root(); cfg.metaCacheDir = cfg.mcppHome / "cache"; cfg.logDir = cfg.mcppHome / "log"; cfg.configFile = cfg.mcppHome / "config.toml"; @@ -647,7 +648,7 @@ void print_env(const GlobalConfig& cfg) { std::println("xlings pinned = {}", kXlingsPinnedVersion); std::println("xlings home = {}", cfg.xlingsHome().string()); std::println("config = {}", cfg.configFile.string()); - std::println("BMI cache = {}", cfg.bmiCacheDir.string()); + std::println("build cache = {}", cfg.bmiCacheDir.string()); std::println("meta cache = {}", cfg.metaCacheDir.string()); if (!cfg.defaultToolchain.empty()) std::println("default toolchain = {}", cfg.defaultToolchain); diff --git a/src/doctor.cppm b/src/doctor.cppm index 2e64db30..0105a062 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -150,25 +150,29 @@ export int doctor_report() { mcpp::ui::status("Checking", "std module"); if (tc) { - auto cacheRoot = mcpp::toolchain::default_cache_root(); - auto fpDir = cacheRoot; + // Entries live at /std//{gcm,pcm}.cache/std.*; the + // object sits at the entry root. Look for the object rather than one + // compiler's BMI extension so this reports on clang and MSVC too. + auto stdRoot = mcpp::toolchain::default_cache_root() / "std"; std::error_code ec; - if (std::filesystem::exists(cacheRoot, ec)) { + if (std::filesystem::exists(stdRoot, ec)) { bool found = false; - for (auto& e : std::filesystem::directory_iterator(cacheRoot, ec)) { - auto stdgcm = e.path() / "std.gcm"; - if (std::filesystem::exists(stdgcm)) { - ok(std::format("{} ({})", - stdgcm.string(), - mcpp::bmi_cache::human_bytes(std::filesystem::file_size(stdgcm)))); + for (auto& e : std::filesystem::directory_iterator(stdRoot, ec)) { + for (auto name : {"std.o", "std.obj"}) { + auto obj = e.path() / name; + if (!std::filesystem::exists(obj, ec)) continue; + ok(std::format("{} (entry {})", e.path().string(), + mcpp::bmi_cache::human_bytes( + mcpp::bmi_cache::dir_size(e.path())))); found = true; break; } + if (found) break; } - if (!found) warn("no std.gcm found yet (will be built on first `mcpp build`)"); + if (!found) warn("no std module cached yet (built on first `mcpp build`)"); } else { - warn(std::format("cache root '{}' missing (run `mcpp env` to init)", - cacheRoot.string())); + warn(std::format("no std module cache at '{}' yet " + "(built on first `mcpp build`)", stdRoot.string())); } } @@ -189,11 +193,27 @@ export int doctor_report() { mcpp::ui::status("Checking", "cache health"); auto bmiRoot = mcpp::toolchain::default_cache_root(); auto sz = mcpp::bmi_cache::dir_size(bmiRoot); - if (sz > std::uintmax_t(1) * 1024 * 1024 * 1024) { - warn(std::format("BMI cache occupies {} (run `mcpp cache prune` to GC)", + if (sz > std::uintmax_t(4) * 1024 * 1024 * 1024) { + warn(std::format("build cache occupies {} " + "(`mcpp cache gc --max-size 4GiB` to collect)", mcpp::bmi_cache::human_bytes(sz))); } else { - ok(std::format("BMI cache size = {}", mcpp::bmi_cache::human_bytes(sz))); + ok(std::format("build cache size = {}", mcpp::bmi_cache::human_bytes(sz))); + } + // The pre-v1 cache was keyed by whole-project fingerprint, which folded in + // the consumer's own name and version — so it accumulated one copy of every + // dependency per project configuration and never produced a cross-project + // hit. Nothing reads it now. Report the size, never delete it. + { + auto legacyRoot = mcpp::home::legacy_bmi_root(); + std::error_code lec; + if (std::filesystem::is_directory(legacyRoot, lec)) { + auto lsz = mcpp::bmi_cache::dir_size(legacyRoot); + warn(std::format("pre-v1 cache at '{}' occupies {} and is no longer " + "used — `mcpp cache clean --legacy` reclaims it", + legacyRoot.string(), + mcpp::bmi_cache::human_bytes(lsz))); + } } // Pre-#311 builds could park the std BMI cache in the current working // directory (`.mcpp-bmi/`) whenever neither MCPP_HOME nor HOME resolved — @@ -441,10 +461,11 @@ export int explain_code(std::string_view code) { "The [toolchain] pin in mcpp.toml does not match the detected toolchain.\n" "Either install the pinned toolchain (xlings install ...) or relax the\n" "pin (e.g. \"gcc@>=15\" instead of \"gcc@15.1.0\")."}, - {"E0005", "BMI cache corruption", - "A cached BMI file referenced by manifest.txt is missing on disk. Run\n" - "`mcpp cache prune --older-than 0d` to drop stale entries; the next build\n" - "will repopulate."}, + {"E0005", "build cache corruption", + "A file listed in a cache entry's entry.json is missing on disk. Such an\n" + "entry is treated as a miss and rebuilt, so this is never wrong output —\n" + "only wasted space. `mcpp cache verify` lists every affected entry and\n" + "`mcpp cache gc --older-than 0s` reclaims them."}, {"E0006", "index requires a newer mcpp", "The package index declares (index.toml [index].min_mcpp) that its\n" "descriptors need a newer mcpp than this binary — parsing them would\n" @@ -486,8 +507,9 @@ export int self_init(bool force) { if (!home.empty()) { std::error_code ec; std::filesystem::remove_all(home / "registry", ec); - std::filesystem::remove_all(home / "bmi", ec); - std::filesystem::remove_all(home / "cache", ec); + std::filesystem::remove_all(home / "cache", ec); // index metadata + std::filesystem::remove_all(home / "build-cache", ec); // compiled artifacts + std::filesystem::remove_all(home / "bmi", ec); // pre-v1 build cache } } diff --git a/src/home.cppm b/src/home.cppm index 31138431..0f1b5cb0 100644 --- a/src/home.cppm +++ b/src/home.cppm @@ -25,10 +25,25 @@ export namespace mcpp::home { // $MCPP_HOME. See root() below for the resolution order. std::filesystem::path root(); -// The cross-project BMI cache root ($MCPP_HOME/bmi). Both the std module -// cache (toolchain/stdmod.cppm) and the dep BMI cache (bmi_cache.cppm, via -// GlobalConfig::bmiCacheDir) live here — they MUST agree. -std::filesystem::path bmi_root(); +// The cross-project build cache root ($MCPP_HOME/build-cache/v1). Holds both +// the std module cache (toolchain/stdmod.cppm, under std/) and the per-package +// dependency cache (bmi_cache.cppm, under pkg/) — they MUST agree. +// +// NOT `$MCPP_HOME/cache`: that name is already taken by GlobalConfig's index +// metadata cache, which config.cppm's reset path deletes wholesale. Nesting +// compiled artifacts inside a directory something else clears is a trap that +// would surface as "the build cache empties itself sometimes". +// +// The `v1` segment is the layout version. It exists so replacing the layout +// never has to migrate or delete anything: a new layout is a new segment, and +// the old tree becomes inert until `mcpp cache clean --legacy` collects it. +std::filesystem::path cache_root(); + +// The pre-v1 cache root ($MCPP_HOME/bmi), keyed by whole-project fingerprint. +// Nothing reads or writes it any more; `mcpp doctor` reports it and +// `mcpp cache clean --legacy` removes it. Kept resolvable rather than +// hardcoded at the call sites so "where is the old cache" has one answer too. +std::filesystem::path legacy_bmi_root(); } // namespace mcpp::home @@ -90,7 +105,11 @@ std::filesystem::path root() { return default_home(); } -std::filesystem::path bmi_root() { +std::filesystem::path cache_root() { + return root() / "build-cache" / "v1"; +} + +std::filesystem::path legacy_bmi_root() { return root() / "bmi"; } diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index fb9a6119..332c829e 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -877,6 +877,7 @@ std::expected parse_string(std::string_view content, if (auto v = doc->get_string("build.target")) m.buildConfig.target = *v; if (auto v = doc->get_string("build.default-profile")) m.buildConfig.defaultProfile = *v; else if (auto v = doc->get_string("build.profile")) m.buildConfig.defaultProfile = *v; // accepted alias + if (auto v = doc->get_string("build.cache")) m.buildConfig.cacheMode = *v; // [xlings] — build environment (L-1). Subsections mirror .xlings.json 1:1. if (auto v = doc->get_string_array("xlings.deps")) m.xlings.deps = *v; @@ -900,7 +901,7 @@ std::expected parse_string(std::string_view content, // // MUST stay in sync with the `doc->get_*("build.")` reads above. static constexpr std::string_view kKnownBuildKeys[] = { - "allow_host_libs", "c_standard", "cflags", "cxxflags", + "allow_host_libs", "c_standard", "cache", "cflags", "cxxflags", "default-profile", "defines", "dialect_cxxflags", "flags", "include_dirs", "include_dirs_after", "ldflags", "macos_deployment_target", "profile", "sources", "static_stdlib", @@ -915,7 +916,7 @@ std::expected parse_string(std::string_view content, "[build] has unsupported key '{}' (ignored). Supported keys: " "sources, cflags, cxxflags, ldflags, defines, flags, " "include_dirs, include_dirs_after, dialect_cxxflags, " - "c_standard, target, static_stdlib, allow_host_libs, " + "c_standard, target, static_stdlib, allow_host_libs, cache, " "profile, macos_deployment_target.", key)); } } diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 8f931708..c7d56dfd 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -290,6 +290,12 @@ struct BuildConfig : BuildInputs { // footgun): a project that defaults to dev should pass `--profile release` // when producing a distributable (a pack-time release guard is a follow-up). std::string defaultProfile; + // `[build] cache` — "global" (default) | "local" | "off". Project-level + // default for the global dependency cache; --cache and MCPP_BUILD_CACHE + // both override it. Validated in prepare_build (unknown value: warning, or + // error under --strict) rather than here, so parsing a manifest never + // depends on the build-mode vocabulary. + std::string cacheMode; }; // `[runtime]` — requirements needed when launching built binaries. diff --git a/src/toolchain/fingerprint.cppm b/src/toolchain/fingerprint.cppm index ddbb0405..d21a4140 100644 --- a/src/toolchain/fingerprint.cppm +++ b/src/toolchain/fingerprint.cppm @@ -18,7 +18,7 @@ import mcpp.toolchain.detect; export namespace mcpp::toolchain { -inline constexpr std::string_view MCPP_VERSION = "2026.7.30.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.7.30.2"; struct FingerprintInputs { Toolchain toolchain; diff --git a/src/toolchain/stdmod.cppm b/src/toolchain/stdmod.cppm index d32596f4..3dabf39c 100644 --- a/src/toolchain/stdmod.cppm +++ b/src/toolchain/stdmod.cppm @@ -12,13 +12,21 @@ module; // clang++ -std=c++23 pcm.cache/std.pcm -c -o std.o // // We invoke the compiler in a dedicated cache directory so the produced -// BMI is owned by mcpp and reused across all builds with the same fingerprint. +// BMI is owned by mcpp and reused across every build with the same std identity. // // Output layout: -// // +// /std// // gcm.cache/std.gcm ← GCC BMI // pcm.cache/std.pcm ← Clang BMI // std.o ← linked into final binaries +// +// The directory used to be named by the WHOLE-PROJECT fingerprint, which folds +// in the root package's name, version and flags — none of which reach the std +// module's compile commands. The metadata written next to the artifacts +// (metadata_for below) has always been the correct identity and has always been +// what validates a hit; only the directory name was wrong. On one machine that +// cost 1014 directories holding 15 distinct identities: 16.1 GB where ~0.5 GB +// was needed, and a `mcpp version bump` re-precompiled std from scratch. export module mcpp.toolchain.stdmod; @@ -36,7 +44,7 @@ import mcpp.toolchain.msvc; export namespace mcpp::toolchain { struct StdModule { - std::filesystem::path cacheDir; // // + std::filesystem::path cacheDir; // /std// std::filesystem::path bmiPath; // /gcm.cache/std.gcm std::filesystem::path objectPath; // /std.o std::filesystem::path compatBmiPath; // /pcm.cache/std.compat.pcm @@ -54,7 +62,6 @@ std::filesystem::path default_cache_root(); // arm64-apple-macosxNN triple than the code importing it. std::expected ensure_built( const Toolchain& tc, - std::string_view fingerprint_hex, std::string_view cpp_standard, std::string_view cpp_standard_flag, std::string_view macos_deployment_target = {}, @@ -86,6 +93,24 @@ std::filesystem::path metadata_path(const std::filesystem::path& cacheDir) { return cacheDir / "std-module.json"; } +// The directory segment used while deriving the identity, so the build commands +// folded into it do not contain the identity they are about to produce. +constexpr std::string_view kStdKeyPlaceholder = "@std-key@"; + +// The std module's cache directory name. Derived from the metadata that already +// defines std identity (metadata_for below: compiler, version, driver identity, +// target triple, stdlib, standard + flag, source hashes, build commands) — the +// same object metadata_matches validates a hit against. Naming the directory +// after the whole-project fingerprint instead is what let 15 real identities +// occupy 1014 directories. +// +// Format compatibility is carried by the `build-cache/v1` path segment above +// one, not by an epoch field: a std BMI's validity depends on the toolchain, +// never on mcpp's cache bookkeeping. +std::string std_identity_key(const nlohmann::json& normalized) { + return hash_string("mcpp-std-key-v1\x1f" + normalized.dump()); +} + nlohmann::json metadata_for(const Toolchain& tc, std::string_view cppStandard, std::string_view cppStandardFlag, @@ -182,12 +207,11 @@ std::filesystem::path default_cache_root() { // resolution that predated Windows' USERPROFILE branch and self-contained // installs, so the std BMI cache could land in the *current working // directory* (`.mcpp-bmi/`) while dep BMIs went to $MCPP_HOME/bmi. - return mcpp::home::bmi_root(); + return mcpp::home::cache_root(); } std::expected ensure_built( const Toolchain& tc, - std::string_view fingerprint_hex, std::string_view cpp_standard, std::string_view cpp_standard_flag, std::string_view macos_deployment_target, @@ -200,12 +224,6 @@ std::expected ensure_built( const bool isMsvc = tc.compiler == CompilerId::MSVC; StdModule sm; - sm.cacheDir = cache_root / std::string(fingerprint_hex); - sm.bmiPath = isMsvc ? mcpp::toolchain::msvc::std_bmi_path(sm.cacheDir) - : is_clang(tc) - ? mcpp::toolchain::clang::std_bmi_path(sm.cacheDir) - : mcpp::toolchain::gcc::std_bmi_path(sm.cacheDir); - sm.objectPath = sm.cacheDir / (isMsvc ? "std.obj" : "std.o"); // Build sysroot + include flags for std module precompilation, derived // from the shared toolchain link model (same resolver as flags.cppm — @@ -236,26 +254,62 @@ std::expected ensure_built( // All three providers expose the same command-sequence shape (A5 backend // surface normalization) — no per-compiler arity branching here. - std::vector stdCommands = - isMsvc ? mcpp::toolchain::msvc::std_module_build_commands( - tc, sm.cacheDir, cpp_standard_flag) - : is_clang(tc) - ? mcpp::toolchain::clang::std_module_build_commands( - tc, sm.cacheDir, sm.bmiPath, sysroot_flag, cpp_standard_flag) - : mcpp::toolchain::gcc::std_module_build_commands( - tc, sm.cacheDir, sysroot_flag, cpp_standard_flag); - std::vector compatCommands; - if (!tc.stdCompatSource.empty()) { - if (isMsvc) { - compatCommands = mcpp::toolchain::msvc::std_compat_build_commands( - tc, sm.cacheDir, cpp_standard_flag); - } else if (is_clang(tc)) { - auto compatBmi = mcpp::toolchain::clang::std_compat_bmi_path(sm.cacheDir); - compatCommands = mcpp::toolchain::clang::std_compat_build_commands( - tc, sm.cacheDir, compatBmi, sm.bmiPath, sysroot_flag, cpp_standard_flag); + // + // Everything downstream of the cache directory is derived here, because the + // directory NAME is derived from the metadata and the metadata contains the + // build commands, which contain the directory. Resolving that circularity + // by hand: run this once against a placeholder directory to obtain the + // identity (that is what "normalized" means — the commands then name a + // literal placeholder segment instead of a key-specific path), hash it, + // then run it again against the real directory to get what goes on disk. + struct Derived { + std::filesystem::path bmiPath; + std::filesystem::path objectPath; + std::vector stdCommands; + std::vector compatCommands; + nlohmann::json metadata; + }; + auto derive = [&](const std::filesystem::path& cacheDir) { + Derived d; + d.bmiPath = isMsvc ? mcpp::toolchain::msvc::std_bmi_path(cacheDir) + : is_clang(tc) + ? mcpp::toolchain::clang::std_bmi_path(cacheDir) + : mcpp::toolchain::gcc::std_bmi_path(cacheDir); + d.objectPath = cacheDir / (isMsvc ? "std.obj" : "std.o"); + d.stdCommands = + isMsvc ? mcpp::toolchain::msvc::std_module_build_commands( + tc, cacheDir, cpp_standard_flag) + : is_clang(tc) + ? mcpp::toolchain::clang::std_module_build_commands( + tc, cacheDir, d.bmiPath, sysroot_flag, cpp_standard_flag) + : mcpp::toolchain::gcc::std_module_build_commands( + tc, cacheDir, sysroot_flag, cpp_standard_flag); + if (!tc.stdCompatSource.empty()) { + if (isMsvc) { + d.compatCommands = mcpp::toolchain::msvc::std_compat_build_commands( + tc, cacheDir, cpp_standard_flag); + } else if (is_clang(tc)) { + auto compatBmi = mcpp::toolchain::clang::std_compat_bmi_path(cacheDir); + d.compatCommands = mcpp::toolchain::clang::std_compat_build_commands( + tc, cacheDir, compatBmi, d.bmiPath, sysroot_flag, cpp_standard_flag); + } } - } - auto metadata = metadata_for(tc, cpp_standard, cpp_standard_flag, stdCommands, compatCommands); + d.metadata = metadata_for(tc, cpp_standard, cpp_standard_flag, + d.stdCommands, d.compatCommands); + return d; + }; + + const auto stdRoot = cache_root / "std"; + auto normalized = derive(stdRoot / kStdKeyPlaceholder).metadata; + auto identity = std_identity_key(normalized); + sm.cacheDir = stdRoot / identity; + + auto derived = derive(sm.cacheDir); + sm.bmiPath = derived.bmiPath; + sm.objectPath = derived.objectPath; + const auto& stdCommands = derived.stdCommands; + const auto& compatCommands = derived.compatCommands; + const auto& metadata = derived.metadata; auto metaPath = metadata_path(sm.cacheDir); bool std_cached = std::filesystem::exists(sm.bmiPath) && std::filesystem::exists(sm.objectPath) diff --git a/src/ui.cppm b/src/ui.cppm index 6ca9cd61..5229adb8 100644 --- a/src/ui.cppm +++ b/src/ui.cppm @@ -31,7 +31,11 @@ void status(std::string_view verb, std::string_view message); void info(std::string_view verb, std::string_view message); // Bold green Finished line. -void finished(std::string_view profile, std::chrono::milliseconds elapsed); +// `descriptor` annotates the profile's actual effect (e.g. "optimized", +// "unoptimized + debuginfo"). Empty = print the profile name alone; callers +// that never resolved the profile knobs must not invent one. +void finished(std::string_view profile, std::chrono::milliseconds elapsed, + std::string_view descriptor = {}); // "warning:" / "error:" prefix lines (yellow / red). void warning(std::string_view message); @@ -245,12 +249,20 @@ void info(std::string_view verb, std::string_view message) { } } -void finished(std::string_view profile, std::chrono::milliseconds elapsed) { +void finished(std::string_view profile, std::chrono::milliseconds elapsed, + std::string_view descriptor) { if (g_quiet) return; init(); auto v = verb_padded("Finished"); auto secs = static_cast(elapsed.count()) / 1000.0; - auto msg = std::format("{} [optimized] in {:.2f}s", profile, secs); + // `[optimized]` used to be hardcoded here alongside a hardcoded "release" + // at the only call site, so every build — including `--dev` at -O0 -g — + // announced "Finished release [optimized]". The descriptor is now supplied + // by whoever actually resolved the profile knobs, and omitted by callers + // (the fast path) that never resolved them: no caller has to guess. + auto msg = descriptor.empty() + ? std::format("{} in {:.2f}s", profile, secs) + : std::format("{} [{}] in {:.2f}s", profile, descriptor, secs); if (g_color) { std::println("{}{}{}{} {}", kBold, kBrightGreen, v, kReset, msg); diff --git a/tests/e2e/100_cppfly_reflection.sh b/tests/e2e/100_cppfly_reflection.sh index 8177be71..313edd6d 100755 --- a/tests/e2e/100_cppfly_reflection.sh +++ b/tests/e2e/100_cppfly_reflection.sh @@ -104,10 +104,12 @@ out=$("$binary") } # The std BMI prebuild carries the same dialect (scan/prebuild/compile agree). -metadata="$(find "$MCPP_HOME/bmi" -name std-module.json | head -1)" -grep -q '"std_flag": "-std=c++26 -freflection' "$metadata" || { - cat "$metadata" - echo "FAIL: std-module.json std_flag lacks -std=c++26 -freflection" +# Recursive grep, not `find | head -1`: one MCPP_HOME can hold several std +# identities now, so "the first one" is arbitrary. +grep -rl '"std_flag": "-std=c++26 -freflection' "$MCPP_HOME/build-cache/v1/std" >/dev/null 2>&1 || { + find "$MCPP_HOME/build-cache/v1/std" -name std-module.json \ + -exec grep -H '"std_flag"' {} \; 2>/dev/null + echo "FAIL: no std-module.json records std_flag -std=c++26 -freflection" exit 1 } diff --git a/tests/e2e/10_env_command.sh b/tests/e2e/10_env_command.sh index 5a0eb0ea..77716f71 100755 --- a/tests/e2e/10_env_command.sh +++ b/tests/e2e/10_env_command.sh @@ -14,7 +14,7 @@ out=$("$MCPP" self env 2>&1) # Verify directory tree [[ -d "$MCPP_HOME/bin" ]] || { echo "missing bin/"; exit 1; } [[ -d "$MCPP_HOME/registry" ]] || { echo "missing registry/"; exit 1; } -[[ -d "$MCPP_HOME/bmi" ]] || { echo "missing bmi/"; exit 1; } +[[ -d "$MCPP_HOME/build-cache/v1" ]] || { echo "missing build-cache/v1/"; exit 1; } [[ -d "$MCPP_HOME/cache" ]] || { echo "missing cache/"; exit 1; } [[ -f "$MCPP_HOME/config.toml" ]] || { echo "missing config.toml"; exit 1; } [[ -f "$MCPP_HOME/registry/.xlings.json" ]] || { echo "missing seeded .xlings.json"; exit 1; } diff --git a/tests/e2e/170_bmi_staging_no_cascade.sh b/tests/e2e/170_bmi_staging_no_cascade.sh index 8d365571..32c513ea 100755 --- a/tests/e2e/170_bmi_staging_no_cascade.sh +++ b/tests/e2e/170_bmi_staging_no_cascade.sh @@ -9,7 +9,7 @@ # carried no `restat`, so every importer of `import std` rebuilt whenever # the cache-side BMI got a newer mtime (which happens on any cwd change # before the cache-root fix below). -# 2. The staging SOURCE must live under $MCPP_HOME/bmi. The std module cache +# 2. The staging SOURCE must live under $MCPP_HOME/build-cache/v1/std. That cache # used to resolve its root through a private copy of the home logic that # knew neither %USERPROFILE% nor self-contained installs, so on Windows it # parked the cache in the current working directory as `.mcpp-bmi/`. @@ -56,11 +56,13 @@ DST=$(unescape_ninja "$(echo "$EDGE" | awk '{print $2}')") SRC=$(unescape_ninja "$(echo "$EDGE" | awk '{print $NF}')") echo "staging: $SRC -> $DST" -# ── invariant 2: the cache root is $MCPP_HOME/bmi, never a cwd-local dir ── +# ── invariant 2: the cache root is $MCPP_HOME/build-cache/v1/std, never a cwd-local +# dir (and never the pre-v1 $MCPP_HOME/bmi tree, which nothing reads now) ── HOME_ROOT=$(norm_path "${MCPP_HOME:-$HOME/.mcpp}") case "$(norm_path "$SRC")" in - "$HOME_ROOT"/bmi/*) ;; - *) echo "FAIL: std BMI cache is '$SRC', expected under '$HOME_ROOT/bmi'"; exit 1 ;; + "$HOME_ROOT"/build-cache/v1/std/*) ;; + *) echo "FAIL: std BMI cache is '$SRC', expected under '$HOME_ROOT/build-cache/v1/std'" + exit 1 ;; esac for leftover in "$TMP/.mcpp-bmi" "$TMP/app/.mcpp-bmi"; do [[ -d "$leftover" ]] && { echo "FAIL: legacy cache dir created at $leftover"; exit 1; } diff --git a/tests/e2e/172_build_cache_cross_project.sh b/tests/e2e/172_build_cache_cross_project.sh new file mode 100644 index 00000000..38de385f --- /dev/null +++ b/tests/e2e/172_build_cache_cross_project.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# requires: gcc fresh-sandbox +# 172_build_cache_cross_project.sh — the global build cache must actually save +# work, and it must save it ACROSS projects. +# +# Two defects met here, and this file is the gate for both. +# +# 1. The key was the whole-project fingerprint, which serializes every package +# in the graph INCLUDING the root — its name, its version, its [build] +# flags. So two projects never shared an entry, and bumping a project's own +# version invalidated every dependency it had. (Measured on one machine: +# 26 GB across 1198 fingerprint directories, `compat.zlib@1.3.2` stored 162 +# times.) +# +# 2. Even on a hit, nothing was saved. Artifacts were copied into the build dir +# from inside prepare_build while those paths stayed declared as compile edge +# outputs — and ninja treats an output it has no command line for in +# .ninja_log as dirty ("command line not found in log"), which a fresh build +# dir always is. Every "cached" unit was recompiled while the CLI printed +# "Cached". +# +# So the load-bearing assertion is not "an entry exists" but "the second project +# has ZERO compile edges for the dependency's sources". Read from build.ninja, +# not from the log: a status line is exactly what lied before. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +# ── an offline path index serving one library package ──────────────────────── +INDEX_DIR="$TMP/local-index" +mkdir -p "$INDEX_DIR/pkgs/s" +cat > "$INDEX_DIR/pkgs/s/shared-lib.lua" <<'EOF' +package = { + spec = "1", + name = "shared-lib", + description = "Shared across projects", + licenses = {"MIT"}, + type = "package", + xpm = { + linux = { + ["1.0.0"] = { + url = "https://example.invalid/shared-lib-1.0.0.tar.gz", + sha256 = "0000000000000000000000000000000000000000000000000000000000000000", + }, + }, + }, + mcpp = { + language = "c++23", + import_std = true, + sources = { "src/**/*.cppm" }, + targets = { ["shared-lib"] = { kind = "lib" } }, + deps = {}, + }, +} +EOF + +# `make_project ` — same dependency, and +# deliberately DIFFERENT identity, because the consumer's identity is exactly +# what used to leak into the dependency's cache key. +make_project() { + local dir="$1" name="$2" version="$3" + mkdir -p "$TMP/$dir/src" + mkdir -p "$TMP/$dir/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.0/src" + cat > "$TMP/$dir/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.0/src/lib.cppm" <<'EOF' +export module shared.lib; +export int shared_value() { return 41; } +EOF + cat > "$TMP/$dir/src/main.cpp" <<'EOF' +import std; +import shared.lib; +int main() { std::println("{}", shared_value() + 1); return 0; } +EOF + cat > "$TMP/$dir/mcpp.toml" < build.log 2>&1 || { cat build.log; exit 1; } + +N1="$(find_ninja "$TMP/projone")" +[[ -n "$N1" ]] || { echo "FAIL: projone has no build.ninja"; exit 1; } +[[ "$(dep_compile_edges "$N1")" -gt 0 ]] || { + echo "FAIL: cold build had no compile edges for the dependency" + grep -n 'shared-lib' "$N1" | head + exit 1 +} +"$MCPP" cache list > list1.log 2>&1 +grep -q 'shared-lib' list1.log || { + echo "FAIL: cold build did not populate a cache entry" + cat list1.log; cat build.log + exit 1 +} + +# ── project two: DIFFERENT name and version, same dependency ───────────────── +make_project projtwo projtwo 9.9.9 +cd "$TMP/projtwo" +"$MCPP" build > build.log 2>&1 || { cat build.log; exit 1; } + +N2="$(find_ninja "$TMP/projtwo")" +[[ -n "$N2" ]] || { echo "FAIL: projtwo has no build.ninja"; exit 1; } + +# THE assertion. +edges="$(dep_compile_edges "$N2")" +if [[ "$edges" != "0" ]]; then + echo "FAIL: second project recompiled the dependency ($edges compile edges)" + echo " the cache key is still carrying the consumer's identity" + grep -nE ': (cxx_module|cxx_object|cxx_scan) .*shared-lib' "$N2" | head + exit 1 +fi +staged="$(dep_stage_edges "$N2")" +[[ "$staged" -gt 0 ]] || { + echo "FAIL: dependency has neither compile nor stage edges — it is missing" + grep -n 'shared-lib' "$N2" | head + exit 1 +} + +# The status line must agree, and must carry the unit count. The bare word +# "Cached" was printed for months while every unit was recompiled behind it; a +# number that has to match the skipped edges cannot go quietly wrong that way. +grep -qE 'Cached local-dev\.shared-lib v1\.0\.0 \([0-9]+ unit' build.log || { + echo "FAIL: no 'Cached ... (N units)' line for the reused dependency" + cat build.log + exit 1 +} + +# And it has to WORK: staged artifacts must link and run. +./target/*/*/bin/projtwo > run.log 2>&1 || { cat run.log; exit 1; } +grep -q '^42$' run.log || { echo "FAIL: staged artifacts produced wrong output"; cat run.log; exit 1; } + +# ── the consumer's own version must not invalidate the dependency ──────────── +cd "$TMP/projone" +sed -i.bak 's/version = "0.1.0"/version = "0.2.0"/' mcpp.toml +rm -f mcpp.toml.bak +rm -rf target +"$MCPP" build > bump.log 2>&1 || { cat bump.log; exit 1; } +N3="$(find_ninja "$TMP/projone")" +[[ "$(dep_compile_edges "$N3")" == "0" ]] || { + echo "FAIL: bumping the consumer's own version invalidated the dependency" + grep -nE ': (cxx_module|cxx_object|cxx_scan) .*shared-lib' "$N3" | head + exit 1 +} + +# ── a real change to the dependency MUST invalidate it (no false hits) ─────── +# Bump the dependency's version in the index and in the manifests: a different +# version is a different entry, so it has to be compiled again. +sed -i.bak 's/\["1\.0\.0"\]/["1.0.1"]/' "$INDEX_DIR/pkgs/s/shared-lib.lua" +rm -f "$INDEX_DIR/pkgs/s/shared-lib.lua.bak" +mv "$TMP/projone/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.0" \ + "$TMP/projone/.mcpp/.xlings/data/xpkgs/local-dev.shared-lib/1.0.1" +sed -i.bak 's/"local-dev.shared-lib" = "1.0.0"/"local-dev.shared-lib" = "1.0.1"/' \ + "$TMP/projone/mcpp.toml" +rm -f "$TMP/projone/mcpp.toml.bak" +rm -rf target +"$MCPP" build > newver.log 2>&1 || { cat newver.log; exit 1; } +N4="$(find_ninja "$TMP/projone")" +[[ "$(dep_compile_edges "$N4")" -gt 0 ]] || { + echo "FAIL: a new dependency version was served from the old entry" + cat newver.log + exit 1 +} + +echo "OK" diff --git a/tests/e2e/173_build_profile_isolation.sh b/tests/e2e/173_build_profile_isolation.sh new file mode 100644 index 00000000..516903cf --- /dev/null +++ b/tests/e2e/173_build_profile_isolation.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# requires: gcc fresh-sandbox +# 173_build_profile_isolation.sh — the build profile must be an invalidation +# axis, and the fast path must not hand back another profile's artifacts. +# +# `[profile.]` lands its knobs in buildConfig.optLevel/debug/lto/strip and +# flags.cppm turns them into -O/-g/-flto, but the fingerprint serialized only +# cflags/cxxflags/ldflags. So `--dev`, `--release` and `--profile dist` produced +# ONE fingerprint, hence one target/// directory and one global cache +# entry — a release build could be served -O0 -g dependency objects. +# +# The second half was worse and user-visible without any cache involved: +# `.build_cache` keyed its fast-path entries by target triple alone, and the fast +# path only refuses to run when an EXPLICIT --profile/--dev/--release is passed. +# A bare `mcpp build` after `mcpp build --release` therefore took the fast path +# against the release build.ninja and reported success in 0.00s, leaving -O2 +# artifacts where -O0 -g was asked for. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +cd "$TMP" +mkdir -p app/src +cd app +cat > src/main.cpp <<'EOF' +#include +int main() { std::printf("hi\n"); return 0; } +EOF +cat > mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" +standard = "c++23" + +[targets.app] +kind = "bin" +main = "src/main.cpp" +EOF + +# The -O level recorded in a build dir's build.ninja. Reading the generated graph +# rather than a status line: a status line is what lied here before. +opt_of() { grep -oE '\-O[0-9s]' "$1/build.ninja" | sort -u | tr -d '\n'; } +dir_for_opt() { + local want="$1" + for d in target/*/*/; do + [[ -f "$d/build.ninja" ]] || continue + [[ "$(opt_of "$d")" == "$want" ]] && { printf '%s' "$d"; return 0; } + done + return 1 +} + +# ── each profile gets its own build dir ───────────────────────────────────── +"$MCPP" build --dev > dev.log 2>&1 || { cat dev.log; exit 1; } +"$MCPP" build --release > release.log 2>&1 || { cat release.log; exit 1; } +"$MCPP" build --profile dist > dist.log 2>&1 || { cat dist.log; exit 1; } + +count=$(find target -name build.ninja | wc -l) +[[ "$count" -eq 3 ]] || { + echo "FAIL: expected one build dir per profile, found $count" + for d in target/*/*/; do echo " $d -> $(opt_of "$d")"; done + exit 1 +} +dir_for_opt "-O0" >/dev/null || { echo "FAIL: no -O0 (dev) build dir"; exit 1; } +dir_for_opt "-O2" >/dev/null || { echo "FAIL: no -O2 (release) build dir"; exit 1; } +dir_for_opt "-O3" >/dev/null || { echo "FAIL: no -O3 (dist) build dir"; exit 1; } + +# The announced profile must be the one that was built. +grep -q 'Finished dev' dev.log || { cat dev.log; echo "FAIL: dev not announced"; exit 1; } +grep -q 'Finished release' release.log || { cat release.log; echo "FAIL: release not announced"; exit 1; } +grep -q 'Finished dist' dist.log || { cat dist.log; echo "FAIL: dist not announced"; exit 1; } +# ...and the descriptor must not contradict the flags. "release [optimized]" was +# hardcoded, so a --dev build used to announce itself as an optimized release. +grep -q 'Finished dev \[unoptimized' dev.log || { + cat dev.log + echo "FAIL: dev build described as optimized" + exit 1 +} + +# ── the regression gate: bare build after --release must be dev ───────────── +DEVDIR="$(dir_for_opt "-O0")" +rm -rf target +"$MCPP" build --release > r2.log 2>&1 || { cat r2.log; exit 1; } +"$MCPP" build > bare.log 2>&1 || { cat bare.log; exit 1; } + +grep -q 'Finished dev' bare.log || { + echo "FAIL: bare build after --release did not resolve to the dev profile" + cat bare.log + exit 1 +} +DEVDIR="$(dir_for_opt "-O0")" || { + echo "FAIL: bare build produced no -O0 build dir (fast path served release)" + for d in target/*/*/; do echo " $d -> $(opt_of "$d")"; done + cat bare.log + exit 1 +} +# The dev objects must actually exist and carry debug info: an existing build dir +# with no objects in it would satisfy a path check but not a build. +[[ -f "$DEVDIR/obj/main.o" ]] || { + echo "FAIL: dev build dir has no object" + find "$DEVDIR" -type f | head + exit 1 +} +if command -v readelf > /dev/null 2>&1; then + readelf --debug-dump=info "$DEVDIR/obj/main.o" 2>/dev/null \ + | grep -q 'DW_AT_producer' || { + echo "FAIL: dev object has no debug info (it is a release object)" + exit 1 + } +fi + +# ── switching back is incremental, not a full rebuild ────────────────────── +# Both entries coexist in .build_cache now; keying on the triple alone made each +# profile evict the other, so a dev/release rotation could never be incremental. +"$MCPP" build --release > r3.log 2>&1 || { cat r3.log; exit 1; } +"$MCPP" build > bare2.log 2>&1 || { cat bare2.log; exit 1; } +if grep -qE 'src/main\.cpp' bare2.log; then + echo "FAIL: returning to the dev profile recompiled from scratch" + cat bare2.log + exit 1 +fi +grep -q 'Finished dev' bare2.log || { cat bare2.log; echo "FAIL: wrong profile"; exit 1; } + +echo "OK" diff --git a/tests/e2e/174_cache_modes_and_commands.sh b/tests/e2e/174_cache_modes_and_commands.sh new file mode 100644 index 00000000..dfccd27c --- /dev/null +++ b/tests/e2e/174_cache_modes_and_commands.sh @@ -0,0 +1,380 @@ +#!/usr/bin/env bash +# requires: gcc fresh-sandbox +# 174_cache_modes_and_commands.sh — the three build modes, and the cache +# maintenance surface they need. +# +# mcpp build read + write the global cache (default) +# mcpp build --cache=local neither; every dependency compiles in target/ +# mcpp build --cache=off neither, and target/ is cleared first +# mcpp build --no-cache deprecated alias for --cache=off +# +# `--no-cache` used to be the only switch, and it only ever cleared target/ — +# a name that says nothing about a cache. It stays accepted. +# +# The maintenance half exists because a cache nobody can inspect, size or +# reclaim is a cache nobody can trust. `prune` in particular used to rank +# entries by the directory's mtime, which only records when an entry was +# WRITTEN — so a dependency that hit on every build looked as stale as one +# nobody had touched in a month. `gc` reads entry.json's `accessed` stamp. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +INDEX_DIR="$TMP/local-index" +mkdir -p "$INDEX_DIR/pkgs/m" +cat > "$INDEX_DIR/pkgs/m/mode-lib.lua" <<'EOF' +package = { + spec = "1", + name = "mode-lib", + description = "Cache mode fixture", + licenses = {"MIT"}, + type = "package", + xpm = { + linux = { + ["1.0.0"] = { + url = "https://example.invalid/mode-lib-1.0.0.tar.gz", + sha256 = "0000000000000000000000000000000000000000000000000000000000000000", + }, + }, + }, + mcpp = { + language = "c++23", + import_std = true, + sources = { "src/**/*.cppm" }, + targets = { ["mode-lib"] = { kind = "lib" } }, + deps = {}, + }, +} +EOF + +mkdir -p "$TMP/app/src" \ + "$TMP/app/.mcpp/.xlings/data/xpkgs/local-dev.mode-lib/1.0.0/src" +cat > "$TMP/app/.mcpp/.xlings/data/xpkgs/local-dev.mode-lib/1.0.0/src/lib.cppm" <<'EOF' +export module mode.lib; +export int mode_value() { return 5; } +EOF +cat > "$TMP/app/src/main.cpp" <<'EOF' +import std; +import mode.lib; +int main() { std::println("{}", mode_value()); return 0; } +EOF +cat > "$TMP/app/mcpp.toml" </dev/null | wc -l; } + +# ── mcpp cache dir: where IS it? ──────────────────────────────────────────── +out=$("$MCPP" cache dir 2>&1) +[[ "$out" == "$MCPP_HOME/build-cache/v1"* ]] || { + echo "FAIL: cache dir reported '$out', expected '$MCPP_HOME/build-cache/v1'" + exit 1 +} + +# ── local: nothing read, nothing written ─────────────────────────────────── +"$MCPP" build --cache=local > local.log 2>&1 || { cat local.log; exit 1; } +[[ "$(entry_count)" -eq 0 ]] || { + echo "FAIL: --cache=local wrote $(entry_count) cache entries" + find "$MCPP_HOME/build-cache/v1/pkg" -maxdepth 4 2>/dev/null + exit 1 +} +# The dependency still has to be built, in-project. +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: --cache=local did not compile the dependency locally" + grep -n 'mode-lib' "$NINJA" | head + exit 1 +} +./target/*/*/bin/app > run.log 2>&1 || { cat run.log; exit 1; } +grep -q '^5$' run.log || { echo "FAIL: local build produced wrong output"; cat run.log; exit 1; } + +# ── global (default): writes an entry ────────────────────────────────────── +rm -rf target +"$MCPP" build > global.log 2>&1 || { cat global.log; exit 1; } +[[ "$(entry_count)" -eq 1 ]] || { + echo "FAIL: default build wrote $(entry_count) entries, expected 1" + cat global.log + exit 1 +} + +# ── cache list / list --json / info / verify ─────────────────────────────── +"$MCPP" cache list > list.log 2>&1 +grep -q 'mode-lib' list.log || { cat list.log; echo "FAIL: list omits the entry"; exit 1; } + +"$MCPP" cache list --json > list.json 2>&1 +python3 - list.json > jsoncheck.log 2>&1 <<'PYEOF' || { cat jsoncheck.log; exit 1; } +import json, sys +d = json.load(open(sys.argv[1])) +if "root" not in d or "entries" not in d: + sys.exit(f"FAIL: list --json missing keys: {sorted(d)}") +pkgs = [e for e in d["entries"] if e["kind"] == "pkg"] +if not pkgs: + sys.exit("FAIL: list --json has no package entries") +e = pkgs[0] +for k in ("kind", "label", "key", "dir", "bytes", "files", "accessed", "complete"): + if k not in e: + sys.exit(f"FAIL: entry missing '{k}': {sorted(e)}") +if not e["complete"]: + sys.exit(f"FAIL: fresh entry reported incomplete: {e}") +if e["bytes"] <= 0: + sys.exit(f"FAIL: entry reports {e['bytes']} bytes") +PYEOF + +# info must print the recorded key inputs — that is the whole reason a cache +# entry describes itself, and the only way to diagnose a suspected wrong hit. +"$MCPP" cache info mode-lib > info.log 2>&1 || { cat info.log; exit 1; } +grep -q 'inputs' info.log || { cat info.log; echo "FAIL: info omits key inputs"; exit 1; } +grep -q 'opt_level' info.log || { cat info.log; echo "FAIL: info omits the profile axis"; exit 1; } + +"$MCPP" cache verify > verify.log 2>&1 || { cat verify.log; echo "FAIL: verify failed on a healthy cache"; exit 1; } +grep -q 'all complete' verify.log || { cat verify.log; echo "FAIL: verify message"; exit 1; } + +# verify must FAIL (non-zero) when an artifact is gone. Explicit if/exit rather +# than `! cmd`: under `set -e` a negated command is exempted from the errexit +# check, so `! "$MCPP" cache verify` can never fail this test. +victim="$(find "$MCPP_HOME/build-cache/v1/pkg" -path '*/obj/*' -type f | head -1)" +[[ -n "$victim" ]] || { echo "FAIL: no cached object to remove"; exit 1; } +mv "$victim" "$victim.bak" +rc=0 +"$MCPP" cache verify > verify2.log 2>&1 || rc=$? +if [[ "$rc" -eq 0 ]]; then + echo "FAIL: verify passed with a missing artifact" + cat verify2.log + exit 1 +fi +grep -q 'incomplete' verify2.log || { cat verify2.log; echo "FAIL: verify wording"; exit 1; } +mv "$victim.bak" "$victim" + +# ── off: clears this build's build dir and touches no cache ──────────────── +# The build DIRECTORY (target///), not all of target/: sibling dirs +# for other profiles and other targets are not this invocation's business. +builddir="$(dirname "$(find target -name build.ninja | head -1)")" +marker="$builddir/_off_marker" +touch "$marker" +before="$(entry_count)" +"$MCPP" build --cache=off > off.log 2>&1 || { cat off.log; exit 1; } +[[ ! -f "$marker" ]] || { echo "FAIL: --cache=off did not clear the build dir"; exit 1; } +[[ "$(entry_count)" -eq "$before" ]] || { + echo "FAIL: --cache=off changed the cache ($before -> $(entry_count))" + exit 1 +} +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: --cache=off served the dependency from the cache" + exit 1 +} + +# --no-cache must behave identically (deprecated alias). +touch "$marker" +"$MCPP" build --no-cache > nocache.log 2>&1 || { cat nocache.log; exit 1; } +[[ ! -f "$marker" ]] || { echo "FAIL: --no-cache did not clear the build dir"; exit 1; } + +# An unknown mode is refused, not silently taken as "global". +rc=0 +"$MCPP" build --cache=bogus --strict > bogus.log 2>&1 || rc=$? +[[ "$rc" -ne 0 ]] || { echo "FAIL: --cache=bogus accepted under --strict"; cat bogus.log; exit 1; } +grep -qi 'cache mode' bogus.log || { cat bogus.log; echo "FAIL: unhelpful error"; exit 1; } + +# MCPP_BUILD_CACHE is the env-level equivalent. +rm -rf target +MCPP_BUILD_CACHE=local "$MCPP" build > env.log 2>&1 || { cat env.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: MCPP_BUILD_CACHE=local was ignored" + exit 1 +} + +# [build] cache is the project-level equivalent, and --cache beats it. +cat >> mcpp.toml <<'EOF' + +[build] +cache = "local" +EOF +rm -rf target +"$MCPP" build > manifest.log 2>&1 || { cat manifest.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: [build] cache = local was ignored" + exit 1 +} +rm -rf target +"$MCPP" build --cache=global > override.log 2>&1 || { cat override.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +if grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA"; then + echo "FAIL: --cache=global did not override [build] cache = local" + exit 1 +fi + +# Drop the manifest override again — everything below assumes the default mode, +# and a leftover `cache = "local"` would make the gc section measure an empty +# cache and "pass" for the wrong reason. +python3 - mcpp.toml <<'PYEOF' +import sys +p = sys.argv[1] +lines = open(p).read().splitlines(keepends=True) +out, skip = [], False +for ln in lines: + if ln.strip() == "[build]": + skip = True + continue + if skip and ln.strip().startswith("cache"): + skip = False + continue + out.append(ln) +open(p, "w").write("".join(out)) +PYEOF +if grep -q '^cache' mcpp.toml; then + echo "FAIL: could not strip the [build] cache override" + cat mcpp.toml + exit 1 +fi + +# ── the fast path must honour the declared mode ───────────────────────────── +# A build.ninja generated under `global` contains stage_file edges reading the +# cache. Replaying it for a request that asked for `local` would use the cache the +# manifest just said not to use — and ruling the cache out is `local`'s whole +# purpose. .build_cache therefore records the mode, exactly as it records the +# profile, and a mismatch is a miss. +cat >> mcpp.toml <<'EOF' + +[build] +cache = "local" +EOF +rm -rf target +"$MCPP" build --cache=global > modeglobal.log 2>&1 || { cat modeglobal.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +staged_before=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) +[[ "$staged_before" -gt 0 ]] || { + echo "FAIL: --cache=global did not produce stage edges (cache should be warm here)" + cat modeglobal.log + exit 1 +} +# Bare build: the manifest says local, and the fast path must NOT replay the +# global graph. Note mcpp.toml is untouched between these two builds, so its +# mtime cannot be what saves us — only the recorded mode can. +"$MCPP" build > modebare.log 2>&1 || { cat modebare.log; exit 1; } +NINJA="$(find target -name build.ninja | head -1)" +staged_after=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) +[[ "$staged_after" -eq 0 ]] || { + echo "FAIL: bare build replayed the global-mode graph ($staged_after stage edges)" + echo " [build] cache = local was bypassed by the fast path" + cat modebare.log + exit 1 +} +grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { + echo "FAIL: local mode did not compile the dependency" + exit 1 +} +# Strip the override again for the sections below. +python3 - mcpp.toml <<'PYEOF' +import sys +p = sys.argv[1] +out, skip = [], False +for ln in open(p).read().splitlines(keepends=True): + if ln.strip() == "[build]": + skip = True + continue + if skip and ln.strip().startswith("cache"): + skip = False + continue + out.append(ln) +open(p, "w").write("".join(out)) +PYEOF +rm -rf target +"$MCPP" build > moderestore.log 2>&1 || { cat moderestore.log; exit 1; } + +# ── gc: LRU by last USE, not by when the entry was written ───────────────── +# Backdate the entry's accessed stamp, then confirm an age-bounded gc collects it +# — and that a build afterwards refreshes the stamp so it would survive next time. +entry="$(find "$MCPP_HOME/build-cache/v1/pkg" -name entry.json | head -1)" +python3 - "$entry" <<'PYEOF' +import json, sys +p = sys.argv[1] +d = json.load(open(p)) +d["accessed"] = "1000" # 1970-ish +json.dump(d, open(p, "w"), indent=2) +PYEOF +"$MCPP" cache gc --older-than 1s > gc.log 2>&1 || { cat gc.log; exit 1; } +[[ "$(entry_count)" -eq 0 ]] || { + echo "FAIL: gc did not collect a long-unused entry" + cat gc.log + exit 1 +} +grep -q 'Collected' gc.log || { cat gc.log; echo "FAIL: gc wording"; exit 1; } + +rm -rf target +"$MCPP" build > refill.log 2>&1 || { cat refill.log; exit 1; } +"$MCPP" build > touch.log 2>&1 || { cat touch.log; exit 1; } # a HIT +"$MCPP" cache gc --older-than 1h > gc2.log 2>&1 || { cat gc2.log; exit 1; } +[[ "$(entry_count)" -eq 1 ]] || { + echo "FAIL: gc collected an entry that was just used" + cat gc2.log + exit 1 +} + +# gc requires a budget; no arguments is a usage error, not a silent no-op. +rc=0 +"$MCPP" cache gc > gcnoargs.log 2>&1 || rc=$? +[[ "$rc" -ne 0 ]] || { echo "FAIL: gc with no budget succeeded"; cat gcnoargs.log; exit 1; } + +# gc --max-size evicts package entries to hit a budget, and leaves std entries +# alone: a std BMI is shared by every project on the machine and costs ~30 s to +# rebuild, so trading it for a little disk is the wrong trade. The summary must +# therefore talk about PACKAGE entries — reporting it as "cache now 0 B" would +# read as an empty cache while tens of MB of std BMIs sit next to it. +[[ "$(entry_count)" -eq 1 ]] || { echo "FAIL: expected one package entry before gc"; exit 1; } +[[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { echo "FAIL: no std entries to protect"; exit 1; } +"$MCPP" cache gc --max-size 1B > gcsize.log 2>&1 || { cat gcsize.log; exit 1; } +[[ "$(entry_count)" -eq 0 ]] || { echo "FAIL: gc --max-size kept package entries"; cat gcsize.log; exit 1; } +[[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { + echo "FAIL: gc --max-size evicted std entries" + cat gcsize.log + exit 1 +} +grep -q 'package entries now' gcsize.log || { + cat gcsize.log + echo "FAIL: gc summary must scope its figure to package entries" + exit 1 +} + +# Refill for the clean tests below. +rm -rf target +"$MCPP" build > refill2.log 2>&1 || { cat refill2.log; exit 1; } + +# ── clean --std / --all / --legacy ───────────────────────────────────────── +[[ -d "$MCPP_HOME/build-cache/v1/std" ]] || { echo "FAIL: no std cache dir"; exit 1; } +"$MCPP" cache clean --std > cleanstd.log 2>&1 +[[ ! -d "$MCPP_HOME/build-cache/v1/std" ]] || { + echo "FAIL: clean --std left the std entries" + exit 1 +} +[[ "$(entry_count)" -eq 1 ]] || { echo "FAIL: clean --std dropped package entries too"; exit 1; } + +"$MCPP" cache clean --all > cleanall.log 2>&1 +[[ "$(entry_count)" -eq 0 ]] || { echo "FAIL: clean --all left package entries"; exit 1; } + +# --legacy targets the pre-v1 tree, which nothing reads any more. +mkdir -p "$MCPP_HOME/bmi/deadbeef/deps/idx/pkg@1.0.0" +echo x > "$MCPP_HOME/bmi/deadbeef/deps/idx/pkg@1.0.0/old.o" +"$MCPP" cache clean --legacy > cleanlegacy.log 2>&1 +[[ ! -d "$MCPP_HOME/bmi" ]] || { cat cleanlegacy.log; echo "FAIL: clean --legacy left $MCPP_HOME/bmi"; exit 1; } +grep -q 'pre-v1' cleanlegacy.log || { cat cleanlegacy.log; echo "FAIL: legacy wording"; exit 1; } + +echo "OK" diff --git a/tests/e2e/19_bmi_cache_reuse.sh b/tests/e2e/19_bmi_cache_reuse.sh index 2f495724..e83023d9 100755 --- a/tests/e2e/19_bmi_cache_reuse.sh +++ b/tests/e2e/19_bmi_cache_reuse.sh @@ -1,15 +1,21 @@ #!/usr/bin/env bash # requires: -# 19_bmi_cache_reuse.sh — verify M3.2 BMI persistent cache wiring. +# 19_bmi_cache_reuse.sh — local-source packages must never enter the global +# build cache, DIRECTLY or TRANSITIVELY. # -# 1. Path deps don't populate the cache (correctness invariant from docs/26). -# 2. The cache layout matches docs/26 §二 (~/.mcpp/bmi//deps/...). +# 1. A direct path dep is not cached. +# 2. A path dep reached THROUGH another path dep is not cached either. # -# We don't fetch a real registry dep here (would need network, and existing -# E2E tests deliberately avoid that). The cross-project reuse positive case -# is covered by unit tests around the bmi_cache module's stage/populate -# round-trip; this script verifies the *wiring* into mcpp build doesn't -# break path-dep flows and that bmi/ is created lazily. +# (2) is the regression this file exists for. The old predicate looked the +# package up in the ROOT manifest's dependencies/dev-dependencies and skipped it +# when the spec was path/git — but a transitively-reached package is in neither +# map, so it fell through and WAS cached, with its index name defaulting to the +# default index (a workspace member `B` landed on disk as `mcpplibs/B@0.1.0`). +# Its sources can then change without changing name@version, so the cache key +# cannot see the change: a stale object, served silently. +# +# The cross-project positive case needs a registry dep and lives in +# 172_build_cache_cross_project.sh. set -e TMP=$(mktemp -d) @@ -69,11 +75,12 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; exit 1; } -# bmi/ should exist (env init creates it) but no deps/ entry for path deps. -[[ -d "$MCPP_HOME/bmi" ]] || { echo "missing $MCPP_HOME/bmi"; exit 1; } -if find "$MCPP_HOME/bmi" -path "*/deps/*/mylibA*" 2>/dev/null | grep -q .; then - echo "FAIL: path dep mylibA was populated into BMI cache (must be skipped)" - find "$MCPP_HOME/bmi" -maxdepth 5 +# The cache root exists (env init creates it) but holds no package entry for a +# path dep. +[[ -d "$MCPP_HOME/build-cache/v1" ]] || { echo "missing $MCPP_HOME/build-cache/v1"; exit 1; } +if find "$MCPP_HOME/build-cache/v1/pkg" -path "*mylibA*" 2>/dev/null | grep -q .; then + echo "FAIL: path dep mylibA was populated into the build cache (must be skipped)" + find "$MCPP_HOME/build-cache/v1" -maxdepth 5 exit 1 fi @@ -83,36 +90,79 @@ if grep -q 'Cached mylibA' build.log; then cat build.log; exit 1 fi -# --- Part 2: pre-seed a fake cache entry and verify it's staged into target/ --- -# Capture this build's fingerprint dir so we can drop a fake registry-dep -# cache entry beside it. -cd .. -rm -rf myapp/target -fp_dir="$(ls "$MCPP_HOME/bmi" | head -1)" -[[ -n "$fp_dir" ]] || { echo "no fingerprint dir under bmi/"; exit 1; } +# --- Part 2: a path dep reached THROUGH a path dep is also excluded --------- +cd "$TMP" +"$MCPP" new mylibB > /dev/null +cd mylibB +cat > src/inner.cppm <<'EOF' +export module mylibB.inner; +export auto inner() -> int { return 7; } +EOF +rm src/main.cpp +cat > mcpp.toml <<'EOF' +[package] +name = "mylibB" +version = "0.1.0" +[language] +standard = "c++23" +modules = true +import_std = true +[modules] +sources = ["src/**/*.cppm"] +[targets.mylibB] +kind = "lib" +EOF -# Create a synthetic cache entry as if a previous build produced it. -fake_pkg_dir="$MCPP_HOME/bmi/$fp_dir/deps/mcpplibs/fake.pkg@9.9.9" -mkdir -p "$fake_pkg_dir/gcm.cache" "$fake_pkg_dir/obj" -echo SYN > "$fake_pkg_dir/gcm.cache/fake.pkg.gcm" -echo SYN > "$fake_pkg_dir/obj/fake.m.o" -cat > "$fake_pkg_dir/manifest.txt" <<'EOF' -# Auto-generated by mcpp bmi_cache. Do not edit. -gcm: fake.pkg.gcm -obj: fake.m.o +# mylibA now depends on mylibB by path, so myapp reaches mylibB transitively +# and mylibB appears in NEITHER of myapp's dependency maps. +cd "$TMP/mylibA" +cat > src/greet.cppm <<'EOF' +export module mylibA.greet; +import std; +import mylibB.inner; +export auto greet() -> void { std::println("hi {}", inner()); } +EOF +cat >> mcpp.toml <<'EOF' +[dependencies.mylibB] +path = "../mylibB" EOF -[[ -f "$fake_pkg_dir/manifest.txt" ]] || { echo "synth manifest missing"; exit 1; } -# Smoke check: the manifest format documented in docs/26 §5.2 is parseable -# (one gcm: line + one obj: line + leading comment). We don't run mcpp build -# against this synthetic dep because there's no consuming manifest with a -# version-style [dependencies] entry that resolves without a real registry — -# that would require network. The unit tests cover the parse + stage path. +cd "$TMP/myapp" +rm -rf target +"$MCPP" build > build2.log 2>&1 || { cat build2.log; exit 1; } + +for name in mylibA mylibB; do + if find "$MCPP_HOME/build-cache/v1/pkg" -path "*$name*" 2>/dev/null | grep -q .; then + echo "FAIL: local package $name entered the build cache" + find "$MCPP_HOME/build-cache/v1/pkg" -maxdepth 4 2>/dev/null + cat build2.log + exit 1 + fi +done -# Sanity: parse comments + entries the same way bmi_cache does. -gcm_count=$(grep -c '^gcm: ' "$fake_pkg_dir/manifest.txt") -obj_count=$(grep -c '^obj: ' "$fake_pkg_dir/manifest.txt") -[[ "$gcm_count" == "1" ]] || { echo "expected 1 gcm: line, got $gcm_count"; exit 1; } -[[ "$obj_count" == "1" ]] || { echo "expected 1 obj: line, got $obj_count"; exit 1; } +# And `mcpp cache list` — the user-visible surface — must not list them either. +"$MCPP" cache list > list.log 2>&1 +for name in mylibA mylibB; do + if grep -q "$name" list.log; then + echo "FAIL: cache list shows local package $name" + cat list.log + exit 1 + fi +done + +# Editing the transitive path dep must still take effect (it is compiled, not +# served from a cache that cannot see the change). +cd "$TMP/mylibB" +cat > src/inner.cppm <<'EOF' +export module mylibB.inner; +export auto inner() -> int { return 99; } +EOF +cd "$TMP/myapp" +"$MCPP" run > run.log 2>&1 || { cat run.log; exit 1; } +grep -q 'hi 99' run.log || { + echo "FAIL: edit to a transitive path dep did not reach the binary" + cat run.log + exit 1 +} echo "OK" diff --git a/tests/e2e/22_doctor_cache_publish.sh b/tests/e2e/22_doctor_cache_publish.sh index 6ec661dc..c33960d2 100755 --- a/tests/e2e/22_doctor_cache_publish.sh +++ b/tests/e2e/22_doctor_cache_publish.sh @@ -11,6 +11,14 @@ TMP=$(mktemp -d) trap "rm -rf $TMP" EXIT export MCPP_HOME="$TMP/mcpp-home" +# 0) cache list on a genuinely empty cache: friendly message. +# This has to come BEFORE doctor — doctor resolves a build plan, which +# precompiles the std module, and `cache list` now reports std entries too. +# Hiding them is how 16 GB of duplicated std BMIs went unnoticed, so listing +# them is the fix, not a regression. +out=$("$MCPP" cache list 2>&1) +[[ "$out" == *"empty"* ]] || { echo "cache list on empty cache: '$out'"; exit 1; } + # 1) doctor: should always run, exit 0 or 1 (warn ok), never 2 rc=0 "$MCPP" self doctor > doctor.log 2>&1 || rc=$? @@ -21,9 +29,24 @@ grep -q 'Checking registry' doctor.log || { cat doctor.log; echo "no regis grep -q 'Checking cache health' doctor.log || { cat doctor.log; echo "no cache check"; exit 1; } grep -q 'Doctor result' doctor.log || { cat doctor.log; echo "no result line"; exit 1; } -# 2) cache list (empty): friendly message +# 2) after doctor, the std module it precompiled must be VISIBLE. std entries +# dominate the cache's size, and `cache list` used to walk only dep entries. out=$("$MCPP" cache list 2>&1) -[[ "$out" == *"empty"* ]] || { echo "cache list empty: '$out'"; exit 1; } +[[ "$out" == *"std"* ]] || { echo "cache list omits the std entry: '$out'"; exit 1; } +# The age column must be plausible: file_time_type's epoch is not the Unix +# epoch, and reading it as if it were printed ages like "74509d ago". +if [[ "$out" =~ ([0-9]+)d\ ago ]] && (( BASH_REMATCH[1] > 3650 )); then + echo "cache list reports an implausible age (clock epoch bug): '$out'" + exit 1 +fi + +# 2b) cache dir must point at the cache the rest of the tool uses. +out=$("$MCPP" cache dir 2>&1) +[[ "$out" == "$MCPP_HOME/build-cache/v1"* ]] || { + echo "cache dir '$out' != '$MCPP_HOME/build-cache/v1'"; exit 1; } + +# 2c) verify must pass on a healthy cache. +"$MCPP" cache verify > /tmp/_v.log 2>&1 || { cat /tmp/_v.log; echo "verify failed"; exit 1; } # 3) publish dry-run on a fresh package. Publish uses `git archive` for the # source tarball, so we git-init + commit first. We also need a non-empty diff --git a/tests/e2e/40_llvm_bmi_cache.sh b/tests/e2e/40_llvm_bmi_cache.sh index 41d23eba..698eaa1e 100755 --- a/tests/e2e/40_llvm_bmi_cache.sh +++ b/tests/e2e/40_llvm_bmi_cache.sh @@ -60,17 +60,17 @@ int main() { } EOF -# First build — should compile the dependency -out1=$("$MCPP" build --no-cache 2>&1) +# First build — populates the cache. Deliberately NOT `--no-cache`: that is now +# an alias for `--cache=off`, which means neither read NOR write, so a build that +# used it would leave nothing for the second build to reuse. MCPP_HOME is fresh +# here, so this build is already cold. +out1=$("$MCPP" build 2>&1) echo "$out1" | grep -q "Compiling.*mcpplibs.cmdline" || { - # It's OK if it says "Cached" because global cache may exist - echo "$out1" | grep -q "Cached.*mcpplibs.cmdline" || { - echo "FAIL: mcpplibs.cmdline not mentioned in first build: $out1" - exit 1 - } + echo "FAIL: mcpplibs.cmdline not compiled in the first (cold) build: $out1" + exit 1 } -# Second build (clean target, keep BMI cache) — dependency should be cached +# Second build, clean target dir, cache kept — the dependency must be reused. rm -rf target out2=$("$MCPP" build 2>&1) echo "$out2" | grep -q "Cached.*mcpplibs.cmdline" || { @@ -78,4 +78,27 @@ echo "$out2" | grep -q "Cached.*mcpplibs.cmdline" || { exit 1 } +# ...and reuse must mean "not recompiled". The status line alone used to be +# printed while ninja rebuilt every unit behind it, so assert on the graph. +NINJA="$(find target -name build.ninja | head -1)" +[[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja"; exit 1; } +if grep -qE ': (cxx_module|cxx_object|cxx_scan) .*mcpplibs' "$NINJA"; then + echo "FAIL: cached dependency still has compile edges" + grep -nE ': (cxx_module|cxx_object|cxx_scan) .*mcpplibs' "$NINJA" | head + exit 1 +fi + +# Clang + module partitions: mcpplibs.cmdline has a `:options` partition, which a +# consumer never imports directly. Its stage edge must be sequenced before the +# consumer's compile, or clang fails with `failed to find module file for module +# 'mcpplibs.cmdline:options'` — a race Linux won and macOS lost. +grep -q '_mcpp_staged_cache' "$NINJA" || { + echo "FAIL: staged artifacts are not sequenced before compilation" + exit 1 +} + +# And the whole thing must actually link and run. +out3=$("$MCPP" run 2>&1) || { echo "FAIL: run: $out3"; exit 1; } +echo "$out3" | grep -q 'cache test ok' || { echo "FAIL: run output: $out3"; exit 1; } + echo "OK" diff --git a/tests/e2e/49_bmi_cache_nested_custom_index.sh b/tests/e2e/49_bmi_cache_nested_custom_index.sh index 7a2c915b..cdc8b633 100644 --- a/tests/e2e/49_bmi_cache_nested_custom_index.sh +++ b/tests/e2e/49_bmi_cache_nested_custom_index.sh @@ -93,27 +93,53 @@ if grep -q "bmi cache populate failed" build.log; then exit 1 fi -manifest="$(find "$MCPP_HOME/bmi" -path "*/deps/local-dev/*collision-lib@1.0.0/manifest.txt" | head -1)" -[[ -n "$manifest" ]] || { - echo "FAIL: missing custom-index BMI manifest" - find "$MCPP_HOME/bmi" -maxdepth 6 -type f | sort +entry="$(find "$MCPP_HOME/build-cache/v1/pkg/local-dev" -name entry.json | head -1)" +[[ -n "$entry" ]] || { + echo "FAIL: missing custom-index cache entry" + find "$MCPP_HOME/build-cache/v1" -maxdepth 6 -type f | sort cat build.log exit 1 } -grep -Eq '^obj: .+/foo\.m\.o$' "$manifest" || { - echo "FAIL: manifest did not preserve nested object path" - cat "$manifest" - exit 1 -} +# Read the structured entry instead of grepping a log: entry.json exists so the +# cache describes itself, and every object it lists must be on disk under obj/. +python3 - "$entry" > entrycheck.log 2>&1 <<'PYEOF' || { cat entrycheck.log; exit 1; } +import json, os, sys +entry = sys.argv[1] +d = json.load(open(entry)) +objs = d.get("obj", []) +if not [o for o in objs if "/" in o and o.endswith("foo.m.o")]: + sys.exit(f"FAIL: entry did not preserve a nested object path: {objs}") +root = os.path.dirname(entry) +for o in objs: + p = os.path.join(root, "obj", o) + if not os.path.exists(p): + sys.exit(f"FAIL: listed object missing on disk: {p}") +PYEOF rm -rf target "$MCPP" build > build2.log 2>&1 || { cat build2.log; exit 1; } grep -q "Cached local-dev.collision-lib v1.0.0" build2.log || { - echo "FAIL: second cold build did not reuse BMI cache" + echo "FAIL: second cold build did not reuse the build cache" cat build2.log exit 1 } +# A reused dependency must not be recompiled — that was the whole defect. Its +# outputs are declared by stage_file edges now, so there is no compile edge for +# its sources at all. +NINJA="$(find target -name build.ninja | head -1)" +[[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja"; exit 1; } +if grep -qE ': (cxx_module|cxx_object) .*collision-lib' "$NINJA"; then + echo "FAIL: cached dependency still has compile edges" + grep -nE ': (cxx_module|cxx_object) .*collision-lib' "$NINJA" + exit 1 +fi +grep -qE '^build .* : stage_file .*collision-lib' "$NINJA" || { + echo "FAIL: cached dependency has no stage_file edge" + grep -n 'stage_file' "$NINJA" | head + exit 1 +} + echo "OK" diff --git a/tests/e2e/59_cpp_standard_config.sh b/tests/e2e/59_cpp_standard_config.sh index 6eacd020..8345af61 100644 --- a/tests/e2e/59_cpp_standard_config.sh +++ b/tests/e2e/59_cpp_standard_config.sh @@ -71,11 +71,14 @@ if grep -q -- "-std=c++23" compile_commands.json; then exit 1 fi -metadata="$(find "$MCPP_HOME/bmi" -name std-module.json | head -1)" -[[ -n "$metadata" ]] || { echo "FAIL: std module metadata missing"; exit 1; } -grep -q '"cpp_standard": "c++26"' "$metadata" || { - echo "FAIL: std module metadata missing C++26 standard" - cat "$metadata" +# The entry recording c++26, selected by CONTENT rather than by `find | head -1`: +# one MCPP_HOME can hold several std identities now. +metadata="$(grep -rl '"cpp_standard": "c++26"' \ + "$MCPP_HOME/build-cache/v1/std" 2>/dev/null | head -1)" +[[ -n "$metadata" ]] || { + echo "FAIL: no std module metadata records c++26" + find "$MCPP_HOME/build-cache/v1/std" -name std-module.json \ + -exec grep -H '"cpp_standard"' {} \; 2>/dev/null exit 1 } grep -q '"std_flag": "-std=c++26"' "$metadata" || { diff --git a/tests/e2e/98_reflection_import_std.sh b/tests/e2e/98_reflection_import_std.sh index 86c046a3..4d9430b8 100755 --- a/tests/e2e/98_reflection_import_std.sh +++ b/tests/e2e/98_reflection_import_std.sh @@ -80,7 +80,15 @@ out=$("$MCPP" run 2>&1) || { echo "FAIL: build/run: $out"; exit 1; } [[ "$out" == *"dep-ok"* ]] || { echo "FAIL: dep module output: $out"; exit 1; } # The std BMI build command must carry the dialect flag. -grep -rl '"std_flag": "[^"]*-freflection' "${MCPP_HOME:-$HOME/.mcpp}/bmi/" >/dev/null \ - || { echo "FAIL: std-module.json lacks -freflection in std_flag"; exit 1; } +# Recursive grep across every std entry, not `find | head -1`: one MCPP_HOME can +# now hold several std identities (they are keyed by std identity rather than by +# whole-project fingerprint), so "the first one" is arbitrary. Asserting that SOME +# entry records the flag is the claim that matters. +STD_ROOT="${MCPP_HOME:-$HOME/.mcpp}/build-cache/v1/std" +grep -rl '"std_flag": "[^"]*-freflection' "$STD_ROOT" >/dev/null 2>&1 || { + echo "FAIL: no std-module.json records -freflection in std_flag" + find "$STD_ROOT" -name std-module.json -exec grep -H '"std_flag"' {} \; 2>/dev/null + exit 1 +} echo "PASS: dialect flags reach std BMI + whole module graph (issue #210)" diff --git a/tests/unit/test_bmi_cache.cpp b/tests/unit/test_bmi_cache.cpp index 714bc25d..6fe2c517 100644 --- a/tests/unit/test_bmi_cache.cpp +++ b/tests/unit/test_bmi_cache.cpp @@ -7,6 +7,7 @@ import std; import mcpp.bmi_cache; +import mcpp.libs.json; using namespace mcpp::bmi_cache; @@ -26,13 +27,27 @@ struct Tmp { } }; -CacheKey makeKey(const std::filesystem::path& home) { +nlohmann::json makeInputs(std::string_view flavor = "base") { + nlohmann::json j; + j["epoch"] = 1; + j["toolchain"] = {{"compiler", "gcc"}, {"compiler_version", "16.1.0"}}; + j["profile"] = {{"opt_level", "2"}, {"debug", false}}; + j["package"] = {{"index", "mcpplibs"}, {"name", "mcpplibs.cmdline"}, + {"version", "0.0.1"}}; + j["config"] = {{"cxxflags", nlohmann::json::array({std::string(flavor)})}}; + return j; +} + +CacheKey makeKey(const std::filesystem::path& root, + std::string_view key = "deadbeef0123abcd", + std::string_view flavor = "base") { return CacheKey{ - .mcppHome = home, - .fingerprint = "deadbeef0123abcd", + .cacheRoot = root, .indexName = "mcpplibs", .packageName = "mcpplibs.cmdline", .version = "0.0.1", + .keyHex = std::string(key), + .inputs = makeInputs(flavor), }; } @@ -41,153 +56,233 @@ void writeFile(const std::filesystem::path& p, std::string_view body) { std::ofstream(p) << body; } +std::string readFile(const std::filesystem::path& p) { + std::ifstream is(p); + return std::string((std::istreambuf_iterator(is)), {}); +} + +nlohmann::json readJson(const std::filesystem::path& p) { + std::ifstream is(p); + nlohmann::json j; + is >> j; + return j; +} + +DepArtifacts oneOfEach() { + return DepArtifacts{ .bmiFiles = {"lib.gcm"}, .objFiles = {"lib.m.o"} }; +} + +void seedProject(const std::filesystem::path& project) { + writeFile(project / "gcm.cache" / "lib.gcm", "G"); + writeFile(project / "obj" / "lib.m.o", "O"); +} + } // namespace -TEST(BmiCache, KeyDirLayoutMatchesDocs26) { - auto k = makeKey("/home/u/.mcpp"); - auto expected = std::filesystem::path("/home/u/.mcpp/bmi/deadbeef0123abcd/deps/mcpplibs/mcpplibs.cmdline@0.0.1"); +TEST(BmiCache, EntryDirIsKeyedByPackageIdentityAndKeyNotByProject) { + auto k = makeKey("/home/u/.mcpp/build-cache/v1"); + auto expected = std::filesystem::path( + "/home/u/.mcpp/build-cache/v1/pkg/mcpplibs/mcpplibs.cmdline@0.0.1/deadbeef0123abcd"); EXPECT_EQ(k.dir(), expected); - EXPECT_EQ(k.manifestFile().filename().string(), "manifest.txt"); - EXPECT_EQ(k.bmiDir().filename().string(), "gcm.cache"); - EXPECT_EQ(k.objDir().filename().string(), "obj"); + EXPECT_EQ(k.entryFile().filename().string(), "entry.json"); + EXPECT_EQ(k.bmiDir().filename().string(), "bmi"); + EXPECT_EQ(k.objDir().filename().string(), "obj"); +} + +// The layout version must be a path segment: replacing the layout has to be a +// new tree, never a migration of (or a deletion of) the old one. +TEST(BmiCache, EntryDirCarriesTheLayoutVersion) { + auto k = makeKey("/home/u/.mcpp/build-cache/v1"); + auto s = k.dir().generic_string(); + EXPECT_NE(s.find("/build-cache/v1/pkg/"), std::string::npos) << s; + // And nothing may land under the pre-v1 root. + EXPECT_EQ(s.find("/.mcpp/bmi/"), std::string::npos) << s; } -TEST(BmiCache, IsCachedFalseWhenManifestMissing) { +TEST(BmiCache, IsCachedFalseWhenEntryMissing) { Tmp t; - auto k = makeKey(t.path); - EXPECT_FALSE(is_cached(k)); + EXPECT_FALSE(is_cached(makeKey(t.path))); } -TEST(BmiCache, PopulateThenStageRoundTrip) { +TEST(BmiCache, PopulateWritesSelfDescribingEntry) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); - - writeFile(project / "gcm.cache" / "mcpplibs.cmdline.gcm", "GCM-A"); - writeFile(project / "gcm.cache" / "mcpplibs.cmdline-options.gcm", "GCM-B"); - writeFile(project / "obj" / "cmdline.m.o", "OBJ-A"); - - DepArtifacts arts { - .bmiFiles = { "mcpplibs.cmdline.gcm", "mcpplibs.cmdline-options.gcm" }, - .objFiles = { "cmdline.m.o" }, - }; + seedProject(project); auto k = makeKey(home); - auto pop = populate_from(k, project, arts); + auto pop = populate_from(k, project, oneOfEach()); ASSERT_TRUE(pop) << pop.error(); - EXPECT_TRUE(std::filesystem::exists(k.manifestFile())); - EXPECT_TRUE(std::filesystem::exists(k.bmiDir() / "mcpplibs.cmdline.gcm")); - EXPECT_TRUE(std::filesystem::exists(k.bmiDir() / "mcpplibs.cmdline-options.gcm")); - EXPECT_TRUE(std::filesystem::exists(k.objDir() / "cmdline.m.o")); + ASSERT_TRUE(std::filesystem::exists(k.entryFile())); + EXPECT_TRUE(std::filesystem::exists(k.bmiDir() / "lib.gcm")); + EXPECT_TRUE(std::filesystem::exists(k.objDir() / "lib.m.o")); EXPECT_TRUE(is_cached(k)); - // Round-trip: stage into a fresh project dir. - auto project2 = t.path / "proj2" / "target"; - auto staged = stage_into(k, project2); - ASSERT_TRUE(staged) << staged.error(); - EXPECT_EQ(staged->bmiFiles.size(), 2u); - EXPECT_EQ(staged->objFiles.size(), 1u); - EXPECT_TRUE(std::filesystem::exists(project2 / "gcm.cache" / "mcpplibs.cmdline.gcm")); - EXPECT_TRUE(std::filesystem::exists(project2 / "obj" / "cmdline.m.o")); + // The entry has to carry the key AND the full inputs: a hit is validated + // field by field, so a cache whose entries only listed files could never be + // audited when a wrong hit was suspected. + auto j = readJson(k.entryFile()); + EXPECT_EQ(j.value("key", std::string{}), "deadbeef0123abcd"); + EXPECT_EQ(j["inputs"], makeInputs()); + EXPECT_EQ(j["bmi"].size(), 1u); + EXPECT_EQ(j["obj"].size(), 1u); + EXPECT_TRUE(j.contains("created")); + EXPECT_TRUE(j.contains("accessed")); +} - // Staged file content must match original. - std::ifstream is(project2 / "obj" / "cmdline.m.o"); - std::string body((std::istreambuf_iterator(is)), {}); - EXPECT_EQ(body, "OBJ-A"); +// The whole point of recording inputs: equal hashes are never trusted on their +// own. Same key directory, different inputs ⇒ miss. +TEST(BmiCache, IsCachedFalseWhenRecordedInputsDiffer) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto written = makeKey(home, "samekey00000000", "base"); + ASSERT_TRUE(populate_from(written, project, oneOfEach())); + EXPECT_TRUE(is_cached(written)); + + auto probed = makeKey(home, "samekey00000000", "different-flag"); + EXPECT_FALSE(is_cached(probed)); } -TEST(BmiCache, StageIntoDoesNotTouchIdenticalOutputs) { +// An entry written by an older mcpp may carry extra keys, but every field the +// current build cares about must be present. Missing ⇒ mismatch, never a pass. +TEST(BmiCache, IsCachedFalseWhenARequiredInputFieldIsAbsent) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); + seedProject(project); - writeFile(project / "gcm.cache" / "mcpplibs.cmdline.gcm", "GCM-A"); - writeFile(project / "obj" / "cmdline.m.o", "OBJ-A"); + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); - DepArtifacts arts { - .bmiFiles = { "mcpplibs.cmdline.gcm" }, - .objFiles = { "cmdline.m.o" }, - }; + auto j = readJson(k.entryFile()); + j["inputs"].erase("profile"); + std::ofstream(k.entryFile()) << j.dump(2); + + EXPECT_FALSE(is_cached(k)); +} + +TEST(BmiCache, IsCachedFalseWhenSchemaDiffers) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); auto k = makeKey(home); - ASSERT_TRUE(populate_from(k, project, arts)); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); - auto staged = stage_into(k, project); - ASSERT_TRUE(staged) << staged.error(); - auto gcmTime = std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"); - auto objTime = std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"); + auto j = readJson(k.entryFile()); + j["schema"] = kEntrySchema + 1; + std::ofstream(k.entryFile()) << j.dump(2); - auto stagedAgain = stage_into(k, project); - ASSERT_TRUE(stagedAgain) << stagedAgain.error(); - EXPECT_EQ(std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"), gcmTime); - EXPECT_EQ(std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"), objTime); + EXPECT_FALSE(is_cached(k)); } -TEST(BmiCache, StageIntoDoesNotOverwriteExistingOutputs) { +TEST(BmiCache, IsCachedFalseWhenSentinelExistsButFileMissing) { Tmp t; - auto home = t.path / "home"; - auto cacheProject = t.path / "cache-proj" / "target"; - auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(cacheProject / "gcm.cache"); - std::filesystem::create_directories(cacheProject / "obj"); - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); - writeFile(cacheProject / "gcm.cache" / "mcpplibs.cmdline.gcm", "CACHE-GCM"); - writeFile(cacheProject / "obj" / "cmdline.m.o", "CACHE-OBJ"); + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + ASSERT_TRUE(is_cached(k)); - DepArtifacts arts { - .bmiFiles = { "mcpplibs.cmdline.gcm" }, - .objFiles = { "cmdline.m.o" }, - }; + std::filesystem::remove(k.objDir() / "lib.m.o"); + EXPECT_FALSE(is_cached(k)); +} + +TEST(BmiCache, ResolveCachedReportsTheArtifactsWithoutCopying) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); auto k = makeKey(home); - ASSERT_TRUE(populate_from(k, cacheProject, arts)); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); - writeFile(project / "gcm.cache" / "mcpplibs.cmdline.gcm", "PROJECT-GCM"); - writeFile(project / "obj" / "cmdline.m.o", "PROJECT-OBJ"); - auto gcmTime = std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"); - auto objTime = std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"); + auto arts = resolve_cached(k); + ASSERT_TRUE(arts) << arts.error(); + EXPECT_EQ(arts->bmiFiles, std::vector{"lib.gcm"}); + EXPECT_EQ(arts->objFiles, std::vector{"lib.m.o"}); - auto staged = stage_into(k, project); - ASSERT_TRUE(staged) << staged.error(); + // resolve_cached must not write into a project dir. Staging is a ninja edge + // now: copying artifacts in from outside the graph is exactly what made the + // old cache a no-op — ninja rebuilds any output it has no command-line + // record for, so the copies were recompiled over on every fresh build dir. + auto project2 = t.path / "proj2" / "target"; + (void)resolve_cached(k); + EXPECT_FALSE(std::filesystem::exists(project2)); +} - { - std::ifstream is(project / "gcm.cache" / "mcpplibs.cmdline.gcm"); - std::string body((std::istreambuf_iterator(is)), {}); - EXPECT_EQ(body, "PROJECT-GCM"); - } - { - std::ifstream is(project / "obj" / "cmdline.m.o"); - std::string body((std::istreambuf_iterator(is)), {}); - EXPECT_EQ(body, "PROJECT-OBJ"); - } - EXPECT_EQ(std::filesystem::last_write_time(project / "gcm.cache" / "mcpplibs.cmdline.gcm"), gcmTime); - EXPECT_EQ(std::filesystem::last_write_time(project / "obj" / "cmdline.m.o"), objTime); +TEST(BmiCache, CachedPathsPointIntoTheEntry) { + auto k = makeKey("/home/u/.mcpp/build-cache/v1"); + EXPECT_EQ(cached_bmi_path(k, "lib.gcm"), k.bmiDir() / "lib.gcm"); + EXPECT_EQ(cached_obj_path(k, "sub/lib.m.o"), k.objDir() / "sub" / "lib.m.o"); } -TEST(BmiCache, IsCachedFalseWhenSentinelExistsButFileMissing) { +// Nested object paths (mcpp#233's per-package object prefixes) must round-trip. +TEST(BmiCache, PopulateHandlesNestedObjectPaths) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); - writeFile(project / "gcm.cache" / "lib.gcm", "G"); - writeFile(project / "obj" / "lib.m.o", "O"); + writeFile(project / "obj" / "pkg_zlib" / "zlib-1.3" / "compress.o", "NESTED"); - DepArtifacts arts { .bmiFiles = {"lib.gcm"}, .objFiles = {"lib.m.o"} }; auto k = makeKey(home); + DepArtifacts arts { .objFiles = {"pkg_zlib/zlib-1.3/compress.o"} }; ASSERT_TRUE(populate_from(k, project, arts)); - ASSERT_TRUE(is_cached(k)); + EXPECT_EQ(readFile(k.objDir() / "pkg_zlib" / "zlib-1.3" / "compress.o"), "NESTED"); + EXPECT_TRUE(is_cached(k)); +} - // Delete one cached file → is_cached must return false even though manifest still exists. - std::filesystem::remove(k.objDir() / "lib.m.o"); - EXPECT_FALSE(is_cached(k)); +// touch_accessed is what makes `cache gc` an LRU rather than "drop what was +// populated long ago". It must move the stamp and leave the artifacts alone — +// their mtimes participate in ninja's restat handling. +TEST(BmiCache, TouchAccessedMovesTheStampAndNotTheArtifacts) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + + auto j0 = readJson(k.entryFile()); + auto created0 = j0.value("created", std::string{}); + auto objTime0 = std::filesystem::last_write_time(k.objDir() / "lib.m.o"); + auto bmiTime0 = std::filesystem::last_write_time(k.bmiDir() / "lib.gcm"); + + // Rewrite the stamp to something clearly old, then touch. + j0["accessed"] = "1000"; + std::ofstream(k.entryFile()) << j0.dump(2); + touch_accessed(k); + + auto j1 = readJson(k.entryFile()); + EXPECT_NE(j1.value("accessed", std::string{}), "1000"); + EXPECT_EQ(j1.value("created", std::string{}), created0) + << "touch must not reset the creation stamp"; + EXPECT_EQ(std::filesystem::last_write_time(k.objDir() / "lib.m.o"), objTime0); + EXPECT_EQ(std::filesystem::last_write_time(k.bmiDir() / "lib.gcm"), bmiTime0); + EXPECT_TRUE(is_cached(k)) << "touching must not invalidate the entry"; +} + +TEST(BmiCache, RepopulatePreservesCreatedStamp) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto k = makeKey(home); + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + auto j0 = readJson(k.entryFile()); + j0["created"] = "12345"; + std::ofstream(k.entryFile()) << j0.dump(2); + + ASSERT_TRUE(populate_from(k, project, oneOfEach())); + EXPECT_EQ(readJson(k.entryFile()).value("created", std::string{}), "12345"); } TEST(BmiCache, PopulateFailsIfBuildOutputMissing) { @@ -202,39 +297,49 @@ TEST(BmiCache, PopulateFailsIfBuildOutputMissing) { EXPECT_NE(pop.error().find("expected build output missing"), std::string::npos); } +// Two keys for the same package@version are independent entries — that is what +// lets one machine hold a dev-profile and a release-profile build of the same +// dependency at once. +TEST(BmiCache, DifferentKeysAreIndependentEntries) { + Tmp t; + auto home = t.path / "home"; + auto project = t.path / "proj" / "target"; + seedProject(project); + + auto a = makeKey(home, "aaaaaaaaaaaaaaaa", "opt"); + auto b = makeKey(home, "bbbbbbbbbbbbbbbb", "debug"); + ASSERT_TRUE(populate_from(a, project, oneOfEach())); + ASSERT_TRUE(populate_from(b, project, oneOfEach())); + EXPECT_NE(a.dir(), b.dir()); + EXPECT_TRUE(is_cached(a)); + EXPECT_TRUE(is_cached(b)); +} + #if !defined(_WIN32) -// M4 #9: when an external holder takes the .lock, populate_from must skip -// (returns success but does NOT clobber the directory). -// Uses flock() which is POSIX-only. +// When an external holder takes the .lock, populate_from must skip (returns +// success but does NOT clobber the entry). Uses flock(), POSIX-only. TEST(BmiCache, PopulateSkipsWhenLockHeld) { Tmp t; auto home = t.path / "home"; auto project = t.path / "proj" / "target"; - std::filesystem::create_directories(project / "gcm.cache"); - std::filesystem::create_directories(project / "obj"); - writeFile(project / "gcm.cache" / "lib.gcm", "G2"); - writeFile(project / "obj" / "lib.m.o", "O2"); + seedProject(project); auto k = makeKey(home); - // Take the lock manually before populate runs. std::filesystem::create_directories(k.dir()); auto lockPath = k.dir() / ".lock"; int fd = ::open(lockPath.c_str(), O_CREAT | O_RDWR, 0644); ASSERT_GE(fd, 0); ASSERT_EQ(::flock(fd, LOCK_EX | LOCK_NB), 0); - DepArtifacts arts { .bmiFiles = {"lib.gcm"}, .objFiles = {"lib.m.o"} }; - auto pop = populate_from(k, project, arts); + auto pop = populate_from(k, project, oneOfEach()); EXPECT_TRUE(pop) << "should silently skip when lock is held"; - // manifest.txt must NOT have been written by the second writer. - EXPECT_FALSE(std::filesystem::exists(k.manifestFile())); + EXPECT_FALSE(std::filesystem::exists(k.entryFile())); ::flock(fd, LOCK_UN); ::close(fd); - // After lock released, a fresh populate should succeed. - auto pop2 = populate_from(k, project, arts); + auto pop2 = populate_from(k, project, oneOfEach()); ASSERT_TRUE(pop2) << pop2.error(); - EXPECT_TRUE(std::filesystem::exists(k.manifestFile())); + EXPECT_TRUE(std::filesystem::exists(k.entryFile())); } #endif // !defined(_WIN32) diff --git a/tests/unit/test_build_profile.cpp b/tests/unit/test_build_profile.cpp new file mode 100644 index 00000000..df1cc3b8 --- /dev/null +++ b/tests/unit/test_build_profile.cpp @@ -0,0 +1,172 @@ +// The profile axis: profile selection, and the invariant that the resolved +// profile knobs participate in the fingerprint. +// +// Both halves were broken together. `[profile.]` lands its knobs in +// buildConfig.optLevel/debug/lto/strip and flags.cppm turns them into +// -O/-g/-flto, but canonical_compile_flags serialized only cflags/cxxflags/ +// ldflags — so `--dev`, `--release` and `--profile dist` produced ONE +// fingerprint, hence one target/// directory and one global cache +// entry. Meanwhile `.build_cache` keyed its fast-path entries by target triple +// alone and the fast path only refused to run for an EXPLICIT profile flag, so +// `mcpp build --release` followed by a bare `mcpp build` reported success in +// 0.00s and left the release artifacts in place. + +#include + +import std; +import mcpp.build.prepare; +import mcpp.manifest; + +namespace { + +mcpp::manifest::Manifest base() { + mcpp::manifest::Manifest m; + m.package.name = "demo"; + m.package.version = "0.1.0"; + m.package.standard = "c++23"; + m.buildConfig.optLevel = "2"; + return m; +} + +} // namespace + +// ── resolve_profile_name: the ONE rule, shared with the fast paths ─────────── + +TEST(BuildProfile, OverrideBeatsManifestDefault) { + auto m = base(); + m.buildConfig.defaultProfile = "release"; + EXPECT_EQ(mcpp::build::resolve_profile_name(m, "dev"), "dev"); +} + +TEST(BuildProfile, ManifestDefaultBeatsGlobalDefault) { + auto m = base(); + m.buildConfig.defaultProfile = "release"; + EXPECT_EQ(mcpp::build::resolve_profile_name(m, ""), "release"); +} + +TEST(BuildProfile, GlobalDefaultIsDev) { + EXPECT_EQ(mcpp::build::resolve_profile_name(base(), ""), "dev"); +} + +TEST(BuildProfile, CustomProfileNamePassesThrough) { + EXPECT_EQ(mcpp::build::resolve_profile_name(base(), "contracts"), "contracts"); +} + +// ── the fingerprint invariant ──────────────────────────────────────────────── + +TEST(BuildProfile, OptLevelIsInTheCanonicalFlags) { + auto a = base(); + auto b = base(); b.buildConfig.optLevel = "0"; + EXPECT_NE(mcpp::build::canonical_compile_flags(a), + mcpp::build::canonical_compile_flags(b)); +} + +TEST(BuildProfile, DebugLtoStripAreInTheCanonicalFlags) { + auto ref = mcpp::build::canonical_compile_flags(base()); + { auto m = base(); m.buildConfig.debug = true; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.lto = true; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.strip = true; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } +} + +// The exact shape the three built-in profiles resolve to (prepare_build's +// profile block). Asserting on the flag string rather than on a fingerprint hex +// keeps the failure legible when someone adds a knob and forgets to serialize it. +TEST(BuildProfile, BuiltInProfilesProduceDistinctCanonicalFlags) { + auto dev = base(); + dev.buildConfig.optLevel = "0"; + dev.buildConfig.debug = true; + + auto release = base(); + release.buildConfig.optLevel = "2"; + + auto dist = base(); + dist.buildConfig.optLevel = "3"; + dist.buildConfig.strip = true; + + auto fdev = mcpp::build::canonical_compile_flags(dev); + auto frelease = mcpp::build::canonical_compile_flags(release); + auto fdist = mcpp::build::canonical_compile_flags(dist); + + EXPECT_NE(fdev, frelease); + EXPECT_NE(frelease, fdist); + EXPECT_NE(fdev, fdist); +} + +TEST(BuildProfile, CanonicalFlagsStillCoverTheNonProfileKnobs) { + auto ref = mcpp::build::canonical_compile_flags(base()); + { auto m = base(); m.package.standard = "c++26"; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.cxxflags = {"-DFOO"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.cflags = {"-DBAR"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.ldflags = {"-lm"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.dialectCxxflags = {"-freflection"}; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } + { auto m = base(); m.buildConfig.cStandard = "c17"; + EXPECT_NE(mcpp::build::canonical_compile_flags(m), ref); } +} + +// ── cache-mode parsing ────────────────────────────────────────────────────── + +TEST(BuildProfile, CacheModeParsesTheThreeModes) { + using mcpp::build::CacheMode; + EXPECT_EQ(mcpp::build::parse_cache_mode("global"), CacheMode::Global); + EXPECT_EQ(mcpp::build::parse_cache_mode("local"), CacheMode::Local); + EXPECT_EQ(mcpp::build::parse_cache_mode("off"), CacheMode::Off); + // `none` accepted as a synonym for off; anything else is refused rather + // than silently meaning "global", which would be the worst default for a + // typo to land on. + EXPECT_EQ(mcpp::build::parse_cache_mode("none"), CacheMode::Off); + EXPECT_FALSE(mcpp::build::parse_cache_mode("on").has_value()); + EXPECT_FALSE(mcpp::build::parse_cache_mode("").has_value()); + EXPECT_FALSE(mcpp::build::parse_cache_mode("GLOBAL").has_value()); +} + +TEST(BuildProfile, CacheModeNameRoundTrips) { + using mcpp::build::CacheMode; + for (auto m : {CacheMode::Global, CacheMode::Local, CacheMode::Off}) + EXPECT_EQ(mcpp::build::parse_cache_mode(mcpp::build::cache_mode_name(m)), m); +} + +// ── cache-mode resolution ─────────────────────────────────────────────────── +// Same shape as resolve_profile_name and for the same reason: the fast paths +// skip prepare_build, so they must settle the mode from one shared pure rule. +// Without that, a graph generated under `global` (which contains stage_file +// edges reading the cache) got replayed for a request that asked for `local` — +// and ruling the cache out is `local`'s entire purpose. + +TEST(BuildProfile, CacheModeOverrideBeatsManifest) { + auto m = base(); + m.buildConfig.cacheMode = "local"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, "global"), + mcpp::build::CacheMode::Global); +} + +TEST(BuildProfile, CacheModeManifestBeatsDefault) { + auto m = base(); + m.buildConfig.cacheMode = "off"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, ""), mcpp::build::CacheMode::Off); +} + +TEST(BuildProfile, CacheModeDefaultsToGlobal) { + EXPECT_EQ(mcpp::build::resolve_cache_mode(base(), ""), + mcpp::build::CacheMode::Global); +} + +// An unparseable value must not silently mean "global" at THIS level either: it +// falls through to the next source, and prepare_build reports it separately. +TEST(BuildProfile, UnknownManifestCacheModeFallsThroughToDefault) { + auto m = base(); + m.buildConfig.cacheMode = "bogus"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, ""), + mcpp::build::CacheMode::Global); + // ...and an unparseable override does not shadow a valid manifest value. + m.buildConfig.cacheMode = "local"; + EXPECT_EQ(mcpp::build::resolve_cache_mode(m, "bogus"), + mcpp::build::CacheMode::Local); +} diff --git a/tests/unit/test_cache_key.cpp b/tests/unit/test_cache_key.cpp new file mode 100644 index 00000000..529732a1 --- /dev/null +++ b/tests/unit/test_cache_key.cpp @@ -0,0 +1,265 @@ +#include + +import std; +import mcpp.build.cache_key; +import mcpp.libs.json; +import mcpp.manifest; +import mcpp.modgraph.scanner; +import mcpp.toolchain.detect; + +namespace ck = mcpp::build::cache_key; + +namespace { + +ck::BuildAxes axes() { + ck::BuildAxes b; + b.compilerId = "gcc"; + b.compilerVersion = "16.1.0"; + b.driverIdentity = "0123456789abcdef"; + b.targetTriple = "x86_64-linux-gnu"; + b.stdlibId = "libstdc++"; + b.stdlibVersion = "16.1.0"; + b.cppStandard = "c++23"; + b.cppStandardFlag = "-std=c++23"; + b.cStandard = "c11"; + b.optLevel = "2"; + b.debug = false; + return b; +} + +ck::PackageAxes pkg() { + ck::PackageAxes p; + p.indexName = "compat"; + p.packageName = "compat.zlib"; + p.version = "1.3.2"; + p.cflags = {"-D_GNU_SOURCE"}; + p.sources = {"zlib-1.3.2/adler32.c", "zlib-1.3.2/deflate.c"}; + return p; +} + +mcpp::modgraph::PackageRoot rootAt(const std::filesystem::path& at) { + mcpp::modgraph::PackageRoot r; + r.root = at; + r.manifest.package.name = "zlib"; + r.manifest.package.namespace_ = "compat"; + r.manifest.package.version = "1.3.2"; + return r; +} + +} // namespace + +// The defect this key exists to fix: the consumer's identity used to be folded +// into every dependency's cache path (the whole-project fingerprint serializes +// every package in the graph, including the root's name and version), so +// `mcpp version bump` invalidated the whole cache and two projects never shared +// an entry. Axes A–F contain nothing about the consumer, so there is no input +// this test could vary to demonstrate the old behaviour — the closest positive +// statement is that a dependency's key is a pure function of its own axes. +TEST(CacheKey, IsStableAcrossCallsForTheSameInputs) { + EXPECT_EQ(ck::key_hex(axes(), pkg()), ck::key_hex(axes(), pkg())); +} + +TEST(CacheKey, KeyIsSixteenHexChars) { + auto k = ck::key_hex(axes(), pkg()); + EXPECT_EQ(k.size(), 16u); + for (char c : k) + EXPECT_TRUE(std::isxdigit(static_cast(c))) << k; +} + +// ── C axis: the profile. This is the one that was missing entirely, and the +// reason a --release build could be served -O0 -g dependency objects. ──────── +TEST(CacheKey, ProfileOptLevelChangesTheKey) { + auto a = axes(); + auto b = axes(); b.optLevel = "0"; + EXPECT_NE(ck::key_hex(a, pkg()), ck::key_hex(b, pkg())); +} + +TEST(CacheKey, ProfileDebugLtoStripEachChangeTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto b = axes(); b.debug = true; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.lto = true; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.strip = true; EXPECT_NE(ck::key_hex(b, pkg()), base); } +} + +// ── A axis: toolchain identity ─────────────────────────────────────────────── +TEST(CacheKey, ToolchainIdentityChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto b = axes(); b.compilerId = "clang"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.compilerVersion = "15.1.0"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.targetTriple = "x86_64-linux-musl"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.stdlibId = "libc++"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + // Two payloads that both call themselves "gcc 16.1.0" but are not the same + // build must not share an entry. + { auto b = axes(); b.driverIdentity = "ffffffffffffffff"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } +} + +// ── B axis: language and dialect ───────────────────────────────────────────── +TEST(CacheKey, LanguageAndDialectChangeTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto b = axes(); b.cppStandard = "c++26"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.cppStandardFlag = "-std=c++2c"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.dialectFlags = {"-freflection"}; + EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.cStandard = "c17"; EXPECT_NE(ck::key_hex(b, pkg()), base); } + { auto b = axes(); b.macosDeploymentTarget = "14.0"; + EXPECT_NE(ck::key_hex(b, pkg()), base); } +} + +// ── D axis: package identity ───────────────────────────────────────────────── +TEST(CacheKey, PackageIdentityChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto p = pkg(); p.version = "1.3.1"; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.packageName = "compat.zstd"; + EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.indexName = "other"; EXPECT_NE(ck::key_hex(axes(), p), base); } +} + +// ── E axis: the package's own config ───────────────────────────────────────── +TEST(CacheKey, OwnBuildConfigChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + { auto p = pkg(); p.cflags.push_back("-DEXTRA"); EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.cxxflags = {"-fno-rtti"}; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.defines = {"FOO=1"}; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.globFlags = {"glob:*.c"}; EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.generatedFiles = {"cfg.h=1"};EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.includeDirs = {"pub:/x"}; + EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.sources.push_back("zlib-1.3.2/gzread.c"); + EXPECT_NE(ck::key_hex(axes(), p), base); } + { auto p = pkg(); p.features = {"main"}; EXPECT_NE(ck::key_hex(axes(), p), base); } +} + +// ── F axis: upstream keys (Merkle) ─────────────────────────────────────────── +// A dependency's artifacts are bound to the exact upstream artifacts they were +// built against — GCC embeds a CRC of an imported module's BMI into the +// importer's BMI, so a stale importer BMI fails with `module 'X' CRC mismatch`. +// The key therefore folds in the upstream's key, not a summary of its interface. +TEST(CacheKey, UpstreamKeyChangesTheKey) { + auto base = ck::key_hex(axes(), pkg()); + auto p = pkg(); + p.upstreamKeys = {"1111111111111111"}; + auto withUp = ck::key_hex(axes(), p); + EXPECT_NE(withUp, base); + + p.upstreamKeys = {"2222222222222222"}; + EXPECT_NE(ck::key_hex(axes(), p), withUp); +} + +TEST(CacheKey, UpstreamOrderDoesNotMatterWhenSorted) { + auto p1 = pkg(); p1.upstreamKeys = {"1111111111111111", "2222222222222222"}; + auto p2 = pkg(); p2.upstreamKeys = {"1111111111111111", "2222222222222222"}; + EXPECT_EQ(ck::key_hex(axes(), p1), ck::key_hex(axes(), p2)); +} + +// Length-prefixed field joining: without it, ("a","bc") and ("ab","c") collide, +// and for a cache key a collision means serving one package's objects for +// another's. +TEST(CacheKey, AdjacentFieldsCannotCollide) { + auto p1 = pkg(); p1.packageName = "a"; p1.version = "bc"; + auto p2 = pkg(); p2.packageName = "ab"; p2.version = "c"; + EXPECT_NE(ck::key_hex(axes(), p1), ck::key_hex(axes(), p2)); +} + +TEST(CacheKey, ListBoundariesCannotCollide) { + auto p1 = pkg(); p1.cflags = {"-DA", "-DB"}; + auto p2 = pkg(); p2.cflags = {"-DA-DB"}; + EXPECT_NE(ck::key_hex(axes(), p1), ck::key_hex(axes(), p2)); +} + +// entry.json carries these inputs verbatim so a hit can be validated field by +// field. Anything in the key must therefore also be in the JSON, or validation +// would pass on an input the key distinguishes. +TEST(CacheKey, JsonCarriesEveryAxis) { + auto j = ck::to_json(axes(), pkg()); + EXPECT_EQ(j["epoch"], ck::kCacheEpoch); + for (auto k : {"toolchain", "language", "profile", "package", "config", "upstream"}) + EXPECT_TRUE(j.contains(k)) << k; + EXPECT_EQ(j["toolchain"]["compiler"], "gcc"); + EXPECT_EQ(j["profile"]["opt_level"], "2"); + EXPECT_EQ(j["package"]["version"], "1.3.2"); + for (auto k : {"features", "cflags", "cxxflags", "ldflags", "defines", + "glob_flags", "generated_files", "include_dirs", + "source_globs", "sources"}) + EXPECT_TRUE(j["config"].contains(k)) << k; +} + +TEST(CacheKey, JsonDiffersWheneverTheKeyDiffers) { + auto b = axes(); b.optLevel = "0"; + EXPECT_NE(ck::to_json(axes(), pkg()), ck::to_json(b, pkg())); +} + +// Payload include dirs are absolute paths under /registry/data/xpkgs. +// Left absolute, every entry would miss after a home relocation or on another +// machine — which is most of the point of a cross-project cache. +TEST(CacheKey, IncludeDirsAreRelativizedAgainstTheStoreRoot) { + auto makeKeyFor = [](const std::filesystem::path& store) { + auto pkgRoot = rootAt(store / "compat-x-compat.zlib" / "1.3.2"); + pkgRoot.manifest.buildConfig.includeDirs = { + store / "compat-x-compat.zlib" / "1.3.2" / "include", + }; + ck::PackageAxes p; + p.indexName = "compat"; p.packageName = "compat.zlib"; p.version = "1.3.2"; + ck::fill_package_config(p, pkgRoot, store); + return std::pair{ck::key_hex(axes(), p), p.includeDirs}; + }; + auto [k1, dirs1] = makeKeyFor("/home/alice/.mcpp/registry/data/xpkgs"); + auto [k2, dirs2] = makeKeyFor("/opt/ci/mcpp/registry/data/xpkgs"); + EXPECT_EQ(k1, k2); + ASSERT_FALSE(dirs1.empty()); + EXPECT_NE(dirs1.front().find(""), std::string::npos) << dirs1.front(); + EXPECT_EQ(dirs1, dirs2); +} + +// A path that is under the package root but not under the store (a generated +// directory, say) gets a package-relative placeholder for the same reason. +TEST(CacheKey, PackageRootRelativeDirsAreAlsoRelativized) { + std::filesystem::path store = "/home/u/.mcpp/registry/data/xpkgs"; + auto pkgRoot = rootAt("/somewhere/else/zlib"); + pkgRoot.manifest.buildConfig.includeDirs = {"/somewhere/else/zlib/generated"}; + ck::PackageAxes p; + ck::fill_package_config(p, pkgRoot, store); + ASSERT_FALSE(p.includeDirs.empty()); + EXPECT_NE(p.includeDirs.front().find(""), std::string::npos) + << p.includeDirs.front(); +} + +TEST(CacheKey, FillPackageConfigCarriesFlagsAndGeneratedFiles) { + std::filesystem::path store = "/home/u/.mcpp/registry/data/xpkgs"; + auto pkgRoot = rootAt(store / "compat-x-compat.zlib" / "1.3.2"); + pkgRoot.manifest.buildConfig.cflags = {"-D_GNU_SOURCE"}; + pkgRoot.manifest.buildConfig.cxxflags = {"-fno-exceptions"}; + pkgRoot.manifest.buildConfig.defines = {"ZLIB_CONST"}; + pkgRoot.manifest.buildConfig.cStandard = "c11"; + pkgRoot.manifest.buildConfig.generatedFiles = {{"cfg.h", "#define A 1"}}; + + ck::PackageAxes p; + ck::fill_package_config(p, pkgRoot, store); + EXPECT_NE(std::ranges::find(p.cflags, "-D_GNU_SOURCE"), p.cflags.end()); + EXPECT_NE(std::ranges::find(p.cxxflags, "-fno-exceptions"), p.cxxflags.end()); + EXPECT_NE(std::ranges::find(p.defines, "ZLIB_CONST"), p.defines.end()); + ASSERT_EQ(p.generatedFiles.size(), 1u); + EXPECT_EQ(p.generatedFiles.front(), "cfg.h=#define A 1"); + // A package's own C standard reaches its own C units, so it must be in the + // key even though the whole-graph C standard is on the B axis. + bool sawCStd = false; + for (auto& f : p.cflags) if (f.find("c_standard=c11") != std::string::npos) sawCStd = true; + EXPECT_TRUE(sawCStd); +} + +// Generated files are a map; iteration order must not leak into the key. +TEST(CacheKey, GeneratedFilesAreOrderIndependent) { + std::filesystem::path store = "/home/u/.mcpp/registry/data/xpkgs"; + auto build = [&](std::vector> entries) { + auto pkgRoot = rootAt(store / "p" / "1"); + for (auto& [k, v] : entries) + pkgRoot.manifest.buildConfig.generatedFiles[k] = v; + ck::PackageAxes p; + ck::fill_package_config(p, pkgRoot, store); + return ck::key_hex(axes(), p); + }; + EXPECT_EQ(build({{"a.h", "1"}, {"b.h", "2"}}), + build({{"b.h", "2"}, {"a.h", "1"}})); +} diff --git a/tests/unit/test_home.cpp b/tests/unit/test_home.cpp index 5bab58d5..7a2cd029 100644 --- a/tests/unit/test_home.cpp +++ b/tests/unit/test_home.cpp @@ -43,13 +43,30 @@ class ScopedEnv { TEST(Home, ExplicitMcppHomeWins) { ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test"); EXPECT_EQ(mcpp::home::root(), std::filesystem::path("/tmp/mcpp-home-test")); - EXPECT_EQ(mcpp::home::bmi_root(), - std::filesystem::path("/tmp/mcpp-home-test") / "bmi"); + EXPECT_EQ(mcpp::home::cache_root(), + std::filesystem::path("/tmp/mcpp-home-test") / "build-cache" / "v1"); } -TEST(Home, BmiRootIsAlwaysRootSlashBmi) { +TEST(Home, CacheRootIsAlwaysRootSlashBuildCacheV1) { ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test-2"); - EXPECT_EQ(mcpp::home::bmi_root(), mcpp::home::root() / "bmi"); + EXPECT_EQ(mcpp::home::cache_root(), mcpp::home::root() / "build-cache" / "v1"); +} + +// The layout version has to be a path segment, not an implicit convention: +// replacing the layout must never require migrating or deleting the old tree. +TEST(Home, CacheRootCarriesTheLayoutVersion) { + ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test-2b"); + EXPECT_EQ(mcpp::home::cache_root().filename(), std::filesystem::path("v1")); + EXPECT_EQ(mcpp::home::cache_root().parent_path().filename(), + std::filesystem::path("build-cache")); + // Must NOT nest inside /cache: GlobalConfig's index metadata cache + // owns that name and its reset path removes the whole directory. + EXPECT_NE(mcpp::home::cache_root().parent_path().filename(), + std::filesystem::path("cache")); + // The pre-v1 root must stay resolvable and must be a DIFFERENT tree, so + // `doctor` can report it and `cache clean --legacy` can reclaim it. + EXPECT_EQ(mcpp::home::legacy_bmi_root(), mcpp::home::root() / "bmi"); + EXPECT_NE(mcpp::home::legacy_bmi_root(), mcpp::home::cache_root()); } // The user-home fallback must land on `.mcpp`, never on the legacy flat @@ -63,21 +80,21 @@ TEST(Home, FallbackUsesDotMcppNotLegacyFlatDir) { #endif auto root = mcpp::home::root(); EXPECT_EQ(root.filename(), std::filesystem::path(".mcpp")); - EXPECT_EQ(mcpp::home::bmi_root().filename(), std::filesystem::path("bmi")); - EXPECT_NE(mcpp::home::bmi_root().filename(), std::filesystem::path(".mcpp-bmi")); + EXPECT_NE(mcpp::home::cache_root().filename(), + std::filesystem::path(".mcpp-bmi")); } -// #311's D2 regression gate: the std module cache and the dep BMI cache must +// #311's D2 regression gate: the std module cache and the dep cache must // resolve to ONE root. `default_cache_root()` used to be a private copy of the // home resolution that drifted (no USERPROFILE, no self-contained detection), // so on Windows the std BMI landed in the current working directory. -TEST(Home, StdModuleCacheRootEqualsHomeBmiRoot) { +TEST(Home, StdModuleCacheRootEqualsHomeCacheRoot) { { ScopedEnv home("MCPP_HOME", "/tmp/mcpp-home-test-3"); - EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::bmi_root()); + EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::cache_root()); } { ScopedEnv mcppHome("MCPP_HOME", nullptr); - EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::bmi_root()); + EXPECT_EQ(mcpp::toolchain::default_cache_root(), mcpp::home::cache_root()); } } diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 2649cac8..7afea22d 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -926,3 +926,220 @@ TEST(NinjaBackend, FilterKeepsStagingDiagnosticsAndFailedTarget) { EXPECT_EQ(filtered.find("stage --output"), std::string::npos) << filtered; EXPECT_EQ(filtered.find("ninja: Entering directory"), std::string::npos) << filtered; } + +// ── Cache-served units: stage edges, not compile edges ────────────────────── +// +// The global dependency cache used to copy artifacts into the build dir from +// inside prepare_build, while those same paths stayed declared as the outputs of +// compile edges. Ninja treats an output it has no command line for in +// .ninja_log as dirty ("command line not found in log"), so on a fresh build dir +// every "cached" unit was recompiled — the cache saved nothing and the CLI still +// printed "Cached". Staging has to be an edge for ninja to have a record. + +TEST(NinjaBackend, CachedUnitEmitsStageEdgeAndNoCompileEdge) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.cppm", + .object = "obj/dep.m.o", + .packageName = "dep", + .providesModule = "dep", + .servedFromCache = true, + .cachedObject = "/bc/v1/pkg/idx/dep@1.0.0/key/obj/dep.m.o", + .cachedBmi = "/bc/v1/pkg/idx/dep@1.0.0/key/bmi/dep.gcm", + }); + + auto ninja = emit_ninja_string(plan); + + EXPECT_NE(ninja.find("build obj/dep.m.o : stage_file " + "/bc/v1/pkg/idx/dep@1.0.0/key/obj/dep.m.o"), + std::string::npos) << ninja; + // The BMI is staged too, at the path a compile edge would have produced. + EXPECT_NE(ninja.find("stage_file /bc/v1/pkg/idx/dep@1.0.0/key/bmi/dep.gcm"), + std::string::npos) << ninja; + // No compile edge, and no scan edge, for this unit. + EXPECT_EQ(ninja.find("cxx_module /store/dep/src/dep.cppm"), std::string::npos) + << ninja; + EXPECT_EQ(ninja.find("cxx_object /store/dep/src/dep.cppm"), std::string::npos) + << ninja; + EXPECT_EQ(ninja.find("dep.cppm.ddi"), std::string::npos) << ninja; +} + +// `--verify size` for the same reason the std artifacts use it: the entry +// directory is named by a key covering the toolchain, dialect, profile, the +// package's own config and its dependencies' keys, so under one key equal size +// IS equivalence — and size comes from directory metadata, which stays readable +// even when a holder denies opening the file. +TEST(NinjaBackend, CachedStageEdgesUseSizeVerification) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/a.c", + .object = "obj/a.o", + .packageName = "dep", + .servedFromCache = true, + .cachedObject = "/cache/obj/a.o", + }); + + auto ninja = emit_ninja_string(plan); + auto at = ninja.find("build obj/a.o : stage_file /cache/obj/a.o"); + ASSERT_NE(at, std::string::npos) << ninja; + EXPECT_NE(ninja.find("verify = --verify size", at), std::string::npos) << ninja; +} + +// A cached unit sits next to uncached ones; only the cached one changes shape. +TEST(NinjaBackend, UncachedUnitsStillCompileAlongsideCachedOnes) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.c", + .object = "obj/dep.o", + .packageName = "dep", + .servedFromCache = true, + .cachedObject = "/cache/obj/dep.o", + }); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + + auto ninja = emit_ninja_string(plan); + EXPECT_NE(ninja.find("build obj/dep.o : stage_file /cache/obj/dep.o"), + std::string::npos) << ninja; + EXPECT_NE(ninja.find("build obj/main.o : cxx_object src/main.cpp"), + std::string::npos) << ninja; +} + +// The default path must be byte-identical to what it was: `servedFromCache` is +// false everywhere unless a cache hit set it. +TEST(NinjaBackend, NoStageEdgesWithoutCachedUnits) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + + auto ninja = emit_ninja_string(plan); + EXPECT_EQ(ninja.find("build obj/main.o : stage_file"), std::string::npos) << ninja; + EXPECT_NE(ninja.find("build obj/main.o : cxx_object src/main.cpp"), + std::string::npos) << ninja; +} + +// A cached unit whose cachedObject was never filled in is a bug upstream, not a +// reason to emit a stage edge with an empty input — it must fall back to being +// compiled rather than producing an unbuildable graph. +TEST(NinjaBackend, CachedUnitWithoutCachedObjectPathIsStillCompiled) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + .servedFromCache = true, + }); + + auto ninja = emit_ninja_string(plan); + EXPECT_EQ(ninja.find("stage_file \n"), std::string::npos) << ninja; + EXPECT_EQ(ninja.find("build obj/main.o : stage_file"), std::string::npos) << ninja; +} + +// A cache-served unit keeps its compile_commands.json entry. The units stay in +// the plan (only their EDGE changes shape) precisely so clangd does not lose the +// dependency's sources — a cache that silently degraded IDE navigation would be +// a bad trade for build time. +TEST(NinjaBackend, CachedUnitsStillAppearInCompileCommands) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.c", + .object = "obj/dep.o", + .packageName = "dep", + .servedFromCache = true, + .cachedObject = "/bc/v1/pkg/idx/dep@1.0.0/key/obj/dep.o", + }); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + + auto flags = compute_flags(plan); + auto cdb = emit_compile_commands(plan, flags); + + EXPECT_NE(cdb.find("/store/dep/src/dep.c"), std::string::npos) << cdb; + EXPECT_NE(cdb.find("src/main.cpp"), std::string::npos) << cdb; +} + +// Replacing a package's compile edges with stage edges also removes the ordering +// those compile edges carried. A module PARTITION is the case that breaks: a +// consumer imports `pkg`, so its dyndep declares `pkg`'s BMI and nothing else — +// the partition BMI `pkg:part` was reached only because `pkg`'s own compile edge +// depended on it. With independent stage edges ninja may start the consumer while +// the partition is still unstaged, and Clang fails with `failed to find module +// file for module 'pkg:part'`. Observed on macOS CI while Linux won the race, +// which is why it is pinned here rather than left to scheduling. +TEST(NinjaBackend, NonCachedEdgesOrderAfterEveryStagedArtifact) { + auto plan = minimal_plan(); + // A cached package with a primary interface and a partition. + plan.compileUnits.push_back({ + .source = "/store/dep/src/dep.cppm", + .object = "obj/dep.m.o", + .packageName = "dep", + .providesModule = "dep", + .servedFromCache = true, + .cachedObject = "/bc/obj/dep.m.o", + .cachedBmi = "/bc/bmi/dep.gcm", + }); + plan.compileUnits.push_back({ + .source = "/store/dep/src/part.cppm", + .object = "obj/part.m.o", + .packageName = "dep", + .providesModule = "dep:part", + .servedFromCache = true, + .cachedObject = "/bc/obj/part.m.o", + .cachedBmi = "/bc/bmi/dep-part.gcm", + }); + // The consumer, which imports only the primary module. + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + .imports = {"dep"}, + }); + + auto ninja = emit_ninja_string(plan); + + // One phony aggregating every staged artifact — a per-edge list would repeat + // it and grow ninja lines without bound (mcpp#274). + auto phony = ninja.find("build _mcpp_staged_cache : phony"); + ASSERT_NE(phony, std::string::npos) << ninja; + auto phonyLine = ninja.substr(phony, ninja.find('\n', phony) - phony); + for (auto* art : {"obj/dep.m.o", "obj/part.m.o", + "gcm.cache/dep.gcm", "gcm.cache/dep-part.gcm"}) { + EXPECT_NE(phonyLine.find(art), std::string::npos) + << art << " missing from: " << phonyLine; + } + + // The consumer's edge orders after it... + auto consumer = ninja.find("build obj/main.o"); + ASSERT_NE(consumer, std::string::npos) << ninja; + auto consumerLine = ninja.substr(consumer, ninja.find('\n', consumer) - consumer); + EXPECT_NE(consumerLine.find("|| _mcpp_staged_cache"), std::string::npos) + << consumerLine; + + // ...as an ORDER-ONLY dependency, not an implicit one: the real content + // dependency is still declared where it always was, so this must add + // sequencing without making a staged BMI dirty its consumers. + auto bar = consumerLine.find("|| _mcpp_staged_cache"); + EXPECT_EQ(consumerLine.find("| _mcpp_staged_cache"), bar + 1) + << "staged phony must appear only after ||, never as an implicit input: " + << consumerLine; +} + +TEST(NinjaBackend, NoStagedPhonyWhenNothingIsCached) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "objc_rule_test", + }); + auto ninja = emit_ninja_string(plan); + EXPECT_EQ(ninja.find("_mcpp_staged_cache"), std::string::npos) << ninja; +}