fix(batch): enforce indexed byte and chunk budgets

[Tomodad]

Publication status: Ready for Review; coordinator verification and all cross-platform CI checks passed.

Summary

Fixes #961 by bounding ctx_batch_execute indexed ingestion independently from its Markdown display response.

• enforce UTF-8 body-byte budgets per command and per batch
• enforce a true generated-chunk cap
• require explicit opt-in for limits above the defaults
• index stdout/stderr as structured plain chunks instead of indexing the assembled display Markdown
• report complete, partial, and not-indexed decisions without silent truncation
• expose captured/indexed/dropped bytes and generated/indexed/dropped chunk counts
• bound the Indexed Sections response inventory to 50 metadata-only rows without rehydrating stored chunk bodies

Default policy

max_bytes_per_command    = 8 MiB
max_total_indexed_bytes  = 16 MiB
max_generated_chunks     = 4,608
max_chunk_bytes          = 4 KiB
inventory display rows   = 50

Limits are inclusive. Byte budgets count UTF-8 stdout+stderr body bytes after execution capture and before any display-Markdown join or FTS write. Requested increases above any default require allow_large_ingestion=true.

The 8/16 MiB and 4,608-chunk defaults are benchmark-derived, remain configurable, and can be revisited with cross-platform production evidence.

Coordinator review fixes

This revision addresses all three blockers recorded in stage-1-coordinator-verification.md:

1. Replaces getChunksBySource() response inventory hydration with formatBatchSectionInventory(plan.chunks, totalChunks) and caps display at 50 rows with an explicit omitted count.
2. Adds tests/batch-ingestion-handler.test.ts, which invokes the registered real MCP ctx_batch_execute handler and checks the bounded response behavior.
3. Makes max_generated_chunks a true generation cap; public generated_chunks never exceeds the configured maximum.

The benchmark now measures the inventory construction included in the production handler path.

Display/index separation and compatibility

• MCP content remains Markdown and remains present.
• Commands, status, duration, labels, budget summary, bounded section inventory, and query results remain visible.
• Indexed chunks retain command label, command text, exit status, duration, timeout state, stream, part number, and body.
• stdout and stderr remain distinct and command/source boundaries remain traceable.
• runBatchCommands.outputs remains enabled by default for existing callers.
• runBatchCommands.commands, structuredContent.indexing, and ContentStore.indexPreparedChunks() are additive.
• The batch handler alone sets includeDisplayOutputs: false to avoid a second batch-sized display/index join.
• No existing input field was removed or retyped.

No silent truncation

Markdown and structured metrics expose:

• complete / partially indexed / not indexed
• captured / indexed / dropped body bytes
• metadata-inclusive stored bytes
• generated / indexed / dropped / partial chunks
• triggered budget names
• opt-in requested/applied state
• per-command exit, duration, timeout, status, bytes, chunks, and triggers

A partial payload is never labeled complete.

TDD evidence

Corrective RED:

Test Files  2 failed (2)
Tests       3 failed | 17 passed (20)

expected generated_chunks 2, received 11
expected generated_chunks 1, received 2
real handler response did not contain the bounded 50-row/omitted inventory marker

Evidence:

C:\Users\Zhuyixiang\.codex\visualizations\2026\07\16\019f6b86-f029-7950-a7e7-ec7cbe78675f\stage-1-coordinator-red.txt

Corrective GREEN, rerun after rebasing onto refreshed origin/next:

node_modules\.bin\vitest.cmd run tests\batch-ingestion-budgets.test.ts tests\batch-ingestion-handler.test.ts --maxWorkers=1 --no-file-parallelism
# 2 files passed, 20 tests passed

Related regression validation:

runBatchCommands server tests: 24 passed, 475 skipped
ctx_batch_execute server tests: 20 passed, 479 skipped
store-bytecap + store: 127 passed, 1 skipped
complete server suite: 499 passed
npm.cmd run typecheck: passed
npm.cmd run build: passed
npm.cmd run assert-bundle: passed for all six bundles
npm.cmd run assert-asymmetric-drift: passed
git diff --check origin/next...HEAD: passed

All Vitest runs in the resumed session used --maxWorkers=1 --no-file-parallelism. The default all-files parallel suite was deliberately not rerun after two confirmed Windows Wof.sys kernel crashes during broad concurrent filesystem enumeration. Directly affected and complete server/store suites passed under serialized execution.

Benchmark

Command:

node --expose-gc --import tsx tests\batch-ingestion-benchmark.ts --repeats 3

Environment: Node v24.13.0, Windows x64, three repeats per scenario/mode.

Raw result:

C:\Users\Zhuyixiang\.codex\visualizations\2026\07\16\019f6b86-f029-7950-a7e7-ec7cbe78675f\stage-1-benchmark-corrective.json

Selected default-policy medians:

• small batch: 3.811 ms; 131,072 indexed bytes; 34 chunks; inventory 34 rows / 1,048 bytes
• 8 MiB minus 1 byte: 190.051 ms; all 8,388,607 bytes indexed; largest intermediate 4,096 bytes; inventory 50 rows / 1,407 bytes
• 9 MiB command: 190.484 ms; 8 MiB indexed and 1 MiB dropped; inventory 50 rows / 1,407 bytes
• exact 16 MiB batch: 410.306 ms; all 16,777,216 bytes indexed in 4,158 chunks; inventory 50 rows / 1,857 bytes
• 18 MiB batch: 403.724 ms; 16 MiB indexed and 2 MiB dropped; generated chunks capped at 4,608
• 5,000-section explosion: 68.838 ms; generated/indexed 4,608; dropped 392; inventory 50 rows / 1,506 bytes instead of 5,000 rows / 98,910 bytes

The coordinator's pre-fix default-policy reproduction returned 4,156 inventory lines and about 144 KiB. The corrected handler returns at most 50 rows and an explicit omitted count.

Orthogonality

PR #898 remains open and unmerged. It changes ContentStore.#chunkMarkdown() plus tests/store-bytecap.test.ts and generated bundles to enforce a per-row Markdown cap.

This PR does not modify #chunkMarkdown() and does not depend on #898. Its src/store.ts changes add prepared-chunk ingestion around indexPreparedChunks() and iterable vocabulary extraction. File overlap is limited to src/store.ts and generated bundles; source hunks are separate. If both land, retain both behaviors and regenerate bundles through the repository build.

PR #963 remains open and unmerged. It changes bounded search-result hydration. This PR changes ingestion. Shared src/store.ts/bundle overlap is semantically orthogonal.

Files

• src/batch-ingestion.ts
• src/server.ts
• src/store.ts
• tests/batch-ingestion-budgets.test.ts
• tests/batch-ingestion-handler.test.ts
• tests/batch-ingestion-benchmark.ts
• server.bundle.mjs
• cli.bundle.mjs

Commits

598df57ff8c2f6adb87078b69abccc486c00c2e6 fix(batch): enforce indexed byte and chunk budgets
3e21a12bd7baccf4fb0cb4a9070f50ea89489235 fix(batch): bound section inventory and generated chunks

Base: origin/next@e1d9448.

Checklist

• Real RED before corrective production changes
• Acceptance and real handler integration tests
• Related server and store regressions
• Typecheck
• Build and generated bundle assertions
• Reproducible benchmark with three repeats
• git diff --check
• Worktree clean
• fix(store): cap oversized markdown chunks #898/fix(store): bound FTS search result hydration for oversized rows #963 orthogonality audited
• Coordinator re-verification
• User approval to push/create Draft PR
• Cross-platform CI after authorized publication
• User approval to mark Ready for Review