perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]

[robert3005]

Rationale for this change

Stacked on #8964 — review that first.

A nested builder learns about validity from two sources: one row at a time as scalars are appended, and a whole array's worth at a time as arrays are. Only the first needs a null buffer. LazyBitBufferBuilder treated both the same, so every appended array paid

self.nulls.append_validity_mask(&array.validity()?.execute_mask(array.len(), ctx)?);

— executing the array's validity into a Mask and copying its bits into a running buffer.

StructArray::try_concat has never done that. It hands its per-chunk validities to Validity::concat, which returns AllValid/AllInvalid/NonNullable when they are uniform and a Validity::Array(ChunkedArray<Bool>) otherwise — no bitmap, no execution. That gap is the one thing standing between the builders and the swizzle helpers they are meant to replace: routing chunked struct canonicalization through StructBuilder today would trade a lazy validity for a materialized one on every nullable chunked struct, which is the kind of regression that shows up in a scan rather than in a test.

What changes are included in this PR?

ValidityBuilder (vortex-array/src/builders/validity.rs) mirrors ChildBuilder: appended validity is kept as a run, bits appended one at a time go into a LazyBitBufferBuilder, and finish flushes the buffer into the run list and concatenates. StructBuilder, ListBuilder, ListViewBuilder and FixedSizeListBuilder hold their validity this way; the leaf builders keep LazyBitBufferBuilder unchanged.

The call sites lose their execute_mask:

self.nulls.append_validity(array.validity()?, array.len());

append_validity needs neither the ExecutionCtx nor a VortexResult any more, since nothing is executed on this path.

Two details worth calling out for review:

• However few values a run covers, it is kept as it arrived, so a builder's validity splits on exactly the boundaries its children do (feat(array): builders no longer canonicalize their children [builders-child-stack] #8964). Uniform runs still collapse, so a builder fed an array at a time does not pay a bool array for validity it never had.
• Validity::concat treats NonNullable and AllValid as different kinds and falls back to a bool array when both appear — which happens as soon as a non-nullable array is appended next to a scalar. Both mean "no nulls", so finish_with_nullability checks definitely_no_nulls across the runs first and answers from the declared nullability instead. Without that, appending an array and then a scalar to a non-nullable builder produced a Validity::Array and tripped the nullability assertion.

Seven unit tests on ValidityBuilder (runs preserved, uniform runs collapsing, a one-row validity earning a run too, bits and runs interleaving in order, set_validity replacing runs, non-nullable finishing non-nullable, finish resetting), plus a builder-level test asserting that two appended nullable struct arrays come back with a chunked bool validity rather than one copied buffer.

What APIs are changed? Are there any user-facing changes?

None — ValidityBuilder is pub(crate). The observable change is representational: a nested builder that consumed nullable arrays now returns Validity::Array(ChunkedArray<Bool>) where it previously returned a single BoolArray. That is the same shape StructArray::try_concat has always produced.

Checks

• cargo nextest run --workspace — 7168 passed, 560 skipped
• cargo clippy --all-targets --all-features — clean
• cargo +nightly fmt --all --check — clean

---

Generated by Claude Code

Stacked PR Chain: builders-child-stack

PR	Title	Merges Into
#9046	perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack]	N/A
#8964	feat(array): builders no longer canonicalize their children [builders-child-stack]	#9046
#8965	feat(array): let callers choose a nested builder's chunk threshold [builders-child-stack]	#8964
#8966	perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]	#8965
#8967	refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]	#8966

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will not alter performance

✅ 1842 untouched benchmarks
⏩ 55 skipped benchmarks¹

---

_{Comparing claude/builders-lazy-validity-9ze0t6 (dd4420b) with claude/builders-canonical-children-9ze0t6 (c9bacae)}

Footnotes

1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

Do we still need LazyBitBufferBuilder?

[robert3005]

On LazyBitBufferBuilder: yes, it is still carrying its weight, in two roles.

1. It is ValidityBuilder::pending — the bits that arrive one row at a time from append_scalar / append_n_nulls. Its laziness (no buffer at all until the first null) is exactly what makes an all-valid builder cost nothing, so ValidityBuilder would have to reimplement it to drop it.
2. The flat builders use it directly: BoolBuilder, PrimitiveBuilder, DecimalBuilder, VarBinViewBuilder, plus varbinview::compute::zip. Those need bit-level operations a run cannot express — set_bit at an arbitrary index (PrimitiveBuilder's uninit ranges) and append_buffer.

What this PR does remove is the nested builders' use of it: ListBuilder, ListViewBuilder, StructBuilder and FixedSizeListBuilder all hold a ValidityBuilder now, and nothing outside ValidityBuilder builds a bit buffer for a nested builder.

I did not migrate the flat builders on purpose: a flat builder copies its values into one contiguous buffer, so there is no chunk boundary for validity runs to line up with, and keeping runs there would hand downstream a chunked bool array for an otherwise flat array. Say the word if you would rather have one accumulator everywhere and I will do it.