Zstd: append values in bulk and harden the frame metadata reads - #9099
Zstd: append values in bulk and harden the frame metadata reads#9099robert3005 wants to merge 1 commit into
Conversation
| row = valid_index + 1; | ||
| // Null rows carry an empty view, so scatter the stored values into their rows. | ||
| let mut views = BufferMut::<BinaryView>::zeroed(slice.n_rows); | ||
| for (view, index) in valid_views.into_iter().zip_eq(valid_indices) { |
There was a problem hiding this comment.
we should use for_each_valid_index if possible
There was a problem hiding this comment.
Done. The scatter now matches on mask.bit_buffer() and only the AllOr::Some arm walks bits, via for_each_set_index with a running valid-row counter. That drops the mask.indices() materialization — the views were its only consumer — and the zip_eq with it. The all-false case is an early mask.all_false() return instead of an indices() match arm.
There was a problem hiding this comment.
this shouldn't be necessary
There was a problem hiding this comment.
Removed. Right — after match_each_varbin_builder! (all four widths) and VarBinViewBuilder, there is nothing left for a Utf8/Binary dtype, and canonicalizing would only decompress to a VarBinView for VarBinView::append_to_builder to reject the same remainder. It is now a vortex_bail!, matching what VarBinView already does, with a comment saying why there is no fallback.
Applied the same removal to FSST (#9097) and OnPair (#9100), which had the identical block. Dict keeps its fallback since its dtype is not restricted to varbin.
`append_to_builder` walked the decompressed frames one value at a time, appending each through a single-value call and interleaving null runs by hand. It now derives the value byte range from the slice's own metadata and hands the whole region to `append_value_slices`, which sizes the offsets, byte storage and validity once — one offset store plus one `memcpy` per value. Adds a `VarBinViewBuilder` path as well: the frames already hold the values contiguously, so the views can reference them in place instead of going through the canonical array and rewriting every view a second time to rebase its buffer index. `reconstruct_views` therefore takes the buffer index the views should start at. Frame metadata comes straight off disk, so the arithmetic it drives is now checked and surfaced as errors rather than panics or wrapped lengths: value counts, frame sizes, length prefixes and the offsets they walk to. The missing `n_values` fallback no longer reads a byte count as a value count for variable-width values, where that mis-attributes values to frames — it is accepted only for the single-frame case that is still recoverable. Decompression also writes through `WriteBuf` into uninitialized spare capacity rather than a `&mut [u8]` over memory nothing has written yet. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
31c8739 to
1c85ef5
Compare
Merging this PR will improve performance by 13.18%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | take_map[(0.1, 0.5)] |
1,109.4 µs | 980.2 µs | +13.18% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/zstd-append-to-builder (1c85ef5) with rk/dict-gather-into-builder (c82aa58)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
1 participant
Write zstd values directly into the builder without additional copy