Fix table UDF result block splitting - #18333
Open
ColinLeeo wants to merge 6 commits into
Open
Conversation
4 tasks
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18333 +/- ##
=========================================
Coverage ? 43.45%
Complexity ? 374
=========================================
Files ? 5366
Lines ? 383022
Branches ? 49823
=========================================
Hits ? 166428
Misses ? 216594
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses oversized TsBlock outputs from table-model UDFs by enforcing configured TsBlock size/row limits in TableFunctionOperator, preventing large blocks from propagating through the query pipeline and risking RPC frame overflows.
Changes:
- Add final-result TsBlock splitting in
TableFunctionOperator(after pass-through columns are appended) based on configured TsBlock constraints. - Add a unit test covering variable-width outputs with/without pass-through columns.
- Add an integration-test UDF plus an IT validating large UDF results are returned without loss/duplication and with pass-through alignment under a small TsBlock limit.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/execution/operator/process/tvf/TableFunctionOperatorTest.java | Adds unit coverage for splitting variable-width UDF output blocks with optional pass-through columns. |
| iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/function/TableFunctionOperator.java | Implements final-result TsBlock splitting and adjusts operator completion/return-size reporting. |
| integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/udf/IoTDBUserDefinedTableFunctionIT.java | Adds end-to-end validation that large table-UDF results are split without data loss and keep pass-through columns aligned. |
| integration-test/src/main/java/org/apache/iotdb/db/query/udf/example/relational/LargeResultTableFunction.java | Introduces a table UDF used by the integration test to generate large variable-width outputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
64
to
65
| private static final long INSTANCE_SIZE = | ||
| RamUsageEstimator.shallowSizeOfInstance(AggregationMergeSortOperator.class); |
Comment on lines
423
to
426
| @Override | ||
| public long calculateMaxReturnSize() { | ||
| return Math.max(DEFAULT_MAX_TSBLOCK_SIZE_IN_BYTES, properBlockBuilder.getRetainedSizeInBytes()); | ||
| return maxTsBlockSizeInBytes; | ||
| } |
Comment on lines
+264
to
+277
| /** | ||
| * Splits the final result using the same logical in-memory size accounting as {@link | ||
| * TsBlockBuilder}. | ||
| * | ||
| * <p>Serializing candidate regions to find their exact sizes would write every value into | ||
| * temporary buffers, only for the exchange layer to serialize the selected regions again. | ||
| * Rebuilding the result with a size-tracking {@link TsBlockBuilder} would avoid that temporary | ||
| * serialization, but it would turn the UDF's batched column output into row-by-row, | ||
| * column-by-column copies. | ||
| * | ||
| * <p>Instead, fixed-width values are accounted for directly from their data types, while only the | ||
| * retained sizes of variable-width values are inspected. This deliberately estimates the | ||
| * in-memory TsBlock size rather than its serialized size because the two representations are not | ||
| * equivalent. The resulting regions are views over the original columns and do not copy their |
shuwenwei
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
A table-model UDF can generate a large result while processing a partition, especially when one device contains many rows or the UDF expands each input row into multiple output rows. Previously,
TableFunctionOperatorcould return the generatedTsBlockwithout applying the configured TsBlock row-count and size targets to the final result.The final result may also contain pass-through columns. Returning it as one large block increases memory pressure in the query pipeline and the risk that a downstream serialized block exceeds an RPC frame limit.
Design
TableFunctionOperatornow splits each final result inbuildTsBlock()after pass-through columns have been appended. Splitting at this point ensures that proper columns and pass-through columns consume the same block-size budget and remain aligned in every result fragment.The implementation estimates the logical in-memory size of each result region using the same per-position accounting model as
TsBlockBuilder:Binaryvalue.For blocks containing only fixed-width columns, the maximum position count is calculated directly from the per-position size, so no row scan is needed. For blocks containing variable-width columns, each position in those columns is inspected once because payload sizes can differ by row. This costs
O(positionCount * variableWidthColumnCount), but it only reads null states,Binaryreferences, and retained sizes. It does not serialize values or traverse/copy payload bytes.The selected fragments are created with
TsBlock.getRegion(), so they are views over the original columns rather than rebuilt row by row. Fragments are queued in their original order and returned by subsequentnext()calls.This deliberately uses an in-memory size estimate instead of serializing candidate regions. Serializing candidates would write the values into temporary buffers only for the exchange layer to serialize the selected fragments again. Rebuilding the result through a size-tracking
TsBlockBuilderwould avoid temporary serialization but would turn the UDF's batched column output into row-by-row, column-by-column copies.The configured size is therefore a logical in-memory target, not an exact serialized-size guarantee. It reduces the risk of oversized serialized blocks without adding serialization work to the UDF execution path.
Configuration and corner cases
This PR introduces no new configuration. It uses:
max_tsblock_size_in_bytesmax_tsblock_line_numberBoth constraints are applied to the final result after pass-through columns are appended. A row is indivisible; if one row alone exceeds
max_tsblock_size_in_bytes, it is returned by itself so execution can continue. Result ordering and empty-output behavior are unchanged.Tests
The unit tests cover:
The integration test configures a 1 KB TsBlock target and runs a table UDF that expands four input rows into 256 variable-width output rows. It verifies:
(time, repeat_index)pair is returned;The following targeted tests passed:
mvn test -pl iotdb-core/datanode -am \ -Dtest=TableFunctionOperatorTest \ -DfailIfNoTests=false \ -Dsurefire.failIfNoSpecifiedTests=false mvn verify -Drat.skip=true -DskipUTs \ -Dit.test=IoTDBUserDefinedTableFunctionIT#testLargeResultIsSplitWithoutDataLoss \ -DfailIfNoTests=false \ -Dfailsafe.failIfNoSpecifiedTests=false \ -pl integration-test -am \ -PTableSimpleIT -P with-integration-testsThis PR has:
Key changed/added classes
TableFunctionOperatorTableFunctionOperatorTestLargeResultTableFunctionIoTDBUserDefinedTableFunctionIT