[Pipe] Fix historical TsFile schema recovery race - #18314
Open
Caideyipi wants to merge 1 commit into
Open
Conversation
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18314 +/- ##
============================================
- Coverage 43.10% 42.93% -0.17%
Complexity 374 374
============================================
Files 5362 5363 +1
Lines 381676 381774 +98
Branches 49568 49585 +17
============================================
- Hits 164504 163903 -601
- Misses 217172 217871 +699 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jt2594838
reviewed
Jul 27, 2026
Comment on lines
+117
to
+118
| private final boolean isAutoCreateSchemaAllowed; | ||
| private final boolean isAutoCreateSchema; |
Contributor
There was a problem hiding this comment.
Too confusing, give better names.
Comment on lines
+126
to
+139
| default ISchemaTree fetchSchemaList( | ||
| final List<PartialPath> devicePaths, | ||
| final List<String[]> measurementsList, | ||
| final MPPQueryContext context) { | ||
| DataNodeSchemaLockManager.getInstance() | ||
| .takeReadLock(context, SchemaLockType.VALIDATE_VS_DELETION_TREE); | ||
| final PathPatternTree patternTree = new PathPatternTree(); | ||
| for (int i = 0; i < devicePaths.size(); i++) { | ||
| for (final String measurement : measurementsList.get(i)) { | ||
| patternTree.appendFullPath(devicePaths.get(i), measurement); | ||
| } | ||
| } | ||
| return fetchSchema(patternTree, true, context, true); | ||
| } |
Contributor
There was a problem hiding this comment.
Where is the lock released?
Add a comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem and observed behavior
Historical Pipe events from ConfigRegion, SchemaRegion, and DataRegion are transferred independently. A historical TsFile can therefore reach the receiver before the schema and template events that describe it, making the result dependent on event arrival order.
The timing-dependent failure is as follows:
SHOW PATHS SET DEVICE TEMPLATE aligned_templatedoes not return the expectedroot.sg_aligned.device_aligned.aligned_templateonroot.sg_aligned.device_alignedfailed because ordinary timeseries had already been created under that path.The early TsFile load auto-creates ordinary timeseries. When the historical template-set event arrives later, it conflicts with those timeseries and the receiver cannot restore the original template binding.
Fix
This avoids imposing cross-region transfer ordering: data loading simply waits and retries until the schema history has been applied.
Compatibility
The new behavior is opt-in through an internal system parameter and defaults to disabled on the wire. Existing behavior is preserved for external sources, incomplete schema inclusion, general-write requests,
mark-as-pipe-request=false, legacy requests, and ordinaryLOAD TSFILEoperations. The existing default that permits per-load schema auto-creation also remains unchanged unless the receiver gets the new marker.Tests
IoTDBPipeAutoConflictITto assert thatroot.sg_aligned.device_alignedis bound toaligned_templateon the receiver.OK (76 tests)across 6 test classes.template-nodeartifact.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
PipeDataNodeTaskBuilder: determines whether a Pipe has complete schema history and injects the internal wait marker.PipeTransferTsFileSealWithModReqand DataRegion sinks: propagate the marker across all V2 transfer paths.IoTDBDataNodeReceiver: configures marked sync and async loads to verify without auto-creating schema.LoadTsFileAnalyzer,TreeSchemaAutoCreatorAndVerifier, andSchemaValidator: support per-load schema auto-creation control and retryable missing-schema handling.