fix(objectstorage): retry enabling the project on 409 conflict - #1645
Open
FabianHardt wants to merge 1 commit into
Open
fix(objectstorage): retry enabling the project on 409 conflict#1645FabianHardt wants to merge 1 commit into
FabianHardt wants to merge 1 commit into
Conversation
bucket, credential and credentials group each enable object storage for the
project before creating their own object. When two of them are created in the
same apply, Terraform runs them in parallel and the API rejects the losing
call:
Error: Enabling object storage project before creation: failed to create
object storage project: 409 Conflict
([{project.create_conflict Two concurrent calls try to create the same
project}]), status code 409
The apply fails, although nothing is wrong - the competing call enables the
project a moment later. The comment in enableProject already assumed the call
to be idempotent ("Creation will also be successful if the project is already
enabled"), which holds for sequential calls but not for concurrent ones.
enableProject now retries on 409 and leaves every other error untouched, so an
apply no longer depends on the order in which Terraform happens to start the
resources. Users can work around it today with depends_on, but that requires
knowing about an implicit call that the resource documentation does not
mention.
The retry is deliberately narrow rather than utils.RetryRequest: that helper
also retries errors that are not API errors, which would slow down the
existing unit tests.
Signed-off-by: Fabian Hardt <fabian.hardt@opitz-consulting.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
stackit_objectstorage_bucket,stackit_objectstorage_credentialandstackit_objectstorage_credentials_groupeach enable object storage for theproject before creating their own object. When two of them are created in the
same apply and nothing forces an order, Terraform runs them in parallel and the
API rejects the losing call:
The apply fails although nothing is actually wrong — the competing call enables
the project a moment later.
Minimal reproducer (both resources in one config, no reference between them):
The comment in
enableProjectalready assumes idempotency — "Creation willalso be successful if the project is already enabled" — which holds for
sequential calls but not for concurrent ones.
Change
enableProjectretries on409and leaves every other error untouched. Anapply no longer depends on the order in which Terraform happens to start the
resources.
depends_onworks around it today, but that requires knowing about an implicitAPI call the resource documentation does not mention — the error message points
at object storage projects, not at a missing dependency.
The retry is deliberately narrow rather than
utils.RetryRequest: that helperalso retries errors which are not
GenericOpenAPIError, which would slow theexisting unit tests down.
Tests
Added to
credentialsgroup/resource_test.go:TestEnableProjectRetriesOnConflict— succeeds immediately, one conflict thensuccess, and conflicts until the attempts are used up; asserts the number of
API calls in each case
TestEnableProjectDoesNotRetryOtherErrors— a403must fail on the firstattempt
The retry delay is a package variable so tests can shorten it.
Note
The three copies of
enableProjectare identical; I kept the duplication tokeep the diff reviewable. Happy to extract it into
objectstorage/utilsinstead if you prefer.