fix(ci): publish system-tests image via S3 + GH Actions dispatch - #4065
fix(ci): publish system-tests image via S3 + GH Actions dispatch#4065MilanGarnier wants to merge 8 commits into
Conversation
|
#4013 pushed the per-branch system-tests image to GHCR directly from GitLab CI, authenticating with an octo-sts token scoped `packages: write`. That token comes from a GitHub App installation, and GHCR refuses package writes from installation tokens (permission_denied: installation not allowed to Write organization package). The job has allow_failure: true, so it has been failing silently since #4013 merged and no image has actually been published. Replace it with a two-step, two-runtime pipeline: 1. GitLab CI ("publish docker image for system tests (upload)") uploads the built tarballs and installer to s3://dd-trace-php-builds/ci/<ref-slug>/, then ("... (dispatch)") mints an actions: write-scoped octo-sts token and dispatches a new GitHub Actions workflow. 2. .github/workflows/publish-system-tests-image.yml downloads the artifacts from S3 and pushes to GHCR using the workflow's native GITHUB_TOKEN, which is actually allowed to write org packages. Update gitlab-ci-publish-packages.sts.yaml from packages: write to actions: write to match, since nothing mints a packages-scoped token through this policy anymore.
f4dedbc to
97f2a40
Compare
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97f2a40fe0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- Scope both the S3 upload and the GHCR fetch by CI_COMMIT_SHORT_SHA, not just the branch slug. Two concurrent pipelines for the same branch were sharing one S3 prefix, so a slower upload could overwrite a faster one's objects mid-read on the GitHub Actions side, mixing old and new artifacts into one image. - Fail the GitLab dispatch job loudly if the PR-lookup request itself errors, instead of treating any non-2xx response the same as "zero open PRs" and silently skipping the publish. - Move package cleanup in the GH Actions workflow from before the push to after it, and match on untagged versions instead of the tag being retargeted. The old ordering deleted-by-tag before pushing, which 404s on the very first publish ever (no package exists yet) and, since Actions runs steps with `bash -eo pipefail`, took the whole job down before it reached the push itself. Cleanup is also now best-effort (continue-on-error): it's storage hygiene, not a precondition for publishing. - Add an org.opencontainers.image.source label to the built image so GHCR links it to this repo, which is what lets the repo's own GITHUB_TOKEN manage the package going forward.
The upload job is allow_failure: true so the pipeline doesn't go red over this optional publish, but that also makes needs: run the dispatch job even when upload failed or only partially uploaded. Re-verify the three expected S3 objects exist for this commit before dispatching, instead of trusting the upload job's (possibly falsely-green) status.
A failed or partial publish should be visible and block merge like any other job, rather than going green silently. This also makes the needs: chain behave normally: dispatch simply won't run if upload fails, no allow_failure/needs interaction to reason about.
needs: already refuses to run dispatch if upload failed now that upload isn't allow_failure. The only case this check still caught (retrying dispatch alone from the GitLab UI, which bypasses needs:) isn't worth the extra script complexity.
Drop CI_COMMIT_SHORT_SHA/commit_slug from the S3 upload path and the workflow_dispatch inputs, keeping CI_COMMIT_REF_SLUG as the sole identifier.
Drop two comments that just restated the adjacent config (packages: write, continue-on-error) instead of explaining non-obvious behavior.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e12d4abed
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| gh api "${PACKAGE_PATH}/versions" \ | ||
| --jq '.[] | select(.metadata.container.tags == []) | .id' \ | ||
| | xargs -r -I {} gh api "${PACKAGE_PATH}/versions/{}" -X DELETE |
There was a problem hiding this comment.
When this workflow publishes the linux/amd64,linux/arm64 tag, Buildx creates a tagged image index plus untagged per-platform manifests; GHCR exposes those child manifests as package versions with empty metadata.container.tags (the same failure mode has been reported for GHCR untagged cleanup: actions/delete-package-versions#162). This selector deletes every untagged version immediately after the push, including the current amd64/arm64 manifests referenced by ${REF_SLUG}, so the branch tag can exist but fail to pull in system-tests; delete only known stale versions for this tag or skip the untagged cleanup.
Useful? React with 👍 / 👎.
| aws s3 cp packages/dd-library-php-*-x86_64-linux-gnu.tar.gz "${DEST}/dd-library-php-x86_64-linux-gnu.tar.gz" | ||
| aws s3 cp packages/dd-library-php-*-aarch64-linux-gnu.tar.gz "${DEST}/dd-library-php-aarch64-linux-gnu.tar.gz" | ||
| aws s3 cp packages/datadog-setup.php "${DEST}/datadog-setup.php" |
There was a problem hiding this comment.
For any non-default branch that has no open GitHub PR, this upload job still copies the branch tarballs to the public S3 prefix that the Actions workflow later downloads anonymously. The PR gate only runs in the following dispatch job, so internal/throwaway branch pipelines now publish build artifacts even though the image publish is skipped; move the open-PR check before these aws s3 cp calls or make the upload job conditional on the same gate.
Useful? React with 👍 / 👎.
| PAYLOAD=$(jq -n --arg ref "${CI_DEFAULT_BRANCH}" --arg slug "${CI_COMMIT_REF_SLUG}" \ | ||
| '{ref: $ref, inputs: {ref_slug: $slug}}') | ||
| HTTP=$(curl -s -o /tmp/dispatch.json -w "%{http_code}" -X POST \ | ||
| -H "Authorization: Bearer ${GITHUB_TOKEN}" -H "Accept: application/vnd.github+json" \ | ||
| "https://api.github.com/repos/DataDog/dd-trace-php/actions/workflows/publish-system-tests-image.yml/dispatches" \ |
There was a problem hiding this comment.
When this change is exercised from the PR branch that adds publish-system-tests-image.yml, the dispatch targets ${CI_DEFAULT_BRANCH} even though that branch does not contain the new workflow file yet; GitHub documents that a workflow_dispatch workflow must be on the default branch before it can be triggered (https://docs.github.com/en/actions/how-tos/manage-workflow-runs/manually-run-a-workflow#configuring-a-workflow-to-run-manually). Since this dispatch job is no longer allowed to fail, open-PR pipelines for this PR will fail with a 404 instead of validating the fix; land the workflow on the default branch first or keep this dispatch optional until it exists there.
Useful? React with 👍 / 👎.
Benchmarks [ tracer ]Benchmark execution time: 2026-07-28 14:17:37 Comparing candidate commit 6e12d4a in PR branch Found 2 performance improvements and 0 performance regressions! Performance is the same for 192 metrics, 0 unstable metrics.
|
1 participant
Description
#4013 pushed the per-branch system-tests image to GHCR directly from GitLab
CI, authenticating with an octo-sts token scoped
packages: write. Thattoken is minted from a GitHub App installation, and GHCR refuses package
writes from installation tokens (
permission_denied: installation not allowed to Write organization package). The job hasallow_failure: true,so it's been failing silently since #4013 merged — no image has actually
been published.
This replaces that job with a two-step, two-runtime pipeline:
publish docker image for system tests (upload)) uploads thebuilt tarballs + installer to
s3://dd-trace-php-builds/ci/<ref-slug>/,then (
... (dispatch)) mints anactions: write-scoped octo-sts tokenand dispatches a new GitHub Actions workflow.
.github/workflows/publish-system-tests-image.ymldownloads theartifacts from S3 and pushes to GHCR using the workflow's native
GITHUB_TOKEN, which — unlike the octo-sts token — is actually allowedto write org packages.
.github/chainguard/gitlab-ci-publish-packages.sts.yamlis updated frompackages: writetoactions: writeto match (nothing mints apackages-scoped token through this policy anymore).
Reviewer checklist