Skip to content

chore(ci): retry lstk interception and pin the lstk CLI version - #112

Merged
DrisDary merged 2 commits into
mainfrom
chore/ci-lstk-interception-retry
Jul 30, 2026
Merged

chore(ci): retry lstk interception and pin the lstk CLI version#112
DrisDary merged 2 commits into
mainfrom
chore/ci-lstk-interception-retry

Conversation

@DrisDary

Copy link
Copy Markdown
Contributor

Motivation

Every sample job is currently failing at setup, on every branch, with:

LocalStack is already running.
Configuring Azure CLI for LocalStack...
[DEBUG] Starting lstk az interception...
Error: LocalStack Azure Emulator is not running
make: *** [Makefile:26: test] Error 1

This is not caused by any sample. It happens in run-samples.sh before a single sample executes, and it hit two unrelated branches within a minute of each other (fix/eventhubs-dashboard-totals and feat/eventhubs-eventgrid-sample), both of which were green the day before with no workflow change in between.

The emulator is genuinely running when this fires: the image is pulled, localstack wait has already returned, and run-samples.sh prints "LocalStack is already running." immediately before. So lstk is reporting a live emulator as absent. That is a reproducible class of behaviour rather than a theory — locally, against a running Azure emulator, lstk status answers LocalStack AWS Emulator is not running. The flavour-specific detection can produce false negatives.

Two things changed upstream on the same day, and both were investigated and eliminated: localstack/localstack-azure:latest was re-pushed at 06:54, and @localstack/lstk 0.20.0 was published at 11:10. With the freshly pulled image, lstk 0.20.0 and an empty AZURE_CONFIG_DIR (the state a CI runner is in), interception succeeds locally every time. The failure has not been reproduced outside CI, so this PR makes the step survivable and self-diagnosing rather than claiming a root cause it cannot demonstrate.

Two properties of the existing code turn a transient into a total outage: run-samples.sh runs under set -euo pipefail, and lstk az start-interception is the only command in that setup block without a failure guard — every neighbouring az call already has || true. One bad detection therefore kills the job outright.

Changes

  • run-samples.sh retries lstk az start-interception up to five times with increasing backoff, and on the final failure prints the lstk version, the container state, the emulator health endpoint and the tail of lstk.log.
  • .github/workflows/run-samples.yml pins the CLI via LSTK_VERSION: "0.20.0" instead of installing the floating latest.

The step is deliberately still fatal after the last attempt, and || true was rejected: without interception the Azure CLI targets real Azure, so continuing would point the suite's az group create / az group delete calls at the cloud. Failing fast is the correct behaviour; the only defect was failing on the first transient.

The pin is hygiene, not the fix, and is called out as such: it pins the version that demonstrably works, and brings this install in line with the rest of the workflow, where every action is already SHA-pinned. An unpinned dependency lands in every run simultaneously with no commit to bisect — precisely the position this failure left us in.

Tests

  • bash -n run-samples.sh passes; the workflow YAML parses.
  • The retry loop was exercised under set -euo pipefail with a stubbed command: a transient failure recovers on attempt 3 and exits 0, and a persistent failure gives up after 5 attempts and exits 1 — confirming it cannot silently fall through to real Azure.
  • The modified block was extracted verbatim from run-samples.sh and executed against a live emulator and a real lstk, succeeding on the first attempt.
  • run-samples.sh --list is unchanged (36 entries), so sharding is unaffected.

Since the CI failure could not be reproduced locally, this cannot be proven to clear it. If the cause is deterministic rather than transient the job will still fail — but the diagnostics will then identify why, which is the information the current failure does not provide.

Related

Unblocks #109 and #110, both of which are red on this failure rather than on their own content.

@DrisDary DrisDary self-assigned this Jul 30, 2026

@paolosalvatori paolosalvatori left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense , I noticed the same issue when running lstk az start-interception locally. No need for a Claude Code review. Thanks for the fix!

@anisaoshafi

anisaoshafi commented Jul 30, 2026

Copy link
Copy Markdown

ℹ️ Fix is shipped in lstk v0.20.1. Thanks Idris for raising the lstk PR. Feel free to unpin the lstk version if you prefer!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants