Skip to content

AIR CLI: implement air run --watch (stream logs until completion) - #6080

Merged
riddhibhagwat-db merged 2 commits into
air-clifrom
air-run-watch
Jul 29, 2026
Merged

AIR CLI: implement air run --watch (stream logs until completion)#6080
riddhibhagwat-db merged 2 commits into
air-clifrom
air-run-watch

Conversation

@riddhibhagwat-db

Copy link
Copy Markdown
Contributor

Changes & Why

After submitting a workload, --watch follows the run's logs to completion and exits with the run's outcome, reusing the same Bricklens-with-MLflow-fallback pipeline as air logs.

  • Text mode: prints "Submitted run", the dashboard link, and "Monitoring run and streaming logs...", then streams the logs.
  • JSON mode: emits a SUBMITTED event with the run id, a STATUS event on each lifecycle transition, the streamed LOG/ALERT events, and a closing terminal-status envelope (SUCCESS/FAILED/CANCELED) — matching the Python CLI's --watch JSONL contract.
  • Without --watch, the plain submit path now prints a tip about --watch.
  • STATUS events are watch-scoped (opt-in via logRequest.onStatusChange), so the merged air logs output is unchanged.

--dry-run still takes precedence over --watch (nothing is submitted or streamed).

Tests

Unit tests (experimental/air/cmd/)

  • logbricklens_test.go: Bricklens client query/path serialization + time_unix_nano parsing
  • logstream_test.go: fallback classification, status projection, --minutes/tail math, page dedup/ordering, retry-then-fallback, JSONL/ALERT emit, Ctrl-C exit
  • logmlflow_test.go: MLflow chunk discovery/listing, attempt-prefix layout, no-logs exit-code parity
  • logs_test.go: air logs command: flag validation, completed-run tail, Bricklens→MLflow fallback, past-retry static view
  • run_watch_test.go: air run --watch: text stream, JSON SUBMITTED→STATUS→LOG→terminal envelope, failed-run exit code, dry-run precedence

Acceptance tests (acceptance/experimental/air/)

  • logs/ : text/JSON streaming, --minutes, --lines, --lines 0, --retry, mutual-exclusion errors, invalid id, negative node, --download-to
  • logs-mlflow-fallback/ : Bricklens FEATURE_DISABLED → MLflow fallback → no-logs (text & JSON)
  • run/ : dry-run, --override, config validation, --watch ignored under --dry-run
  • run-submit/ : real submit payload + --watch tip line
  • help/ : air --help, air logs --help command-tree pins

Manual verification:
Properly monitors and outputs logs from runs on manual test:
Screenshot 2026-07-27 at 3 00 15 PM

After submitting a workload, --watch follows the run's logs to completion and
exits with the run's outcome, reusing the same Bricklens-with-MLflow-fallback
pipeline as `air logs`.

- Text mode: prints "Submitted run", the dashboard link, and "Monitoring run
  and streaming logs...", then streams the logs.
- JSON mode: emits a SUBMITTED event with the run id, a STATUS event on each
  lifecycle transition, the streamed LOG/ALERT events, and a closing
  terminal-status envelope (SUCCESS/FAILED/CANCELED) — matching the Python CLI's
  --watch JSONL contract.
- Without --watch, the plain submit path now prints a tip about --watch.
- STATUS events are watch-scoped (opt-in via logRequest.onStatusChange), so the
  merged `air logs` output is unchanged.

--dry-run still takes precedence over --watch (nothing is submitted or streamed).

Co-authored-by: Isaac
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: e74adb3

Run: 30400984356

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
❌​ aws-cli-is linux 8 1 3 4 320 1035 7:42
❌​ aws-cli-is windows 8 1 3 4 322 1033 17:43
❌​ azure-cli-is linux 8 1 3 4 320 1034 7:24
❌​ azure-cli-is windows 8 1 3 4 322 1032 19:44
❌​ gcp-cli-is linux 8 4 4 319 1036 10:52
❌​ gcp-cli-is windows 8 4 4 321 1034 17:11
16 interesting tests: 8 FAIL, 4 KNOWN, 4 SKIP
Test Name aws-cli-is linux aws-cli-is windows azure-cli-is linux azure-cli-is windows gcp-cli-is linux gcp-cli-is windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/deploy/mlops-stacks ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/generate/auto-bind ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/generate/auto-bind/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
❌​ TestAccept/bundle/resources/grants/volumes ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K

@maggiewang-db maggiewang-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you do a manual test with the fallback mlflow path?

Comment thread experimental/air/cmd/run.go Outdated
// envelope after streaming. Mirrors the Python CLI's --watch JSONL contract.
out := cmd.OutOrStdout()
printSubmittedEvent(out, runIDStr, dashboardURL)
terminalStatus := "FAILED"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac claims this terminalStatus is never updated in the fallback mlflow code path. Can you double check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, thanks for the catch!

The closing --watch JSON envelope derived its status from the onStatusChange
callback, which only fires on the Bricklens path. On the MLflow fallback (or an
active run that terminates without a Bricklens-observed transition) the status
stayed at its "FAILED" default, mislabeling a successful run.

Derive the terminal status from the run's actual state (re-resolved after
streaming) instead, correct regardless of which backend served the logs. Add a
regression test that forces the MLflow fallback and asserts the envelope reports
SUCCESS.

Co-authored-by: Isaac
@riddhibhagwat-db
riddhibhagwat-db merged commit 5fddda2 into air-cli Jul 29, 2026
25 of 26 checks passed
@riddhibhagwat-db
riddhibhagwat-db deleted the air-run-watch branch July 29, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants