Skip to content

ci: Add nightly OpenAPI spec sync and make model codegen reproducible - #983

Draft
vdusek wants to merge 2 commits into
masterfrom
ci/nightly-openapi-spec-sync
Draft

ci: Add nightly OpenAPI spec sync and make model codegen reproducible#983
vdusek wants to merge 2 commits into
masterfrom
ci/nightly-openapi-spec-sync

Conversation

@vdusek

@vdusek vdusek commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Replaces the docs-PR-triggered model regeneration with a nightly job, and makes the generated files reproducible from committed state so drift can no longer slip in unnoticed.

What changes

  • spec/openapi.json - a committed snapshot of the published spec, now the only input codegen reads. .gitattributes marks it linguist-generated so it collapses in diffs (one click away) instead of burying the model diff, and doesn't skew the repository language stats.
  • scripts/update_openapi_spec.py + poe update-spec - downloads the published spec, refuses anything that isn't a plausible OpenAPI document (size, required top-level members) so an error page can't overwrite the snapshot, and normalizes the formatting. Key order is deliberately preserved: keep_model_order ties generated model order to the spec.
  • poe generate-models now generates from the snapshot instead of the live URL. generate-models-from-file is redundant with that and is removed - to try a candidate spec, overwrite the snapshot and regenerate.
  • poe check-models (new, part of check-code and the new Models check CI job) regenerates, compares, and restores the working tree. It never leaves the generated files modified and doesn't care whether they're committed, so it's safe to run mid-change.
  • on_schedule_regenerate_models.yaml - nightly at 02:00 UTC plus manual dispatch, modelled on Crawlee's update_playwright_version.yaml. Refreshes the snapshot, regenerates, and opens a PR when the models change. Scheduled failures go to Slack, following Crawlee's on_schedule_tests.yaml. manual_regenerate_models.yaml is deleted.
  • _models.py loses the Plan.available_proxy_groups docstring - see below.

Why

Two independent problems, both visible in #979:

The generated files weren't a function of committed state. The spec was fetched live and the codegen toolchain floated, so nobody could tell whether _models.py was current. Renovate moved datamodel-code-generator 0.68.1 -> 0.69.0 (#968) -> 0.71.0 (#975) and ruff to 0.16 (#978); master's generated output no longer matched what that toolchain produces. check-models closes this: a codegen or formatter upgrade that changes output now turns its own dependency-bump PR red, so that PR carries the regenerated files instead of the drift contaminating an unrelated spec change later. The accumulated drift is the Plan.available_proxy_groups docstring removal in this PR - the description is still in the spec, on AvailableProxyGroups.additionalProperties, so this is purely codegen catching up.

The regeneration branch rotted. The manual branch was permanent and never rebased, and the workflow checked it out before regenerating - so it generated with the codegen tooling from that stale branch. In #979 that produced _literals.py in the pre-#941 closed Literal[...] form; merging it would have silently reverted the enum relaxation. The workflow's own comment stated the intent ("regenerating on master keeps the codegen tooling current") but that only held on the branch-creation path.

The new workflow fixes both structurally:

Notes

  • Merge ci: Drop cross-repo model regeneration dispatch to apify-client-python apify-docs#2835 first - it removes the dispatch that still calls the manual_regenerate_models.yaml deleted here.
  • Needs a SLACK_WEBHOOK_URL repository secret for the failure notification. Without it the alert step fails, but only on a run that has already failed.
  • Because this PR already carries the regenerated output for the current published spec, the first nightly run should find no changes and open nothing.
  • Plan.available_proxy_groups loses its description because collapse_root_models discards the $ref-ed schema's docs, and postprocess doesn't carry them over. That predates this PR (it came with the generator bump) and applies to every $ref-ed root model, so it's worth fixing separately in postprocess_generated_models.py.

The second commit applies review follow-ups: the "already up for review" gate now also requires an open PR (a leftover branch would otherwise silently suppress every future regeneration), the sync script validates info.version before overwriting the snapshot, retries transient download failures, writes LF endings, resolves its path from the repo root, and reports errors on stderr.

✍️ Drafted by Claude Code

@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Jul 29, 2026
@vdusek vdusek self-assigned this Jul 29, 2026
@github-actions github-actions Bot added this to the 146th sprint - Tooling team milestone Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.62%. Comparing base (b49fb35) to head (7539cdd).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #983      +/-   ##
==========================================
- Coverage   94.62%   94.62%   -0.01%     
==========================================
  Files          58       58              
  Lines        5248     5247       -1     
==========================================
- Hits         4966     4965       -1     
  Misses        282      282              
Flag Coverage Δ
integration 92.28% <ø> (+0.01%) ⬆️
unit 83.89% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants