Skip to content

Fix three recent cuda.core build and CI issues - #2470

Open
rwgk wants to merge 5 commits into
NVIDIA:mainfrom
rwgk:stop_using_cuda-bindings_from_pypi
Open

Fix three recent cuda.core build and CI issues#2470
rwgk wants to merge 5 commits into
NVIDIA:mainfrom
rwgk:stop_using_cuda-bindings_from_pypi

Conversation

@rwgk

@rwgk rwgk commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes three unrelated issues that recently slipped into the cuda.core build and CI paths:

  1. Same-checkout CI builds could Cythonize cuda.core against a published cuda-bindings wheel from PyPI instead of the wheel produced by the current workflow.
  2. The graph memcpy host-source cast no longer compiled against the current in-tree cuda-bindings declaration.
  3. The cuda.core API check could fail with SIGPIPE under pipefail even after finding a valid release tag.

The issues happened to surface together while validating recent changes, but each fix is independent and remains in a separate commit.

1. Build cuda.core against the exact same-checkout cuda-bindings wheel

The cuda.core build backend requests cuda-bindings==<CUDA major>.*. CI previously made its newly built development wheel available only through PIP_FIND_LINKS.

PIP_FIND_LINKS adds candidates without prioritizing them over the package index. A published stable wheel can therefore beat a same-checkout development wheel when prereleases are not enabled. Enabling prereleases is not sufficient either, because a different, newer index candidate could still win. Even an exact version pin would leave the artifact source ambiguous if a tagged build were rerun after publication.

As a result, CI could successfully build current cuda.core sources against declarations from an older or otherwise unrelated published cuda-bindings package. The standard wheel tests install the current bindings wheel only after the core wheel has already been Cythonized, so they did not detect this mismatch. This is the false-green behavior described in #2468.

The build backend now recognizes the CI-only CUDA_CORE_BUILD_BINDINGS_WHEEL_DIR environment variable:

  • when unset, it preserves the existing cuda-bindings==<CUDA major>.* requirement
  • when set, it requires the directory to contain exactly one matching cuda_bindings-<major>.*.whl
  • it returns that wheel as a direct PEP 508 cuda-bindings @ file://... build requirement
  • it fails with a descriptive error for an empty setting, a missing wheel, or multiple matching same-major wheels
  • it reports both the selected wheel and the imported bindings version and location in the build log

Filtering by CUDA major allows current- and previous-major wheels to coexist in one artifact directory while still making the selected dependency unambiguous.

The local artifact directory is wired into every same-checkout pip/PEP 517 cuda.core build in CI:

  • current-major production wheels on Linux and Windows
  • previous-major production wheels on Linux and Windows
  • Linux and Windows wheel-from-sdist tests
  • Linux and Windows coverage builds

Linux coverage now retains and installs a locally built bindings wheel before building core. Existing PIP_FIND_LINKS and PIP_PRE settings remain only where they are needed to keep the same-checkout prerelease cuda-pathfinder eligible; bindings selection no longer depends on pip candidate ranking.

Intentional compatibility coverage is unchanged. The explicit BINDINGS_SOURCE=published rows continue testing released same-major, older-minor cuda-bindings wheels from PyPI, and isolated Pixi environments that intentionally resolve published packages are unaffected. The analogous static cuda-pathfinder build-dependency selection is outside this PR.

Closes #2468.

2. Fix the graph memcpy host-source pointer cast

The current cuda-bindings declaration exposes CUDA_MEMCPY3D.srcHost as void *, but a refactor in #2395 unintentionally restored a const void * cast in cuda.core. Cython correctly rejected assigning that const-qualified pointer to the mutable field because it discards the qualifier.

The assignment now casts the source address to void *, matching the current declaration. A mutable pointer remains assignable when building against an older declaration that accepts const void *, so this keeps the intentional published-bindings compatibility path valid as well.

This source incompatibility had been masked by the first issue: CI was Cythonizing against the older published declaration rather than the current same-checkout PXD. Selecting the exact local bindings wheel exposed the failure and made the one-line correction necessary. The change does not alter structure layout, ABI, ownership, or runtime copy behavior.

3. Avoid SIGPIPE in the cuda.core API check

The API check selected the newest cuda-core-v* release tag with:

gh api ... --paginate | head -1

The workflow runs with set -o pipefail. Once head received the first matching tag, it closed the pipe while gh was still paginating. gh then received SIGPIPE and returned status 141, causing the job to fail even though tag selection had succeeded.

The pipeline now uses sed -n '1p'. It prints the first matching tag while continuing to consume the complete paginated stream, allowing gh to finish normally. A genuine gh api failure still propagates through pipefail, and the existing explicit check for an empty tag remains unchanged.

Validation

  • TestVenv/bin/python -m pytest -v --noconftest cuda_core/tests/test_build_hooks.py ci/tools/tests (61 passed)
  • pre-commit run --all-files, including ruff, mypy, YAML validation, and actionlint
  • pip dry-run using the generated direct local-wheel requirement
  • build-hook coverage for the unchanged fallback, a development wheel in a path containing spaces, co-located CUDA 12 and 13 wheels, an empty setting, missing artifacts, and ambiguous same-major artifacts
  • live API-tag selection plus a simulated producer failure to confirm that real gh api errors still propagate

Coordination with #2464

PR #2464 independently evolved to address overlapping parts of these build and CI issues as part of its broader selective-wheel-build work. The implementations were cross-checked, and the detailed comparison is recorded in the PR discussion. Whichever PR lands second can rebase and reconcile the overlapping changes.

@rwgk rwgk added this to the cuda.core next milestone Jul 31, 2026
@rwgk rwgk added bug Something isn't working P0 High priority - Must do! CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module labels Jul 31, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk rwgk self-assigned this Jul 31, 2026
@rwgk

rwgk commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

rwgk commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Added follow-up commit 71502ad to fix the unrelated API-check failure exposed by this PR.

It is not a flake:

The command used gh api --paginate | head -1 under pipefail. After reading the first tag, head closed the pipe while gh was still paginating, so gh received SIGPIPE and the step failed.

The commit replaces head -1 with sed -n '1p'. sed prints only the first (newest) matching tag but continues consuming the full stream, allowing gh to complete normally. This remains fail-closed: a genuine gh api failure still propagates through pipefail, and the existing empty-tag check is unchanged.

I kept this as a separate commit because the same issue is also being handled in #2464; it can be dropped easily if that PR lands first.

Validation:

  • the updated command selected cuda-core-v1.1.1 from the live repository tags
  • a simulated producer failure propagated its nonzero status through the pipeline
  • pre-commit run --all-files passed, including actionlint

@rwgk

rwgk commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@github-actions

Copy link
Copy Markdown

@rwgk

rwgk commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @kkraus14 for pointing me to #2464. I did a fresh cross-check before continuing with this PR, since #2464 now overlaps with #2470 in three areas:

  • selecting the exact same-checkout cuda-bindings wheel when building cuda.core
  • fixing the srcHost Cython cast
  • avoiding the API-check head -1 / SIGPIPE failure

The exact-wheel work in #2464 is a valid solution for the production wheel workflow. Its direct-file constraints prevent an index candidate from taking precedence, and an earlier CI run surfaced the original Cython mismatch, which is useful confirmation that the local development wheel was selected. That approach also fits #2464's broader selective-build and artifact-reuse work, and it additionally constrains cuda-pathfinder.

#2470 takes a different approach for issue #2468: the cuda.core build hook selects an exact local bindings wheel when the CI-only environment variable is set. That selection is currently wired into the current- and previous-major production wheel builds, the Linux and Windows sdist tests, and the Linux and Windows coverage builds. It also provides explicit selection/import diagnostics and focused unit tests, while leaving ordinary builds unchanged when the variable is unset.

The srcHost changes in the two PRs are equivalent. The API-check fixes differ slightly: #2470 uses sed -n '1p' so the complete paginated stream is consumed while a genuine gh api failure still propagates through pipefail; #2464 currently uses process substitution with || true.

Based on this cross-check, I think it still makes sense to continue #2470 as the focused fix for #2468. In particular, #2464 does not currently apply its exact bindings constraint to the sdist or coverage workflows. At the same time, its constraint-based implementation remains useful within the selective-build work.

Whichever PR lands second will need a small rebase and de-duplication of the overlapping cast and API-check changes. I wanted to record the comparison here so it is clear that #2470 is continuing with the current state of #2464 taken into account.

@rwgk

rwgk commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@rwgk rwgk changed the title CI: build cuda.core against the exact local cuda-bindings wheel Fix three recent cuda.core build and CI issues Aug 1, 2026
@rwgk
rwgk marked this pull request as ready for review August 1, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: CI may build cuda.core against published cuda-bindings instead of the same-commit wheel

1 participant