coverage: capture what the Windows coverage crash actually is - #2426
Open
rluo8 wants to merge 12 commits into
Open
coverage: capture what the Windows coverage crash actually is#2426rluo8 wants to merge 12 commits into
rluo8 wants to merge 12 commits into
Conversation
The "Run cuda.core tests (with 8MB stack)" step on the Windows coverage runner exits 139 on every nightly, and its log holds nothing else: three lines, no pytest output at all. The step before it runs cuda.bindings through the same wrapper, the same 8 MB thread and the same coverage wheel and finishes normally with 479 passed, so the crash is specific to cuda.core -- and with no "collected N items" line to go on, we cannot even tell whether it reaches a test. PYTHONFAULTHANDLER is set job-wide because it costs nothing; the rest is scoped to the one step that crashes. No behaviour change. - PYTHONFAULTHANDLER names the exception and dumps every thread's Python stack. Unlike pytest's built-in faulthandler plugin, the env var also covers tests that spawn their own `python -c` children. PYTHONUNBUFFERED keeps the final lines out of the block buffer of a non-tty pipe. - run_pytest_with_stack.py --isolate runs pytest in a child process so a surviving parent can report its raw exit status. The shell only shows a translation of that status; the parent names the NTSTATUS outright. The child's status is passed through unchanged. - pytest_crashlog records session progress to a line-buffered file, so the last line survives a native fault and localises it to interpreter startup, conftest import, a named module's collection, a named test, or interpreter shutdown. The wrapper opens the log before calling pytest.main, because pytest imports the initial conftests -- and for cuda_core that pulls in cuda.core -- before pytest_configure runs. A no-op unless PYTEST_CRASHLOG is set. Verified on a Windows GPU runner against the linetrace coverage wheel: identical test results with and without the change. Signed-off-by: Rui Luo <ruluo@nvidia.com>
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
Hi @mdboom , could you please help trigger a coverage run with this draft PR ? The Windows coverage run always failed when running cuda.core tests. This is to add some debug logs for triage. |
Contributor
|
/ok to test 8291f92 |
Contributor
|
/ok to test 9ed18a7 |
|
TEMPORARY -- not for merge. The exit-139 crash this branch instruments only reproduces on the Windows coverage runner, and coverage.yml is reachable only from the nightly schedule and from workflow_dispatch on the default branch. Every iteration of the diagnostics has therefore needed a maintainer to sync the mirror branch by hand. copy-pr-bot already pushes this PR to pull-request/<n> on the upstream repo, and a push event reads the workflow from the pushed commit, so the trigger takes effect from the PR itself without merging. ci.yml uses the same trigger. A PR run covers the Windows leg alone -- coverage-vars, build-wheel-windows, coverage-windows -- roughly 28 minutes, 18 of them on an A100. The two skipped jobs are skipped for different reasons: coverage-linux has no bearing on a Windows crash, and combine-and-deploy would both fail on the missing .coverage.linux and publish a half report to gh-pages. always() does not stop the latter, so its condition is gated as well. concurrency cancels superseded runs, which would otherwise queue another A100 job per push. The nightly is unchanged: on refs/heads/main both conditions hold and all five jobs run as before. Signed-off-by: Rui Luo <ruluo@nvidia.com>
rluo8
marked this pull request as ready for review
July 29, 2026 02:34
The first round of diagnostics placed the fault: the crash log holds `open pid=...` and `entering pytest.main` and nothing else, so the process dies while pytest loads its initial conftests -- before collection, and far before any test runs. What it cannot say is which module. Three additions, none of which touch machine state: PYTHONVERBOSE makes the interpreter announce every import as it begins one, so the last line written before the process dies names the module it died in. Its output is stderr, so the session is redirected to a file rather than left in the step log: a crash takes the tail of a pipe with it, while each unbuffered write to a file has already reached the OS. The Windows event log carries an Application Error record for every process the OS kills, naming the faulting module and the offset within it, and it does so whether or not crash dumps are configured. That is the one fact no Python-level log can produce. Reading it needs no registry key set and nothing restored afterwards, which a LocalDumps setup would have on a machine shared with other jobs. PYTHONFAULTHANDLER has so far written nothing at all, which leaves two very different readings: the handler produced a traceback the dying process failed to flush, or it never ran because the fault happened outside any Python frame. run_pytest_with_stack.py now points faulthandler at its own unbuffered file when PYTEST_FAULTLOG is set, so still empty there means the handler genuinely had nothing to say. The three logs upload together as an artifact. No behaviour change: the step already carries continue-on-error, and every addition is inert on a run that does not crash. Signed-off-by: Rui Luo <ruluo@nvidia.com>
The stack from the last round places the fault inside an extension module's initialisation, two exec_module frames below cuda.core._context, but does not name it. PYTHONVERBOSE was meant to: it announces every import, so its last line would be the module. It could not deliver -- its output goes through the interpreter's C-level stdio, which buffers when redirected to a file, and the child's tail stopped at `import 'colorama'` several hundred lines before the crash. The crash log does not have that problem. It is line buffered, and both of its lines survived every run so far, so the same question goes through that file instead. A meta path finder records each name and returns None, leaving the real finders to do the work; because the write lands before the module begins executing, a fault during an extension's init leaves that module as the last IMPORT line. sys.unraisablehook goes to the same file. A Cython `cdef ... noexcept` function cannot propagate an exception, so one raised inside it is printed and cleared and whatever it was computing keeps its default -- in _resource_handles.pyx, a NULL driver function pointer that faults only when something later calls through it. Recording unraisables in sequence with the IMPORT lines puts a swallowed error next to the module that swallowed it. The previous hook still runs, so stderr is unchanged. Both are off unless PYTEST_CRASHLOG is set, and tracing starts before pytest.main because the fault happens before pytest_configure would run.
Import tracing put the fault in cuda.core._event, reached through _stream from _context, but the smallest thing that reproduces it is still a pytest session over 3869 tests. These probes strip that down: one fresh interpreter per module, importing exactly one name, under coverage and nothing else. The linetrace wheel and the ctracer stay because they are the two conditions test-wheel-windows.yml lacks while running the same tests on the same runner without faulting. The order is leaf-first, so the first line that fails to print PROBE OK is the smallest unit that reproduces. Importing _event directly also enters the _context/_stream/_event cimport cycle from the opposite side, which says whether the direction matters -- worth knowing, since the same cycle exists on Linux and on our own runner without faulting, so circularity cannot be the whole story. They run after cuda.bindings rather than before pathfinder because that is the machine state the crash is known to occur in. A clean-machine placement is the follow-up if these come back clean, which would itself say the earlier suites leave something behind. COVERAGE_FILE points the probes at their own data file; `coverage run` would otherwise truncate the one the suites are appending to. Each exit code is captured rather than left to `set -e`, so one crash does not hide the modules after it.
rluo8
marked this pull request as draft
July 29, 2026 09:29
faulthandler stops exactly where the answer starts. The fault is inside an extension module's initialiser, so the frames that matter are the ones it cannot reach, and every report ends `<cannot get C stack on this system>`. Every ordinary way to see past that is closed on this runner. cdb is not installed: the SDK is present without the Debugging Tools feature, so Windows Kits ships the debugger DLLs and none of the executables. procdump would mean an outbound download on a runner whose agent exists to police outbound requests. A postmortem debugger under AeDebug never fires, because this image does not record user-process crashes with WER, which is why the crash-dumps directory came back empty when that was tried. And there are no PDBs in any case, since build_hooks.py refuses debuggable builds on Windows. dbgcore.dll in System32 is always there though, and exports MiniDumpWriteDump, so the process reports on itself. A vectored handler writes the exception record, the faulting address resolved to module + RVA, the native return chain resolved the same way, the module map, and a minidump, then returns EXCEPTION_CONTINUE_SEARCH so the process still dies with the exit code CI would have seen. Symbols are absent, but which binary faulted is the open question and module + RVA answers it. crash_probe.py drives four runs. A selftest faults on purpose, so a clean report proves the handler works here and an empty one is never mistaken for "nothing crashed". A run under coverage reproduces the nightly's own conditions on the 8 MB stack the pytest wrapper uses. A run with a bare trace function instead of coverage arms Cython's linetrace the same way, so a fault there means any tracer will do rather than coverage specifically, and the last line of its breadcrumb names the .pyx statement that was executing. env_fingerprint.py records the environment around the import -- driver, CPU, and the loaded-module map -- which is what comparing against a machine that does not fault requires. This replaces the per-submodule probes, which measured nothing: importing cuda.core._resource_handles executes cuda/core/__init__.py on the way in, so all six walked the same path and all six failed identically. Verified on two Windows runners that do not fault, where the selftest produced a resolved address, a 44-frame backtrace and a 300 KB dump, and the breadcrumb recorded .pyx line numbers through to the end of the import.
rluo8
marked this pull request as ready for review
July 30, 2026 08:34
rluo8
marked this pull request as draft
July 30, 2026 09:11
Windows coverage has produced no wheel since 2026-07-29, so the job fails before Coverage (Windows) can start and the crash diagnostics on this branch have nowhere to run. PIP_PRE is set here so pip will consider the locally built cuda-bindings at all: it carries a .devN version, which counts as a pre-release. The side effect is that PyPI's pre-releases become eligible too, and cuda-bindings 13.4.0b1, published that same day, outranks the local build for `cuda-bindings==13.*`. So the wheel built one step earlier is discarded and a 13.4 wheel is downloaded instead. Its cydriver.pxd is generated from CTK 13.4 headers, where CUmemLocation has a `localized` field, while cuda.core compiles against the 13.3.0 mini-CTK, where it does not. Cython generates a struct converter over every field, and MSVC rejects it: _managed_memory_ops.cpp(18447): error C2039: 'localized' is not a member of 'CUmemLocation_st' Linux is unaffected, and build-wheel.yml escapes it as well, because cibuildwheel sets PIP_FIND_LINKS without PIP_PRE. Constraining cuda-bindings to the version just built keeps what PIP_PRE was for and drops what it let in. The version is read from the wheel filename rather than hardcoded, with the local segment removed: PEP 440 has a public `==` specifier ignore a candidate's local label, so `==13.3.2.dev139` still selects 13.3.2.dev139+g0123456, and constraints files stay free of local versions, which pip has objected to before. TEMPORARY, like the pull-request push trigger above it. The real fix belongs upstream; this only exists so the diagnostics can run before that lands.
rluo8
marked this pull request as ready for review
July 31, 2026 03:00
The fault is a null read inside msvcp140's mutex lock, taken by a std::lock_guard that _stream.pyx:422 executes unconditionally, so instrumentation should have nothing to do with it. Two of the three combinations have been run and both fault: coverage, and a bare sys.settrace. The third -- no coverage, no trace function -- has never been tried on the machine that faults. It matters because cov and nocov wheels differ in a way nobody had checked: the cibuildwheel-built wheel carries its own msvcp140 14.44 under cuda_core.libs, put there by delvewheel, while the coverage build's plain `pip wheel` leaves the process to load the system copy, which on this image was built in 2016 and predates the constexpr std::mutex change. If that is the real variable, linetrace was never a condition either, and this probe says so in one line.
The msvcp140.dll this crash blames is currently dated only by a TimeDateStamp read out of a minidump, which says August 2016 but is not a version number. A report asking for the runner image to be updated needs the version, so read it off the file. Both machines print it, because they are not the same machine and only one of them matters. Wheels are built on a GitHub-hosted windows-2022 image and run on the self-hosted GPU runner, and a plain `pip wheel` performs no delvewheel repair, so nothing from the build machine's runtime travels with the wheel. Seeing a current version on the builder next to a 2015-era one on the tester is the shape of the problem in two lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is a draft pr for diagnostics. Please do not merge.
The
Run cuda.core tests (with 8MB stack)step ofCI: Coveragealways exited 139. Its log is three lines — the traced command,Segmentation fault, exit 139 — with no pytest header, nocollected N items, no test ids and no faulthandler traceback. I couldn't reproduce this failures on my local systems.Three diagnostics, no behaviour change:
PYTHONFAULTHANDLER/PYTHONUNBUFFERED— names the exception and dumps every thread's stack, and unlike pytest's built-in plugin it reaches tests that spawn their ownpython -cchildren.--isolate— runs pytest in a child process so a surviving parent reports the raw NTSTATUS instead of the shell's translation of it (Git Bash renders an access violation asSegmentation fault/ 139). The child's status is passed through unchanged.pytest_crashlog— line-buffered progress log whose last line survives a native fault, localising it to interpreter startup, conftest import, a module's collection, a test, or shutdown. Opened beforepytest.main, since pytest imports the conftests — which pull incuda.core— beforepytest_configure. A no-op unlessPYTEST_CRASHLOGis set.