feat(cc): introduce py_extension rule for C/C++ Python extension modules - #3973
Draft
rickeylev wants to merge 88 commits into
Draft
feat(cc): introduce py_extension rule for C/C++ Python extension modules#3973rickeylev wants to merge 88 commits into
rickeylev wants to merge 88 commits into
Conversation
...and sycophantic toasters are bad collaborators
…into py-extension
This builds on the existing `py-extension` branch. It adds support for platform and abi tags in the resulting library filename. The `:py_extension_test`, `:py_extension_analysis_tests`, and `:py_limited_api_tests` test targets in `//tests/cc/py_extension` now build and pass. Relates to #3283
…tension (#3875) Per feedback from #3851, this PR re-works the platform detection logic to rely on platform constraints instead of the Python runtime/toolchain. ### Description This PR contains updates to the experimental `py_extension` implementation. It re-works the platform tag detection to rely on modern platform constraints, and refactors the compilation and linking mechanism to delegate to Bazel's native `cc_shared_library` and `cc_library` rules. This PR targets the main repository's `py-extension` branch (not `main`). ### Key Changes #### 1. Platform Tag & ABI Tag Derivation * **Introduced `abi_tag` to `PyCcToolchainInfo`**: Added the `abi_tag` field to the `PyCcToolchainInfo` provider, populated by the `py_cc_toolchain` rule. It defaults to deriving the tag from `python_version` (e.g. `cpython-311`) for backward compatibility. * **Constraint-Based Detection**: Replaced legacy parsing of `cc_toolchain` CPU names with direct lookup in rules_python's central `PLATFORMS` registry using platform constraints. * **Limited API Configuration**: Updated how limited API (`.abi3.so`) suffixes are appended based on the toolchain configuration. #### 2. Compilation & Linking Delegation (Refactor to `cc_shared_library`) * **Macro Fusion**: Replaced the custom linking logic in the `py_extension` rule with a macro of the same name. It now accepts C/C++ source/header files directly (`srcs`, `hdrs`, `copts`, `defines`), implicitly wrapping them in a private `cc_library` under the hood. * **Consolidated Attributes**: Consolidated static linkage under standard `deps` (removing the redundant `static_deps` attribute) and aligned linker arguments with `cc_shared_library`'s `user_link_flags`. * **Wrapper Rule (`_py_extension_wrapper`)**: Added a lightweight, private rule that wraps the `cc_shared_library` output to: * Query the Python toolchain for platform/ABI tags to resolve PEP 3149 compliant filenames (e.g. `.cpython-311-x86_64-linux-gnu.so` or `.abi3.so`). * Create a cheap `symlink` from the CSL output to the PEP 3149 name. * Propagate `PyInfo` (for python rules) and `CcSharedLibraryInfo` (for dynamic C++ dependencies). * **ODR Validation**: Leveraging native `cc_shared_library` ensures strict analysis-time validation against One Definition Rule (ODR) violations (e.g., duplicate static linkage in dynamic chains). #### 3. Runfiles and Data Support * **Runfiles Propagation**: Re-implemented runfiles collection using the repository's standard `builders.RunfilesBuilder` to support runtime assets (`data` attribute) and dynamic library dependency propagation. ### Verification & Testing * Added parity test cases in `dependency_graph_tests.bzl` to verify dynamic dependency chains and static sharing behavior. * Added runfiles validation in `py_extension_tests.bzl` to verify data asset propagation. * Updated limited API tests to match CSL output structure. * All 14 tests in `//tests/cc/py_extension/...` are compiling and passing. ────── TAG=agy CONV=23f8a5e8-2d99-401a-9903-1256b7d42b0e --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Why: Merge the latest upstream changes into py-extension to keep the branch up to date and compatible with Starlarkified rules_cc in Bazel 9. How: - Resolve conflict in dev/pyproject.toml and remove stale docs/requirements.txt. - Explicitly load CcSharedLibraryInfo from rules_cc. - Use cc_common.link without main_output to avoid private API restrictions, naming the intermediate library with a leading underscore and .pyextimpl suffix. - Add missing bzl_library targets and fix header includes in tests.
Why: Integrate upstream/py-extension (#3875) platform detection logic and cc_shared_library macro wrapper architecture into our local branch. How: - Resolve conflict in py_extension_rule.bzl in favor of upstream wrapper implementation. - Preserve runfiles dynamic lookup test in py_extension_test.py. - Fix function docstring header formatting in py_cc_toolchain_rule.bzl.
…e type annotations
Why:
Mark py_extension as a public but volatile API in documentation and improve docstring parameter type readability.
How:
- Add volatile API inclusion block to py_extension macro docstring.
- Cross-reference py_library.imports using {attr}.
- Add {type} annotations for all parameters and remove redundant Optional wording.
…3971) Per feedback from #3875 , this PR makes a few changes: * Removed an unnecessary hasattr check on py_cc_toolchain * Renamed the arbitrary "os" and "cpu" identifiers to terms standardized in PEP 508. Also marks `py_extension` as experimental. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Why: The py_cc_toolchain_rule.bzl loads flags.bzl, but :flags was missing from the deps attribute of //python/private:py_cc_toolchain_rule, breaking starlark doc extraction in docs build test. How: Add :flags to deps of //python/private:py_cc_toolchain_rule in python/private/BUILD.bazel.
…brary Why: py_cc_toolchain_macro.bzl loads //python/private/pypi:pep508_env.bzl, but the dependency was missing from the py_cc_toolchain_macro target in //python/private:BUILD.bazel. How: Add //python/private/pypi:pep508_env to deps of py_cc_toolchain_macro in python/private/BUILD.bazel.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
rickeylev
marked this pull request as draft
July 29, 2026 03:48
…D file Why: CI failed because tests/cc/py_extension/BUILD.bazel failed Buildifier checks and python/private/cc/BUILD.bazel referenced invalid bzl_library target names from @rules_cc. How: - Run buildifier to format tests/cc/py_extension/BUILD.bazel. - Update py_extension_macro and py_extension_rule bzl_library deps in python/private/cc/BUILD.bazel to use correct target names.
Why: WORKSPACE mode builds failed on Buildkite because bzlmod-only target names were used for rules_cc bzl_library dependencies. How: Use @rules_cc//cc:core_rules and @rules_cc//cc/common in python/private/cc/BUILD.bazel, which are supported in both WORKSPACE mode and bzlmod mode.
Why: Linking C extension shared libraries (.dylib) on macOS failed with undefined Python C symbols because ld requires -undefined dynamic_lookup when symbols are resolved at runtime. How: Add default user_link_flags with -undefined dynamic_lookup for @platforms//os:osx and @platforms//os:macos in py_extension_macro.bzl.
Why: GitHub actions CI failed on the ruff format check due to an extra blank line in py_extension_pkg_test.py. How: Reformat tests/cc/py_extension/py_extension_pkg_test.py using ruff format.
Collaborator
Author
|
cc @rsartor-cmd : something that you might run into integrating on the protobuf side: 65886db tldr, some flags necessary for building on mac are missing (that commit adds them) |
Why: py_extension_bzl is superseded by the py_extension bzl_library target in //python/cc:BUILD.bazel. How: Remove the py_extension_bzl bzl_library target definition from python/cc/BUILD.bazel.
Why: pyelftools is ELF-specific and fails when parsing Mach-O (.dylib on macOS) or PE (.dll on Windows) dynamic libraries. How: Decorate test_inspect_elf with @unittest.skipIf(sys.platform != 'linux', ...) in tests/cc/py_extension/py_extension_test.py.
…ames Why: Analysis tests in py_extension_tests.bzl and py_cc_toolchain_tests.bzl failed on macOS because extension file basenames and toolchain target platform tags vary across operating systems. How: Set explicit platform_machine and sys_platform on fake_py_cc_toolchain_impl in tests/support/cc_toolchains/BUILD.bazel, and use file_path_matches pattern matching for extension filenames in tests/cc/py_extension/py_extension_tests.bzl.
Why: - Remove copyright header from new test helper file per repository rules. - Add comments explaining -fPIC requirement, restored empty target comment, and CPython import library exports_filter behavior. How: - Strip copyright header in tests/cc/py_extension/py_extension_build_test.bzl. - Add explanatory comments for copts, empty target, and import library exports_filter in py_extension_macro.bzl.
Why: Formatting and comment styling adjustments in py_extension_macro.bzl per code review. How: - Format copts comment inside the list above -fPIC. - Rephrase empty target comment to 'cc_shared_library requires a dependency, so use an empty library when none are given'.
Why: Ensure explanation comment is not lost if alias targets are relocated. How: Duplicate explanatory comment above both current_py_cc_headers_private_alias and current_py_cc_libs_private_alias targets.
…_py_cc_libs Why: - Eliminate redundant user_link_flags $(locations) on Windows since CcInfo in deps automatically passes python3xx.lib to MSVC link.exe. - Remove manual depset flattening (linker_inputs.to_list()) and .dll string checking in current_py_cc_libs. How: - Remove $(locations @rules_python//python/cc:current_py_cc_libs) and additional_linker_inputs in py_extension_macro.bzl. - Simplify _current_py_cc_libs_impl to directly return py_cc_toolchain.libs.providers_map.values().
… current_py_cc_libs" This reverts commit 71df325.
Why: - Remove redundant @platforms//os:osx alias in select block. - Place platform-specific link flag comments directly above their respective select entries for clarity. How: - Move macOS comment above @platforms//os:macos entry in select. - Move Windows comment above @platforms//os:windows entry in select. - Remove @platforms//os:osx entry.
Why: Allow external rules and packages to load and depend on //python/cc:py_extension. How: Add visibility = ["//visibility:public"] to bzl_library(name = "py_extension") in python/cc/BUILD.bazel.
Why: - Update docstring in python/cc/py_extension.bzl to refer to Python C extension modules and link to official CPython C API guide. - Remove redundant explicit data arg in py_extension macro so kwargs propagates it. How: - Update docstring in python/cc/py_extension.bzl with Python C API link. - Remove data parameter, docstring entry, and conditional assignment in py_extension_macro.bzl.
Why: Clarify module docstring scope per code review. How: Change docstring in python/private/cc/py_extension_rule.bzl to 'Implementation of rules supporting py_extension'.
Why: - Address code review feedback on py_extension macro/rule arguments and docstrings. - Standardize toolchain type constants and imports processing. - Remove obsolete _bzl targets managed by Gazelle. How: - Add local_defines and additional_linker_inputs to py_extension macro. - Remove linkshared, linkstatic, and visibility explicit args from py_extension macro. - Use CC_TOOLCHAIN_TYPE constant and get_imports helper in py_extension_rule. - Remove _bzl targets from python/private/cc/BUILD.bazel.
Why: - Align py_cc_toolchain with py_runtime by supporting abi_flags resolution. - Move libc default resolution into py_cc_toolchain rule implementation. - Use LibcFlag constants across cc toolchain macros/rules. How: - Add abi_flags attribute to py_cc_toolchain rule and provider field to PyCcToolchainInfo. - Add get_value helper method to LibcFlag in flags.bzl. - Move libc resolution from py_cc_toolchain_macro to py_cc_toolchain_rule.
Why: - Organize py_extension starlark test suites into dedicated subpackage directories. How: - Create dependency_graph, py_extension, and py_limited_api subdirectories under tests/cc/py_extension. - Move test suites and associated C test sources into subdirectories without Bazel copyright headers in new BUILD.bazel files. - Set explicit subpackage visibility on required target fixtures in tests/cc/py_extension/BUILD.bazel.
Why: - Remove visibility requirements from targets in tests/cc/py_extension/BUILD.bazel. How: - Update py_extension_tests.bzl to create helper targets locally using util.helper_target. - Remove visibility attributes from targets in tests/cc/py_extension/BUILD.bazel.
Why: - Restore target visibility on py_extension test targets in tests/cc/py_extension/BUILD.bazel. - Keep py_extension_tests.bzl simple by pointing directly to the build test targets. How: - Add visibility = ["//tests/cc/py_extension:__subpackages__"] to ext_static, ext_with_data, and ext_shared in tests/cc/py_extension/BUILD.bazel. - Remove exports_files declaration. - Revert py_extension_tests.bzl to target //tests/cc/py_extension targets directly.
Why: - Use package(default_visibility = ...) instead of setting visibility individually on targets. How: - Add package(default_visibility = ["//tests/cc/py_extension:__subpackages__"]) in tests/cc/py_extension/BUILD.bazel. - Remove target-level visibility attributes.
Why: - current_py_cc_libs is already added to deps of the internal cc_library on Windows. - CcInfo automatically propagates the CPython import libraries from deps to cc_shared_library. How: - Remove and additional_linker_inputs from py_extension_macro.bzl.
Remove unused libc parameter and attribute from py_extension macro and wrapper rule since platform tag resolution is handled by py_cc_toolchain. Use PyInfoBuilder in py_extension_wrapper rule implementation. Fix string formatting in error message and simplify libc tag mapping in py_cc_toolchain_rule. Clean up hasattr guards in dependency_graph_tests.bzl.
Collect test functions into a global _tests list as they are defined and pass the list to test_suite.
…shared_library deps and exports_filter on Windows On Windows MSVC, link.exe requires python3.lib/python3xx.lib when linking extension DLLs because MSVC embeds /DEFAULTLIB:python3.lib into object files. Passing current_py_cc_libs in cc_shared_library deps and default exports_filter on Windows ensures the import library is included on the link command line.
Document LNK1104, Bzlmod exports_filter, PyMODINIT_FUNC dllexport, LNK1107, and current_py_cc_libs propagation solutions.
…ion_macro Why: - Avoid duplicate label errors in cc_shared_library deps when srcs/hdrs are omitted. - Ensure exports_filter uses canonical Bzlmod Label strings so current_py_cc_libs is matched and retained by link.exe on Windows. How: - Wrap alias target labels with str(Label(...)) for canonical string expansion. - Only append py_cc_libs_win_alias to csl_deps_with_win when srcs or hdrs are provided.
…dows
Why:
- Bazel dereferences alias rules during cc_shared_library dependency processing.
- exports_filter matching failed because it contained the alias target label rather than the dereferenced actual target label (//python/cc:current_py_cc_libs), causing current_py_cc_libs to be pruned and link.exe to fail with LNK1104.
How:
- Add str(Label("//python/cc:current_py_cc_libs")) to win_exports_filter in py_extension_macro.bzl.
… in exports_filter
Define _PY_CC_HEADERS_ALIAS, _PY_CC_LIBS_ALIAS, and _PY_CC_LIBS_TARGET as uppercase module constants.
Evaluating str(Label(...)) at top-level module load time bakes in @@//... instead of caller's canonical Bzlmod repository context (e.g. @@rules_python+...). Evaluating Label inside def py_extension(...) ensures exports_filter matches cc_shared_library canonical target labels on Windows.
Why:
- Matching exact target labels in exports_filter across Bzlmod repository canonical names can fail if the dereferenced target label format differs from the macro stringification.
How:
- Use str(Label("//python/cc:__subpackages__")) and str(Label("//python/private/cc:__subpackages__")) in win_exports_filter.
Why: - Adding restrictive package label strings to exports_filter prevented matching external toolchain targets (e.g. @@+python+python_3_11_15...//:python_lib). How: - Pass csl_deps_with_win directly to exports_filter when exports_filter is None.
…ilter Why: - Passing str(Label(...)) pre-evaluated the label string as @@//... instead of letting rules_cc resolve string literals inside the target caller's Bzlmod context (@@rules_python+...). How: - Use plain string literals "//python/cc:current_py_cc_libs" and "//python/private/cc:current_py_cc_libs_private_alias" in exports_filter.
Why:
- CPython import libraries (python3xx.lib) are imported with system_provided = True in cc_import, which causes Bazel's CcInfo provider to suppress propagating the .lib path to link.exe.
How:
- Pass in user_link_flags and Label("//python/cc:current_py_cc_libs") in additional_linker_inputs on Windows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(cc): introduce py_extension rule for C/C++ Python extension modules
Building C/C++ Python extension modules currently requires manually
configuring complex
cc_binaryorcc_shared_librarytargets andsetting up platform, ABI3, and extension tags. The new
py_extensionrule provides a high-level Starlark API for compilingnative extensions that integrate seamlessly with
rules_pythontoolchains and
py_librarydependencies.Closes #3283.