Princeton RSE Peer network first meeting
0 likes296 views
The document summarizes Henry Schreiner's work on several Python and C++ scientific computing projects. It describes a scientific Python development guide built from the Scikit-HEP summit. It also outlines Henry's work on pybind11 for C++ bindings, scikit-build for building extensions, cibuildwheel for building wheels on CI, and several other related projects.
 { return i - j; });
}
BONUS project: Iâm not a maintainer, but help out occasionally](https://image.slidesharecdn.com/projectsreport-henryiii-231113194913-485c3b2b/85/Princeton-RSE-Peer-network-first-meeting-10-320.jpg)
![scikit-build
CMake - Python bridge for building
extensions
50% of my time for three years (NSF
grant)
Introduced scikit-build-core last year:
modern, static-conïŹg backend
Over 200 users now (vs. ~900
scikit-build classic users)
Family of tools: scikit-build (classic),
scikit-build-core, cmake, ninja,
moderncmakedomain, etc.
[build-system]
requires = ["scikit-build-core"]
build-backend = "scikit_build_core.build"
[project]
name = "example"
version = "0.0.1"
cmake_minimum_required(VERSION 3.guatda.com/cmx.p15...3.27)
project(example LANGUAGES C)
find_package(Python REQUIRED COMPONENTS
Interpreter Development.Module REQUIRED)
Python_add_library(_module MODULE src/module.c WITH_SOABI)
install(TARGETS _module DESTINATION example)
pyproject.toml
CMakeLists.txt](https://image.slidesharecdn.com/projectsreport-henryiii-231113194913-485c3b2b/85/Princeton-RSE-Peer-network-first-meeting-11-320.jpg)
![cibuildwheel
Builds wheels on CI and locally
CPython 3.6-3.12, PyPy 3.7-3.10
macOS, Windows, manylinux, musllinux
All OSâs: 64-bit Intel, 32-bit Intel, ARM
Linux: ppc64le & s390x too (emulated)
Tested on GHA, Azure, Travis, AppVeyor,
Circle CI, GitLab CI, Cirrus CI
ConïŹg via env or pyproject.toml
Can run tests after building wheel
See my blog for recipe to auto-generate
build matrix!
on: [push, pull_request]
jobs:
build_wheels:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
steps:
- uses: actions/checkout@v4
- uses: pypa/cibuildwheel@v2.16.2
- uses: actions/upload-artifact@v3
with:
path: ./wheelhouse/*.whl
.github/workflows/wheel.yml
pyproject.toml
[tool.cibuildwheel]
test-extras = ["test"]
test-command = "pytest {project}/tests"](https://image.slidesharecdn.com/projectsreport-henryiii-231113194913-485c3b2b/85/Princeton-RSE-Peer-network-first-meeting-14-320.jpg)
Princeton RSE Peer network first meeting
- 1. Projects report - @henryiii Henry Schreiner - 11-13-2023 https://iscinumpy.dev
- 2. ScientiïŹc-Python Development Guide Built from the Scikit-HEP work at the 2023 ScientiïŹc-Python Developerâs Summit Merged with cookie & sp-repo-review from Scikit-HEP (later slides) Over a dozen in-depth guide page on topics like producing compiled wheels in GitHub Actions Kept up to date with the latest recommendations (like PyPI Trusted Publishers and ruff-format tabs) Recently featured on Python Bytes and The Real Python Podcast!
- 3. ScientiïŹc-Python Development Guide Built from the Scikit-HEP work at the 2023 ScientiïŹc-Python Developerâs Summit Merged with cookie & sp-repo-review from Scikit-HEP (later slides) Over a dozen in-depth guide page on topics like producing compiled wheels in GitHub Actions Kept up to date with the latest recommendations (like PyPI Trusted Publishers and ruff-format tabs) Recently featured on Python Bytes and The Real Python Podcast!
- 4. ScientiïŹc-Python Development Guide Built from the Scikit-HEP work at the 2023 ScientiïŹc-Python Developerâs Summit Merged with cookie & sp-repo-review from Scikit-HEP (later slides) Over a dozen in-depth guide page on topics like producing compiled wheels in GitHub Actions Kept up to date with the latest recommendations (like PyPI Trusted Publishers and ruff-format tabs) Recently featured on Python Bytes and The Real Python Podcast!
- 5. ScientiïŹc-Python cookie pipx run cookiecutter gh:scientific-python/cookie Optional, but learn pipx if you donât already use it daily! Or copier, or cruft
- 6. ScientiïŹc-Python cookie Over 10 build backends, including compiled C++ or Rust Selection of license and VCS versioning (most backends) In-sync with Development Guide (some parts actually rendered into the guide) Generation tested with nox in CI Pre-commit & GHA updated weekly
- 7. repo-review & sp-repo-review repo-review: unbiased Python 3.10+ framework for checking repo conïŹguration (no checks) sp-repo-review: plugin with checks based on the dev guide (part of the same repo, like cookie) Checks are linked to matching badges throughout the guide Runs in WebAssembly too, so embedded directly in the guide too! Validate-pyproject also has plugin
- 8. Iâm working on the Python & C++ development story And a lot has and is happeningâŠ
- 9. Overall Picture Iâm working on each part. Bindings to CPython / PyPy Build system C++ code Binary Redistributable binaries Build tool(s) Build conïŹguration
- 10. pybind11 (& nanobind) The âmissingâ C++ bindings for CPython/PyPy Header-only, zero dependency Supports NumPy without the headers Great CMake & Meson support Three example repos (setuptools, CMake, scikit-build-core build systems) Extensive CI testing & static checks Used by SciPy, PyTorch, Google, etc. C++17, CPython 3.8+ Only simple C++ constructs (design for bindings) Limited API / Stable ABI 3.12+ #include <pybind11/pybind11.h> namespace py = pybind11; int add(int i, int j) { return i + j; } PYBIND11_MODULE(python_example, m) { m.def("add", &add); m.def("subtract", [](int i, int j) { return i - j; }); } BONUS project: Iâm not a maintainer, but help out occasionally
- 11. scikit-build CMake - Python bridge for building extensions 50% of my time for three years (NSF grant) Introduced scikit-build-core last year: modern, static-conïŹg backend Over 200 users now (vs. ~900 scikit-build classic users) Family of tools: scikit-build (classic), scikit-build-core, cmake, ninja, moderncmakedomain, etc. [build-system] requires = ["scikit-build-core"] build-backend = "scikit_build_core.build" [project] name = "example" version = "0.0.1" cmake_minimum_required(VERSION 3.guatda.com/cmx.p15...3.27) project(example LANGUAGES C) find_package(Python REQUIRED COMPONENTS Interpreter Development.Module REQUIRED) Python_add_library(_module MODULE src/module.c WITH_SOABI) install(TARGETS _module DESTINATION example) pyproject.toml CMakeLists.txt
- 12. scikit-build-core: features Powerful conïŹg system Impl: declarative dataclasses + transformer - Pyproject, conïŹg-settings, and env - Generates JSONSchema - Generates readme docs Overrides (like cibuildwheel & mypy) Automatic cmake/ninja Only adds the wheels as build dependencies if required (not on system already) - Supports WebAssembly, BSD, etc. - Can set min CMake version required Features hard with setuptools Conditional Stable ABI / Limited ABI macOS compatibility tags Write to optional wheel dirs (ex: headers) Experimental editable mode, with auto-rebuild Extra goodies Auto inclusion of site-packages in preïŹx path Entry points for custom CMake modules FindPython auto-backport Dynamic metadata plugin system
- 13. build pipx run build The oïŹcial replacement for python setup.py sdist and pip wheel . Released 1.0! Supports any backend
- 14. cibuildwheel Builds wheels on CI and locally CPython 3.6-3.12, PyPy 3.7-3.10 macOS, Windows, manylinux, musllinux All OSâs: 64-bit Intel, 32-bit Intel, ARM Linux: ppc64le & s390x too (emulated) Tested on GHA, Azure, Travis, AppVeyor, Circle CI, GitLab CI, Cirrus CI ConïŹg via env or pyproject.toml Can run tests after building wheel See my blog for recipe to auto-generate build matrix! on: [push, pull_request] jobs: build_wheels: runs-on: ${{ matrix.os }} strategy: matrix: os: [ubuntu-latest, windows-latest, macos-latest] steps: - uses: actions/checkout@v4 - uses: pypa/cibuildwheel@v2.16.2 - uses: actions/upload-artifact@v3 with: path: ./wheelhouse/*.whl .github/workflows/wheel.yml pyproject.toml [tool.cibuildwheel] test-extras = ["test"] test-command = "pytest {project}/tests"
- 15. My other projects github.com/henryiii iscinumpy.dev scikit-hep.org Boost-histogram, Hist, UHI, and uproot-browser (Histograms in Python) CLI11: C++âs most powerful argument parser