# Seamless operability between C++11 and Python

pybind11 — Seamless operability between C++11 and Python

Warning

Combining older versions of pybind11 (< 2.6.0) with Python 3.9.0 will trigger undefined behavior that typically manifests as crashes during interpreter shutdown (but could also destroy your data. You have been warned.)

We recommend that you update to the latest patch release of Python (3.9.1), which includes a fix that resolves this problem. If you do use Python 3.9.0, please update to the latest version of pybind11 (2.6.0 or newer), which includes a temporary workaround specifically when Python 3.9.0 is detected at runtime.

pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. Its goals and syntax are similar to the excellent Boost.Python library by David Abrahams: to minimize boilerplate code in traditional extension modules by inferring type information using compile-time introspection.

The main issue with Boost.Python—and the reason for creating such a similar project—is Boost. Boost is an enormously large and complex suite of utility libraries that works with almost every C++ compiler in existence. This compatibility has its cost: arcane template tricks and workarounds are necessary to support the oldest and buggiest of compiler specimens. Now that C++11-compatible compilers are widely available, this heavy machinery has become an excessively large and unnecessary dependency.

Think of this library as a tiny self-contained version of Boost.Python with everything stripped away that isn’t relevant for binding generation. Without comments, the core header files only require ~4K lines of code and depend on Python (2.7 or 3.5+, or PyPy) and the C++ standard library. This compact implementation was possible thanks to some of the new C++11 language features (specifically: tuples, lambda functions and variadic templates). Since its creation, this library has grown beyond Boost.Python in many ways, leading to dramatically simpler binding code in many common situations.

Tutorial and reference documentation is provided at pybind11.readthedocs.io. A PDF version of the manual is available here. And the source code is always available at github.com/pybind/pybind11.

## Core features

pybind11 can map the following core C++ features to Python:

• Functions accepting and returning custom data structures per value, reference, or pointer
• Instance methods and static methods
• Instance attributes and static attributes
• Arbitrary exception types
• Enumerations
• Callbacks
• Iterators and ranges
• Custom operators
• Single and multiple inheritance
• STL data structures
• Smart pointers with reference counting like std::shared_ptr
• Internal references with correct reference counting
• C++ classes with virtual (and pure virtual) methods can be extended in Python

## Goodies

In addition to the core functionality, pybind11 provides some extra goodies:

• Python 2.7, 3.5+, and PyPy/PyPy3 7.3 are supported with an implementation-agnostic interface.
• It is possible to bind C++11 lambda functions with captured variables. The lambda capture data is stored inside the resulting Python function object.
• pybind11 uses C++11 move constructors and move assignment operators whenever possible to efficiently transfer custom data types.
• It’s easy to expose the internal storage of custom data types through Pythons’ buffer protocols. This is handy e.g. for fast conversion between C++ matrix classes like Eigen and NumPy without expensive copy operations.
• pybind11 can automatically vectorize functions so that they are transparently applied to all entries of one or more NumPy array arguments.
• Python’s slice-based access and assignment operations can be supported with just a few lines of code.
• Everything is contained in just a few header files; there is no need to link against any additional libraries.
• Binaries are generally smaller by a factor of at least 2 compared to equivalent bindings generated by Boost.Python. A recent pybind11 conversion of PyRosetta, an enormous Boost.Python binding project, reported a binary size reduction of 5.4x and compile time reduction by 5.8x.
• Function signatures are precomputed at compile time (using constexpr), leading to smaller binaries.
• With little extra effort, C++ types can be pickled and unpickled similar to regular Python objects.

## Supported compilers

1. Clang/LLVM 3.3 or newer (for Apple Xcode’s clang, this is 5.0.0 or newer)
3. Microsoft Visual Studio 2015 Update 3 or newer
4. Intel classic C++ compiler 18 or newer (ICC 20.2 tested in CI)
5. Cygwin/GCC (previously tested on 2.5.1)
6. NVCC (CUDA 11.0 tested in CI)
7. NVIDIA PGI (20.9 tested in CI)

This project was created by Wenzel Jakob. Significant features and/or improvements to the code were contributed by Jonas Adler, Lori A. Burns, Sylvain Corlay, Eric Cousineau, Ralf Grosse-Kunstleve, Trent Houliston, Axel Huebl, @hulucc, Yannick Jadoul, Sergey Lyskov Johan Mabille, Tomasz Miąsko, Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim Schellart, Henry Schreiner, Ivan Smirnov, Boris Staletic, and Patrick Stewart.

We thank Google for a generous financial contribution to the continuous integration infrastructure used by this project.

### Contributing

See the contributing guide for information on building and contributing to pybind11.

pybind11 is provided under a BSD-style license that can be found in the LICENSE file. By using, distributing, or contributing to this project, you agree to the terms and conditions of this license.

###### pybind
Seamless operability between C++ and Python
• #### error_already_set::what() is now constructed lazily

The original motivation for this PR was a performance optimization. See the original PR description below.

After nearly 3 years, work on this PR has led to a few related but distinct behavior changes:

• Already merged PR #3971 — Fixes formatting of messages for originally unnormalized Python errors.

This PR:

• Secondary errors during normalization of the Python error are clearly reported (see test_flaky_exception_failure_point_init in test_exceptions.py).
• Secondary errors obtaining the error message are clearly reported (see test_flaky_exception_failure_point_str in test_exceptions.py).
• Calling error_already_set::restore() multiple times is clearly reported (preempts accidentally clearing an error, or accidentally restoring the same error twice; see test_error_already_set_double_restore in test_exceptions.py).
• The error_already_set copy constructor is no longer racy.
• detail::error_string() no longer restores the error message. This fixes (in practice inconsequential) bugs in detail/class.h.

The performance gain for exceptions with long tracebacks can be very substantial. See https://github.com/pybind/pybind11/pull/1895#issuecomment-1127700599 for concrete numbers.

A review of the error_already_set development history is under https://github.com/pybind/pybind11/pull/1895#issuecomment-1134206370.

# Original PR description below

Prior to this PR throwing error_already_set was expensive due to the eager construction of the error string. The PR changes error_already_set::what to construct the error string on first call.

Before:

1.7 µs ± 9.03 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

After:

1.14 µs ± 1.93 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Benchmark:

const py::int_ foo() {
py::dict d;
try {
return d["foo"];
} catch (const py::error_already_set &) {
return py::int_(42);
}
}

PYBIND11_MODULE(foo, m) {
m.def("foo", &foo);
}


Note that the running time of the benchmark is dominated by stack unwinding, therefore the speedup is only ~30%.

## Suggested changelog entry:

error_already_set is now safer and more performant, especially for exceptions with long tracebacks.


## Description

Eigen::Tensor allows you to handle np.arrays with more than two dimensions, which is quite handy in many cases.

I also added tests for all the relevant behavior.

Note that the Eigen::TensorMap implementation could be better if we allow std::optional, but I'm not sure if we are allowed to require that in this codebase.

Closes #1377 Closes #2119

## Suggested changelog entry:

pybind11/eigen/tensor.h adds converters to and from Eigen::Tensor and Eigen::TensorMap


## Description

This PR adds a PYBIND11_SIMPLE_GIL_MANAGEMENT option (cmake, C++ define). The new option may be useful to try when debugging GIL-related issues, to determine if the more complex default implementation is or is not to blame. See comments here for background.

This PR was triggered by https://github.com/pytorch/pytorch/issues/83101. While we could not reproduce the original problem in a unit test, many tests were added to test_gil_scoped.py trying to do so. That work exposed that test_gil_scoped.py has been finding deadlocks for a long time (years), but those were thus far ignored as "flakes". test_gil_scoped.py was updated to make it crystal clear when a DEADLOCK is detected. It turns out deadlocks are detected at a fairly high rate (ballpark 1 in 1000). This needs to be worked on separately. To not continue to pollute the CI results in the meantime, SKIP_IF_DEADLOCK = True was added in test_gil_scoped.py. The GitHub Actions CI logs can be harvested in the future to determine what platforms exactly are affected, and at what rate. Hopefully the information accumulating over time will lead to a better understanding and fixes, so that SKIP_IF_DEADLOCK can be removed eventually.

WARNING: Please be careful to not create ODR violations when using the new option: everything that is linked together with mutual symbol visibility needs to be rebuilt.

## Suggested changelog entry:

* A PYBIND11_SIMPLE_GIL_MANAGEMENT option was added (cmake, C++ define), along with many additional tests in test_gil_scoped.py. The option may be useful to try when debugging GIL-related issues, to determine if the more complex default implementation is or is not to blame. See #4216 for background. WARNING: Please be careful to not create ODR violations when using the option: everything that is linked together with mutual symbol visibility needs to be rebuilt.

• #### Support NVidia HPC SDK (PGI)

In this PR:

• Support NVidia HPC SDK with PG compilers
• Factor out CMake filter function
• Allow CMake to build even if pytest is missing (still shows warning)

Closes #2214 and closes #2407.

• #### WIP: PyPy support

This commit includes a few minor modifications to pybind11 that are needed to get simple hello-world style functions to compile and run on the latest PyPy. Types are now supported as well.

The test suite compiles but crashes when executed.

• #### Intel ICC C++17 compatibility

• ICC can’t handle complex enable if’s inline, like enable_if_t<all_of<is_positional<Args>...>::value>>, but if it is refactored into a separate constexpr function, it works.
• But MSVC doesn't like it if the function is in the return type.
• Fold expressions cause an internal compiler error, so we have to avoid turning them on even in C++17 mode
• Plain noexcept can cause weird errors with following syntax, noexcept(true) works fine, workaround in tests

This also fixes the NVIDIA HPC SDK compiler in C++17 mode.

## Description

The Intel compiler has trouble with pybind11 when compiling in C++17 mode. The first two examples in the code below result in errors like "no instance of function template [...] matches the argument list", while the third one triggers an internal compiler error.

#include <pybind11/pybind11.h>

void test1() {
pybind11::print("Hello World");
}

void test2(pybind11::object obj) {
obj();
}

class C {
};

void test3(pybind11::module_ &m) {
pybind11::class_<C>(m, "C");
}


Details can be found in #2707. @tobiasleibner implemented the fix for the template mismatch, while the workaround for the internal compiler error is mine. Part of @tobiasleibner's work went into #2573 already, which hasn't been merged yet. Since he currently doesn't have time, he suggested I post a pull request.

Related to #2573 Fixes #2714 Fixes #2707

## Suggested changelog entry:

* Support ICC and NVIDIA HPC SDK in C++17 mode

• #### CMake installation path consistency

Problem:

1. In a given installation PREFIX, pip install pybind11 or conda install pybind11 will install pybind11 headers into a new pybind11 subdirectory of the python include directory, while cmake + make install results in pybind11 headers being put into PREFIX/include/pybind11.

For example, with a conda installation, pip will give

PREFIX/include/pythonX.Y/pybind11 while cmake yields PREFIX/include/pybind11.

2. It would be nice to have a consistent installation scheme between the two methods, plus the benefit of having the cmake project config files being made available, in the context of conda.

Solution:

This changes the destination of the headers of pybind into PYTHON_INCLUDE_DIRS/pybind11.

The generated project config cmake files are installed in the same location as before, but reflect the new location of the headers.

• #### CI: Intel icc/icpc via oneAPI

Add testing for Intel icc/icpc via the oneAPI images. Intel oneAPI is in a late beta stage, currently shipping oneAPI beta09 with ICC 20.2.

We are adding // [workaround(intel)] near the workarounds to it will be easy to search them out later, and disable them to see if they have been fixed by the Intel compiler team.

Here are the workarounds here:

• py::args() -> py::args{}, in tests, regression in ICC 20+, workaround in tests
• = default instead of {} doesn’t always work (was a recent modernization, so might have been a problem before too), workaround in tests

## Suggested changelog entry:

* Support Intel OneAPI compiler (ICC 20.2) and add to CI.

• #### WIP: Multiple inheritance support

This is an early snapshot of a potential way to support multiple inheritance in a minimally intrusive way. Standard single inheritance is supported using a fast fallback that matches the current implementation.

Somewhat surprisingly, the change and related simplifications actually reduce the size of generated binaries by 4.4% :).

There are probably a ton of cases that I didn't think of, so this is up for public review now.

• #### Fix cmake configuration when lto flags not available

CMake current fails to run with Clang on linux:

$CC=clang CXX=clang++ cmake .. ... (various configuration) ... -- Configuring done CMake Error at tools/pybind11Tools.cmake:135 (target_compile_options): Error evaluating generator expression:$<:-flto>

Expression did not evaluate to a known generator expression
Call Stack (most recent call first):


This commit fixes the generator expressions when the lto flags are not found (they will be empty strings, but the generator expression needs a 0).

#### Motivation

Acquiring the pybind11 repo (for large projects where the pybind11_add_module isn't appropriate) is admittedly not hard either as a git submodule or as a CMake ExternalProject plus writing a Findpybind11.cmake file to search out the header location. But for the ExternalProject case, the better CMake route is to provide a pybind11Config.cmake in the installation so that consuming projects can use pre-built (installed) pybind11 without writing their own detection.

#### PR Changes

• [x] Builds a minimal interface library target (pb11) and exports it so that pybind11Config.cmake and pybind11Targets.cmake are installed along with the headers. Instructions for detection and use given in tools/pybind11Config.cmake.in
• [x] Allows passing -DCMAKE_INSTALL_INCLUDEDIR to specify non-standard install location. Defaults to include. Note that this isn't affecting the pybind11_add_module function since it's not exporting interface include dirs.

#### Questions

• [x] Used pybind11::pybind11 for imported target name. If you want any other capitalization pattern for the project name, let me know and I'll adjust everything.
• [x] I've hard-coded the version at 1.8, but the write_basic_package_version_file(...) four lines and the ConfigVersion.cmake installation can be commented out, if you don't want to deal with versioning in development mode.
• I'm medium proficient at CMake, so please let me know if you see any holes in the procedure, would like some edits, or want the PR split.

#### CMake Version

• Checked that GNUInstallDirs and CMakePackageConfigHelpers present at 2.8.12 (https://gitlab.kitware.com/cmake/cmake/blob/v2.8.12/Modules/)
• Checked that the \$<INSTALL_INTERFACE generator present at 2.8.12 (https://cmake.org/cmake/help/v2.8.12/cmake.html#section_Generators)
• Couldn't find a 2.8.12 to install, but can confirm the exported target exports at 3.0.1

## Description

This PR actually matters

• only for Python 3.6 (according to the Python documentation),
• only with -DPYBIND11_SIMPLE_GIL_MANAGEMENT=ON (evidently, based on CI flakes observed since PR #4216 was merged),
• and only if embed.h is used.

Hypothesis: -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF covers up that PyEval_InitThreads() was misplaced.

This PR almost does not matter anymore at all:

Changed in version 3.7: This function is now called by Py_Initialize(), so you don’t have to call it yourself anymore.

Note that Python 3.6 EOL was 2021-12-23 (https://endoflife.date/python), but since PR #4216 was merged we had distracting CI flakes that this PR aims to get rid of, e.g.:

• 🐍 3.6 • windows-2022 • x64
Run cmake --build .  --target cpptest -j 2
MSBuild version 17.3.1+2badb37d1 for .NET Framework
Checking File Globs
external_module.vcxproj -> D:\a\pybind11\pybind11\tests\test_embed\external_module.cp36-win_amd64.pyd
test_embed.vcxproj -> D:\a\pybind11\pybind11\tests\test_embed\Debug\test_embed.exe
CUSTOMBUILD : Fatal Python error : This thread state must be current when releasing [D:\a\pybind11\pybind11\tests\test_embed\cpptest.vcxproj]

Current thread 0x0000[14](https://github.com/pybind/pybind11/actions/runs/3511087576/jobs/5881522679#step:13:15)84 (most recent call first):

Thread 0x00000fc4 (most recent call first):
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(247,5): error MSB8066: Custom build for 'D:\a\pybind11\pybind11\CMakeFiles\bf7d1[16](https://github.com/pybind/pybind11/actions/runs/3511087576/jobs/5881522679#step:13:17)f1[18](https://github.com/pybind/pybind11/actions/runs/3511087576/jobs/5881522679#step:13:19)90487c30aedc390ae4de7\cpptest.rule;D:\a\pybind11\pybind11\tests\test_embed\CMakeLists.txt' exited with code -1073740791. [D:\a\pybind11\pybind11\tests\test_embed\cpptest.vcxproj]
Error: Process completed with exit code 1.


## Suggested changelog entry:

Bug fix affecting only Python 3.6 under very specific, uncommon conditions: correction of misplaced PyEval_InitThreads() call.

• #### Change include_guard to check target only

include_guard seems to break all contained logic to handle use case of building for multiple Python versions.

## Description

Use case: trying to create multiple targets for different Python versions

Issue: some variables are not updated on subsequent targets, most noticeably the SUFFIX, resulting in (as example) all .pyd files being named .cp39-win_amd64.pyd (despite linking against 3.9, 3.10, etc.)

There seems to be a lot of logic to handle this case on subsequently entering this file, however a fix (likely related to changed in Conan) added an include guard to prevent errors about creating the same target twice. This also sabotages all of that above logic, since it will not execute another time and nothing gets updated after switching Python versions.

Removing the include_guard and instead only create the target if it doesn't exist already seems to fix that. However, someone with deeper understanding of the dependencies and flows should verify that this won't have unexpected and fatal side effects.

### Discussed in https://github.com/pybind/pybind11/discussions/4331

Originally posted by Tomcat-42 November 14, 2022 First of all, Hello friends!!! :wave:

Despite knowing this is more of a pip build systems issue rather than a pybind itself one, I think someone here might have encountered a simillar issue and can help me.

I have two separate repositories: libdaa and daa . The first one, libdaa , is a c++ code base that generates various targets. One of those targets, daa.so, is a pybind11 python module. I can build this target using CMake (or plain make), and use daa.so correctly as a python module. The next picture shows libdaa repo:

The second repo, daa, is intended to be a pip package that wraps daa.so (built on libdaa repo) and will be published on PyPi. Here, I have libdaa repo as a submodule on daa/libdaa. The picture shows this:

Said that, how can I configure daa repo for building the daa.so target that is inside the daa/libdaa repo as a python package?

Thanks, Pablo.

## Description

In some cases it is convenient to have an opportunity to construct a scoped_interpreter with a custom PyConfig instance supplied.

PyConfig my_config{};
//...
pybind11::scoped_interpreter{my_config}


This PR intends to query if such a feature is something the maintainers would support and accept (after the feature is properly implemented).

## Suggested changelog entry:

* overloaded constructor for scoped_interpreter

###### Structy is an irresponsibly dumb and simple struct serialization/deserialization library for C, Python, and vanilla JavaScript.

Structy Structy is an irresponsibly dumb and simple struct serialization/deserialization library for C, Python, and vanilla JavaScript. You can think

Sep 13, 2022
###### Import C++ files directly from Python!

If you've used cppimport version 0.0.*, some new features for you! Compiler arguments, multiple source files, bug fixes! Read on. Import C or C++ file

Nov 16, 2022
###### The most widely used Python to C compiler

Welcome to Cython! Cython is a language that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports mo

Nov 16, 2022
###### Sol3 (sol2 v3.0) - a C++ <-> Lua API wrapper with advanced features and top notch performance - is here, and it's great! Documentation:

sol2 sol2 is a C++ library binding to Lua. It currently supports all Lua versions 5.1+ (LuaJIT 2.0+ and MoonJIT included). sol2 aims to be easy to use

Nov 25, 2022
###### Libraries and examples to support Pimoroni Pico add-ons in C++ and MicroPython.

Pimoroni Pico Libraries and Examples Welcome to the brave new world of Pico! This repository contains the C/C++ and MicroPython libraries for our rang

Nov 24, 2022
###### Tools and libraries to glue C/C++ APIs to high-level languages

CppSharp is a tool and set of libraries which facilitates the usage of native C/C++ code with the .NET ecosystem. It consumes C/C++ header and library

Nov 23, 2022
###### A tool for generating cross-language type declarations and interface bindings.

Djinni Djinni is a tool for generating cross-language type declarations and interface bindings. It's designed to connect C++ with either Java or Objec

Nov 19, 2022
###### Duktape - embeddable Javascript engine with a focus on portability and compact footprint

Duktape ⚠️ Master branch is undergoing incompatible changes for Duktape 3.x. To track Duktape 2.x, follow the v2-maintenance branch. Introduction Dukt

Nov 16, 2022
###### SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages.

SWIG (Simplified Wrapper and Interface Generator) Version: 4.1.0 (in progress) Tagline: SWIG is a compiler that integrates C and C++ with languages

Nov 25, 2022
###### A minimalist and mundane scripting language.

Drift Script A minimalist and mundane scripting language. I like all simple things, simple and beautiful, simple and strong. I know that all developme

Nov 14, 2022
###### Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS) Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and d

Nov 4, 2022
###### DotX64Dbg aims to provide a seamless way to write and test plugins for X64Dbg using .Net 5.0 and C#.

DotX64Dbg (EARLY ALPHA) Plugins and Scripting with C# for x64Dbg. Create Plugins for X64Dbg with ease DotX64Dbg aims to provide a seamless way to writ

Oct 16, 2022
###### A seamless, gasketmounted QAZ-like 40% keyboard

QEZ A seamless, gasketmounted QAZ-like 40% keyboard STEP files for the case KiCAD PCB files Gerbers for PCB production JLCPCB BOM JLCPCB CPL Vial-QMK

Nov 18, 2022
###### Inter-process communication library to enable allocation between processes/threads and send/receive of allocated regions between producers/consumer processes or threads using this ipc buffer.

This is a relatively simple IPC buffer that allows multiple processes and threads to share a dynamic heap allocator, designate "channels" between processes, and share that memory between producer/consumer pairs on those channels.

Aug 20, 2022
###### A Binary Clock. Written 3 different ways. C and SDL, Python and PyGame, Python and PyGame Zero.

Super Clock A Binary Clock. Written 3 different ways. Python with PyGame Zero, Python with PyGame and C with SDL2. Time is displayed in 3 groups of 8

Dec 8, 2021
###### BLLIP reranking parser (also known as Charniak-Johnson parser, Charniak parser, Brown reranking parser) See http://pypi.python.org/pypi/bllipparser/ for Python module.

BLLIP Reranking Parser Copyright Mark Johnson, Eugene Charniak, 24th November 2005 --- August 2006 We request acknowledgement in any publications that

Oct 22, 2022
###### A run-time C++ library for working with units of measurement and conversions between them and with string representations of units and measurements

Units What's new Some of the CMake target names have changed in the latest release, please update builds appropriately Documentation A library that pr

Oct 6, 2022
###### Cross-platform, efficient, customizable, and robust asynchronous HTTP/WebSocket server C++14 library with the right balance between performance and ease of use

What Is RESTinio? RESTinio is a header-only C++14 library that gives you an embedded HTTP/Websocket server. It is based on standalone version of ASIO

Nov 25, 2022
###### Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths.

WDT Warp speed Data Transfer Design philosophy/Overview Goal: Lowest possible total transfer time - to be only hardware limited (disc or network bandw

Nov 14, 2022
###### The missing bridge between Java and native C++

JavaCPP Commercial support: Introduction JavaCPP provides efficient access to native C++ inside Java, not unlike the way some C/C++ compilers interact

Nov 26, 2022