Skip to content

Last updated：2026-10-12

Accelerating Python with pybind11

Usually, for time-consuming operations we prefer to implement them in C++ and then have Python call the corresponding interface. Or, because of an underlying library, we need to expose a Python API to the outside. In these cases we typically need to support accessing C++ from Python — how do we do that?

There are quite a few ways. This section uses pybind11 as the example to introduce a complete project template. If you have this kind of need later, you can modify it based on the template.

Note: (lazy version) all the code for this section is also available in the Knowledge Planet community.

1. Macro Expansion

Suppose we take a rather time-consuming sort as the example. If we want to write a sort in C++ and then use it in Python, how do we write it?

The C++ program is fairly simple: expose an interface that does the sorting, for example:

std::vector<int> sort_vector(const std::vector<int>& input) {
  std::vector<int> result = input;
  std::sort(result.begin(), result.end());
  return result;
}

So here comes the question: how does pybind11 use this interface?

Just use a pybind11 macro to define a module, for example:

PYBIND11_MODULE(sort_module, m) {
    m.doc() = "pybind11 example plugin"; // Optional module docstring

    m.def("sort_vector", &sort_vector, "A function that sorts a list of integers");
}

After the macro expands, it is PyInit_sort_module.

When we import an xxx library in Python, the Python interpreter automatically calls the entry function for loading an extension module written in C/C++, and that entry function is the PyInit_sort_module below:

static ::pybind11::module_::module_def pybind11_module_def_sort_module [[maybe_unused]];

[[maybe_unused]] static void pybind11_init_sort_module(::pybind11::module_ &);

extern "C" [[maybe_unused]] __attribute__((visibility("default"))) PyObject *PyInit_sort_module();

extern "C" __attribute__((visibility("default"))) PyObject *PyInit_sort_module() {
    {
        const char *compiled_ver = "3" "." "11";
        const char *runtime_ver = Py_GetVersion();
        size_t len = std::strlen(compiled_ver);

        if (std::strncmp(runtime_ver, compiled_ver, len) != 0 || (runtime_ver[len] >= '0' && runtime_ver[len] <= '9')) {
            PyErr_Format(PyExc_ImportError,
                         "Python version mismatch: module was compiled for Python %s, "
                         "but the interpreter version is incompatible: %s.",
                         compiled_ver, runtime_ver);
            return nullptr;
        }
    }

    pybind11::detail::get_internals();
    auto m = ::pybind11::module_::create_extension_module("sort_module", nullptr, &pybind11_module_def_sort_module);

    try {
        pybind11_init_sort_module(m);
        return m.ptr();
    } catch (pybind11::error_already_set & e) {
        pybind11::raise_from(e, PyExc_ImportError, "initialization failed");
        return nullptr;
    } catch (const std::exception &e) {
        ::pybind11::set_error(PyExc_ImportError, e.what());
        return nullptr;
    }
}

void pybind11_init_sort_module(::pybind11::module_ & m) {
    m.doc() = "pybind11 example plugin";

    m.def("sort_vector", &sort_vector, "A function that sorts a list of integers");
}

So if we don't use this macro, importing the library will raise an error:

>>> import sort_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_sort_module)

2. CMake + pybind11

pybind11 is used in C++ as follows:

  • Headers
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
  • Macro
PYBIND11_MODULE(sort_module, m) {
    // 你的实现
}

So how do you download and link pybind11 in CMake?

Just use the FetchContent module, for example:

FetchContent_Declare(
    pybind11
    URL https://github.com/pybind/pybind11/archive/v2.12.0.tar.gz
    )

Then add your module in CMakeLists:

pybind11_add_module(sort_module sort_module.cc)

CMake will download pybind11 for you and build it at the same time.

3. Usage

The above generates an so file. Install it into Python's site-packages directory, and it will be recognized on import. Below is a demo of our custom module:

import sort_module

def test_sort():
    input_list = [5, 2, 3, 1, 4]
    sorted_list = sort_module.sort_vector(input_list)
    print(f"Original list: {input_list}")
    print(f"Sorted list: {sorted_list}")

if __name__ == "__main__":
    test_sort()

Run:

(base) ➜  build python ../test.py
Original list: [5, 2, 3, 1, 4]
Sorted list: [1, 2, 3, 4, 5]

Comments