Skip to content

Last updated：2026-10-12

CPython Extension Development Notes

This article takes a recent feature I submitted to Apache as its background, and talks about the problems I ran into with CPython while trying to write a library from scratch. The code is also available in the Knowledge Planet community — interested readers can download it to study.

0. Background

Recently, a feature I submitted to Apache Arrow changed a C++ interface in a way that also changed interfaces in other languages, and some tests needed fixing accordingly.

Anyone familiar with PyArrow knows this well: this interface change also required modifying that library, since it lives in the same repository — otherwise CI would not pass. PyArrow's implementation is built on CPython. I had never studied CPython in depth before, and I ran into a pile of problems. Two of them are quite important, and I'm recording them here.

Problem 1: default constructions of other classes are not supported inside initialization functions.

Example:

 def __init__(self, mode="only_valid", filter=Expression._scalar(True)):
        pass

The error:

TypeError: descriptor '_scalar' for 'pyarrow._compute.Expression' objects doesn't apply to a 'bool' object

You can see it wasn't recognized. In reality, Expression._scalar(True) is legal. Let's look at the implementation inside:

@staticmethod
def _scalar(value):
    cdef:
        Scalar scalar

    if isinstance(value, Scalar):
        scalar = value
    else:
        scalar = lib.scalar(value)

    return Expression.wrap(CMakeScalarExpression(scalar.unwrap()))

As you can see, a normal bool type is supported inside. I suspected this was a CPython limitation, so changing it to the following form worked:

def __init__(self, mode="only_valid", filter=None):
    if filter is None:
        filter = Expression._scalar(True)

Problem 2: definition order

When I used the _true created later, the default value passed in every time was empty. This is fairly easy to understand: everything eventually gets translated into an xxx.cpp file when compiled, and by C++ rules, what is read earlier is naturally empty.

def __init__(self, mode="only_valid", filter=_true):
        pass


cdef CExpression _true = CMakeScalarExpression(
    <shared_ptr[CScalar]> make_shared[CBooleanScalar](True)
)

Alright. Based on the above background, I also wanted to write an example myself — for instance, writing a class in C++ that wraps sort and sum, then calling it from Python.

1. A Complete CPython Example

  1. Create a .h file
void sort(std::vector<int>& nums) {
    std::sort(nums.begin(), nums.end());
}
int sum(std::vector<int>& nums) {
    int sum = 0;
    for (int num : nums) {
        sum += num;
    }
    return sum;
}

2) Create foo.pyx

The key point: the vector above needs:

from libcpp.vector cimport vector

Then define a class that calls the C++ interface.

cdef class PyFoo:
    cdef Foo* f

    def __cinit__(self):
        self.f = new Foo()

    def __dealloc__(self):
        del self.f

    def sort(self, nums):
        cdef vector[int] c_nums = nums
        self.f.sort(c_nums)

    def sum(self, nums):
        cdef vector[int] c_nums = nums
        return self.f.sum(c_nums)
  1. Create the setup.py file
ext = Extension('Foo', sources=["foo.pyx"], language="c++", include_dirs=[numpy.get_include()])

setup(name="Foo", ext_modules = cythonize([ext]))
  1. Run
python3 setup.py build_ext --inplace

Finally, you can write a test script to use the Python interface you wrote yourself.

import Foo

f = Foo.PyFoo()
nums = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5]
f.sort(nums)
print("Sorted nums:", nums)
print("Sum of nums:", f.sum(nums))

CPython is used in quite a few projects. Go learn it~

Run:

➜  cpython_examples python3 test.py 
Sorted nums: [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5]
Sum of nums: 44

Comments