Estimate project
← All posts

Extending Python code with C/C++

This article shows several examples of integrating C/C++ extensions with Python code, along with the approaches available for doing it.

3 min read
Machine LearningPerformancePythonC++

You might want to write a C/C++ extension for your Python code for several reasons. Such an extension can add new built-in object types, make system calls, or invoke functions from C/C++ libraries. As an ML engineer, you’ll most probably rely on that last capability to make your algorithms faster. The majority of popular algorithms already exist as heavily optimized libraries such as OpenCV, SciPy, Skimage, Numpy, or Pandas, so writing your own extensions is usually unnecessary. Still, particularly in computer vision, you can run into problems that a C/C++ extension helps solve. At work, I needed a flood-filling algorithm to segment 3D volumes. Skimage does provide flood-filling, but it can’t be customized. My own flood-filling algorithm written in Python was painfully slow. So I chose to implement my own Python extension in C. The C version ran 100 times faster than its Python counterpart. Quite impressive.

You can combine C/C++ code with Python in a number of ways:

Below, I’ll quickly walk through a few of these approaches and demonstrate how to create a binding between C/C++ and Python.

Ctypes

Ctypes is the most straightforward option - ideal when you want to pass a matrix to C, run heavy computations, and receive the results. Its biggest downside is that c_types works with C only, so to use C++ you either have to build a C bridge to C++ or pick another solution.

Ctypes is suitable only for small extensions.

Create the ctypes_example.cfile:

Create a makefile:

Execute make in the terminal. This produces a py_ext.so file in the current directory.

Create ctypes_example.py:

Then run the Python file. A basic script that edits a Numpy array is done.

Python C API

This is the approach Python currently recommends for writing C and C++ extensions. Another option is Cython, which sits somewhere between Python and C. Cython lets you write code quickly – getting things done in Cython clearly takes less code than in C. The price, however, is speed: Cython code often struggles to reach the performance of C code.

To build a Python C API extension, create the py_objects.c file:

Compile the C code using the command below:

Lastly, create the python file py_objects.py:

Boost Python

Boost offers a strongly C++-style interface between C++ and Python. It maps std::vector<> and std::map<> directly onto Python lists and dictionaries. In addition, C++ code automatically shares smart-pointer reference counts with Python’s reference counts, and Python classes can override C++ virtual methods.

On Linux, Boost can be installed with:

sudo apt-get install libboost-all-dev -lboost_python3 -lboost_numpy3

Create boost_python.cpp:

Create a makefile:

where

-lboost_python{PYTHON_VERSION_SUFFIX}

and

-lboost_numpy{PYTHON_VERSION_SUFFIX}

use a PYTHON_VERSION_SUFFIX

that matches the system python version and is typically found by default in

‍_usr/lib_ and _-I/home/<user>/anaconda3/envs/sample_env/include/python3.10,_

is the path to the python include directory.

Create boost_python.py:

Then run the Python file. A basic script that edits a Numpy array is done. The advantage of Boost is that we can work in C++ in a completely natural manner.

Using setup.py instead of a makefile

When building a Python project, we’d rather not compile C code by hand. The makefile can be swapped for the familiar setup.py Python script.

After that, all we need to do is install the setup.py file. Building and testing the extension code can be wrapped in a brief makefile, as shown below:

Conclusions

This article covered why you might want C/C++ extensions and showed several examples of integrating them with Python code. For a small function that does heavy computation, Ctypes is the simplest choice. The pythonic way of writing an extension is typically the Python C API or Cython. Cython is quick to write but may fall short of C code performance. The Python C API delivers the best speed at the cost of writing more code. Finally, boost offers integration with Python code that feels very much like C++. It lets Python classes override virtual methods, maps Python lists directly onto C++ vectors, and more. Every use case deserves individual consideration and the most suitable method.