Right, so you want to talk to Python directly, in its mother tongue: C. You’re tired of pure Python’s speed limits, or maybe you need to wrap some arcane C library that doesn’t have a decent Python binding. Welcome to the club. Writing a CPython extension module is the most “bare-metal” way to do this. It’s powerful, it’s fast, and it will absolutely, 100%, make you appreciate the cleanliness of Python syntax. We’re about to get our hands dirty.

The core concept is simple: you’re going to write a C function that follows a very specific, slightly verbose, protocol. This function will receive PyObject pointers (which represent every Python object you’ve ever loved) and it will return a PyObject pointer (which is how you give something back to the Python interpreter). The Python C API is your toolbox for converting between C types and PyObjects, and for doing things like raising exceptions.

The Obligatory, “Hello World” But It’s in C

Let’s start with the canonical example. You want to create a C function that can be called from Python. Here’s the entire, terrifying, beautiful thing. Save this as spammodule.c.

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject* spam_system(PyObject *self, PyObject *args) {
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command)) {
        return NULL; // This tells Python the argument parsing failed, and it will raise a TypeError.
    }
    sts = system(command);
    return PyLong_FromLong(sts);
}

static PyMethodDef SpamMethods[] = {
    {"system",  spam_system, METH_VARARGS, "Execute a shell command."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

static struct PyModuleDef spammodule = {
    PyModuleDef_HEAD_INIT,
    "spam",   /* name of module */
    NULL,     /* module documentation, may be NULL */
    -1,       /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
    SpamMethods
};

PyMODINIT_FUNC PyInit_spam(void) {
    return PyModule_Create(&spammodule);
}

Now, to build this glorious mess, you need a setup.py file. The setuptools library does the heavy lifting of finding your C compiler and invoking it with the right, obnoxiously long, flags.

from setuptools import setup, Extension

module = Extension('spam', sources=['spammodule.c'])

setup(
    name='spam',
    version='1.0',
    description='A simple example of a C extension',
    ext_modules=[module],
)

Run python setup.py build_ext --inplace. If you don’t hear your compiler scream, it worked. You’ll get a spam.cpython-39-x86_64-linux-gnu.so file (or a .pyd on Windows). Now, in Python:

import spam
status = spam.system('ls -la')  # It's just a Python function now. Magic.

The Devil is in the Details (Mostly Memory Management)

See that PyArg_ParseTuple line? That’s your first real point of failure. It takes the tuple of arguments Python received (args) and a format string ("s") and tries to unpack them into C variables. "s" means “a string”, and it gives us a const char*. The list of format units is long and weird; bookmark the official docs. "i" for integer, "d" for double, "O" for a generic PyObject, and so on. If this function fails (returns 0), you must return NULL to propagate the error up to the Python interpreter.

The other critical line is return PyLong_FromLong(sts);. You can’t just return the C integer sts. Python wouldn’t know what to do with it. You have to manufacture a new Python integer object from it. This pattern is everywhere: you take C data, and you use a function like PyLong_FromLong, PyUnicode_FromString, or PyFloat_FromDouble to create a new PyObject that Python can handle. The key word here is new. Which brings us to the single biggest headache…

Reference Counting: Your New Best Frenemy

Python uses reference counting (and garbage collection) to manage memory. When you write a C extension, you are now a co-conspirator in this scheme. You must manage the reference counts of PyObjects you work with.

The rules are simple in theory, brutal in practice:

  • When you receive a PyObject from Python (e.g., as an argument), it is “borrowed”. You don’t own it. Don’t decrement its refcount.
  • When you create a new PyObject (e.g., with PyLong_FromLong), you own the reference. You are responsible for eventually Py_DECREF-ing it.
  • If you want to keep a reference to an object you received, you must Py_INCREF it to claim ownership, and then Py_DECREF it when you’re done.

The most common rookie mistake is to Py_DECREF an object you shouldn’t, causing a segfault when Python tries to use it later. The second most common is to not Py_DECREF something you created, causing a memory leak. There are tools like valgrind and tracemalloc to help you find these bugs, but they are a pain. This is why most people use Cython or ctypes for anything non-trivial these days.

Why Would You Actually Do This?

Given the complexity, why bother? Three reasons:

  1. Absolute Control: You can optimize a tight loop down to the last CPU cycle. You’re not going through any intermediate abstraction.
  2. Existing C Libraries: Need to talk to a massive, complex, well-tested C library? This is how you do it without a middleman.
  3. It’s The Foundation: Understanding this makes you appreciate the other tools (Cython, ctypes) infinitely more. You know what they’re abstracting away.

It’s the difference between building a car from scratch and being a master mechanic. One is a huge undertaking, but the depth of knowledge is unmatched. Just be prepared for the grease and the occasional smashed thumb.