Build1 publisher2 min readPublished
mypyc roughly doubles h11's speed once its annotations pass strict mypy
A dev.to walkthrough compiles the HTTP/1.1 library h11 into a C extension for roughly twice the speed, and most of the work is reconciling bytearray returns, written to avoid a copy, with annotations that promise bytes.
The Engineer · Build desk
What happened
- mypyc is an ahead-of-time compiler that reads a module's type hints and emits a CPython C extension, a .so or .pyd file that imports like any other Python module.
- A dev.to post reports compiling h11, a small HTTP/1.1 library with 710k dependents on GitHub, with mypyc and getting roughly a factor of two in speed.
- Running mypy in strict mode on h11 first reported 13 errors in 3 files, out of 11 source files checked.
- h11's ReceiveBuffer returns bytearray from its extract methods to avoid another copy of the data, while the validators and constructors it feeds are annotated to accept bytes.
- The post's route runs in three stages: clear strict mypy, get mypyc to compile everything, then confirm the compiled module imports and the tests pass.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The speedup arrives as a build step that emits platform-specific binaries, so the maintainer takes on a compile-and-publish pipeline for every platform the package supports.
- constraint Anything mypy cannot type in strict mode is out of reach of the compiler, so the dynamic corners of a library stay interpreted no matter how hot they are.
- exposure Downstream callers passing a near-enough type into a compiled module get a TypeError at the boundary where the interpreter used to accept it.
- decision h11's deliberate bytearray returns force a choice between widening the annotations to admit bytearray and converting to bytes, which reinstates the copy those methods were written to avoid.
The annotations pay for three specific things at compile time. An integer can be held as a plain `int32_t` at the C level instead of a `PyObject` [3]. Calls and attribute access resolve when the module is built [4]. And an attribute read stops walking `__dict__`, the MRO, descriptors, `__getattribute__` and `__getattr__`, because the compiler already recorded which struct field to read [5].
h11 parses bytes, so the unboxed-integer path has little to work with there, and the other two are where a parser would get its time back. That is the part of the roughly twofold claim [2] worth testing first on another codebase: a hot loop already spending its time inside C-implemented builtins gives early binding nothing to remove. The post frames mypyc as the alternative to rewriting parts of a library in Rust with PyO3 [15]. The walkthrough breaks off after the error listing, so it does not include the workload or the hardware behind the figure [16].
Before the compiler runs at all, mypy has to be satisfied [7]. The post's `pyproject.toml` sets four options under `[tool.mypy]`, and `strict = true` is the one that decides how much work follows [7]. On h11 the result was "Found 13 errors in 3 files (checked 11 source files)" [8]. The author reads those as a couple of redundant `# type: ignore` comments and a few type mismatches [13].
Ten of the thirteen are the same disagreement, repeated down `h11/_readers.py` at lines 53, 84, 87, 104, 114, 134, 161, 182, 204 and 218 [11]. `ReceiveBuffer.maybe_extract_lines()` and `maybe_extract_at_most()` return `bytearray`, and the post says that is deliberate: it avoids copying the data one more time [10]. The functions they feed, `validate()`, `_decode_header_lines()`, `_obsolete_line_fold()` and the `Data()` constructor, are annotated to take `bytes` [10]. That leaves three errors in the rest of the library [14].
The count describes one particular target, too. h11 supports Python back to 3.8, and the post fixes the annotations as if only 3.10 and later had to work, using mypy 2.3.1 [12]. A maintainer who still has to satisfy 3.8 is annotating against an older typing surface and should expect a different list.
What to watch
- Whether the follow-up widens the annotations to admit bytearray or converts to bytes and pays the copy back, and whether the roughly twofold figure survives either fix.
- A published benchmark for the compiled h11, with workload, request mix and hardware, so the figure can be reproduced.
- Whether h11 upstream accepts annotation changes that admit bytearray at its internal API boundaries.