Build1 publisher3 min readPublished
PYTHONUTF8=1 in a .zshrc hid the encoding bug from every local test run
A dev.to write-up traces two nights of hex dumps to a shell that exported PYTHONUTF8=1 on the laptop and nothing on the clean Linux box, where open() fell back to the POSIX locale's ASCII codec.
The Engineer · Build desk
What happened
- A job that read a small UTF-8 CSV and wrote JSON passed pytest on the developer's laptop, then died inside open() on a clean Linux box running the same commit.
- file, hexdump and a raw byte read all agreed the input was legal UTF-8, with c3 a9 for the e-acute in one surname and no BOM in the header.
- Running python under env -i on that same laptop printed utf8_mode 0 and preferred encoding ANSI_X3.4-1968, where the interactive shell had printed 1 and utf-8.
- The laptop's UTF-8 mode came from a user-level sitecustomize.py and PYTHONUTF8=1 in .zshrc, and the server had neither, so it fell back to the POSIX locale encoding.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A shell that exports PYTHONUTF8 or a UTF-8 locale cannot fail the way the server fails. A green local suite tells you nothing about which codec the remote process will pick.
- decision The choice is between passing encoding at every read site and setting the interpreter's mode for the whole process. The first route means finding every implicit decode, including the ones inside subprocess calls.
- exposure Taking the setlocale advice on an image that ships only C and POSIX trades a decode error deep in a read for a locale.Error at process start.
- contradiction The post is disclosed vendor outreach, yet its own account rates the model's answer half right. The discarded half is the one that would have broken on the very box it was meant to repair.
`open()` with no `encoding` argument does not hand you bytes. It asks the interpreter for a default. On this laptop the answer depended on which shell started the process: `sys.flags.utf8_mode` 1 and `utf-8` interactively, 0 and `ANSI_X3.4-1968` under `env -i` with only HOME and PATH kept [6]. That fallback is the POSIX locale encoding, ASCII in practice [7]. The byte `0xc3` is the first half of the e-acute, the `ascii` codec has nothing to map it to, and it raises [8].
The local reproduction is three lines: write `b"id,name\n1,Ren\xc3\xa9\n"` to a file, call `read_text()` with no encoding, run it under `env -i PATH LANG=C LC_ALL=C` [8]. The file was never the problem. `file`, `hexdump` and a raw read all said legal UTF-8 [4], and rewriting the exporter, which is cheaper than doubting your own runtime, changed nothing [5].
Adding `encoding="utf-8"` to the CSV reader cleared the frame named in the traceback. Then a `Path.read_text()` call and a `subprocess` pipeline with `text=True` went on using the process default [13]. Of the three implicit decode sites the post names, that change covered one [17].
For this to be your bug, the development shell has to switch UTF-8 mode on out of band: here, a user-level `sitecustomize.py` plus `PYTHONUTF8=1` in `.zshrc` [7]. The target also has to run under C or POSIX. On the scratch box, `locale` printed `LANG=C` and `LC_ALL=C`, and `locale -a` listed two entries, C and POSIX [16].
The write-up carries a disclosure that it was prepared as part of MonkeyCode's product outreach. The author pasted the traceback into MonkeyCode's free model access asking for a fix rather than a hypothesis list [9]. "The reply was fluent, confident, and half right, which is a dangerous combination when you are already tired," the author wrote [10]. The right half was `encoding="utf-8"` at the call site [11]. The other half was `locale.setlocale(locale.LC_ALL, "en_US.UTF-8")` at process start. The post labels that one the incomplete fix the author almost shipped, and annotates it with its failure mode, `locale.Error` on a box that never ran locale-gen [11][12]. On a box whose only installed locales are C and POSIX, that call raises [18].
What ended the two nights was one command that stripped the environment [6]. The author wrote that if your local shell exports PYTHONUTF8, LC_ALL=en_US.UTF-8, or a conda activation script, you do not have a local reproduction; you have a comfort setting [14]. The scratch Linux box came from MonkeyCode's free server option. The post says a spare VPS or a throwaway container would have taught the same lesson: "The point was not the brand; the point was a process that had never seen my .zshrc" [15]. The post does not include the final change that shipped [20].
What to watch
- Whether this class of bug gets answered in code, with encoding passed at every read site, or in the CI image by exporting PYTHONUTF8=1, since only the first travels with the repository.
- Whether the next post in MonkeyCode's disclosed outreach series rates its model's answer the same way this one did.