Python on Windows decodes UTF-8 config files with the ANSI code page

cp1252 is a single-byte codec, and it maps almost every byte to some character, so decoding UTF-8 through it raises nothing ^[6]. The author of the dev.to post encoded "café-" to UTF-8, got the bytes 63 61 66 c3 a9 e2 80 94, and decoded those bytes as cp1252: caféâ€" ^[5]. Five characters in, eight out ^[1]. What comes back is a valid str, so the dict key is now café and the lookup for café misses ^[6].

Other code pages fail louder. The same bytes under GBK give 'gbk' codec can't decode byte 0x94 in position 7: incomplete multibyte sequence, because GBK expects a valid lead byte followed by a valid trail byte, and a UTF-8 sequence is neither ^[9]. cp1252 has five holes of its own, at 0x81, 0x8D, 0x8F, 0x90 and 0x9D, and one of those in the stream produces 'charmap' codec can't decode byte 0x81 in position 0: character maps to <undefined> ^[10]. 0x81 is a UTF-8 continuation byte, so it sits inside ordinary characters, and U+2041 encodes as E2 81 81 ^[11].

chcp 65001 does not help because it sets the console code page, while Python's default text encoding comes from the system ANSI code page that GetACP returns ^[15]. For a pipe there is no console in the path at all ^[15].

A fourth failure lives in binary-detection code. Tools sniff the first N bytes to guess whether a file is binary, and 1000 is a popular sample size. Cut a UTF-8 file at byte 1000 and the cut can land inside a multibyte character, giving 'utf-8' codec can't decode byte 0xe6 in position 999: unexpected end of data ^[12]. The rest of the file decodes cleanly and there is no NUL byte anywhere. The tool still calls the file binary and refuses to display it, and the author reports hitting that on a plain CJK text file ^[13].

The fix is mechanical. The post says to pass encoding=utf-8 at every point where bytes become text, in open(), Path.read_text() and Path.write_text(); for children, subprocess.run(..., text=True, encoding=utf-8, errors=replace); for your own output, sys.stdout.reconfigure(encoding=utf-8, errors=replace) ^[16]. It flags two details: errors=replace never raises, which is also why it can hide a real mismatch, and errors=surrogateescape round-trips arbitrary bytes without losing any ^[18]. To find the sites in code that already exists, run with PYTHONWARNDEFAULTENCODING=1, or python -X warn_default_encoding, and Python 3.10 and later emits an EncodingWarning everywhere the default is relied on ^[19]. PEP 597 added that warning ^[19].

Pinning UTF-8 is the right default when the files really are UTF-8, and the post's premise is that most config, most code and most modern tooling now are ^[20]. I would not pin it blind on a codebase that reads files some other Windows program wrote. Those can genuinely be in the platform encoding, and encoding="locale" on 3.10 and up declares that without guessing ^[17].