Build1 publisher3 min readPublished
A fullwidth comma slips an emptied payee name past qbofile's empty-string check
NFKD maps U+FF0C onto an ordinary comma, so the name 北京，上海 sanitizes down to a single comma and never trips a check for the empty string. The transaction imports with the right amount and a blank payee.
The Engineer · Build desk
What happened
- qbofile reduces every payee name to printable ASCII for its QBO output, running NFKD, stripping combining marks, applying a fallback map, then replacing anything still outside printable ASCII with a space.
- Its warning code counted a sanitized result of '' as a name that had been emptied, and anything else as a name that had merely lost some of its characters.
- The name 北京，上海 comes out of that sanitizer as a single comma, because the fullwidth comma has an NFKD mapping into ASCII and the han characters do not.
- The user was therefore shown the partial-loss message, which listed 北 京 上 海 as dropped characters and said amounts and dates were unchanged.
- The check now tests the sanitized result against /[A-Za-z0-9]/, so a punctuation-only survivor counts as emptied while a digit counts as content.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure The person exposed is whoever reconciles the books later: a blank payee arrives attached to a correct date and a correct amount, and no downstream step has a reason to look at it again.
- constraint Any sanitizer that gates on the empty string inherits this hole, because length cannot express whether anything readable is left once a single ASCII-mappable character survives.
- decision Keeping digits inside the definition of content is now a decision that has to survive code review, because narrowing the class to letters alone misreports names that still carry a matchable number.
- cost The protection rests on fixtures that nobody writes before seeing the failure, since each mutation was caught by about 0.2 percent of the test suite.
NFKD is a compatibility decomposition, and fullwidth forms are exactly what compatibility covers. U+FF0C FULLWIDTH COMMA normalizes to U+002C. The fullwidth parentheses become `(` and `)`, the fullwidth colon becomes `:`, the fullwidth exclamation mark becomes `!`, and the ideographic space becomes an ordinary space [11]. Han characters have no such mapping, so they reach the last step of the sanitizer and are replaced with spaces, after which the spaces collapse and `trim()` removes them [6]. What survives 北京，上海 is one comma [9]. The post's author wrote: "I did not put that comma there." [22]
Transliteration in qbofile is deliberately silent, on the reasoning that a warning firing on every é is a warning people learn to dismiss [7]. CAFÉ becomes CAFE, and a typographic apostrophe becomes a straight one [4]. 北京 STORE becomes STORE, and 北京烤鸭 becomes the empty string [5]. Losing the whole name is the case that is supposed to speak up, because otherwise the transaction imports cleanly with the right date and the right amount and nothing in the payee field [7].
The guard read `if (!dropped.length) continue; if (text === '') emptied += 1; else partial += 1` [8]. The empty-string shape is the documented one: slug builders that return an empty slug for a CJK title, sanitizers that hand an empty name to an API requiring a non-empty one, issues filed about both [15]. A name reduced to a comma is not empty, so the `else` branch ran and the user was told this [12]:
"1 description lost characters. QBO files can only hold plain ASCII text. Accented letters were converted (café becomes cafe), but these characters have no ASCII equivalent and had to be dropped: 北 京 上 海. Amounts and dates are unchanged. Use the CSV or PDF output if those characters matter." [13]
Every sentence in that message is accurate, which is the least useful property a message can have. It is the text for a name that lost some characters, and the line for a name that lost all of them never printed [12]. The blind spot fits four ordinary business names: 北京，上海 leaves `,`, 星巴克（北京） leaves `( )`, 全家便利店：朝阳店 leaves `:`, and 株式会社（東京） leaves `( )` [14].
The replacement tests for legible content instead of length: `if (!/[A-Za-z0-9]/.test(text)) emptied += 1; else partial += 1` [16]. Digits stay in the definition because 全家 #123 reduces to `#123`, which the post calls a bad name but one you can still match against a line on your statement [17].
Two mutations were run against a suite of 495 tests, and each failed exactly one test [18][19]. One test in 495 is about 0.2 percent of the suite [1]. The first mutation is the original bug, and the test that catches it feeds in 北京，上海, 星巴克（北京） and 全家便利店：朝阳店, then asserts the message says the payee name is gone and does not say `lost characters` [20]. The second mutation narrows content to `[A-Za-z]`. That edit looks like a tidy-up, and it fails on 全家 #123, 7-ELEVEN 北京 and 北京（STORE） [21].
Those one-test margins describe this suite. They transfer to another codebase only if its fixtures already contain a name whose sole ASCII survivors are punctuation, and a coverage report will not generate that fixture for you. The bug needs two conditions to bite: an output target pinned to ASCII, and names arriving from text that uses fullwidth punctuation. qbofile has the first because its QBO header declares `ENCODING:USASCII` and `CHARSET:1252` [2].
What to watch
- Whether the same content-class test is applied to the CSV and PDF output paths the warning tells users to fall back to.
- Whether the fallback map is extended so more CJK punctuation transliterates instead of being replaced with a space.
- Whether the empty-slug projects that already have issues filed against them adopt a content test rather than a length test.