An unquantized model on the main thread cost a health app its screening feature

Loading a TFLite model does file I/O, and running it does compute. According to the dev.to write-up, either one on the main thread gets you an ANR ^[18]. The fix is a single wrapper class that loads the interpreter once and keeps it alive for reuse, so the cost lands at construction and not on every call ^[18].

Converting float32 weights to int8 shrinks a model by 75 percent, at a small accuracy cost ^[11]. Take the 45 MB the first build added to the APK as mostly model, and int8 leaves about 11 MB, returning roughly 34 MB of download ^[1]. The base TFLite interpreter is around 1.5 MB, about three percent of what that model cost ^[15][3].

Integer-only quantization does not work on the converter defaults alone. It needs `converter.representative_dataset`, a small set of typical inputs the quantizer uses to calibrate value ranges, and the author names skipping the calibration set as the most common conversion failure he sees ^[12]. So the build needs typical inputs wherever it runs. For this app those inputs are skin images, and the reason the model went on-device at all was to keep medical images off the network and out of a GDPR conversation ^[1].

Keeping conversion in CI makes the `.tflite` file a build artifact; the author reports finding three versions of `model.tflite` in one repo because someone forgot to replace the file ^[14]. The failure there is version control. The model belongs in `src/main/assets`, not `res/raw`, and it should not be fetched from a network path at startup ^[17].

"The first decision is not technical; it is architectural," the author wrote ^[19]. On-device inference costs nothing per call and runs offline, and it also fixes the model at install time and caps you at the phone's hardware ^[6]. The health app went on-device and returned its risk score in under 100 ms ^[8].

The article's own standfirst advertises inference at 35 ms ^[9], roughly a third of the latency the case study reports ^[2]. For 35 ms to transfer to your app you need a comparable input size, a warm interpreter, and a device whose hardware actually runs int8 several times faster; the source attaches that hardware condition to the speedup itself ^[11]. GPU delegation is a separate optional dependency, `tensorflow-lite-gpu` ^[16]. The write-up does not name the phone, the model architecture, or the accuracy given up to quantization.

This is one practitioner's account of one launch, rebuilt four times since across about a dozen clients ^[5]. The failures were the load path, the thread and the file size ^[2][3], and nothing in it says the model was wrong.