Build1 publisher3 min readPublished
Llama-3 in the browser: the record stays put, but you ship 4 GB to every device
A WebLLM tutorial splits cheap entity extraction from expensive reasoning with no PHI on the wire. The architecture holds up; the first-load bill and the hardware gate are the parts to cost out.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- A dev.to tutorial published by wellallytech describes building a 100% private, client-side health log analyzer that runs a full Llama-3 instance directly in the browser using WebGPU acceleration and WebLLM.
- The stated architecture routes user health log input into local processing that splits into Transformers.js entity extraction and WebLLM Llama-3-8B reasoning; the resulting structured health data and clinical insights are both written to IndexedDB for local storage and then shown in a privacy-safe UI view.
- WebLLM is described as a high-performance in-browser LLM inference engine that uses the WebGPU API to execute model weights compiled with TVM.
- The tutorial uses the Llama-3-8B-Instruct-q4f16_1-MLC variant, described as optimized for 4-bit quantization to fit in browser memory.
- Setup is a single dependency install (npm install @mlc-ai/web-llm) plus a React hook that constructs a webllm.MLCEngine and awaits engine.reload(selectedModel).
Compiled by The EngineerSomething wrong?How this is made
Why it matters
A tutorial published on dev.to by the WellAlly team wires up a health log analyzer that runs Llama-3-8B-Instruct entirely inside the browser: WebLLM on WebGPU for reasoning, Transformers.js for named entity recognition, IndexedDB for the results [1][2]. For teams holding regulated data, the useful part is not the model, it is the split: tagging and generation are separate jobs with separate costs, and in this design neither one puts a patient record on the wire [2][9].
The mechanics are unglamorous, which is a compliment. WebLLM is an in-browser inference engine that uses the WebGPU API to execute weights compiled with TVM [3]. The build installs one dependency, instantiates an MLCEngine, and calls reload on the model ID [5]. The chosen variant is Llama-3-8B-Instruct-q4f16_1-MLC, picked at 4-bit quantization specifically so it fits in browser memory [4]. Extraction runs a token-classification pipeline on Xenova/bert-base-NER and filters the output to entities labelled MED or SYMPTOM before anything reaches the LLM [8]. Structured entities and generated summaries both land in IndexedDB [2].
Now the operator arithmetic the tutorial does not do. Eight billion parameters at four bits each is roughly 4 GB of weights, before runtime overhead [13]. That is the first-load bill on every device, paid over the user's connection and then held in GPU memory. The code registers a progress callback that reports download and load progress as a percentage, which is itself a tell about how long that wait is [6]. The tutorial also does not say where the weights are served from [16], and for a health application that hosting path is the first thing an architecture review should pin down, because it is the one network dependency the design keeps.
Two details in the sample code will bite. The completion is a single awaited call whose result is read once from reply.choices[0].message.content, with no streaming [12], so a user on a browser-hosted 8B model waits in silence for the whole summary. And the entity filter only returns rows if the model actually emits labels named MED and SYMPTOM [14]; verify the label set of whatever NER checkpoint you ship, or the extraction stage will quietly hand back an empty array and the LLM will do all the work unassisted.
Reach is the harder constraint. The stated requirements are a WebGPU-capable GPU on Chrome 113 or later, Edge, or Safari Technology Preview, with React and Vite around it, and the author rates the difficulty as advanced [7]. Safari Technology Preview is a preview build, so by the tutorial's own list, devices on shipping Safari are out [15]. A privacy architecture that only functions on part of your fleet needs a server-side fallback, and that fallback is where the compliance work actually lives.
On that point the source is candid rather than complete. The author states that not a single byte of PHI leaves the device [9], and also that running a model is not enough for production healthcare software, naming state management, local encryption and prompt engineering as the gaps [10]. The pointer for closing those gaps is a company blog described as bridging demos and HIPAA-compliant local software [11]. There are no latency, memory or accuracy measurements anywhere in the piece [17].
Watch three numbers if you prototype this: first-load abandonment against the roughly 4 GB weight fetch [13], the share of your users whose browsers clear the WebGPU bar [7], and how many extracted entities the NER stage actually produces on real logs [8].