Build1 publisher3 min readPublished
Choosing the embedding model first locks the vec0 table to a fixed 768-dimension schema
A dev.to walkthrough replaces a managed vector store and a hosted embedding API with sqlite-vec and a local model on port 11434. Its schema declares float[768], so changing model dimension means re-embedding everything.
The Engineer · Build desk
What happened
- A dev.to walkthrough argues that a RAG corpus under about a million chunks on one machine needs no managed vector store or hosted embedding API, and puts the local pipeline at roughly 50ms per query on a laptop.
- The stack is SQLite with the loadable sqlite-vec extension and a local model served by Ollama, with vectors held in a vec0 virtual table whose embedding column is declared as float[768].
- It prices text-embedding-3-small at $0.02 per million tokens and nomic-embed-text as free after the download, citing repeated re-embeds of a 500MB corpus as the recurring bill.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A deployment under a clause that data never leaves the customer VPC is in breach the first time a contract paragraph is embedded. That makes the hosted embedding call the first thing to pull out of the path.
- constraint The declared float[768] width binds the whole corpus. Any move to a wider model, like any change of chunking strategy, is a new virtual table plus a full re-embedding pass, so the model choice behaves like a schema migration.
- contradiction The $0.004 per query in the article's own scenario works out to 200,000 tokens of embedding, so most of that money is going to the generation call, and a local embedding model does not reduce it.
- decision Anyone weighing a few MTEB points against an interactive latency floor has to run the comparison on their own documents, because the published gap was measured on tasks that may not resemble theirs.
The commitment sits in the CREATE TABLE line. The vec0 virtual table takes the vector width as part of the declaration, `embedding float[768]`, because sqlite-vec stores vectors in a packed binary format [4]. Pick nomic-embed-text and 768 is the table shape. Moving later to a 3072-dimension model means a new virtual table and a pass over every chunk to re-embed it [4]. A chunking change costs that same pass. That is the author's stated reason for not wanting to pay per token to embed [6].
Of the three reasons the article gives, the cost case is the weakest. text-embedding-3-small is priced at $0.02 per million tokens [6]. At roughly four characters per token, a 500MB corpus is about 125 million tokens, so one full re-embed runs about $2.50 [1]. The $0.004 per query in the opening scenario [7] is not embedding spend either: at $0.02 per million, that would be 200,000 tokens in a single query [2]. The generation call is carrying that bill.
Residency is the firmer argument. Sending a paragraph of a merger agreement to text-embedding-3-small is a data transfer to OpenAI, so a contract promising that data never leaves the customer VPC is broken by the embedding call itself [9]. The author wrote that the air-gapped deployment story has "a hole in it the size of an HTTPS connection to api.openai.com" [10]. The walkthrough covers chunking, embedding, storage and retrieval [17]. A hosted answer model would still sit in the path after the embedding call runs locally.
Latency, at least, you can check against the article's own numbers. A hosted round trip is 80 to 200ms before any work happens, against 5 to 15ms for a local embedding on an M2 Mac [8]. End to end, the article claims about 50ms per query on a laptop [2], which comes back about 30ms before the fastest hosted embedding call [3]. That transfers only if Ollama is already running on :11434 with the model resident [3], the machine is not also serving the application, and the index stays under the stated ceiling of roughly a million chunks on a single machine [1].
The article concedes the benchmark gap itself: text-embedding-3-large at 3072 dimensions still beats nomic-embed-text at 768 on MTEB by a few points, and on domain-specific corpora that gap often shrinks [11]. For those few points to matter on your data, your retrieval task would have to resemble MTEB's tasks. The author's advice is to test before assuming [11].
Read the chunking defaults closely. `chunk_text` targets 1800 characters with 200 of overlap, splitting on paragraph boundaries first, then sentences, then hard-wrapping [12]. nomic-embed-text accepts 8192 tokens, but the author reports retrieval quality degrading on long inputs and calls 1500 to 2000 characters "the sweet spot in my testing" [13]. Overlap should be about one sentence [14], and a fixed 512-token window will split sentences, code blocks and table rows [15]. Embedding goes through one HTTP call per batch to Ollama's /api/embed, not one per chunk [16], and metadata filtering needs a regular index alongside the virtual table, because the `+` prefix on text, source and chunk_index stores those columns without indexing them for vector search [5].
What to watch
- Retrieval comparisons of nomic-embed-text against text-embedding-3-large on legal or clinical corpora. That would settle whether the MTEB gap holds where these deployments live.
- A query timing near the stated million-chunk ceiling rather than a laptop-sized index. A timing there would test the 50ms figure at the top of the supported range.
- Any measurement of vec0 scan cost when a WHERE filter on source narrows the candidate set, given that the auxiliary columns are stored without a vector index.