EmbeddingGemma 2 maps five modalities into one 768-dimension vector space

EmbeddingGemma 2 is built on Gemma 4's transformer backbone, with 24 layers, both grouped-query and multi-query attention, a vocabulary of 262,144 tokens, and outputs mean-pooled into a projection layer that takes them from 512 dimensions to 768 ^[13]. Every modality projects into those same 768 dimensions. A text query and an audio clip are then compared with plain cosine similarity ^[4]. Inputs can be interleaved, text and images in one pass, with placeholder tokens marking each media item's position ^[7]. All five modalities draw on one 8,192-token context window, and each spends it at a fixed rate ^[6]. The first model's window was 2,048 tokens ^[21].

The packaging deserves credit. The module sizes add up exactly: a 130M backbone and a 140M embedder for text, a 170M vision encoder, a 300M audio encoder, 740M in total ^[17]. Audio is the largest single piece, about 41% of the full model ^[18]. An app that only does text-to-image retrieval never loads it ^[5]. On the Pixel 11 Pro figures, the vision and audio encoders together add about 376MB of active RAM over the text-only build ^[19].

Matryoshka Representation Learning is what makes the vectors cuttable. The first N dimensions of any output are a valid embedding on their own, so 768 dimensions can be truncated to 512, 256 or 128 without retraining ^[15]. The dev.to writeup's storage figures hold up: one million 768-dimension vectors in bfloat16 come to roughly 1.5GB, and about 250MB at 128 dimensions ^[23]. That is 1,536 bytes a vector at full width ^[24]. One line in its quality table conflicts with them. The table, attributed to the model card, puts 256 dimensions at about 95% of full multimodal quality with a ~6x storage cut ^[22]. At two bytes a dimension, 256 dimensions is 512 bytes a vector, or 512MB per million, a 3x cut ^[24]. The 6x figure fits the 128-dimension row, or a float32 baseline ^[25][26]. At 128 dimensions, quality holds near 90% on text and code and falls to about 75% on multimodal retrieval ^[22].

Google reports its scores on the full-precision model ^[8]. The Pixel footprints are for a quantized build ^[10], and the writeup does not include retrieval scores for that build. Code shows the clearest gain: MTEB Code rose 9.92 points over EmbeddingGemma 1, to 78.68 ^[20][8]. The media scores sit lower, 57.28 on MMEB v2 image retrieval, 50.67 on video retrieval and 49.39 on MAEB audio ^[9], but those are separate benchmarks on separate scales. According to Google's reported numbers, the model beats some specialist models more than twice its size on audio and visual benchmarks ^[27]. For any of this to transfer, your queries have to resemble the benchmark queries, and quantization has to cost little on your hardware.

Integration cost is low. The model runs in transformers, sentence-transformers 6.1.0 and later, vLLM, llama.cpp, SGLang and Ollama ^[11]. Browsers get it through transformers.js and WebGPU, phones through MediaPipe and LiteRT ^[11]. The case for collapsing per-modality models into this one checkpoint is sound on architecture and unproven on quality outside Google's own tables. In my context, a team running separate text and image encoders, I'd load the text-and-vision configuration, embed a sample of the corpus at 256 dimensions, and compare recall against the current pipeline before retiring anything.