1. Contrastive loss $\mathcal L_C$: pull related texts together, push unrelated ones apart#

Building on standard InfoNCE, the loss for a single sample is:

$\ell_i=-\log\frac{e^{\operatorname{sim}(q_i,p_i^+)/\tau}}{e^{\operatorname{sim}(q_i,p_i^+)/\tau}+w_i\,e^{\operatorname{sim}(q_i,p_i^-)/\tau}+\sum_{j\ne i}\mathbb 1_{\text{TN}}(i,j)\,e^{\operatorname{sim}(q_i,p_j^+)/\tau}}$

Here $\operatorname{sim}$ is cosine similarity and $\tau$ is the temperature. The higher the positive scores relative to all candidates, the lower the loss. Besides the positive itself, the denominator contains two kinds of negatives:

• Hard negatives: weighted by $w_i=\exp\left(\alpha\,\mathrm{sg}(\operatorname{sim}(q_i,p_i^-))\right)$ with $\alpha=5.0$. The more a negative resembles the query, the larger its weight and the more attention the model pays to it; $\mathrm{sg}$ is stop-gradient, so the weight only reflects the current difficulty and is not differentiated itself. Hard negatives are optional: the large, noisy pre-finetuning stage uses only (query, target) pairs, and they are added during fine-tuning.
• In-batch negatives: the positives $p_j^+$ of the other samples in the same batch. Each batch is drawn from a single dataset, which makes these negatives harder to tell apart and more valuable; $\mathbb 1_{\text{TN}}$ masks out entries whose query or positive duplicates the current sample’s, so they aren’t mistaken for negatives.

2. Spread-out regularizer $\mathcal L_S$: keep vectors from crowding into a few directions#

For distinct pairs of queries and distinct pairs of positives within a batch, it penalizes the square of their inner products:

$\mathcal L_S= \frac{\sum_{i\ne j}(q_i^\top q_j)^2}{B(B-1)} + \frac{\sum_{i\ne j}((p_i^+)^\top p_j^+)^2}{B(B-1)}$

The closer the inner product is to 0 (nearly orthogonal), the smaller the penalty, so the vectors spread out across the space as a whole. The idea comes from GOR (global orthogonal regularizer): make the vectors of two random texts match, in the mean and second moment of their inner product, two points sampled independently and uniformly on the unit sphere; the paper keeps only the second-moment term.

It constrains overall statistical properties and does not require semantically similar texts to be orthogonal to each other: pulling related texts together is still the job of the contrastive loss, and the two balance each other. The authors also expect this distribution to make the model more robust to quantization and better suited to approximate nearest neighbor (ANN) search.

3. Embedding matching $\mathcal L_D$: learn the teacher’s vectors directly#

The teacher is the larger Gemini Embedding. Most earlier distillation work only has the student imitate the teacher’s query-document relevance scores; here, following EmbedDistill, the student’s vectors are aligned directly with the teacher’s, so the student learns the geometry of the teacher’s entire vector space.

Unlike EmbedDistill, the paper aligns not only queries and positives but also hard negatives, and finds that this clearly improves results. Intuitively, it plays the same role as hard negatives in the contrastive loss: the student learns how the teacher separates a query from its hard negatives. The three terms are added with equal weight:

$\mathcal L_D=\mathcal L_D^{Q}+\mathcal L_D^{P^+}+\mathcal L_D^{P^-}$

Addendum: MRL#

$\mathcal L_C$ and $\mathcal L_S$ also apply MRL (Matryoshka Representation Learning): the losses are computed separately on several nested prefix dimensions (768 / 512 / 256 / 128) and simply summed. Vectors trained this way can be truncated to lower dimensions and used directly.