Vol.181 · Scaling Laws for Dense Retriever Fine-Tuning, Re-Ranking Depth as a Query-Specific Property
Vol.181 for Nov 02 - Nov 08, 2026: Structured digest of 10 papers including Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search; Item-Space On-Policy Distillation for Efficient LLM Reranking; Reinjecting Codebook Structure into Semantic ID Decoders. Includes a quick-reference paper list, one-sentence practical use cases, figure screenshots, and related literature.
Paper List & One-Sentence Practical Guide
This issue (Vol.181 for Nov 02 - Nov 08, 2026) covers 10 papers across information retrieval, generative recommendation, Semantic IDs, LLM reranking, and agentic search. Start with the quick-reference list below to see what each paper is useful for and which related papers to read alongside it, or jump directly to any paper's detailed section and architecture screenshot.
Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search (from Martinico et al.) — Paper
- What it's useful for: Use this framework (from Martinico et al.) to cut the memory footprint, CPU–GPU transfer overhead, and MaxSim scoring latency of ColBERT-style multi-vector retrievers while retaining fine-grained token matching.
- Related papers: ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction · Knowing When to Intervene: RL, Search Agents, and RAG · Also in this issue: #2 Item-Space On-Policy Distillation for Efficient LLM Reranking
Item-Space On-Policy Distillation for Efficient LLM Reranking (from Meta) — Paper
- What it's useful for: Use this technique (from Meta) to replace slow autoregressive LLM reranking with calibrated single-pass scoring, reusable document prefixes, or query-adaptive reranking depth.
- Related papers: Search Agents Need Better Decision Boundaries (SearchJev) · RankZephyr / Listwise LLM Reranking · Also in this issue: #1 Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search
Reinjecting Codebook Structure into Semantic ID Decoders (from Kuaishou) — Paper
- What it's useful for: Use this method (from Kuaishou) to construct and decode discrete Semantic IDs that preserve collaborative and semantic item topology without codebook collapse or item-churn drift.
- Related papers: TIGER: Recommender Systems with Generative Retrieval · A Semantic Tokenizer Cannot Be Judged Without Its Decoder (FLASH) · Also in this issue: #4 Scaling Laws for Dense Retriever Fine-Tuning
Scaling Laws for Dense Retriever Fine-Tuning (from Fu et al.) — Paper
- What it's useful for: Use this architecture (from Fu et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #3 Reinjecting Codebook Structure into Semantic ID Decoders
Learning Semantic IDs End-to-End with Collaborative Signals for CTR Prediction (from Jiang et al.) — Paper
- What it's useful for: Use this method (from Jiang et al.) to construct and decode discrete Semantic IDs that preserve collaborative and semantic item topology without codebook collapse or item-churn drift.
- Related papers: TIGER: Recommender Systems with Generative Retrieval · A Semantic Tokenizer Cannot Be Judged Without Its Decoder (FLASH) · Also in this issue: #3 Reinjecting Codebook Structure into Semantic ID Decoders
A Two-Stage Model-Centric Retriever for Billion-Scale Content Search (from Tencent) — Paper
- What it's useful for: Use this agentic search recipe (from Tencent) to train or control multi-step research agents with grounded rewards, fast sufficiency judgments, and programmatic evidence filtering.
- Related papers: Search Agents Need Better Decision Boundaries, Not Just Bigger Models · Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG · Also in this issue: #10 Residual Self-Distillation for Retrieval-Free Parametric Memory
Predicting Per-Query Re-Ranking Depth from First-Stage Retrieval (from Huo et al.) — Paper
- What it's useful for: Use this architecture (from Huo et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #3 Reinjecting Codebook Structure into Semantic ID Decoders
Reasoning Traces as Attention Queries for Listwise Reranking (from Yan et al.) — Paper
- What it's useful for: Use this technique (from Yan et al.) to replace slow autoregressive LLM reranking with calibrated single-pass scoring, reusable document prefixes, or query-adaptive reranking depth.
- Related papers: Search Agents Need Better Decision Boundaries (SearchJev) · RankZephyr / Listwise LLM Reranking · Also in this issue: #1 Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search
How BERT-based Cross-Encoders Model Relevance in Information Retrieval? (from Vast et al.) — Paper
- What it's useful for: Use this technique (from Vast et al.) to replace slow autoregressive LLM reranking with calibrated single-pass scoring, reusable document prefixes, or query-adaptive reranking depth.
- Related papers: Search Agents Need Better Decision Boundaries (SearchJev) · RankZephyr / Listwise LLM Reranking · Also in this issue: #1 Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search
Residual Self-Distillation for Retrieval-Free Parametric Memory (from Han et al.) — Paper
- What it's useful for: Use this study (from Han et al.) to decide when structured agent memory or parametric self-distillation pays off over fast raw-history search, avoiding unnecessary memory extraction overhead.
- Related papers: Lineage Is Memory: Parent Inheritance and Feedback Disclosure · Harness²: What an Agent Can Learn by Changing Its Harness · Also in this issue: #6 A Two-Stage Model-Centric Retriever for Billion-Scale Content Search
Detailed Paper Sections
[1] Modular and Efficient Approximate Nearest Neighbors Search with Vectorium and kANNolo
- Newsletter Headline: Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search (from Martinico et al.)
- Links: arXiv:2611.03927 · Digest Section · Code / Resource · Code / Resource · Code / Resource
- Research Area: Retrieval-Augmented Generation · Late-Interaction & Multi-Vector Retrieval
Main Content & Technical Findings
This paper from Martinico et al. presents Vectorium and kANNolo, two open-source Rust libraries that split approximate nearest neighbor search into a data-management layer and an indexing layer. The authors argue that existing libraries such as FAISS hard-code one class per index and quantizer combination, and that most of them handle only dense embeddings. They instead abstract the vector, distance function, quantizer, and dataset independently, across dense, sparse, and multi-vector representations. Vectorium provides the data layer with 11 encoders, ranging from 32-bit floats to sub-bit codes. These include reimplementations of Product Quantization and RaBitQ, plus the authors’ own encoders for learned sparse vectors. kANNolo builds HNSW and IVF indexes on top of it that are generic over representation, encoder, and metric. It also adds a two-stage RerankIndex that pairs any first-stage index with any rescoring dataset, such as 1-bit RaBitQ codes over an HNSW graph rescored on f16 vectors. Rust specializes every combination at compile time, so the modularity adds no run-time cost, though compilation takes longer and binaries are larger. The individual components match or beat their reference implementations. Their PQ matches FAISS in accuracy at competitive speed, their RaBitQ scan runs 1.4–2.4x faster than the original code, and their IVF beats FAISS.

Related Papers
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction — Residual compression baseline for token-level late interaction.
- Knowing When to Intervene: RL, Search Agents, and RAG — LearnLLM digest on retrieval efficiency and adaptive pooling.
- #2 in this issue: Item-Space On-Policy Distillation for Efficient LLM Reranking (from Meta) — Companion study in this newsletter.
[2] DecisionRank: On-Policy Distillation for Decision-Model Rerankers
- Newsletter Headline: Item-Space On-Policy Distillation for Efficient LLM Reranking (from Meta)
- Links: arXiv:2610.42128 · Digest Section
- Research Area: Retrieval-Augmented Generation · LLM & Cross-Encoder Reranking
Main Content & Technical Findings
This paper from Meta studies how to distill a generative LLM reranker into a decision model. The decision model ranks all candidates in one forward pass by sorting the first-token logits of document IDs, the same approach FIRST uses. The problem is that the teacher states its preferences as an autoregressive sequence of IDs, while the student only produces a static score vector. Offline teacher rankings never give feedback on the student’s own ranking, and standard token-level on-policy distillation needs autoregressive rollouts that the student never uses at inference. DecisionRank builds the student’s rollout by sorting its scores, takes the top-K candidates, and feeds that order to a frozen teacher as a teacher-forced answer. A single forward pass gives the teacher’s conditional choice distribution over the remaining documents at each step. The student’s distribution at each step comes from renormalizing its same scores over that remaining set. The two are matched with reverse KL over candidate items rather than vocabulary tokens, so the teacher and student can come from different model families. This on-policy loss is trained jointly with a weighted RankNet loss on the teacher’s offline sliding-window rankings. The students are Qwen3 models at 1.7B, 4B, and 8B; the teacher is Qwen3.8-27B, and training uses only 16K queries. All three students beat the teacher on TREC DL19–23. The 8B student has the best averages on TREC DL, BEIR, and BRIGHT among the compared rerankers and runs 24.5x faster than the listwise DirectRank-8B on TREC DL and 16.7x faster on BRIGHT.

Related Papers
- Search Agents Need Better Decision Boundaries (SearchJev) — LearnLLM review of calibrated single-token System-1 decision models.
- RankZephyr / Listwise LLM Reranking — Baseline autoregressive listwise LLM reranking.
- #8 in this issue: Reasoning Traces as Attention Queries for Listwise Reranking (from Yan et al.) — Companion study in this newsletter.
[3] BRIDGE: Bridging the Semantic Quantization Gap for Generative Recommendation
- Newsletter Headline: Reinjecting Codebook Structure into Semantic ID Decoders (from Kuaishou)
- Links: arXiv:2611.07425 · Digest Section
- Research Area: Recommender Systems · Semantic IDs & Codebook Quantization
Main Content & Technical Findings
This paper from Kuaishou looks at generative recommendation, where an RQ-VAE turns each item into a hierarchical sequence of Semantic IDs (SIDs) and a decoder generates them level by level from the user’s history. The authors argue that the decoder only sees discrete code indices and loses the continuous structure the quantizer used to build them, a mismatch they call the Semantic Quantization Gap. They split this gap into three problems. First, the decoder has no semantic anchor when it predicts the first SID level, which is the hardest level to predict. Second, one-hot cross-entropy treats every wrong SID as equally wrong and does nothing about exposure bias from teacher forcing. Third, a standard decoder spends equal compute on every level even though later residual levels are much easier. Their solution, BRIDGE, handles each problem with a separate component, and all three share one looped decoder. A latent reasoning pass runs before SID generation, and two losses pull its output toward frozen coarse prototypes made by re-clustering the level-1 codebook centroids. An EMA copy of the model acts as a teacher that also sees the mean embedding of the target item’s nearest semantic neighbors. Its soft distributions are distilled into a student that decodes from its own predicted prefixes. On Amazon Beauty, Sports and Toys, BRIDGE beats item-ID baselines and SID-based methods including TIGER, ReaRec, COBRA and GR4AD.

Related Papers
- TIGER: Recommender Systems with Generative Retrieval — Foundational RQ-VAE Semantic ID generative recommender.
- A Semantic Tokenizer Cannot Be Judged Without Its Decoder (FLASH) — LearnLLM deep dive on SimHash parallel decoding & alignment.
- Reachability Is Not Ranking: Finite Beams & Continuous Queries — LearnLLM analysis of beam reachability vs. ranking in SID models.
- #5 in this issue: Learning Semantic IDs End-to-End with Collaborative Signals for CTR Prediction (from Jiang et al.) — Companion study in this newsletter.
[4] Scaling Laws For Data-Constrained Dense Retrieval
- Newsletter Headline: Scaling Laws for Dense Retriever Fine-Tuning (from Fu et al.)
- Links: arXiv:2611.04979 · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
This paper from Fu et al. looks at how dense retrieval performance scales when fine-tuning data is limited and reused over many epochs (a setting that standard scaling laws don’t cover because they assume abundant unique data). The authors measure the minimum achievable evaluation loss (MAEL), meaning the lowest evaluation loss a run reaches for a given model size and unique-data budget. A learning-rate sweep on BERT-Medium shows that the learning rate mostly controls how fast a run reaches its minimum, not how low it gets. Once training steps are rescaled by learning rate, the loss curves line up, MAEL varies by only about 3% of the data-scaling range, and CKA analysis shows the best checkpoints learn nearly identical representations. Across 35 BERT models from Tiny to Base, trained with InfoNCE on 5K to 320K MS MARCO pairs, MAEL fits a multiplicative power law in non-embedding parameters and unique samples (R² of 0.99). MAEL also tracks retrieval quality closely, with Spearman correlations near −0.996 against MRR@10, nDCG@10, Recall@100 and MAP@10. The authors fit a second law for the step at which MAEL is reached. They also find that the best weight decay corresponds to a roughly constant AdamW EMA timescale, so ηλU should stay fixed as the data budget changes, and this tuning gives small but consistent gains both in-domain and on NQ, HotpotQA and FiQA. Combining the loss and step laws, they estimate that scaling up the pretrained model is about 1.46 times as compute-efficient as adding unique data.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #7 in this issue: Predicting Per-Query Re-Ranking Depth from First-Stage Retrieval (from Huo et al.) — Companion study in this newsletter.
[5] SIDRec: End-to-End Semantic ID Representation and Modeling for Recommendation
- Newsletter Headline: Learning Semantic IDs End-to-End with Collaborative Signals for CTR Prediction (from Jiang et al.)
- Links: arXiv:2611.01136 · Digest Section
- Research Area: Recommender Systems · Semantic IDs & Codebook Quantization
Main Content & Technical Findings
This paper from Jiang et al. introduces SIDRec, a framework that learns semantic IDs (SIDs) jointly with a discriminative CTR ranker instead of building them offline and freezing them. The authors see two problems with current SID methods. SIDs built from content embeddings miss collaborative signal, since items that look alike can attract different users and vice versa. Fixed SIDs also mean the ranking loss never shapes the code assignments. To handle the first problem, SIDRec fuses fixed multimodal item embeddings with collaborative embeddings initialized from a trained ranking model. A learnable sigmoid gate controls the fusion, the collaborative table keeps updating during training, and the fused vector feeds a three-level RQ-VAE tokenizer. To handle the second, the model uses hard codes in the forward pass and Gumbel-perturbed soft assignments in the backward pass, so CTR gradients reach the tokenizer’s encoder and codebooks. The ranker runs a causal Transformer over the SID tokens of a user’s history and applies DIN target attention against the candidate. An MMoE with four shared experts and per-task gates trains click prediction together with an auxiliary task that predicts the positive target item’s SID, and a reconstruction loss keeps the codes tied to item content. On Amazon Electronics, Amazon Beauty, and MovieLens-20M, SIDRec beats fifteen CTR baselines on both AUC and GAUC. Offline tests on two Kuaishou domains show gains for exposure, click, and conversion prediction.

Related Papers
- TIGER: Recommender Systems with Generative Retrieval — Foundational RQ-VAE Semantic ID generative recommender.
- A Semantic Tokenizer Cannot Be Judged Without Its Decoder (FLASH) — LearnLLM deep dive on SimHash parallel decoding & alignment.
- Reachability Is Not Ranking: Finite Beams & Continuous Queries — LearnLLM analysis of beam reachability vs. ranking in SID models.
- #3 in this issue: Reinjecting Codebook Structure into Semantic ID Decoders (from Kuaishou) — Companion study in this newsletter.
[6] WeSearch: A Unified Model-Centric Retrieval Framework for WeChat Search
- Newsletter Headline: A Two-Stage Model-Centric Retriever for Billion-Scale Content Search (from Tencent)
- Links: arXiv:2611.00530 · Digest Section
- Research Area: LLM Agents · Agentic Search & Deep Research Control
Main Content & Technical Findings
This paper from Tencent describes WeSearch, the retrieval framework that now serves all production traffic in WeChat Search. The authors argue that generative retrieval with hierarchical semantic IDs is a poor fit for content search. Articles and graphic-text posts often cover several topics, so they don’t sit cleanly on a single path, and an early decoding error drops every relevant document under the correct branch. WeSearch keeps clusters but uses them only to shrink the candidate space. A query encoder, initialized from a dual-tower retriever, scores the query against two-level RQ-KMeans cluster centroids, and the members of the top 10,000 clusters form the candidate pool. A coarse ranker shortlists candidates using OPQ-compressed document codes and asymmetric distance computation. A small FFN then re-ranks the shortlist from the query embedding, the reconstructed document embedding, and bucketed document features such as quality, click statistics, and recency. The OPQ rotation and codebooks are trained as model parameters, and the loss combines a cluster-level softmax, a listwise loss over the shortlist, and a pairwise margin loss. Positives come from relevance labels, satisfaction signals, and user behavior. Hard negatives come from lower-graded, weakly clicked, and wrongly retrieved documents. Rolling training and an incremental step that assigns new documents to the nearest existing cluster let the team refresh the model weekly or even daily, where it used to be monthly. In production, WeSearch replaces several parallel lexical and dense paths and runs next to one lexical syntax tree. On a corpus of hundreds of millions of documents, a semantic ID generative retriever trails both the old system and WeSearch at every recall cutoff. An online, one-month full-traffic test shows +0.68% CTR and +2.27% per-user article clicks, with 41% lower retrieval resource cost and 19 ms lower graphic-text retrieval latency.

Related Papers
- Search Agents Need Better Decision Boundaries, Not Just Bigger Models — LearnLLM deep dive on SearchJev System-1/System-2 delegation.
- Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG — LearnLLM digest on T-Search, Programmatic Search Agents, and AutoRAG.
- #7 in this issue: Predicting Per-Query Re-Ranking Depth from First-Stage Retrieval (from Huo et al.) — Companion study in this newsletter.
[7] Optimal Re-Ranking Depth
- Newsletter Headline: Predicting Per-Query Re-Ranking Depth from First-Stage Retrieval (from Huo et al.)
- Links: arXiv:2611.05908 · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
This paper from Huo et al. asks whether neural re-rankers should re-rank every query to the same fixed depth. The authors use the UMBRELA framework with GPT-4o-mini to build dense relevance judgments for 1,000 MSMARCO v1 dev queries, replicate the setup on TREC DL 2019, and trace DCG@10 against re-ranking depth for each query. The pipelines pair BM25 and SPLADE with MonoT5 and MonoELECTRA. Many queries have a clear optimal depth, defined as the shallowest depth that reaches their best DCG@10. Re-ranking past that depth often stalls or hurts effectiveness, sometimes falling below the first-stage ranking. The authors then try to predict this depth before re-ranking. They bin training queries by a predictor score computed on the first-stage list and give each test query a chosen percentile of the optimal depths in its bin. They sweep the percentile to find the shallowest setting that matches full re-ranking to depth 1000. Eleven standard QPP methods, including NQC, WIG, UEF, and Dense-QPP, mostly fail at this task, and the best of them (RSD) only halves the depth. Their own predictor, IR-DCG@10, is the DCG@10 of the first-stage top 10 as scored by the LLM judge. Without losing effectiveness against full re-ranking, it cuts average depth by a factor of three for BM25 with MonoT5 and by about 15 for SPLADE with MonoELECTRA.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #4 in this issue: Scaling Laws for Dense Retriever Fine-Tuning (from Fu et al.) — Companion study in this newsletter.
[8] Re³Rank: Listwise Reranking with Reasoning-Conditioned Retrieval Heads
- Newsletter Headline: Reasoning Traces as Attention Queries for Listwise Reranking (from Yan et al.)
- Links: arXiv:2611.02550 · Digest Section
- Research Area: Retrieval-Augmented Generation · LLM & Cross-Encoder Reranking
Main Content & Technical Findings
This paper from Yan et al. proposes RE³Rank, a listwise reranker that extends attention-based reranking with query-focused retrieval heads. Those earlier methods score each candidate by the attention mass that a few selected heads route to it from the question tokens. The question often lacks the bridge entities and intermediate facts needed to find the evidence, so RE³Rank uses a generated reasoning trace as the attention query instead. A generator reads the question and the top 50 candidates and writes a trace. A separate scorer on the same backbone reads that input plus the trace and scores each candidate by the attention mass that two selected heads (RRHeads) send to it from the trace tokens, with no permutation decoded. The authors train both models jointly with a contrastive cross-entropy loss and a soft Recall@k surrogate. A straight-through estimator carries the ranking gradient through the discrete trace into the generator. They build the model on Qwen3 at 0.6B, 4B, and 8B and train it on 38,000 questions. Averaged over nine benchmarks covering multi-hop, open-domain, long-narrative, and conversational-memory QA, it beats 11 baselines.

Related Papers
- Search Agents Need Better Decision Boundaries (SearchJev) — LearnLLM review of calibrated single-token System-1 decision models.
- RankZephyr / Listwise LLM Reranking — Baseline autoregressive listwise LLM reranking.
- #2 in this issue: Item-Space On-Policy Distillation for Efficient LLM Reranking (from Meta) — Companion study in this newsletter.
[9] How BERT-based Cross-Encoders Model Relevance in Information Retrieval?
- Newsletter Headline: How BERT-based Cross-Encoders Model Relevance in Information Retrieval? (from Vast et al.)
- Links: arXiv:2610.41425 · Digest Section · Code / Resource
- Research Area: Retrieval-Augmented Generation · LLM & Cross-Encoder Reranking
Main Content & Technical Findings
This paper from Vast et al. studies how BERT-based cross-encoders compute relevance. It compares MiniLM-L12, MonoBERT-base and MonoBERT-large on TREC-DL 2019–2022 as in-domain data and on several BEIR datasets as out-of-domain data. The authors combine several interpretability methods and cross-check them against each other: Neuron Integrated Gradients that separate query, document, and special-token segments, attention-mask ablations, an Information Bottleneck variant, attention patterns compared between relevant and non-relevant pairs, and Linear Discriminant Analysis of hidden states. All three models follow the same broad process. Early layers contextualize query and document tokens, and information moves in both directions between them. Specialized attention heads then detect exact lexical matches in early-to-middle layers and semantic matches between contextualized tokens in later layers. A Query-Key analysis shows these heads prefer cross-attention between similar tokens. Relevance information collects in the query tokens and reaches [CLS] only in the last few layers, so before that [CLS] acts as a no-op sink, as [SEP] does. The models differ in the final stage. MonoBERT-large builds a relevance subspace inside [CLS] from raw signals. The shallower MiniLM and MonoBERT-base first refine signals in the query and document tokens, and MiniLM seems to look for evidence of non-relevance rather than relevance. The mechanism stays mostly the same under domain shift. MonoBERT does rely on some components that matter only out of domain, and all three models handle out-of-domain hard negatives poorly, likely because they lean on lexical matches. The authors also find signs that MonoBERT is over-parameterized and recommend MiniLM for its better efficiency and robustness.
Related Papers
- Search Agents Need Better Decision Boundaries (SearchJev) — LearnLLM review of calibrated single-token System-1 decision models.
- RankZephyr / Listwise LLM Reranking — Baseline autoregressive listwise LLM reranking.
- #2 in this issue: Item-Space On-Policy Distillation for Efficient LLM Reranking (from Meta) — Companion study in this newsletter.
[10] From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory
- Newsletter Headline: Residual Self-Distillation for Retrieval-Free Parametric Memory (from Han et al.)
- Links: arXiv:2611.00450 · Digest Section · Code / Resource · Code / Resource
- Research Area: LLM Agents · Agentic & Parametric Memory Systems
Main Content & Technical Findings
This paper from Han et al. introduces ResMem, which stores retrieved knowledge in a parametric memory so an LLM can use it without retrieving anything at inference time. Earlier approaches such as MLP Memory and Memory Decoder train a separate memory to imitate a kNN retriever’s next-token distribution. The memory and the frozen base model are combined only at inference, so the memory can override answers the base already got right. ResMem instead uses the frozen base model as its own teacher. Offline, the model reads BM25-retrieved Wikipedia passages together with the text, and its top-64 token probabilities are cached as targets. The student is the same frozen model without the evidence. An eight-layer MLP memory reads one of the base model’s last-layer hidden states and adds a logit residual to the student’s logits. The KL and cross-entropy losses are computed on this combined prediction. As a result, the memory learns a correction to the base model rather than a second copy of the retriever, and a scale γ sets how much the residual shifts the base model’s predictions at inference. Because cached teacher targets replace the kNN datastore, retained storage on a Wikipedia subset drops from 602 GiB to 32 GiB. With Mistral-7B and a 1.5B-parameter memory, ResMem raises mean F1 across five QA datasets from 32.83 to 34.48. It outperforms LoRA, Memory Decoder, MLP Memory and BM25-RAG, and its margin over MLP Memory widens on long-tail and multi-hop questions.
🤗 https://huggingface.co/RandHan/ResMem-Mistral-7B-v0.3

Related Papers
- Lineage Is Memory: Parent Inheritance and Feedback Disclosure — LearnLLM deep dive on prompt lineage as externalized memory (arXiv:2610.41157).
- Harness²: What an Agent Can Learn by Changing Its Harness — LearnLLM review of experience libraries in recursive agent harnesses.
- #1 in this issue: Composable Rust Libraries for Dense, Sparse, and Multi-Vector ANN Search (from Martinico et al.) — Companion study in this newsletter.