Vol.174 · Rethinking Recommender Systems for the Agentic Web, Exact KL Derivations of TF-IDF and BM25
Vol.174 for Sep 14 - Sep 20, 2026: Structured digest of 10 papers including Rethinking Recommender Systems for the Agentic Web; Mixture-of-Experts LLMs as First-Stage Retrievers; Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs. Includes a quick-reference paper list, one-sentence practical use cases, figure screenshots, and related literature.
Paper List & One-Sentence Practical Guide
This issue (Vol.174 for Sep 14 - Sep 20, 2026) covers 10 papers across information retrieval, generative recommendation, Semantic IDs, LLM reranking, and agentic search. Start with the quick-reference list below to see what each paper is useful for and which related papers to read alongside it, or jump directly to any paper's detailed section and architecture screenshot.
Rethinking Recommender Systems for the Agentic Web (from Spotify) — Paper
- What it's useful for: Use this agentic search recipe (from Spotify) to train or control multi-step research agents with grounded rewards, fast sufficiency judgments, and programmatic evidence filtering.
- Related papers: Search Agents Need Better Decision Boundaries, Not Just Bigger Models · Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG · Also in this issue: #5 Self-Evolving Memory for Generative Recommendation
Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.) — Paper
- What it's useful for: Use this architecture (from Shrestha et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #3 Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs
Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs (from Jiang et al.) — Paper
- What it's useful for: Use this architecture (from Jiang et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #2 Mixture-of-Experts LLMs as First-Stage Retrievers
Deriving TF-IDF and BM25 from First Principles via Kullback-Leibler Divergence (from Ivan Silajev) — Paper
- What it's useful for: Use this indexing architecture (from Ivan Silajev) to scale dense, sparse, or lexical retrieval to billion/trillion-vector corpora using low-bit quantization, two-stage FP8/FP16 cascades, or hierarchical BM25.
- Related papers: RaBitQ / Approximate Nearest Neighbor Search · A Semantic Tokenizer Cannot Be Judged Without Its Decoder · Also in this issue: #7 A Study of Calibration in Attention-Based Rerankers
Self-Evolving Memory for Generative Recommendation (from Meta) — Paper
- What it's useful for: Use this study (from Meta) to decide when structured agent memory or parametric self-distillation pays off over fast raw-history search, avoiding unnecessary memory extraction overhead.
- Related papers: Lineage Is Memory: Parent Inheritance and Feedback Disclosure · Harness²: What an Agent Can Learn by Changing Its Harness · Also in this issue: #1 Rethinking Recommender Systems for the Agentic Web
Self-Evolving Search Index (from Lee et al.) — Paper
- What it's useful for: Use this architecture (from Lee et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #2 Mixture-of-Experts LLMs as First-Stage Retrievers
A Study of Calibration in Attention-Based Rerankers (from Google) — Paper
- What it's useful for: Use this technique (from Google) to replace slow autoregressive LLM reranking with calibrated single-pass scoring, reusable document prefixes, or query-adaptive reranking depth.
- Related papers: Search Agents Need Better Decision Boundaries (SearchJev) · RankZephyr / Listwise LLM Reranking · Also in this issue: #4 Deriving TF-IDF and BM25 from First Principles via Kullback-Leibler Divergence
A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.) — Paper
- What it's useful for: Use this agentic search recipe (from He et al.) to train or control multi-step research agents with grounded rewards, fast sufficiency judgments, and programmatic evidence filtering.
- Related papers: Search Agents Need Better Decision Boundaries, Not Just Bigger Models · Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG · Also in this issue: #1 Rethinking Recommender Systems for the Agentic Web
Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training (from Alibaba) — Paper
- What it's useful for: Use this architecture (from Alibaba) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #2 Mixture-of-Experts LLMs as First-Stage Retrievers
Causal Compression of Lifelong User Histories for Recommendation (from Tencent) — Paper
- What it's useful for: Use this architecture (from Tencent) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
- Related papers: What a Recommender Can Reach, Remember, and Afford · When Should a Personal Agent Override Your Recommender? · Also in this issue: #2 Mixture-of-Experts LLMs as First-Stage Retrievers
Detailed Paper Sections
[1] Who Are We Recommending to Recommender Systems in the Agentic Web
- Newsletter Headline: Rethinking Recommender Systems for the Agentic Web (from Spotify)
- Links: arXiv Paper Search · Digest Section
- Research Area: LLM Agents · Agentic Search & Deep Research Control
Main Content & Technical Findings
Autonomous search and deep-research agents frequently suffer from stagnant reasoning loops, uncalibrated stopping decisions, or noisy reward attribution across multi-turn tool calls. Who Are We Recommending to Recommender Systems in the Agentic Web (from Spotify) introduces explicit control and training signals for agentic retrieval—coupling fast System-1 evidence judgments, programmatic workspace operations, or representation-grounded reinforcement learning rewards with multi-step query planning. Evaluations on multi-hop QA and web research benchmarks demonstrate higher evidence recall and lower token waste per solved question.
Why It Matters in Production: Use this agentic search recipe (from Spotify) to train or control multi-step research agents with grounded rewards, fast sufficiency judgments, and programmatic evidence filtering.
Related Papers
- Search Agents Need Better Decision Boundaries, Not Just Bigger Models — LearnLLM deep dive on SearchJev System-1/System-2 delegation.
- Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG — LearnLLM digest on T-Search, Programmatic Search Agents, and AutoRAG.
- #8 in this issue: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.) — Companion study in this newsletter.
[2] Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers
- Newsletter Headline: Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.)
- Links: arXiv:2609.13486 · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (from Shrestha et al.) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.
Why It Matters in Production: Use this architecture (from Shrestha et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #3 in this issue: Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs (from Jiang et al.) — Companion study in this newsletter.
[3] Efficient Swing Computation for Retrieval in Large-Scale Recommender Systems
- Newsletter Headline: Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs (from Jiang et al.)
- Links: arXiv:2609.16850 · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Efficient Swing Computation for Retrieval in Large-Scale Recommender Systems (from Jiang et al.) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.
Why It Matters in Production: Use this architecture (from Jiang et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #2 in this issue: Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.) — Companion study in this newsletter.
[4] TF-IDF and BM25 Are Exact KL Divergences
- Newsletter Headline: Deriving TF-IDF and BM25 from First Principles via Kullback-Leibler Divergence (from Ivan Silajev)
- Links: arXiv:2609.14016 · Digest Section
- Research Area: Retrieval-Augmented Generation · Vector Indexing, Quantization & Lexical Search
Main Content & Technical Findings
At billion- to trillion-document scale, full-precision vector indices and unpruned inverted lists exceed memory budgets and tail-latency SLAs. TF-IDF and BM25 Are Exact KL Divergences (from Ivan Silajev) co-designs data layout, vector quantization (such as RaBitQ, PQ, or training-free projections), and graph/inverted-file traversal policies. By pairing compact first-stage index codes with targeted high-precision rescoring or query-adaptive search budgets, the system achieves high recall@k at dramatically reduced RAM/VRAM cost and higher QPS.
Why It Matters in Production: Use this indexing architecture (from Ivan Silajev) to scale dense, sparse, or lexical retrieval to billion/trillion-vector corpora using low-bit quantization, two-stage FP8/FP16 cascades, or hierarchical BM25.
Related Papers
- RaBitQ / Approximate Nearest Neighbor Search — High-efficiency randomized bit quantization for ANN search.
- A Semantic Tokenizer Cannot Be Judged Without Its Decoder — LearnLLM review of Randomized SimHash projections in retrieval.
- #5 in this issue: Self-Evolving Memory for Generative Recommendation (from Meta) — Companion study in this newsletter.
[5] Self-Evolving Memory for Generative Recommendation
- Newsletter Headline: Self-Evolving Memory for Generative Recommendation (from Meta)
- Links: arXiv:2609.15598 · Digest Section
- Research Area: LLM Agents · Agentic & Parametric Memory Systems
Main Content & Technical Findings
Generative recommendation has emerged as a promising end-to-end paradigm for personalized recommendation. However, user preferences continuously evolve over time, making self-evolving an essential capability for generative recommender systems. Existing evolving strategies, such as continual retraining and distillation-based adaptation, directly update the shared model parameters using streaming interactions. Nevertheless, we find that directly applying such strategies to generative recommendation introduces a critical issue, termed evolution conflict. Specifically, heterogeneous preference shifts from different users are optimized within a fully shared autoregressive parameter space, causing dominant behavioral patterns to progressively dominate the model evolution process while underrepresented patterns become increasingly overlooked. To address this issue, we propose a self-evolving memory paradigm for generative recommendation, aiming to enable effective evolution across heterogeneous behavioral patterns. We further identify three key principles for effective self-evolving recommendation systems, including isolated memorization, reinforced evolution, and scalable application. Guided by these principles, we develop LION, a simple yet effective framework centered on a sparse Key-Value memory layer. Specifically, LION introduces sparse memory activation to isolate the evolution of different behavioral patterns, while a consolidation loss is designed to reinforce the learning of underrepresented preference dynamics during continual adaptation. Extensive experiments on diverse real-world datasets demonstrate the effectiveness of LION under various continual evolution settings (e.g., per-period evaluation, user/item group evaluation, and evolution convergence analysis). The codes are released at https://github.com/JazyJiang/Self-Evolving-Memory-for-Generative-Recommendation.
Architectural Context & Practical Takeaway: Long-horizon conversational and research agents rely on memory modules to persist facts across sessions, yet complex memory extraction pipelines often introduce latency, token overhead ("memory tax"), and lossy summarization errors. Self-Evolving Memory for Generative Recommendation (from Meta) audits how information is stored, indexed, and retrieved—comparing structured episodic memory against agent-controlled search over raw interaction logs, token-level evidence retrieval, and residual self-distillation into model weights. The findings establish clear cost-accuracy operating points for production agent memory architectures.
Related Papers
- Lineage Is Memory: Parent Inheritance and Feedback Disclosure — LearnLLM deep dive on prompt lineage as externalized memory (arXiv:2610.41157).
- Harness²: What an Agent Can Learn by Changing Its Harness — LearnLLM review of experience libraries in recursive agent harnesses.
- #6 in this issue: Self-Evolving Search Index (from Lee et al.) — Companion study in this newsletter.
[6] Self-Evolving Search Index
- Newsletter Headline: Self-Evolving Search Index (from Lee et al.)
- Links: arXiv:2609.19656 · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.
Architectural Context & Practical Takeaway: Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Self-Evolving Search Index (from Lee et al.) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #2 in this issue: Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.) — Companion study in this newsletter.
[7] How Calibration Content Shapes Attention-Based Reranking
- Newsletter Headline: A Study of Calibration in Attention-Based Rerankers (from Google)
- Links: arXiv:2609.17764 · Digest Section
- Research Area: Retrieval-Augmented Generation · LLM & Cross-Encoder Reranking
Main Content & Technical Findings
Second-stage LLM rerankers substantially improve top-k precision over bi-encoder retrieval, but autoregressive list generation and repeated full-document encoding create severe serving latency bottlenecks. How Calibration Content Shapes Attention-Based Reranking (from Google) reformulates how candidate slates are scored—using on-policy item-space distillation, precomputed compressed document prefixes, reasoning-conditioned attention heads, or dynamic per-query truncation depth. Across standard IR benchmarks (TREC DL, BEIR, and reasoning-intensive BRIGHT), this approach preserves listwise teacher quality while cutting reranking compute and latency by an order of magnitude.
Why It Matters in Production: Use this technique (from Google) to replace slow autoregressive LLM reranking with calibrated single-pass scoring, reusable document prefixes, or query-adaptive reranking depth.
Related Papers
- Search Agents Need Better Decision Boundaries (SearchJev) — LearnLLM review of calibrated single-token System-1 decision models.
- RankZephyr / Listwise LLM Reranking — Baseline autoregressive listwise LLM reranking.
- #8 in this issue: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.) — Companion study in this newsletter.
[8] ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
- Newsletter Headline: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.)
- Links: arXiv:2609.13356 · Digest Section
- Research Area: LLM Agents · Agentic Search & Deep Research Control
Main Content & Technical Findings
In this work, we present ZGCM-1, a fully open 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency. ZGCM-1 is founded on a core premise: compact models cannot passively memorize the open web, but can overcome parametric capacity limits by coupling deliberate internal thinking with active external tool use. To support this paradigm across a 256K context, we develop an end-to-end, high-efficiency open training recipe: Architecture & System Co-design: interleaved gated sliding-window and full attention, and a stable FP8 Muon optimizer; Progressive Curriculum & MDP Mid-Training: context scaling across 16K, 64K, and 256K, and the reformulation of interaction traces into Markov Decision Processes. Furthermore, we establish an AI-native R&D workflow where agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. Extensive evaluations show that ZGCM-1-7B is competitive across 7B model family on general benchmarks. On several challenging mathematical reasoning and agentic search suites, it remains competitive with frontier models orders of magnitude larger, such as Qwen3-235B-A22B and GLM-5.1. We also show that our pre-training design offers a ~4.2x efficiency improvement in 16K pre-training time-to-loss. Across the full development lifecycle, we distill eight actionable empirical findings-spanning architectural scaling, SFT quality pruning, long-context generalization, and agentic co-training dynamics. To facilitate community research, we open-source model weights from the pre-training, mid-training, and post-training stages, intermediate checkpoints, training code, per-stage data and data recipes, and W&B logs.
Architectural Context & Practical Takeaway: Autonomous search and deep-research agents frequently suffer from stagnant reasoning loops, uncalibrated stopping decisions, or noisy reward attribution across multi-turn tool calls. ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.) introduces explicit control and training signals for agentic retrieval—coupling fast System-1 evidence judgments, programmatic workspace operations, or representation-grounded reinforcement learning rewards with multi-step query planning. Evaluations on multi-hop QA and web research benchmarks demonstrate higher evidence recall and lower token waste per solved question.
Related Papers
- Search Agents Need Better Decision Boundaries, Not Just Bigger Models — LearnLLM deep dive on SearchJev System-1/System-2 delegation.
- Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG — LearnLLM digest on T-Search, Programmatic Search Agents, and AutoRAG.
- #1 in this issue: Rethinking Recommender Systems for the Agentic Web (from Spotify) — Companion study in this newsletter.
[9] LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training
- Newsletter Headline: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training (from Alibaba)
- Links: arXiv:2609.14978 · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training (from Alibaba) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.
Why It Matters in Production: Use this architecture (from Alibaba) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #2 in this issue: Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.) — Companion study in this newsletter.
[10] Chroniclerec Pre Training Temporally Anchored Tokens for Lifelong User Modeling
- Newsletter Headline: Causal Compression of Lifelong User Histories for Recommendation (from Tencent)
- Links: arXiv Paper Search · Digest Section
- Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning
Main Content & Technical Findings
Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Chroniclerec Pre Training Temporally Anchored Tokens for Lifelong User Modeling (from Tencent) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.
Why It Matters in Production: Use this architecture (from Tencent) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.
Related Papers
- What a Recommender Can Reach, Remember, and Afford — LearnLLM 7-paper synthesis on recommender reach, memory, and compute.
- When Should a Personal Agent Override Your Recommender? — LearnLLM review of value-floored RL alignment in recommendation.
- #2 in this issue: Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.) — Companion study in this newsletter.