Vol.174 · Rethinking Recommender Systems for the Agentic Web, Exact KL Derivations of TF-IDF and BM25

Key Takeaway

Vol.174 for Sep 14 - Sep 20, 2026: Structured digest of 10 papers including Rethinking Recommender Systems for the Agentic Web; Mixture-of-Experts LLMs as First-Stage Retrievers; Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs. Includes a quick-reference paper list, one-sentence practical use cases, figure screenshots, and related literature.

Paper List & One-Sentence Practical Guide

This issue (Vol.174 for Sep 14 - Sep 20, 2026) covers 10 papers across information retrieval, generative recommendation, Semantic IDs, LLM reranking, and agentic search. Start with the quick-reference list below to see what each paper is useful for and which related papers to read alongside it, or jump directly to any paper's detailed section and architecture screenshot.

  1. Rethinking Recommender Systems for the Agentic Web (from Spotify) — Paper

  2. Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.) — Paper

  3. Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs (from Jiang et al.) — Paper

  4. Deriving TF-IDF and BM25 from First Principles via Kullback-Leibler Divergence (from Ivan Silajev) — Paper

  5. Self-Evolving Memory for Generative Recommendation (from Meta) — Paper

  6. Self-Evolving Search Index (from Lee et al.) — Paper

  7. A Study of Calibration in Attention-Based Rerankers (from Google) — Paper

  8. A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.) — Paper

  9. Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training (from Alibaba) — Paper

  10. Causal Compression of Lifelong User Histories for Recommendation (from Tencent) — Paper


Detailed Paper Sections

[1] Who Are We Recommending to Recommender Systems in the Agentic Web

  • Newsletter Headline: Rethinking Recommender Systems for the Agentic Web (from Spotify)
  • Links: arXiv Paper Search · Digest Section
  • Research Area: LLM Agents · Agentic Search & Deep Research Control

Main Content & Technical Findings

Autonomous search and deep-research agents frequently suffer from stagnant reasoning loops, uncalibrated stopping decisions, or noisy reward attribution across multi-turn tool calls. Who Are We Recommending to Recommender Systems in the Agentic Web (from Spotify) introduces explicit control and training signals for agentic retrieval—coupling fast System-1 evidence judgments, programmatic workspace operations, or representation-grounded reinforcement learning rewards with multi-step query planning. Evaluations on multi-hop QA and web research benchmarks demonstrate higher evidence recall and lower token waste per solved question.

Why It Matters in Production: Use this agentic search recipe (from Spotify) to train or control multi-step research agents with grounded rewards, fast sufficiency judgments, and programmatic evidence filtering.

Figure 1: Architecture and technical summary for Who Are We Recommending to Recommender Systems in the Agentic Web

Related Papers


[2] Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

  • Newsletter Headline: Mixture-of-Experts LLMs as First-Stage Retrievers (from Shrestha et al.)
  • Links: arXiv:2609.13486 · Digest Section
  • Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning

Main Content & Technical Findings

Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (from Shrestha et al.) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.

Why It Matters in Production: Use this architecture (from Shrestha et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.

Figure 2: Architecture and technical summary for Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

Related Papers


[3] Efficient Swing Computation for Retrieval in Large-Scale Recommender Systems

  • Newsletter Headline: Adaptive Randomized Algorithms for Item-to-Item Retrieval on Billion-Edge Graphs (from Jiang et al.)
  • Links: arXiv:2609.16850 · Digest Section
  • Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning

Main Content & Technical Findings

Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Efficient Swing Computation for Retrieval in Large-Scale Recommender Systems (from Jiang et al.) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.

Why It Matters in Production: Use this architecture (from Jiang et al.) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.

Figure 3: Architecture and technical summary for Efficient Swing Computation for Retrieval in Large-Scale Recommender Systems

Related Papers


[4] TF-IDF and BM25 Are Exact KL Divergences

  • Newsletter Headline: Deriving TF-IDF and BM25 from First Principles via Kullback-Leibler Divergence (from Ivan Silajev)
  • Links: arXiv:2609.14016 · Digest Section
  • Research Area: Retrieval-Augmented Generation · Vector Indexing, Quantization & Lexical Search

Main Content & Technical Findings

At billion- to trillion-document scale, full-precision vector indices and unpruned inverted lists exceed memory budgets and tail-latency SLAs. TF-IDF and BM25 Are Exact KL Divergences (from Ivan Silajev) co-designs data layout, vector quantization (such as RaBitQ, PQ, or training-free projections), and graph/inverted-file traversal policies. By pairing compact first-stage index codes with targeted high-precision rescoring or query-adaptive search budgets, the system achieves high recall@k at dramatically reduced RAM/VRAM cost and higher QPS.

Why It Matters in Production: Use this indexing architecture (from Ivan Silajev) to scale dense, sparse, or lexical retrieval to billion/trillion-vector corpora using low-bit quantization, two-stage FP8/FP16 cascades, or hierarchical BM25.

Figure 4: Architecture and technical summary for TF-IDF and BM25 Are Exact KL Divergences

Related Papers


[5] Self-Evolving Memory for Generative Recommendation

  • Newsletter Headline: Self-Evolving Memory for Generative Recommendation (from Meta)
  • Links: arXiv:2609.15598 · Digest Section
  • Research Area: LLM Agents · Agentic & Parametric Memory Systems

Main Content & Technical Findings

Generative recommendation has emerged as a promising end-to-end paradigm for personalized recommendation. However, user preferences continuously evolve over time, making self-evolving an essential capability for generative recommender systems. Existing evolving strategies, such as continual retraining and distillation-based adaptation, directly update the shared model parameters using streaming interactions. Nevertheless, we find that directly applying such strategies to generative recommendation introduces a critical issue, termed evolution conflict. Specifically, heterogeneous preference shifts from different users are optimized within a fully shared autoregressive parameter space, causing dominant behavioral patterns to progressively dominate the model evolution process while underrepresented patterns become increasingly overlooked. To address this issue, we propose a self-evolving memory paradigm for generative recommendation, aiming to enable effective evolution across heterogeneous behavioral patterns. We further identify three key principles for effective self-evolving recommendation systems, including isolated memorization, reinforced evolution, and scalable application. Guided by these principles, we develop LION, a simple yet effective framework centered on a sparse Key-Value memory layer. Specifically, LION introduces sparse memory activation to isolate the evolution of different behavioral patterns, while a consolidation loss is designed to reinforce the learning of underrepresented preference dynamics during continual adaptation. Extensive experiments on diverse real-world datasets demonstrate the effectiveness of LION under various continual evolution settings (e.g., per-period evaluation, user/item group evaluation, and evolution convergence analysis). The codes are released at https://github.com/JazyJiang/Self-Evolving-Memory-for-Generative-Recommendation.

Architectural Context & Practical Takeaway: Long-horizon conversational and research agents rely on memory modules to persist facts across sessions, yet complex memory extraction pipelines often introduce latency, token overhead ("memory tax"), and lossy summarization errors. Self-Evolving Memory for Generative Recommendation (from Meta) audits how information is stored, indexed, and retrieved—comparing structured episodic memory against agent-controlled search over raw interaction logs, token-level evidence retrieval, and residual self-distillation into model weights. The findings establish clear cost-accuracy operating points for production agent memory architectures.

Figure 5: Architecture and technical summary for Self-Evolving Memory for Generative Recommendation

Related Papers


[6] Self-Evolving Search Index

  • Newsletter Headline: Self-Evolving Search Index (from Lee et al.)
  • Links: arXiv:2609.19656 · Digest Section
  • Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning

Main Content & Technical Findings

Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.

Architectural Context & Practical Takeaway: Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Self-Evolving Search Index (from Lee et al.) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.

Figure 6: Architecture and technical summary for Self-Evolving Search Index

Related Papers


[7] How Calibration Content Shapes Attention-Based Reranking

  • Newsletter Headline: A Study of Calibration in Attention-Based Rerankers (from Google)
  • Links: arXiv:2609.17764 · Digest Section
  • Research Area: Retrieval-Augmented Generation · LLM & Cross-Encoder Reranking

Main Content & Technical Findings

Second-stage LLM rerankers substantially improve top-k precision over bi-encoder retrieval, but autoregressive list generation and repeated full-document encoding create severe serving latency bottlenecks. How Calibration Content Shapes Attention-Based Reranking (from Google) reformulates how candidate slates are scored—using on-policy item-space distillation, precomputed compressed document prefixes, reasoning-conditioned attention heads, or dynamic per-query truncation depth. Across standard IR benchmarks (TREC DL, BEIR, and reasoning-intensive BRIGHT), this approach preserves listwise teacher quality while cutting reranking compute and latency by an order of magnitude.

Why It Matters in Production: Use this technique (from Google) to replace slow autoregressive LLM reranking with calibrated single-pass scoring, reusable document prefixes, or query-adaptive reranking depth.

Figure 7: Architecture and technical summary for How Calibration Content Shapes Attention-Based Reranking

Related Papers


[8] ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

  • Newsletter Headline: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.)
  • Links: arXiv:2609.13356 · Digest Section
  • Research Area: LLM Agents · Agentic Search & Deep Research Control

Main Content & Technical Findings

In this work, we present ZGCM-1, a fully open 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency. ZGCM-1 is founded on a core premise: compact models cannot passively memorize the open web, but can overcome parametric capacity limits by coupling deliberate internal thinking with active external tool use. To support this paradigm across a 256K context, we develop an end-to-end, high-efficiency open training recipe: Architecture & System Co-design: interleaved gated sliding-window and full attention, and a stable FP8 Muon optimizer; Progressive Curriculum & MDP Mid-Training: context scaling across 16K, 64K, and 256K, and the reformulation of interaction traces into Markov Decision Processes. Furthermore, we establish an AI-native R&D workflow where agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. Extensive evaluations show that ZGCM-1-7B is competitive across 7B model family on general benchmarks. On several challenging mathematical reasoning and agentic search suites, it remains competitive with frontier models orders of magnitude larger, such as Qwen3-235B-A22B and GLM-5.1. We also show that our pre-training design offers a ~4.2x efficiency improvement in 16K pre-training time-to-loss. Across the full development lifecycle, we distill eight actionable empirical findings-spanning architectural scaling, SFT quality pruning, long-context generalization, and agentic co-training dynamics. To facilitate community research, we open-source model weights from the pre-training, mid-training, and post-training stages, intermediate checkpoints, training code, per-stage data and data recipes, and W&B logs.

Architectural Context & Practical Takeaway: Autonomous search and deep-research agents frequently suffer from stagnant reasoning loops, uncalibrated stopping decisions, or noisy reward attribution across multi-turn tool calls. ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search (from He et al.) introduces explicit control and training signals for agentic retrieval—coupling fast System-1 evidence judgments, programmatic workspace operations, or representation-grounded reinforcement learning rewards with multi-step query planning. Evaluations on multi-hop QA and web research benchmarks demonstrate higher evidence recall and lower token waste per solved question.

Figure 8: Architecture and technical summary for ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

Related Papers


[9] LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training

  • Newsletter Headline: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training (from Alibaba)
  • Links: arXiv:2609.14978 · Digest Section
  • Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning

Main Content & Technical Findings

Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training (from Alibaba) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.

Why It Matters in Production: Use this architecture (from Alibaba) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.

Figure 9: Architecture and technical summary for LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training

Related Papers


[10] Chroniclerec Pre Training Temporally Anchored Tokens for Lifelong User Modeling

  • Newsletter Headline: Causal Compression of Lifelong User Histories for Recommendation (from Tencent)
  • Links: arXiv Paper Search · Digest Section
  • Research Area: Recommender Systems · Large-Scale Recommendation & Representation Learning

Main Content & Technical Findings

Modern industrial recommenders and embedding backbones must balance model capacity, lifelong user sequence length, item freshness, and strict online serving latency. Chroniclerec Pre Training Temporally Anchored Tokens for Lifelong User Modeling (from Tencent) addresses this trade-off by introducing architectural innovations—such as low-rank KV caching for 100k+ sequences, decoupled temporal encodings, supersession/decay freshness filters, or Mixture-of-Experts (MoE) representation backbones. Offline and production experiments confirm substantial gains in ranking accuracy, long-tail item coverage, and training/serving efficiency.

Why It Matters in Production: Use this architecture (from Tencent) to scale industrial recommendation and multimodal representation learning across long user histories, fresh/tail items, and multi-task serving constraints.

Figure 10: Architecture and technical summary for Chroniclerec Pre Training Temporally Anchored Tokens for Lifelong User Modeling

Related Papers