I ran the same logs through the costUSD that Claude Code itself writes into each result. The ranking flips.

The table

Three follow-up questions after forking a session at N = 5, 15 or 30 turns, Sonnet 5.5, 1-hour cache TTL, three reps each. Cost is what Claude Code billed, per call, from its own modelUsage.

The /compact row includes the compaction call itself, which is where the money goes.

Why fewer tokens cost more

Three prices are in play, and raw token counts treat them as one.

• Carrying context is almost free. A cache read is 0.1× the input price. The 155k context at N = 30 costs about 3 cents per model call to carry.

• The summary is written at the most expensive price. /compact makes the model write a summary at the output price (5×), then writes the new prefix into a 1-hour cache at 2×. At N = 30 that one call is $0.071, more than twice what it saved over the next three questions ($0.027).

• The summary forgets what was open. The follow-ups after /compact re-read files the old context already held, so the per-question cost after compaction ($0.066–0.071) is barely under the uncompacted one ($0.074–0.080). Divide the compaction fee by that small gap and you get the break-even column: seven or eight more questions before /compact has paid for itself.

/clear wins at N ≥ 15 because it pays neither the summary nor the re-reads of a half-remembered context. It just starts over, and a fresh start on this task was cheap.

The subagent result is the same story

A subagent starts with a cold cache and writes its own prefix. That is the $0.052 → $0.093 jump on the main call. In exchange the parent’s context stays lighter, which saves about $0.002 per follow-up question. Twenty follow-ups to pay back one delegation.

Two things the logs show that the article did not look for

I also ran kvleak over the raw logs, which prices every cache miss in a transcript.

An 8 h 36 m pause inside two 30-turn chains. Rate limiting paused the runs between turn 28 (08:07 UTC) and turn 29 (16:43 UTC). Turn 29 rebuilt 142k tokens of context at the write price: $0.629 for one call, against $0.092 for the call before it. The session still reports a 95.9% cache hit rate. On Sonnet 5.5 with a 1-hour cache, one miss costs as much as nineteen hits of the same context, and the hit rate cannot see it.

$14 of the $79 benchmark bill was the cache-write premium on prefixes nothing read back. Every claude -p run writes its prefix into the 1-hour cache, paying 2× instead of the plain 1×, then exits. The 5-minute cache charges 1.25×, so for one-shot harnesses CLAUDE_CODE_PROMPT_CACHE_TTL=5m cuts that premium by three quarters, to about $3.50.

What to do with this

1. Do not compact small contexts. At 51k of context /compact never paid back here; at 107k it took about seven more questions. Claude Code’s auto-compact fires near the window limit, which is usually right; the mistake is manual /compact as hygiene.

2. If you will ask fewer than seven more questions, /clear beats /compact. Or just keep going. Carrying is cheap.

3. Delegate for isolation or parallelism, not for savings. A subagent costs a cold start.

4. Set the 5-minute TTL on batch runs. CLAUDE_CODE_PROMPT_CACHE_TTL=5m for anything that exits right after its answer.

5. Count dollars, not tokens. Three prices, three orders of magnitude between a cache read and a summary. Any tool that reports “tokens saved” is reporting the cheapest line on the bill.

Caveats

Sonnet 5.5 only; three reps per cell, so single-digit percentages are noise; one codebase; the follow-up questions were short and answerable, which favours /clear. A long agentic task that needs the history would move the break-even. The compaction fee and the cold-start jump are not noise, they are the price list.

Reproduce it

Dataset: kenseii1/claude-code-limits-tests (runs/session/*.json, runs/subagent*/*.json). Each log carries a tok block with the per-call delta of costUSD (--resume makes the raw total_cost_usd cumulative; the harness already subtracts the previous call).

<code><code>import json, glob, re, collections, statistics as st
R = "claude-code-limits-tests/runs"
for N in ("N05", "N15", "N30"):
    rows = {}
    for arm in ("continue", "clear", "compact"):
        per_rep = collections.defaultdict(float)
        for f in glob.glob(f"{R}/session/{N}-{arm}-*q*-r*.json"):
            rep = re.search(r"-r(\d+)\.json$", f).group(1)
            per_rep[rep] += json.load(open(f))["tok"]["cost"]
        rows[arm] = st.mean(per_rep.values())
    rows["compact"] += st.mean(json.load(open(f))["tok"]["cost"]
                               for f in glob.glob(f"{R}/session/{N}-compactcmd-*.json"))
    base = rows["continue"]
    print(N, {a: f"{c:.4f} ({(c/base-1)*100:+.0f}%)" for a, c in rows.items()})</code></code>

The miss pricing: uvx kvleak --path claude-code-limits-tests/runs, then uvx kvleak session 600996ac --path … for the 8-hour pause.

An interactive version, with the token-versus-dollar charts, is on the kvleak blog.

Harness² makes that surrounding system editable during task execution. It runs alternative harnesses on the same task, examines what changed, and retains experience that can guide later edits. The executor’s model weights stay fixed. [1]

The difficult part is deciding which edits deserve to survive. A cleaner trajectory can still produce a wrong answer. Harness²’s execution comparisons provide evidence about behavior, but the system does not receive the task’s ground-truth grade while adapting. That distinction determines how I would evaluate it.

The editable object is larger than a prompt

The project exposes eight harness components: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. Together, they shape which context the model receives, how it calls tools, what it delegates, and what happens inside the execution loop. [1]

These components can solve different kinds of problems. A prompt can remind an agent to inspect an artifact. A script can perform a repeatable check. A skill can describe a domain procedure. A subagent configuration can change who performs an independent review.

Consider a spreadsheet task. If an agent returns a workbook without checking its formulas, a harness edit could require reopening the saved file and checking calculated cells. That is a hypothetical example of a useful intervention, not a reproduced Harness² result. The important question is whether executing the edited harness reveals that the check actually happened and affected the deliverable.

The released code separates the main responsibilities. `core/recursion.py` schedules proposal, probe, and composition work. `core/context.py` assembles the evidence available to the editor. `core/library.py` stores reusable harnesses and experience, while `core/selection.py` handles output selection and adoption. The official repository documents these implementation details; the paper supplies the experimental claims. [1]

Compare executions before composing an edit

A basic Harness² step has three parts.

Propose. First run the current harness to obtain a reference trajectory and deliverable. An editor then proposes two candidate harnesses. One addresses a deficiency that might recur across domains, such as planning or delegation. The other addresses the current domain’s procedures and constraints. These are executable edits to harness components. [1]

Probe. Run both candidates on the same task under matched model and environment configurations. Compare their trajectories and deliverables with the reference. Did the proposed check execute? Did it expose a missing input? Did the new instruction cause the agent to spend its budget on an irrelevant detour?

Compose. Read the candidate diffs and execution evidence, then form a successor harness. Composition can merge changes, revise conflicts, select one candidate, or preserve the current harness. Execute that successor as well. The result includes the revised harness, its actual execution, and the observations used to justify the changes. [1]

Figure 3 from Harness². The left side adapts the execution harness for one task through proposals, probes and composition. The right side records supported experience for future tasks; it does not update the executor’s model weights. Source: arXiv:2610.14224, CC BY 4.0.

Read the figure from left to right. The candidate probes are the point where a proposed improvement is exposed to execution. The return path matters too: experience changes how future harness edits are made. It does not mean the next task automatically inherits everything that happened on this one.

This design gives composition more information than a free-form request to “improve the prompt.” It still leaves a causal ambiguity. Agent executions vary. If a candidate performs better once, the harness change may have helped, or the executor may simply have taken a more fortunate path. The paper explicitly identifies stochasticity and shared model biases as limits on these behavioral contrasts. [1]

A library carries lessons across tasks, not a permanently mutated agent

After a task, a separate reflection step inspects the improvement record. The library retains an adopted executable harness together with an `EXPERIENCE.md` entry describing the applicable situation, lesson, and evidence. If the evidence is insufficient, reflection need not add an entry. [1]

The next task starts from the common base execution harness. Earlier components can be reused or adapted during proposal, guided by the library. They are not unconditionally installed as the new default.

That reset is useful. A procedure that helps generate a legal document may be irrelevant to a coding task. Even within a domain, a successful edit may depend on a particular file format or tool. Storing the conditions under which an edit helped gives a future editor something more useful than a generic instruction to “be careful.”

There are therefore two changing objects: the execution harness being tested on the current task, and the improvement library that informs later refinement. The improvement process does not freely rewrite every part of itself. In the paper’s formulation, model weights, stage instructions, and the decision rule stay fixed while the library evolves. [1]

The recursion budget buys width or depth

The budget parameter k controls two different schedules.

Parallel recursion collects k base executions, proposes k invariant and k domain-specific candidates, probes them, and performs k independent compositions using the shared candidate pool. It increases the width of a population update.

Sequential recursion begins with the base evidence and chains k refinement steps. Each step can use the preceding harness and accumulated execution history. It increases the depth of refinement. At k=1, both reduce to the basic contrastive step. [1]

Figure 4 from Harness². Parallel recursion explores candidate edits and compositions across a shared pool. Sequential recursion carries the revised harness and accumulated evidence into the following step. Equal k does not imply equal wall-clock latency. Source: arXiv:2610.14224, CC BY 4.0.

The paper’s implementation schedules 4k executor rollouts and 3k improvement stages, including the base executions. At k=1, that is four executor rollouts plus three improvement stages. At k=5, it is twenty rollouts plus fifteen improvement stages, before accounting for selection and post-task reflection. Individual sessions may contain many model calls.

For k>1, a selector compares anonymized deliverables with the request and source materials. It returns an existing output; it does not synthesize a fresh answer. Adoption here means “chosen by the selector,” not “verified correct.” [1]

Parallel execution may reduce elapsed time when enough workers are available. It does not remove the computation or API charges. A deployment decision needs total tokens, cost, and latency across the whole pipeline.

What the reported results establish

The paper evaluates frozen Gemini 3.5 Flash and GLM-5 executors under OpenCode and Codex, giving four model and initial-harness configurations. Claude Opus 4.8 is the default harness editor. The main results report means and standard deviations over three task streams with shuffled task order. [1]

The test scope is specific:

• WorkBuddy-Bench uses 40 Code, 35 Web, and 25 Office test tasks. Its task scores combine deterministic checks with model-based judging.

• JobBench reports 34 held-out professional tasks, scored against weighted rubrics.

• LAB reports 100 held-out legal tasks. Its Partial metric is a per-criterion pass rate from a model judge, not a binary measure of legally correct deliverables.

For one concrete configuration, Gemini 3.5 Flash with OpenCode on WorkBuddy-Bench, Table 1 reports the following macro-average across the three domains: [1]

Setting · Reported average score

Base harness · 72.4, SD 0.9

Harness², k = 1 · 77.5, SD 2.5

Harness², parallel k = 5 · 80.8, derived from the reported +3.3 change in mean

The single-step gain is 5.1 score points. The larger budget adds 3.3 more. The recursion row reports a change in mean, so I have not attached an invented standard deviation to the derived 80.8.

The largest one-step gains across the reported configurations are 11.5 WorkBuddy-Bench points, 4.5 JobBench points, and 11.8 percentage points on LAB Partial. Those are different metrics and configurations. They should not be combined into a single “percent improvement.” Harness² also does not lead every metric: Meta-Harness is ahead on LAB Partial for Gemini under Codex before the extra recursion budget. [1]

Ablations provide some evidence for the design choices. For Gemini/OpenCode at k=1, removing the library reduces WorkBuddy average from 77.5 to 74.4 and JobBench from 36.0 to 34.2. Removing composition yields 75.9 and 34.1 respectively. Those results support the usefulness of these components in that setup; they do not show that every retained lesson transfers.

How this relates to reflective memory

Reflexion supplies a useful earlier comparison. Its Actor generates actions, an Evaluator supplies feedback, and a Self-Reflection component turns experience into text retained in episodic memory. Later trials receive those reflections without updating the actor’s weights. Its feedback can include tests or environment signals, so it should not be described as universally label-free. [2]

Agentic Context Engineering, or ACE, organizes context adaptation through a Generator, Reflector, and Curator. It emphasizes structured incremental updates to a playbook so useful detail is not lost through repeated rewriting. I see it as a relevant comparison for maintaining experience, not as evidence that Harness²’s full workflow works. [3]

Harness² extends the editable surface to an executable harness and tests proposed edits through new executions. It also retains experience about the refinement process. The practical distinction is what persists: a reflection about a failed attempt, an evolving context playbook, or executable components accompanied by observations about when they helped.

For a related deployment question, my SearchJev analysis considers when an agent should delegate or stop. Harness² adds a prior decision: whether to spend more computation changing the agent’s operating procedure at all.

The limits I would test before deployment

Behavioral improvement is not correctness feedback. The improver sees trajectories and deliverables, while benchmark graders evaluate the results separately. A convincing explanation or a more complete-looking document can still be wrong. The absence of ground-truth feedback during adaptation is a constraint, not a guarantee that evaluation is unnecessary. [1]

Public evaluation material can leak into execution. The release’s experiment guide warns that JobBench rubrics are public and reports a validation rollout downloading its task’s rubric. The release does not restrict network access by default. This is a documented evaluation-hygiene risk; it does not by itself establish that the paper’s reported scores are contaminated.

Executor steps do not measure pipeline cost. A shorter selected trajectory can coexist with a larger total bill because proposals, probes, composition, selection, and reflection all consume resources. I would not infer cost savings from a reduction in reported executor turns.

Editable guardrails need an external boundary. Guardrails are among the harness components the framework can edit. In a production system, permission policy, secrets, network egress, and deployment approval should remain outside the editable experiment. That is my deployment recommendation, not a feature I have verified in the release.

The repository is Apache-2.0, but benchmark materials have separate terms. Its local deterministic demo can illustrate both recursion schedules without model credentials. That demonstration would test orchestration, not reproduce the paper’s model-backed benchmark gains. I have not run either here.

An equal-cost experiment would decide whether I would use it

I would begin with a small, non-production task stream with independently checkable deliverables. Compare a fixed harness, repeated execution with output selection, Harness² without persistent experience, and Harness² with its online library. Match total model spending across arms, including the editor and reflector.

Keep task order, available tools, source documents, and evaluation access controlled. Record library growth, artifact correctness, regressions, total dollars, and wall-clock time. A useful edit should improve the deliverable under an independent check, not merely make the trajectory easier for another model to praise.

Then test transfer on tasks the library has not seen. Include a deliberately mismatched library from another domain to measure whether experience helps because it is relevant or merely because it adds instructions.

The outcome I would care about is a measured advantage over equal-cost retries, with a library that helps on later tasks without accumulating regressions. That would justify using Harness² as a harness-engineering process. Until then, the code and paper offer a concrete way to investigate how much agent reliability can be improved outside the model weights.

References

[1] Harness²: Recursive Agent Harnessing for an Open World

[2] Reflexion: Language Agents with Verbal Reinforcement Learning

[3] Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models

Our earlier post on FLASH argued that you can’t judge a semantic tokenizer on its own; you have to judge it together with the decoder and the search procedure. This post follows from that. The new paper, Beyond Semantic IDs: Finite-Beam Reachability and Continuous-Query Generation in Generative Recommendation [1], takes the search procedure as its subject.

Semantic-ID (SID) recommenders in the line of TIGER [2] turn each item into a short sequence of discrete codes. A sequence model is then trained to emit the next item’s codes, and inference runs beam search over valid code prefixes. A beam is finite. Even when the held-out item has a valid and almost unique code, its path can be pruned before it reaches the returned list. In that case no downstream ranker can recover the item.

The authors of [1] separate four quantities that people often blur together:

• Unique-code assignment: does each item get its own code?

• Target-conditioned beam support: for a given user, does the held-out target appear anywhere in the returned list of size B?

• Catalog-wide scoreability: can the model assign a score to every item at all?

• Union-of-output coverage: across all users, what fraction of the catalog shows up in at least one top-K list?

The paper’s main claim is that these are different things. In its closer T5/RQ-VAE-style reproduction on Amazon Beauty, 99.48% of generated sequences are valid SIDs, yet the union of top-100 outputs covers only 44.0% ± 13.7% of the catalog. Coverage is 14.0% for the rarest quartile and 88.7% for the most popular [1]. Valid codes don’t buy reachability.

My aim here is to keep two questions apart all the way through: can the target be reached under a finite search budget, and how well is it ranked once it can be. The paper’s numbers answer the two differently.

Mechanism

The SID interface being replaced

Let ℐ be a catalog of N items. User u has a chronological history sᵤ = (i₁, …, iₜ), and the task is to retrieve iₜ₊₁. An SID system maps item i to a code sequence zᵢ = (z_i,1, …, z_i,L) and trains an autoregressive model. In the standard left-to-right factorization:

Here L is the code length and z_i,<ℓ is the prefix decoded so far. Beam search keeps only the highest-scoring prefixes at each level, so whether an item gets through depends on its whole path surviving every pruning step.

In TIGER [2], the codes come from an RQ-VAE over content embeddings. A residual r₀ is matched to its nearest codeword e_c₀ in the level-0 codebook. The residual r₁ = r₀ - e_c₀ is then quantized at the next level, and so on:

Here d indexes the level, e^(d)ₖ is the k-th codeword at level d, and c_d is the selected code. The reported setup uses Sentence-T5 embeddings, three levels of 256 codewords each, and a fourth code to break collisions [2].

To make the support quantity concrete, I’ll write the paper’s verbal definition as a formula. This is my notation, not an equation from [1]:

O_B(u) is the deduplicated, history-filtered list of B items returned for user u, and i^*ᵤ is that user’s held-out target. For a fixed checkpoint and budget, this is the highest Recall that any reranker working only on O_B(u) can reach. The paper is explicit that this ceiling depends on the budget, not on the architecture: raising B raises it [1].

CQG-Single: one continuous query

Both proposed methods share a frozen item matrix E ∈ ℝ^N × d. Each row eᵢ is an L2-normalized item vector, and in the main experiments it comes from a two-layer SASRec pretrained with BPR (d = 64). A generator f_θ maps user context cᵤ (history, plus demographics where available) to a single query:

Top-K comes from exact dense scoring or approximate nearest-neighbor (ANN) search. There is no codebook, no code-to-item lookup, and no beam. Every encoded item can be scored by construction. As the authors point out, that doesn’t mean every item will actually show up in a finite top-K [1].

CQG-AR: autoregression without a token vocabulary

CQG-AR keeps step-by-step decoding but emits continuous residuals instead of tokens:

with ê⁽⁰⁾ᵤ = 0. D_φ is a causal Transformer decoder, W_AR projects the hidden state into item space, and only the final query ê^(L)ᵤ (with L = 3, to match three SID levels) is used for retrieval. The two variants form a controlled pair. CQG-Single removes both the vocabulary and serial decoding. CQG-AR removes only the vocabulary.

Training objective

The primary loss is cross-entropy over the full catalog with temperature τ= 0.07:

q_b is the generated query for example b and e_b⁺ is the target item’s embedding. Every item appears in the denominator of every update, which costs O(N) per step [1]. This matters later: the continuous models see the whole catalog during training, while the SID decoder gets token-level supervision.

Scoped results

Setup. The datasets are ML-1M (6,040 users, 3,416 items), Amazon Beauty (22,363 / 12,101), and Amazon Sports (21,519 / 10,693, a processed split that the authors say differs from the usual 5-core split). Evaluation is leave-one-out, with history items excluded from ranking. The comparator labeled “TIGER” is the authors’ own hierarchical-k-means SID model built on the same frozen SASRec embeddings, decoded with valid-prefix constraints. It is not the published TIGER system, which uses an RQ-VAE over Sentence-T5 content embeddings. The authors state that this baseline is weaker than published SID systems [1].

Ranking quality (Table 3 of [1]). Values are the mean over seeds 42–48 on ML-1M and 42–44 on Beauty and Sports.

Method · Trainable params · ML-1M R@10 · ML-1M N@10 · Beauty R@10 · Sports R@10

SASRec (full-rank, 1 checkpoint) · 0.32–0.88M · 0.2797 · 0.1535 · 0.0558 · 0.0390

TIGER-style SID · 5.78M · 0.2996 · 0.1739 · 0.0477 · 0.0236

CQG-Single · 3.2M · 0.3104 · 0.1822 · 0.0605 · 0.0363

CQG-AR (L=3) · 5.62M · 0.2976 · 0.1735 · 0.0571 · 0.0355

CQG-Single beats the TIGER-style model in all seven ML-1M runs. The mean paired gain in R@10 is 0.0107 (about 1.1 percentage points), with a 95% paired t-interval of [0.0021, 0.0193] and an exact sign-flip p = .0156. It uses 55.3% as many trainable parameters. CQG-AR roughly ties on ML-1M and is higher on both Amazon sets. On Beauty, the training-run intervals for the three models overlap. On Sports, both continuous models’ intervals lie above the TIGER-style interval [1]. Full-rank SASRec is still the best of all on Sports. Removing the beam doesn’t automatically produce a better ranker.

Reachability (Table 2 and §6 of [1]). This is the result I think matters most. On ML-1M seed 42, with returned-list sizes matched, TargetInOutput@100 is 66.7% for TIGER-style, 67.3% for CQG-Single and 66.4% for CQG-AR. In other words, the method with no beam reaches only 0.6 percentage points more targets than the beam-search model, and CQG-AR reaches fewer. About a third of held-out targets are missing from every method’s top-100. At this budget the reachability ceiling is a ceiling on retrieval quality in general, not something specific to SIDs. Separately, raising TIGER’s list from 100 to 200 lifts its support from 66.7% to 76.3% (seven-seed means 0.6667 and 0.7593).

Coverage and popularity. Union-of-output coverage (Table 4 of [1]) depends on the dataset:

• Beauty: CQG covers much more of the catalog (89.5% vs 74.4% at top-10).

• ML-1M: the methods are nearly tied (97.6% vs 97.0% at top-100).

• Sports: it reverses. TIGER-style covers 94.0% at top-10 against CQG-Single’s 63.1%.

Across popularity quartile × depth cells on ML-1M, CQG-Single leads in 38 of 48. It wins every Q2–Q3 cell, while TIGER-style keeps some rare-item (Q1) advantages at shallow depths. That diagnostic uses a separately selected checkpoint, and Beauty doesn’t reproduce the rare-item pattern [1].

Ablations. On ML-1M, the CQG-AR depth sweep from L = 1 to 4 moves R@10 only between 0.2909 and 0.2944, and the trend isn’t monotonic. Training CQG-AR with 256 sampled negatives instead of the full softmax drops R@10 from 0.2982 to 0.2895. So seeing the whole catalog during training explains part of the gap, by the authors’ own account [1].

Cost scope. The reported cost evidence is the parameter counts and the O(N) per-step training cost. Latency, memory, ANN behavior and index-update cost are not measured [1].

How it relates to the other two papers

TIGER [2]: the foundation, and not the baseline

TIGER introduced the interface that [1] analyzes: RQ-VAE semantic IDs, a Transformer encoder–decoder, and beam search over IDs. It already flagged the right concerns. Invalid IDs made up roughly 0.1–1.6% of top-10 predictions, and the suggested fixes were a wider beam plus filtering. The authors also noted that autoregressive beam decoding can cost more than ANN retrieval [2]. What it didn’t measure was whether valid targets survive the beam, and [1] fills that gap.

TIGER is not the comparator in [1]. For context, published TIGER reports Beauty Recall@10 of 0.0648 [2]. That is numerically above both the TIGER-style 0.0477 and CQG-Single’s 0.0605 in [1]. The Beauty user and item counts match, but the item representation (content vs collaborative embeddings), quantizer, filtering and evaluation pipeline all differ, so this isn’t a matched comparison in either direction. The only safe reading is the one the authors give: within one shared frozen item space, continuous queries beat one hierarchical-k-means SID implementation.

FLASH [3]: discrete IDs without a beam

FLASH [3] is the useful contrast, because it removes beam search while keeping discrete IDs. Each item’s frozen LLM text embedding xᵢ is hashed at code position j using B = ⌈log₂ M ⌉ Gaussian hyperplanes r_j,b:

M is the codebook size (a power of two) and m is the number of positions. Separate heads predict all positions in parallel, and inference scores every item from cached per-position distributions:

The total cost is O(mMd + Nm) [3]. Here hₜ₋₁ is the final history state, E^(j) is the position-j code-embedding table, and fⱼ is head j. The product over positions is a modeling factorization, not a claim that the codes are statistically independent. FLASH also adds an alignment loss -(1)/(|S|)Σ_i ∈ Scos(zᵢ, φ(xᵢ)) that ties learned item representations back to the projected LLM embedding.

For reachability, this means the finite-beam ceiling belongs to beam decoding, not to discrete IDs as such. A SID model scored across the whole catalog can score every item just as CQG can. The paper’s title, “Beyond Semantic IDs,” therefore claims more than its experiments isolate. The paired CQG design separates autoregression from a discrete vocabulary. It does not separate a discrete vocabulary from beam search. FLASH’s own numbers (for example, Beauty R@10 of 0.0832 under protocols adapted from TIGER and RPG [3]) come from a different setup and can’t be ranked against [1].

What the evaluation does not show

• Superiority over strong SID systems. There is one TIGER-style comparator, built on k-means over SASRec embeddings. Published TIGER, LETTER and EAGER appear only as context. The closer RQ-VAE reproduction is a coverage diagnostic, and the authors note that its ranking accuracy is below the reference implementation [1].

• That the ranking gain comes from removing the beam. At a matched list of 100, support differs by 0.6 points, while R@10 differs by about 1.1 points. The sampled-negative control shows that full-catalog training accounts for part of the gap. The gain is real for this setup. The mechanism isn’t settled.

• A compute-matched search comparison. Beam width, returned-list size and FLOPs are not traded off against each other on a single curve. List size alone moves TIGER-style support by about 9.6 points.

• Scale or serving cost. ML-1M has 3,416 items, so exact scoring is trivial there. There are no latency, memory or ANN-recall-loss numbers, and no billion-item training. The O(N) softmax is evaluated only at these sizes.

• Updatability and cold start. The item space is a frozen SASRec table. A brand-new item has no collaborative embedding until something produces one, so the claim that the output dimensionality doesn’t change as items are added doesn’t answer how new items are represented. Cold-item accuracy is listed as unmeasured [1].

• Causal popularity effects. The quartile analysis is observational, depends on the checkpoint, and doesn’t fully replicate across datasets.

One concrete experiment

Run a 2 × 2 factorial on a shared item space, with output type crossed with search procedure:

1. Arms. (a) RQ-VAE SID with beam search; (b) the same SID checkpoint scored across the whole catalog by teacher-forcing every valid code path, so each item gets an exact sequence log-probability. At these catalog sizes, a trie-batched pass is feasible; a FLASH-style parallel SID is the cheaper variant; (c) CQG-Single with exact top-K; (d) CQG-Single with production ANN at a fixed recall target.

2. Controls. Use the same backbone width and the same item embedding source (run once with content embeddings and once with collaborative ones). Use the same training-time catalog visibility: either full softmax for every arm, or 256 sampled negatives for every arm.

3. Budget axis. Sweep returned-list size B ∈ {50, 100, 200, 500} and record p50/p99 latency and accelerator-seconds per 1,000 users at each point. Compare arms at matched latency, not only at matched B.

4. Report three numbers separately. Report TargetSupport@B (reachability), Recall/NDCG@10 conditional on the target being reachable (ranking), and catalog coverage by popularity quartile. Include paired intervals over at least five seeds.

5. Decision rule for deployment. In a shadow run, feed each arm’s top-B into the existing production ranker. Ship the continuous-query retriever only if it raises the ranker’s achievable recall at equal p99 latency. If arm (b) closes most of the gap with (c), the fix is to change the SID search procedure, not to remove the IDs.

If (a) → (b) captures most of the support gain while (b) → (c) captures most of the conditional-ranking gain, then reachability and ranking quality have been separated, which neither [1] nor [3] does on its own.

References

[1] Beyond Semantic IDs: Finite-Beam Reachability and Continuous-Query Generation in Generative Recommendation

[2] Recommender Systems with Generative Retrieval

[3] Rethinking Semantic ID Construction for Generative Recommendation: SimHash with Parallel Decoding and Semantic Alignment

FLASH revisits SimHash through that lens. The paper combines training-free hashing, parallel code prediction, and semantic alignment. The useful contribution is a co-design argument: item representation, decoder, and objective should be evaluated together.

1. What “training-free” actually covers

FLASH first computes fixed text embeddings from item metadata. At each semantic-ID position, random hyperplanes turn an embedding’s projection signs into a discrete code. Those hyperplanes are shared across items and remain fixed, so the tokenizer does not require fitting a quantization model. [7]

A compact way to express the hashing step is:

<code><code>b_{j,k}(i)=\mathbf{1}[a_{j,k}^{\mathsf T}x_i\geq0],
\qquad
c_j(i)=1+\sum_{k=0}^{K-1}2^k b_{j,k}(i).</code></code>

Here x_i is the text embedding, j identifies a code position, and its K sign bits index one of M=2^K entries. This is a restatement of the paper’s encoding, not a new algorithm.

The recommendation model still trains. Its code embeddings, history model, prediction heads, and alignment projection are learned. Computing the original item embeddings also costs something. “Training-free tokenizer” therefore should not become “training-free recommender” or “free item ingestion.”

There is an important statistical nuance. Independently sampled projections do not prove that code values are statistically independent across real items. Codes all depend on the same item vector. The relevant architectural observation is that their positions have no intrinsic residual or left-to-right ordering.

2. Why parallel prediction is a plausible match

TIGER assigns semantic codeword tuples to items and autoregressively predicts the next item’s identifier from user history. [8] FLASH asks whether imposing that same sequential prediction structure on unordered hash codes is helpful.

Its predictor factorizes the target-ID probability across code positions and uses a dedicated prediction head for each position. All heads share the history representation and predict simultaneously. [7]

<code><code>p(c_1,\ldots,c_m\mid h)
\approx\prod_{j=1}^{m}p_j(c_j\mid h).</code></code>

This is a modeling assumption. It removes sequential dependencies in the output heads; it does not imply that the underlying joint distribution truly factorizes. The shared history state can influence every head, but the predictor does not condition later code predictions on earlier generated code values.

The paper’s preliminary experiment is useful because it changes the predictor. Replacing autoregressive decoding with parallel decoding improves SimHash-based recommendation, while leaving a residual gap that motivates semantic grounding. [7]

That supports compatibility as one explanation for the earlier gap. It does not prove that autoregression is always inappropriate for every learned or hashed identifier.

3. Recovering a semantic training signal

A hard code preserves only part of the original embedding information. FLASH adds a separate constraint on the learned item representation.

A trainable linear projection maps the frozen text embedding into the recommendation representation space. The alignment objective minimizes negative cosine similarity between this projected embedding and the learned representation of each input item: [7]

<code><code>L_{\mathrm{align}}=-\frac{1}{|S|}\sum_{i\in S}
\cos\big(z_i,\phi(x_i)\big),
\qquad
L=L_{\mathrm{rec}}+\lambda_{\mathrm{align}}L_{\mathrm{align}}.</code></code>

The two objectives supervise different properties. Recommendation loss trains next-item code prediction; alignment encourages the representation to retain information from the text embedding. Better content geometry alone does not establish better modeling of user preferences.

That is where LETTER makes a useful comparison. Its learnable item tokenization explicitly incorporates hierarchical semantics, collaborative information, and code-assignment diversity. [9] FLASH and LETTER differ in more than whether a quantizer is trained. A fair comparison should expose which gains come from collaborative signals, decoding, alignment, or the item encoder.

4. Parallel prediction still has a catalog term

At inference, FLASH computes a probability vector q^{(j)} for each code position once. A valid catalog item’s score is the sum of its code-position log probabilities: [7]

<code><code>\mathrm{score}(i)=\sum_{j=1}^{m}\log q^{(j)}_{c_j(i)}.</code></code>

The paper gives the resulting cost as:

<code><code>O(mMd+Nm),</code></code>

where m is the number of code positions, M their vocabulary size, d the representation dimension, and N the number of candidate items.

The Nm gather-and-sum term remains. The predictor avoids evaluating all possible code combinations, but scoring every catalog item is not independent of catalog size. Removing repeated dense item interactions and removing the catalog scan are different optimizations.

This also clarifies candidate validity: evaluating IDs already attached to real catalog items avoids interpreting arbitrary combinations as products. It does not solve stock availability, policy eligibility, or index freshness. Those constraints still belong in the serving system.

5. The controlled tokenizer comparison is more informative than the headline

On four Amazon review categories, FLASH reports the best results across the displayed Recall and NDCG metrics, with relative NDCG@5 improvements from 5.51% to 14.46% over the best baseline. [7]

Table 2 asks a narrower question: what happens when tokenizers use a shared parallel-prediction and alignment pipeline? SimHash becomes competitive with learned quantizers. On Beauty, its NDCG@10 is 0.0489 versus 0.0498 for OPQ; on CDs, 0.0446 versus 0.0455. It leads those compared tokenizers on Toys and Sports in that table. [7]

Those mixed results are useful. They suggest the full FLASH pipeline’s gains should not be attributed to hashing universally dominating learned quantization.

The measured SimHash tokenization times range from about 0.056 to 0.461 seconds on CPU. The table measures RQ-VAE and RQ-KMeans on GPU, while other methods use CPU. [7] These figures concern code construction from precomputed embeddings in the evaluated datasets, not total ingestion, model training, or online service latency. Hardware and scope belong next to the speed claim.

The paper also reports strong cold-start results when target items are withheld from training, and identifies text-embedding quality, hash collisions, public-benchmark scope, and missing non-text modalities as limitations. [7] That setting is valuable but does not cover every meaning of cold start, such as new users, distribution shifts, or unseen product domains.

6. The reproducibility counterweight

What Makes a Good Semantic ID for Generative Recommendation? reports that no SID design wins universally under its unified evaluation, balanced codebook utilization is insufficient to select the best recommender, and increasing code length or model size does not reliably improve quality. [10]

This is not a direct replication of FLASH. It does challenge the habit of selecting a tokenizer from reconstruction, utilization, or a single benchmark alone.

A useful evaluation would cross tokenizer choice with autoregressive versus parallel prediction and alignment on versus off. Keep the item embeddings, history backbone, time split, tuning budget, and target-item protocol fixed. Report both matched-code configurations and matched-cost configurations, since longer codes can increase representation capacity while changing serving cost.

For a growing catalog, I would additionally record collisions, newly ingested-item ranking, cache refresh time, and agreement between the item eligibility snapshot and scored results. These are proposed measurements, not results already established by FLASH.

What to take into a production experiment

Start with the simplest representation–predictor combination that makes a meaningful comparison. A fast hash baseline under an incompatible decoder is an unfair test of hashing. A stronger predictor with extra semantic supervision is also not evidence that the hash alone caused the improvement.

FLASH supplies a concrete way to separate those effects. The deployment question is whether that separation produces a better quality–cost trade-off on the catalog and behavior distribution you actually serve.

References

[7] Rethinking Semantic ID Construction for Generative Recommendation: SimHash with Parallel Decoding and Semantic Alignment

[8] Recommender Systems with Generative Retrieval

[9] Learnable Item Tokenization for Generative Recommendation

[10] What Makes a Good Semantic ID for Generative Recommendation? A Reproducibility Study

This column covers six recent papers on reinforcement learning, search agents, and RAG applications. The companion column on LLMs and recommender systems is What a Recommender Can Reach, Remember, and Afford. The numbers below are the authors’ reported results on their own benchmarks.

In this column

RL methods

1. Explanatory World Models: credit assignment from explanations

2. RELER: reinforcement learning in embedding space

Search agents

3. T-Search: an open agentic retriever

4. Programmatic Search Agents: control over evidence, not just queries

RAG in practice

5. Relevance versus diversity: diversify only when the pool is redundant

6. Agentic AutoRAG: diagnosis-driven pipeline tuning

RL methods

1. Explanatory World Models: credit assignment from explanations

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Explanatory World Models use a frontier language model to explain which earlier events plausibly caused a delayed outcome, then apply a conservative hierarchical operator that redistributes reward while preserving the undiscounted episode return. The authors test explanation types inspired by actual causation, explanatory simplicity, and hard-to-vary explanations. [1]

Evaluation spans Crafter and WordCraft, the KuaiRand sequential-recommendation benchmark, and offline evaluation of an industry-scale generative recommender, where it improves Recall and behavioral proxies. [1]

Return conservation guarantees the redistribution does not change what the episode was worth. It does not guarantee the explanation assigned credit to the right step. For recommendation, where exposure bias shapes which items could have caused an outcome, that gap matters. The industry evidence is offline.

2. RELER: reinforcement learning in embedding space

Learning to Retrieve via Reinforcement Learning in Embedding Space

RELER treats retrieval as a policy over embeddings. It samples unit-length query and document vectors from von Mises–Fisher distributions centered on encoder outputs, scores the outcomes, and updates the encoder with REINFORCE and a leave-one-out baseline. Conditional-mean projection reduces sampling noise while preserving the policy gradient’s expectation. [2]

Post-training BGE-M3 and Qwen3-Embedding with RELER gives higher average nDCG@10 on BRIGHT than InfoNCE and LambdaLoss. In RAG experiments across seven QA datasets, only the query encoder is adapted while the index and generator stay fixed. [2]

The fixed-index design is clean for attribution and convenient for deployment, because no re-embedding of the corpus is needed. It also means the reported gains do not tell us what happens when document embeddings are trained too.

Search agents

3. T-Search: an open agentic retriever

T-Search: An Open Agentic Retriever and Playground for Hard Multi-Step Search

T-Search runs bounded multi-round search over a fixed corpus and returns ranked evidence chunks with short justifications, leaving answer generation to a separate model. Built on Qwen3.6-35B-A3B, it is trained with adversarially filtered synthetic tasks, round-sliced SFT, and GSPO with a recall reward. [3]

Across seven English and Russian benchmarks with gold evidence, it reports 56.0 Recall@10 with one rollout and 61.3 with three fused rollouts, 14.4 points above its base model for the single rollout. The model, harness, demo, and three benchmarks, including the native-Russian TRuST, are released. [3]

Separating retrieval from answering makes the reward checkable: recall against gold evidence. The trade-off is that recall rewards finding evidence, not deciding when enough has been found — the sufficiency question SearchJev targets.

4. Programmatic Search Agents: control over evidence, not just queries

Programmatic Search Agents: Extending Agentic Search Beyond Query Reformulation

The authors argue that rewriting queries is not enough control: a fixed search interface can retrieve a supporting passage without ever showing it to the agent. Their Programmatic Search Agent runs executable operations over retrieved candidates in a persistent workspace and chooses which evidence to present. [4]

Across five backbones without task-specific training, it raises macro-averaged task success over a query-based agent by 4.00 and 7.56 points on InfoSeek-Eval and BrowseComp-Plus, while within-backbone final-step token use falls by 28.3% and 33.9%. [4]

The token saving is measured at the final step. Programs that filter and deduplicate candidates also cost tokens to write and run, so total cost per question is the comparison to ask for.

RAG in practice

5. Relevance versus diversity: diversify only when the pool is redundant

Finding the Right Balance: Relevance and Diversity in LLM Retrieval

With injected near-duplicates and overlapping chunks, diversification hurts relevance, evidence coverage, and answer quality on clean pools, but can help multi-evidence questions when redundancy makes nearest-neighbor retrieval return repeats. The proposed rule diversifies only when the effective number of distinct documents in the top-k falls below the query’s evidence requirement, and defaults to plain nearest-neighbor retrieval otherwise. [5]

This is the cleanest statement of the column’s theme. Diversity is not a universal objective; it is an intervention with a trigger condition. The rule depends on knowing how many pieces of evidence a query needs, which is itself an estimate in production.

6. Agentic AutoRAG: diagnosis-driven pipeline tuning

Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

After each trial, a Diagnoser labels failed questions as retrieval or generation failures, and a Proposer uses that diagnosis plus model rankings and pricing to pick the next RAG configuration, tracing an accuracy–cost Pareto frontier on a frozen exam. [6]

On a real healthcare corpus it reports 77% median exam accuracy versus 71.5% for the strongest baseline at about 58% of that baseline’s cost per query. [6]

Accuracy here is judged by an LLM on an exam drawn from the corpus. Diagnosis-guided search is a sensible idea; how well the exam predicts production questions is the open variable.

One experiment to try

Does a learned “when to intervene” signal transfer across tasks?

The diversity rule decides when to diversify, SearchJev decides when to escalate, and the personal-agent mediation paper decides when to override a ranking. Each reports gains against a strong default on its own benchmark. A shared evaluation would hold the default fixed and plot the same curve for every method: fraction of cases changed against net benefit, with harms counted separately from gains.

If the curves look alike, “knowing when to act” may be a reusable component. If they differ sharply, each trigger is task-specific and has to be calibrated where it is deployed.

References

[1] Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

[2] Learning to Retrieve via Reinforcement Learning in Embedding Space

[3] T-Search: An Open Agentic Retriever and Playground for Hard Multi-Step Search

[4] Programmatic Search Agents: Extending Agentic Search Beyond Query Reformulation

[5] Finding the Right Balance: Relevance and Diversity in LLM Retrieval

[6] Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

SearchJev asks whether these recurring decisions need the same generative model used for planning and writing. Its answer is a specialized, schema-conditioned decision model, with uncertain cases delegated to a larger model. The interesting question is how that delegation behaves when confidence stops being reliable.

1. Score the decision instead of generating its serialization

SearchJev receives a search state, a decision question, and a finite set of legal options. Each option is associated with a distinct single-token label. It reads the causal LM’s vocabulary logits at the first output position and normalizes only the logits of those labels. [1]

For legal options V_f, the readout is:

<code><code>p_f(v)=\frac{\exp(\ell_f(v))}{\sum_{u\in V_f}\exp(\ell_f(u))}.</code></code>

An option description can be long; the label assigned to it must be one token. The controller maps the selected label back to a structured decision. This avoids generating a JSON object token by token and then parsing it.

That does not make the input free to process. The backbone still processes the supplied search state. The saving targets sequential output generation and decision serialization, not all the computation in a search pipeline. Open-ended query writing remains a different operation.

In the dual-system agent, System 2 retains planning, query generation, and final-answer composition. SearchJev handles bounded judgments, and a search controller converts resolved decisions into actions. [1]

This is a useful design boundary: use a constrained predictor when the output space is enumerable, and keep generation where the task genuinely requires it. If the legal schema excludes the action the agent needs, confident selection among the remaining options will not repair the schema.

2. Why an option probability is not automatically confidence

Soft-Label Learning for Calibrated Decisions, or SLCD, learns from target distributions rather than forcing every example into a certain one-hot label. Its weighted objective combines cross-entropy with a Brier term, followed by held-out temperature fitting. [1]

Writing the target distribution as t_f, the objective is:

<code><code>L_f=w_f\left[-\sum_v t_f(v)\log p_f(v)
+\lambda_B\sum_v\big(p_f(v)-t_f(v)\big)^2\right].</code></code>

The implementation permutes unordered options to reduce position bias, downweights selected weaker supervision sources, and fits temperatures by output type. Positive temperature scaling preserves option order while changing probability sharpness. [1]

The connection to Guo et al.’s calibration study is direct: accurate classifiers can be poorly calibrated, and temperature scaling can improve probability estimates without changing their predicted class. [2] SearchJev adds decision-specific training and typed calibration; its contribution should not be reduced to applying one temperature to an ordinary classifier.

There is a further limitation in the loss itself. A proper scoring rule encourages agreement with the target distribution. If those targets encode an imperfect teacher’s judgment, learning them faithfully does not establish that the resulting probabilities describe real-world correctness. Calibration against labels and calibration against the user’s eventual task outcome are separate questions.

3. Delegation is a risk–coverage problem

SearchJev accepts a local decision when its maximum calibrated option probability exceeds a threshold \delta; otherwise, System 2 resolves the same bounded question. [1]

This connects to selective classification: a predictor can trade coverage for lower risk by rejecting cases it should not handle. [3] In an agent, rejection becomes escalation rather than simply withholding a classification.

A practical analysis would plot:

• Coverage: the fraction of decisions accepted from the small model.

• Selective risk: error among those accepted decisions.

• Escalation cost: extra calls, latency, and tokens paid for rejected decisions.

• Task quality: whether the final answer is correct and supported.

These are suggested deployment measurements, not guarantees supplied by SearchJev. Selective-classification results do not transfer automatically to a changing agent environment.

The threshold also need not be the same for every decision. A false relevance judgment might add a distracting passage; a false sufficiency judgment might end the search before essential evidence is found. Equal confidence can have unequal downstream costs. Comparing per-type thresholds with a single global threshold would test whether that asymmetry matters.

4. What the experiments establish

The paper evaluates 0.8B and 4B models against same-size Qwen3.5 autoregressive JSON baselines. It reports roughly 5.2–5.3 times faster local decisions and mean in-distribution ECE reductions of 74.4% and 40.7%, respectively. [1]

For the agent experiment, the authors sample 100 BrowseComp-Plus questions, run each agent once per question, use Qwen3.5-27B as System 2, and retrieve with BM25. Table 4 reports: [1]

Agent · Answer accuracy · Median active time, seconds · Output tokens, thousands

System 2 only · 45% · 4,080 · 73.7

SearchJev-0.8B + System 2 · 46% · 868 · 15.5

SearchJev-4B + System 2 · 54% · 1,102 · 22.1

The 4B configuration offers a different quality–cost point from the 0.8B configuration. It is more accurate in this sample but slower and more token-intensive. These are median active-search measurements under the paper’s setup, not a universal latency reduction or a p99 serving guarantee. One run on 100 questions also does not establish stability across repeated trajectories.

The harder result is the transfer test. On held-out datasets, decision performance transfers more consistently than calibration. Verification accuracy declines for the 4B model relative to its same-size autoregressive baseline, and calibration gains vary by decision family. [1]

A system can retain useful rankings of options while becoming unreliable about how certain it is. That distinction matters precisely because the controller uses confidence to decide when to escalate.

5. How this differs from training a search policy

ReAct interleaves reasoning with actions that obtain information or interact with an environment. [4] SearchJev changes the division of labor inside such a loop rather than eliminating the loop.

Search-R1 uses reinforcement learning to optimize search-using reasoning trajectories, including query generation and the use of retrieved information. [5] SearchJev’s local probability-learning objective is not that trajectory-level RL objective. A trained search policy and a calibrated decision module could be complementary, but their integration would require its own evaluation.

The cost motivation is real, but comparisons must stay scoped. A separate agentic retrieval study reports 107.4 seconds per query versus 0.67 seconds for standard retrieval, with substantial token consumption. [6] That is a different pipeline and benchmark. It motivates studying repeated decision overhead; it does not establish that SearchJev would remove those costs in that system.

6. A deployment test that would change my confidence

I would freeze the retriever, corpus snapshot, System-2 model, query prompts, and evidence budget, then compare a full-generative controller with SearchJev controllers across several thresholds. The experiment should charge input processing, escalation, retrieval, retries, and final synthesis.

The diagnostic interventions would include incomplete evidence, contradictory passages, option-order permutations, and changes in language or source domain. These are proposed tests. Their purpose is to distinguish useful uncertainty from confidence that merely tracks familiar formatting.

The key outcome would be a quality–cost frontier with uncertainty intervals, broken down by decision type. If the local model remains fast but accepts premature-stop decisions with high confidence, its calibration average can hide the most expensive failure. If it instead escalates those cases while handling routine judgments cheaply, the architectural separation has earned its place.

References

[1] SearchJev: A Fast and Calibrated System-1 Model for Search Agents

[2] On Calibration of Modern Neural Networks

[3] Selective Classification for Deep Neural Networks

[4] ReAct: Synergizing Reasoning and Acting in Language Models

[5] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

[6] Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks

The companion column on RL, agents, and applications is Knowing When to Intervene: Six Papers on RL, Search Agents, and RAG. The numbers below are the authors’ reported results on their own benchmarks.

In this column

Reach

1. Beyond Semantic IDs: can the decoder reach the item?

2. ICE-Fuse: context read from interaction images

Remember and revise

3. MATE: separate memories for recurring and recent taste

4. INTEGER: generative recommenders that hold a conversation

5. Personal-agent mediation: rescue versus harm

Data and cost

6. Evidence Before Sampling: which unobserved items are real negatives?

7. CutBCE: BCE over an 876k-item catalog without dense logits

Reach

1. Beyond Semantic IDs: can the decoder reach the item?

Beyond Semantic IDs: Finite-Beam Reachability and Continuous-Query Generation in Generative Recommendation

The paper separates two properties that are easy to conflate in semantic-ID recommenders: whether an item is scoreable at all, and whether finite-width beam search actually puts a held-out target into the returned list. It proposes CQG-Single, which generates one continuous retrieval query, and CQG-AR, which generates continuous residual updates autoregressively; both score every encoded item through dense or approximate-nearest-neighbor retrieval. [1]

Under a unified protocol with valid-prefix constrained decoding, CQG-Single beats a TIGER-style baseline in all seven runs with 55.3% as many trainable parameters, and extra continuous refinement depth does not monotonically help. [1]

This is a natural sequel to our FLASH deep dive. FLASH asked whether the decoder matches the code structure; this paper asks whether the decoder’s search budget can reach the right codes. The comparison is against one TIGER-style baseline, so it does not establish that continuous queries beat every semantic-ID design.

2. ICE-Fuse: context read from interaction images

Seeing the Context: Enhancing Recommender Systems with Image-Derived Contextual Signals

Most recommenders use images to enrich item or user representations. This paper instead uses a vision-language model to extract physical, social, and modal context from interaction images, and ICE-Fuse integrates those categories into a context-aware recommender. [2]

On TripAdvisor with Review-aware Graph Contrastive Learning, image-derived context does not beat established signals on its own but improves results when combined with them; semantic analysis suggests image and review context capture different aspects of an interaction. [2]

The negative result on its own is useful to report. Evidence comes from one dataset and one recommender, so complementarity is a hypothesis to test elsewhere.

Remember and revise

3. MATE: separate memories for recurring and recent taste

MATE: Adaptive Long- and Short-Term User Memory for LLM-Based Recommendation

MATE checks each new interaction for support from historical behavior and consistency with recent interactions, then uses that temporal evidence to update separate long-term and short-term user memories. A recent-context representation controls how much each memory contributes to the representation used for recommendation. [3]

During online adaptation the shared model is frozen and only the two user memories are updated. On MovieLens-10M, Amazon Luxury Beauty, and KuaiRec it reports a 7.0–13.2% improvement in mean NDCG@10 over the strongest external baseline. [3]

Freezing the model and updating only memory is an attractive serving pattern: per-user state changes, shared weights do not. That also bounds what the experiments show. The online results test memory adaptation, not continual learning of the model itself.

4. INTEGER: generative recommenders that hold a conversation

Adapting Generative Recommenders for Multi-Turn Interaction

INTEGER extends a generative recommender to multi-turn dialogue with a learned routing token that decides when to recommend, history re-anchoring that conditions on both past behavior and the conversation, and behavioral replay with instruction rehearsal to limit forgetting. [4]

On Amazon Beauty and Toys it matches or exceeds the strongest baselines in accuracy with competitive conversation quality, reporting a 13.3% Hit@10 gain on Beauty. The authors note that rejected items are replaced through an intent-agnostic mechanism, leaving attribute-aware feedback to future work. [4]

The routing token is a learned decision about when to act. Two product categories are a narrow base, and an intent-agnostic replacement means “not this one” and “something cheaper” are handled the same way.

5. Personal-agent mediation: rescue versus harm

Personal-Agent Mediated Recommendation with Cross-Platform User History

A platform ranks candidates with local data; a user-authorized personal agent revises the slate using cross-platform history. The authors decompose the change in hit rate into rescues minus harmful overrides, and propose PAMO, which reallocates RL advantage toward rationales that depend on cross-platform history under a value floor. [5]

We covered it in detail in When Should a Personal Agent Override Your Recommender?. The short version: PAMO beats matched GRPO on HR@10 and NDCG@10 across all five test sets, mostly by making fewer harmful overrides rather than more rescues. [5]

MATE, INTEGER, and PAMO all keep a strong model fixed and change something around it: user memory, the decision to recommend, or the final slate. That separation makes each easier to deploy and harder to evaluate in isolation.

Data and cost

6. Evidence Before Sampling: which unobserved items are real negatives?

Evidence Before Sampling: Interpretable Implicit Negative Candidate Discovery for Recommendation

An unobserved user–item pair does not by itself show disinterest. This paper mines observed behavior into symbolic rules, scores them for support, informativeness, and product relevance, ranks the retained rules by evidence, and uses an LLM to interpret them against business objectives and domain knowledge. [6]

Candidate-quality evaluations on an industrial B2B task and three public datasets report higher precision than the evaluated baselines. In the industrial task, symbolic selection improves downstream test PR-AUC by 12.5% over random selection at four negatives per positive. [6]

The downstream comparison is against random negatives, not hard-negative or popularity-based samplers used in production. The more durable idea is the evaluation split: judging whether a negative candidate is valid separately from whether it improves the downstream model. [6]

7. CutBCE: BCE over an 876k-item catalog without dense logits

Cut Binary Cross Entropy: Efficient Large-Vocabulary Loss and Gradient Kernels for Sequential Recommendation

Multi-label BCE over a large catalog normally materializes a dense batch × sequence × vocabulary logits tensor in high-bandwidth memory. CutBCE computes the exact loss and gradient by fusing dense background-loss evaluation with sparse target corrections, using a custom vector-Jacobian product and a Pallas TPU backward kernel that computes logit tiles on-chip. [7]

Training multi-label SASRec on Yambda-50M with 876k items across 8 TPU chips, it reports 65.7% lower peak HBM and 225.9% faster training with comparable accuracy. [7]

This is a systems paper, and its gains are on specific TPU hardware. It matters for modeling because full-catalog losses are often avoided for memory reasons and replaced with sampled losses. Cheaper exact losses widen what can be tested.

One experiment to try

Separate reachability from ranking.

For a held-out target, record two things: whether it enters the candidate set the model can return at all, and where it ranks if it does. Run that for a semantic-ID decoder at several beam widths, a continuous-query generator at several ANN budgets, and a dense retriever, all at matched serving cost.

If most misses are unreachable targets, the fix is search budget or output representation. If most are reachable but ranked low, the fix is the scoring model. A single Recall@K number cannot tell the two apart.

References

[1] Beyond Semantic IDs: Finite-Beam Reachability and Continuous-Query Generation in Generative Recommendation

[2] Seeing the Context: Enhancing Recommender Systems with Image-Derived Contextual Signals

[3] MATE: Adaptive Long- and Short-Term User Memory for LLM-Based Recommendation

[4] Adapting Generative Recommenders for Multi-Turn Interaction

[5] Personal-Agent Mediated Recommendation with Cross-Platform User History

[6] Evidence Before Sampling: Interpretable Implicit Negative Candidate Discovery for Recommendation

[7] Cut Binary Cross Entropy: Efficient Large-Vocabulary Loss and Gradient Kernels for Sequential Recommendation

Personal-Agent Mediated Recommendation with Cross-Platform User History turns that question into a learning problem. A platform produces a ranking; a user-authorized agent sees that ranking plus cross-platform history and outputs the final slate. The useful contribution is not that an LLM can rerank. It is the accounting: every change the agent makes is either a rescue or a harm relative to a strong default.

1. The task: mediation, not reranking from scratch

The platform returns a candidate set, its full ranking over that set, and item metadata. The agent receives that proposal together with within-platform and cross-platform history, writes a visible rationale, then outputs an ordered top-K slate. It does not see platform scores, population interactions, or model state, and the platform never sees the cross-platform history. [1]

That two-sided asymmetry is the whole point. Because returning the platform’s own slate is always valid, the authors frame the problem as selective mediation rather than ranking from scratch. [1]

For a single held-out target Y, the paper measures the agent’s effect against the platform slate P:

<code><code>\Delta(S;P,Y)=U(S,Y)-U(P,Y).</code></code>

With top-K inclusion as the utility, every episode falls into one of four cells: preserved hit, rescue, harmful override, or mutual miss. The platform-relative hit-rate change decomposes exactly as the rescue probability minus the harmful-override probability. [1]

<code><code>\Delta \mathrm{HR}@K=\Pr(\text{rescue})-\Pr(\text{harmful override}).</code></code>

This identity is simple, and it is the part I would carry into any product that lets an assistant edit a ranked list. An average HR@10 hides whether the agent won by fixing misses or lost by breaking hits. Reporting both terms makes that visible.

2. Why access to more history does not solve it

The base Qwen3-4B-Instruct agent falls below the platform on all five test sets, with harmful overrides exceeding rescues throughout. Among proprietary reference models, Haiku roughly matches the platform, while Sonnet and Opus improve HR@10 on all five datasets but still make non-negligible harmful overrides. [1]

The platform here is deliberately not a strawman. It is a target-platform SASRec ranking over 50 candidates, then reranked by Claude Sonnet 4.6 using only within-platform history, and frozen for every agent condition. [1] SASRec is a self-attention sequential recommender that focuses predictions on relevant items in a user’s action history. [2]

So the agent faces an expert default. Overriding it should require specific evidence, and the experiments show that general model capability alone does not deliver that discipline.

3. PAMO: who gets credit for a correct override?

Outcome-only RL rewards a correct slate regardless of why it was correct. The authors point out that a successful correction may genuinely rely on cross-platform history or may come from generic reranking, and outcome rewards cannot tell the two apart. [1]

Personal Attribution Mediation Optimization (PAMO) adds a second signal. For each sampled response, it scores the same visible rationale twice — once with full history and once with cross-platform history masked — and uses the length-normalized log-likelihood ratio as a personal mediation support score. [1]

<code><code>c_g=\frac{1}{|R_g|}\log\frac{\bar\pi_\theta(R_g\mid B,H)}{\bar\pi_\theta(R_g\mid B,H^-)}.</code></code>

The authors are explicit that this score is model-relative: it measures how much a rationale depends on the cross-platform history, not whether that history is correct or useful. [1] That is an important limit. A rationale can lean heavily on irrelevant cross-platform records.

Advantage reallocation with a value floor

PAMO decomposes single-target NDCG into nested top-k cutoffs, computes GRPO-style group-relative advantages at each cutoff, and modifies advantages only for rollouts whose cutoff outcome disagrees with the platform. [1] The group-relative baseline itself comes from GRPO, introduced in DeepSeekMath as a PPO variant that reduces memory use. [3]

Within each disagreement set, PAMO tilts the fixed advantage mass toward high-support responses, subject to a constraint. The reallocated weights must keep at least the average direction-aligned NDCG magnitude of the uniform allocation, which the authors call the value floor; the solution takes an exponential-family form with a dual coefficient for that constraint. [1]

<code><code>w^\star_{g,k}\propto\exp\!\left(\eta c_g+\lambda_k v_{g,k}\right),\quad \lambda_k\ge 0.</code></code>

The floor matters because support alone is not quality. Without the value floor, the support-only variant performs below the κ = 0 baseline across tested support strengths, while full PAMO improves HR@10 and NDCG@10. [1] History dependence is useful only after it is tied back to ranking value.

The theory shows that the reallocation preserves cutoff-level advantage mass and follows a locally optimal first-order direction within a fixed rollout group; the authors note it does not imply global optimality of the policy or a guaranteed held-out improvement. [1] I read the theorem as a design justification for the update rule, not as an accuracy guarantee.

PAMO adds one teacher-forced scoring pass during training, no additional rollouts, and no inference-time component. [1] The deployed agent costs the same to run as the GRPO-trained one.

4. What the results show

All trainable agents share one setup. They use a Qwen3-4B-Instruct backbone; GRPO and PAMO start from the same SFT checkpoint and share data, rollout and update budgets, differing only in advantage reallocation. At test time each model produces one response per episode. [1]

HR@10, rescue, and harmful-override rates at K = 10 (percent):

Test set · Platform · GRPO · PAMO · GRPO rescue / harm · PAMO rescue / harm

Movie · 56.0 · 61.2 · 62.2 · 9.2 / 3.9 · 10.3 / 4.0

Toy · 47.7 · 53.4 · 54.8 · 11.2 / 5.6 · 11.9 / 4.8

Grocery (unseen) · 45.0 · 48.8 · 50.6 · 8.4 / 4.7 · 10.0 / 4.5

Beauty (unseen) · 48.6 · 51.2 · 51.7 · 8.9 / 6.3 · 8.6 / 5.5

OpenPlay (real, test-only) · 40.8 · 47.4 · 47.9 · 13.8 / 7.1 · 13.2 / 6.0

These figures come from Tables 2 and 3. PAMO improves HR@10 and NDCG@10 over GRPO on all five datasets with similar mean intervention, and lowers harmful overrides on four of them. [1]

The pattern is more specific than “PAMO rescues more.” On Beauty and OpenPlay, PAMO rescues slightly less than GRPO; its gain comes from fewer harmful overrides. On Movie, harm is marginally higher while rescues rise. The improvement is a better balance, not a uniform win on both terms.

The OpenPlay gap is small at HR@10 (47.9 vs. 47.4) but larger at the top of the list. On OpenPlay, PAMO’s NDCG@10 is 0.284 versus 0.267 for GRPO, with H@3 of 26.2 versus 22.2. [1] For an interface that shows three items, that is the number that matters.

The masking test addresses the obvious objection that the agent is just a better reranker. Re-evaluating each model with only cross-platform records masked — same candidates, same platform proposal — lowers HR@10 and NDCG@10 for every evaluated model in the five-dataset macro average. [1]

Figure 2 of the paper: HR@10 and NDCG@10 averaged over the five test sets, with cross-platform history masked (open markers) and with full context (filled markers). The dashed line is the platform baseline. Source: arXiv:2610.07588, CC BY 4.0.

5. Where the evidence is thinner than the headline

Sampled candidates. Each episode ranks the held-out target plus 49 popularity-sampled negatives. [1] Krichene and Rendle show that sampled ranking metrics can be inconsistent with exact metrics and may not preserve relative comparisons between recommenders, even in expectation. [4] The paper’s relative comparisons are cleaner than most because every method sees identical candidates, but absolute HR@10 on 50 items should not be read as full-catalog performance.

Proxy platforms. Four of the five environments treat Amazon product categories as proxy platforms, and the authors acknowledge these are not independent services. [1] The real cross-platform test links Steam, Nintendo, and Xbox activity for a 645-user cohort used only for external testing. [1] That external test is the strongest evidence in the paper, and also the smallest.

Point estimates. The main tables report one test-time response per episode. I did not find seed variation or confidence intervals for the five-dataset comparison, so sub-point differences, such as Beauty HR@10, deserve caution.

Rationale faithfulness. The authors annotate 900 PAMO rescues with Claude Sonnet 4.6 and report cross-platform context as the primary driver in 72%, with 25% incidental. [1] That analysis reads the agent’s own rationales. A rationale that cites Led Zeppelin records can still be a post-hoc story for a decision the model would have made anyway. The support score has the same dependency: it scores the rationale text, not the slate.

Position anchoring. The agent sees candidates relabeled by platform rank. LLM rankers can be biased by item positions and popularity in prompts. [5] Some deference may be positional rather than evidential. That does not invalidate the comparison, since every agent sees the same order, but it is a confound to test.

Availability. The authors say MediateRec will be released upon internal approval [1], so the results cannot yet be reproduced externally.

6. The experiment I would add before trusting an agent with my feed

Masking cross-platform history shows the information matters. It does not show that this user’s information matters. I would add a mismatched-history control: give the agent another user’s cross-platform history with the same length and category mix. If performance stays near the full-history level, the agent is exploiting generic extra text. If it drops toward or below the masked condition, the gain is user-specific.

Second, I would break down harm by history length and source overlap. A user with two cross-platform records and one with forty are not the same risk. The agent should defer more often when evidence is sparse, and rescue-harm curves conditioned on history size would show whether it does.

Third, a deployment should treat deferral as the default and override as a decision with a cost. The paper’s identity already provides the metric. A production system could cap harmful overrides at a fixed rate on a held-out set, then maximize rescues under that constraint — the same risk-coverage framing that applies to search-agent escalation.

The broader point is architectural. Once users bring their own agents, platform rankings become proposals rather than final answers. The paper provides a clear way to score that negotiation: count what the agent fixed, count what it broke, and train it to tell the difference.

References

[1] Personal-Agent Mediated Recommendation with Cross-Platform User History

[2] Self-Attentive Sequential Recommendation

[3] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[4] On Sampled Metrics for Item Recommendation (KDD 2020)

[5] Large Language Models are Zero-Shot Rankers for Recommender Systems

阅读英文版

本期目录

LLM 与推荐系统
1. TIGER：用 Semantic ID 生成推荐候选
2. LIFT 与 LIME：信息边界与计算复用

RL 与 Agent
3. OThink-R1：通过混合训练数据学习快慢思考
4. UFO：通过多轮强化学习利用失败反馈

一、LLM 与推荐系统

1. TIGER：用 Semantic ID 生成推荐候选

Recommender Systems with Generative Retrieval 将推荐检索表述为生成物品标识：每个物品由具有语义意义的码字元组表示，序列到序列 Transformer 根据用户交互历史，自回归预测下一个物品的 Semantic ID。

论文摘要报告在所测试数据集上优于对照方法，并报告对没有历史交互的物品有更好的检索表现。这里的结论对应论文的任务与实验设置，不能外推成所有冷启动场景都会改善。

编辑建议：除了推荐指标，还应检查编码冲突、合法物品映射、目录增量更新与服务成本。语义编码的共享有助于迁移，但量化层级不能仅凭示意图就解释为人工类目树。以上是后续评估问题，不是本次独立验证的实验结论。

Paper: Recommender Systems with Generative Retrieval — arXiv:2305.05065

2. LIFT 与 LIME：信息边界与计算复用

LIFT 将每次交互拆为有序的 Request、Item、Context、Action 状态，并按因果序列建模。召回读取 Request 状态，排序读取 Context 状态，让两项任务共享历史建模，同时保留阶段各自的信息边界。摘要介绍了 Role-Conditioned Attention 和 Pre-LN Bias，并报告 ML-20M 与 Taobao 上的联合评测。

LIME 处理长历史、大候选集与模型规模带来的计算成本。低秩 Link Embeddings 解耦用户和候选交互，使注意力权重能够预计算；LIME-XOR 将历史长度相关的注意力复杂度从平方降为线性。摘要报告在大候选集或长序列的测试设置中约十倍推理加速，同时达到与强 Transformer 基线接近的效果。这不是任意业务的通用加速保证。

编辑建议：LIFT 的部署首先要检查事件时间和特征可见性；LIME 则应计入缓存构建、刷新和失效成本。两项方法解决不同问题，不能只按某个提升百分比排序。部署前应在一致预算下联合测量质量、尾时延、缓存新鲜度和回退率。

Paper: LIFT: A Lifecycle-aware Interaction Factorization Transformer for Unified Retrieval and Ranking — arXiv:2610.10556

Paper: LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling — arXiv:2510.18239

二、RL 与 Agent

3. OThink-R1：通过混合训练数据学习快慢思考

OThink-R1 面向推理效率与准确率的权衡。论文使用辅助 LLM judge 识别必要与冗余推理，再构造混合微调数据：裁剪冗余推理得到快思考样本，保留完整推理得到慢思考样本。模型经微调后，根据问题特点自主选择思考模式。

Judge 在这里参与训练数据构造，不能据此声称每次线上推理都必须调用外部 judge。摘要报告在数学与问答基准上减少推理 Token，同时维持有竞争力的准确率。

编辑建议：按任务难度检查计算节约与错误率，不只看平均 Token 数。这个条目介绍的是混合数据微调，不把它误标为一个已验证成功的 RL 算法。

Paper: OThink-R1: Intrinsic Fast/Slow Thinking Mode Switching for Over-Reasoning Mitigation — arXiv:2506.02397

4. UFO：通过多轮强化学习利用失败反馈

A Simple “Try Again” Can Elicit Multi-Turn LLM Reasoning 讨论单轮强化学习后的模型为何可能在多轮交互中重复错误答案。其 Unary Feedback as Observation（UFO）方法在多轮 RL 中，将错误回答后的简短负面反馈作为观察，训练模型利用交互历史修正答案。

摘要报告保留单轮表现并改善多轮推理，还介绍了鼓励更少回合完成任务和不同推理尝试的奖励设计。具体奖励公式和实验设置尚未独立核查，因此不在这里补充未核验细节。

编辑建议：同时测量最终成功率、重复答案比例、总计算成本与错误恢复能力。这是训练方法，不意味着对任何现成模型反复说“再试一次”都会获得相同收益。

Paper: A Simple “Try Again” Can Elicit Multi-Turn LLM Reasoning — arXiv:2507.14295

本期一个实验问题

知识库问答增加结构化输出和引用检查后，能否减少无依据的回答？

这是编辑提出的待实施实验，不是上面五篇论文报告的结果。固定资料库、问题集、模型与检索证据，对比自由文本、带引用 schema、schema 加引用与证据校验三个条件。观察引用存在性、证据支持度、正确率、拒答率、延迟和 Token 成本。

格式正确不代表事实正确；检索漏掉必要证据时，引用检查也无法补回知识，应单独记录检索失败。

Read the Chinese edition.

In this issue

LLMs & recommender systems
1. TIGER: generating recommendation candidates with semantic IDs
2. LIFT and LIME: information boundaries and reusable computation

RL & agents
3. OThink-R1: learning fast/slow thinking from hybrid training data
4. UFO: using failure feedback in multi-turn reinforcement learning

I. LLMs & recommender systems

1. TIGER: generating recommendation candidates with semantic IDs

Recommender Systems with Generative Retrieval formulates retrieval as identifier generation. Each item is represented by a semantically meaningful tuple of codewords. A sequence-to-sequence Transformer learns to autoregressively predict the next item’s semantic ID from the user’s interaction history.

The abstract reports improvements over evaluated baselines and better retrieval for items with no prior interaction history. These claims concern the paper’s tested tasks and settings, not a guarantee across all cold-start scenarios.

Editorial suggestion: examine code collisions, valid-item mapping, catalog updates, and serving cost alongside recommendation quality. Code sharing may support transfer, but quantization levels should not be assumed to form a human-readable category tree. These are evaluation questions, not independently verified findings from this issue.

Paper: Recommender Systems with Generative Retrieval — arXiv:2305.05065

2. LIFT and LIME: information boundaries and reusable computation

LIFT decomposes interactions into ordered Request, Item, Context, and Action states, modeled as a causal sequence. Retrieval reads the Request state and ranking reads the Context state, sharing history modeling while retaining stage-specific information. The abstract describes Role-Conditioned Attention and Pre-LN Bias and reports joint evaluations on ML-20M and Taobao.

LIME addresses the cost of long histories, large candidate sets, and larger models. Low-rank Link Embeddings decouple user and candidate interactions, enabling attention-weight precomputation. LIME-XOR reduces history-length attention complexity from quadratic to linear. The abstract reports roughly tenfold inference acceleration in evaluated large-candidate or long-sequence settings, with near-parity to strong Transformer baselines. This is not a universal serving-speed guarantee.

Editorial suggestion: check event timing and feature availability for LIFT, and include cache construction, refresh, and miss handling for LIME. They solve different problems. Compare quality, tail latency, cache freshness, and fallback rates under consistent budgets rather than ranking methods by a single improvement percentage.

Paper: LIFT: A Lifecycle-aware Interaction Factorization Transformer for Unified Retrieval and Ranking — arXiv:2610.10556

Paper: LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling — arXiv:2510.18239

II. RL & agents

3. OThink-R1: learning fast/slow thinking from hybrid training data

OThink-R1 addresses the trade-off between reasoning efficiency and accuracy. An auxiliary LLM judge identifies essential and redundant reasoning, helping construct hybrid fine-tuning data: redundant reasoning is pruned for fast-thinking examples, while full reasoning is retained for slow-thinking examples. After fine-tuning, the model selects its mode based on the problem.

The judge participates in training-data construction. This does not establish that every inference-time decision requires an external judge. The abstract reports reduced reasoning-token use with competitive accuracy on mathematics and question-answering benchmarks.

Editorial suggestion: measure computation savings and errors across task difficulty, rather than reporting only mean token counts. This entry concerns hybrid-data fine-tuning and should not be labeled a successfully verified RL algorithm.

Paper: OThink-R1: Intrinsic Fast/Slow Thinking Mode Switching for Over-Reasoning Mitigation — arXiv:2506.02397

4. UFO: using failure feedback in multi-turn reinforcement learning

A Simple “Try Again” Can Elicit Multi-Turn LLM Reasoning examines how models trained with single-turn RL can repeat incorrect answers during multi-turn interactions. Unary Feedback as Observation (UFO) incorporates brief negative feedback after incorrect answers as observations during multi-turn RL, training models to revise using interaction history.

The abstract reports retaining single-turn performance while improving multi-turn reasoning. It also describes reward designs that encourage fewer turns and diverse reasoning attempts. Exact reward formulas and experimental settings have not been independently checked, so unverified details are not added here.

Editorial suggestion: evaluate final success, answer repetition, total computation, and error recovery together. This is a training method; repeatedly telling an arbitrary existing model to “try again” does not imply the same gains.

Paper: A Simple “Try Again” Can Elicit Multi-Turn LLM Reasoning — arXiv:2507.14295

An experiment to try

Can structured outputs and citation checks reduce unsupported answers in a knowledge-base assistant?

This is an editorial proposal, not an experiment reported by the five papers above. Hold the corpus, question set, model, and retrieved evidence fixed. Compare free-form answers, answers using a citation schema, and the same schema with citation and evidence checks. Measure citation existence, evidential support, accuracy, abstention, latency, and token cost.

Valid formatting does not establish factual accuracy. Citation checks cannot recover evidence that retrieval missed, so retrieval failures should be recorded separately.

GPTs are distinguished by their vast network size, with some housing billions of parameters and being honed on wide-ranging datasets. This immense size equips them to handle diverse linguistic tasks, ranging from composing emails to distilling intricate ideas into accessible language.

In embarking on the journey to craft a GPT model using Python and NumPy, it’s crucial to grasp the architecture’s underlying elements and their synergy, particularly within the sphere of NLP. We will delve into these fundamental concepts and offer an in-depth breakdown of each aspect to forge an exhaustive tutorial.

Step 1: Setting Up the Environment

First, clone the necessary repository and set up the environment:

GitHub — jaymody/picoGPT: An unnecessarily tiny implementation of GPT-2 in NumPy.

import numpy as np
import sys
!git clone https://github.com/jaymody/picoGPT.git
# Pretrained weights are from https://github.com/jaymody/picoGPT
sys.path.append('./picoGPT')

Step 2: Defining Basic Functions

The building blocks of our GPT are functions like GELU (Gaussian Error Linear Unit), linear layers, softmax, and layer_norm mechanisms:

def gelu(x):
    # GELU activation function applies a non-linear activation to the input 'x'
    # It's used to introduce nonlinearity in the model and helps with learning complex patterns
    return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) 
                                  * (x + 0.044715 * x**3)))
def linear(x, w, b):
    # Linear transformation function that computes the weighted sum of the inputs 'x' plus a bias 'b'
    # 'w' are the weights and 'b' is the bias term, this is a basic building block of neural networks
    return x @ w + b

def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

def ffn(x, c_fc, c_proj):
    a = gelu(linear(x, **c_fc))
    x = linear(a, **c_proj)
    return x

def layer_norm(x, g, b, eps: float = 1e-5):
    # Feedforward network function that applies a GELU activation after a linear transformation
    # followed by another linear transformation
    # It's a sequence of operations commonly used in transformers
    mean = np.mean(x, axis=-1, keepdims=True)
    variance = np.var(x, axis=-1, keepdims=True)
    x = (x - mean) / np.sqrt(variance + eps)
    return g * x + b

2.1 GELU vs other activefunction

GPT employs GELU (Gaussian Error Linear Unit) as its activation function due to its superior performance demonstrated in practice. GELU is a relatively new type of activation function introduced by Hendrycks and Gimpel in 2016. It incorporates characteristics of the ReLU (Rectified Linear Unit) and the probability distribution properties of the normal distribution. The specific expression for the GELU function is:

Some advantages of GELU include:

• Non-saturating: GELU provides non-linear activation, but unlike ReLU, it is not entirely saturated for negative values, which helps with the flow of gradients and reduces the problem of vanishing gradients.
• Smooth Gradients: Due to its smooth nature, GELU can provide continuous gradients, which contributes to stability and performance during the training process.
• Probabilistic Interpretation: Since it emulates the “gating” of input values (i.e., the probabilistic weight of the input value x), it introduces a degree of natural randomness, which can help with regularization of the model and potentially improve its generalization ability.
• Empirical Performance: GELU has shown better performance than other activation functions across various tasks and benchmarks. Particularly in NLP tasks, the combination of GELU with the Transformer model has proven to be highly effective.

2.2 Softmax implementation

We can observe that there are certain differences between the softmax function’s theoretical formula and its implementation in code.

Modifications have been made to the softmax computation within the code, primarily to enhance numerical stability.

def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

• Overflow and Underflow: Direct computation of exp(x_i) can be problematic because the exponentiation can result in very large numbers for moderate values of x_i (overflow) or very small numbers for large negative values of x_i (underflow), leading to numerical instability. Subtracting the max value np.max(x) from each element in x before exponentiation helps to mitigate this risk because the largest value in the adjusted vector will be 0, which, when exponentiated, equals 1. This subtraction makes the largest of the exp values 1, preventing overflow in the numerator and denominator.
• Numerical Stability: By normalizing the values (shifting the inputs to a smaller range), the softmax function becomes more numerically stable. Without this step, the computation of exp(x_i) for large x_i could result in an infinite value in floating-point arithmetic, which would lead to NaN values during division.
• Efficiency: The use of axis=-1 and keepdims=True allows the function to be applied to multi-dimensional arrays and maintain their shape, which is useful when computing softmax across certain dimensions (like batches of data or different features) in more complex models.

2.3 LayerNorm vs BatchNorm

Layer Normalization (LayerNorm) and Batch Normalization (BatchNorm) are both techniques used to stabilize and accelerate the training of deep neural networks, but they work in slightly different ways and are used in different contexts.

Here’s a comparison:

Batch Normalization (BatchNorm):

• Normalizes the output of a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation (hence the name “batch” normalization).
• The normalization is done over the batch dimension, i.e., for each feature, BN computes the mean and variance over the batch of data.
• It is typically applied to the inputs of the activation function in a layer.
• BatchNorm has parameters (scale and shift) that are learnable during training. It requires a sufficiently large batch size to work well since the statistics are computed over the batch.
• It can lead to significant improvements in convergence for convolutional neural networks (CNNs) and fully connected layers.

Layer Normalization (LayerNorm):

• Normalizes the inputs across the features instead of the batch. For each data point in the batch, LayerNorm computes the mean and variance used for normalization across the features.
• It is particularly useful in recurrent neural networks (RNNs) and Transformer models, where the batch size can often be one, or the sequence length can vary.
• LayerNorm also has learnable parameters (gain and bias) that can be adjusted during training. Since LayerNorm does not rely on the batch dimension, it behaves consistently during training and inference.
• It has been found to be more effective than BatchNorm in sequence models where the temporal dynamic is important, as it normalizes across the features at each time step.

In summary, BatchNorm is effective for tasks where the batch size is large and consistent, and it’s commonly used in CNNs. LayerNorm, on the other hand, is more suited to sequence models like RNNs and Transformers because it doesn’t depend on batch statistics and therefore can handle variable sequence lengths and smaller batch sizes.

Step 3: Implementing Multi-Head Attention and Feed-Forward Networks

Multi-head attention and feed-forward networks are crucial for the model:

def attention(q, k, v, mask):
    s = q @ k.T
    s = s / np.sqrt(q.shape[-1]) + mask
    p = softmax(s)
    o = p @ v
    return o

# Multi-head attention
def mha(x, c_attn, c_proj, n_head):
    x = linear(x, **c_attn)
    qkv = np.split(x, 3, axis=-1)
    qkv_heads = list(map(lambda x : np.split(x, n_head, axis=-1), qkv))
    
    causal_mask = (1 - np.tri(x.shape[0], dtype=x.dtype)) * -1e10
    out_heads = [attention(q,k,v, causal_mask) for q, k, v in zip(*qkv_heads)]
    x = np.hstack(out_heads)
    x = linear(x, **c_proj)
    return x

The foundation of GPT lies in the Transformer model, a type of deep learning model designed to handle sequential data like text.

Self-attention Mechanism: This allows the model to weigh the importance of different words in a sentence, providing a dynamic understanding of textual context.

Imagine you’re reading a complex novel, and in each sentence, some words are more important to the meaning than others. For instance, in the sentence “The cat sat on the mat,” words like “cat” and “mat” are more crucial to understanding the sentence than “the” or “on.” Self-attention in a neural network works similarly. It helps the model to focus more on certain words (or parts of the input) when trying to understand or generate text.

In essence, self-attention is like a smart, dynamic highlighter that helps AI models determine which words in a sentence are the most important for understanding the meaning and context, much like how we focus on certain words while reading or listening to make sense of the information.

Here, for each word, we have k, v, q; k and v represent the associations between them. In the self-attention mechanism, “K,” “Q,” and “V” stand for “Key,” “Query,” and “Value,” respectively. These terms come from the field of information retrieval and are used to help the model determine how much attention to give to a given input, such as a word or phrase. Here’s how they work:

• Query: The query represents the current word or item being considered, which can be thought of as a search query looking for other related information.
• Key: The key corresponds to the representation of all other words or items in the database, which can be imagined as index keywords used by the query to find matching items.
• Value: Once a query finds a matching item through a key, the associated value is used to calculate a new representation of the input word or item. The value contains the actual information you want to retain in the output of the attention mechanism.

In the self-attention mechanism, each input element, such as a word, is transformed into representations of query, key, and value. The model allocates attention scores by calculating the compatibility between the query and all keys, typically done by taking their dot product and then applying the softmax function to ensure that all attention scores add up to 1. These scores are then used to weight the corresponding values, resulting in the final output representation of each input element. This output captures the context information of all elements in the input sequence.

In fact, matrix operations are used in the implementation:

• Efficiency: Matrix operations can efficiently utilize modern hardware architectures, especially GPUs (Graphics Processing Units) and TPUs (Tensor Processing Units), which are designed for large-scale parallel computations. Converting the calculations of the attention mechanism into matrix operations can significantly speed up training and inference processes.
• Batch Processing: Matrix operations allow us to process multiple data samples simultaneously, known as batch processing. This means that the network can learn from multiple examples at once, making more effective use of computational resources.
• Vectorization: Matrix and vectorized computations reduce the number of necessary iterations, eliminating the need for explicit loops for each operation. This not only simplifies the complexity of writing code but also reduces computation time.

Step 4: Building the Transformer Blocks

Each transformer block in GPT consists of multi-head attention and feed-forward networks, along with layer normalization:

def transformer_block(x, mlp, attn, ln_1, ln_2, n_head):
    x = x + mha(layer_norm(x, **ln_1), **attn, n_head=n_head)
    x = x + ffn(layer_norm(x, **ln_2), **mlp)
    return x

Step 5: Assembling the GPT Model

Now, we assemble these components to create the GPT model:

GPT is built upon the Transformer architecture, focusing on the decoder part for text generation.

• Decoder Blocks: These are the building blocks of GPT. Each block includes a self-attention layer followed by a feed-forward neural network.
• Layer Normalization and Skip Connections: These components help in stabilizing the learning process and enabling the training of deep networks.

def gpt2(inputs, wte, wpe, blocks, ln_f, n_head):
    x = wte[inputs] + wpe[range(len(inputs))]
    for block in blocks:
        x = transformer_block(x, **block, n_head=n_head)
    x = layer_norm(x, **ln_f)
    return x @ wte.T

Step 6: Generating Text

The process of generating text in GPT involves predicting the next word in a sequence given the previous words.

• Tokenization: The input text is broken into tokens (words or subwords) which are then converted into numerical representations.
• Predictive Modeling: The model predicts the next token based on the previous tokens, using its learned understanding of language patterns.

To generate text, we use the model to predict the next token in a sequence iteratively:

def generate(inputs, params, n_head, n_tokens_to_generate):
    from tqdm import tqdm
    
    for _ in tqdm(range(n_tokens_to_generate), "generating"):
        logits = gpt2(inputs, **params, n_head=n_head)
        next_id = np.argmax(logits[-1])
        inputs.append(int(next_id))
    return inputs[len(inputs) - n_tokens_to_generate :]

Step 7: Running the Model

Finally, we create a function to run our model with a given prompt:

def main(prompt: str, n_tokens_to_generate: int = 40, 
         model_size: str = "124M", models_dir: str = "models"):
    from utils import load_encoder_hparams_and_params
    encoder, hparams, params = load_encoder_hparams_and_params(model_size, models_dir)
    input_ids = encoder.encode(prompt)
    
    assert len(input_ids) + n_tokens_to_generate < hparams["n_ctx"]
    
    output_ids = generate(input_ids, params, hparams["n_head"], n_tokens_to_generate)
    
    output_text = encoder.decode(output_ids)
    return output_text

main("What is LLM?")

2023-10-25 22:49:45.842762: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Fetching checkpoint: 1.00kb [00:00, 6.07Mb/s]                                                       
Fetching encoder.json: 1.04Mb [00:00, 3.13Mb/s]                                                     
Fetching hparams.json: 1.00kb [00:00, 4.89Mb/s]                                                     
Fetching model.ckpt.data-00000-of-00001: 498Mb [00:22, 22.0Mb/s]                                    
Fetching model.ckpt.index: 6.00kb [00:00, 7.37Mb/s]                                                 
Fetching model.ckpt.meta: 472kb [00:00, 1.95Mb/s]                                                   
Fetching vocab.bpe: 457kb [00:00, 1.90Mb/s]                                                         
generating: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:02<00:00, 13.49it/s]

Now you can find you own GPT works~

'\n\nLLM is a programming language that is designed to be used in a variety of applications. It is a programming language that is designed to be used in a variety of applications. It is a'

Conclusions

In conclusion, constructing your own Generative Pre-trained Transformer (GPT) from scratch is a challenging yet incredibly rewarding endeavor that underscores the remarkable capabilities of Python and NumPy in the realm of natural language processing. Throughout this article, we have navigated the foundational elements of GPT architecture, from understanding the intricacies of self-attention and position encoding to grappling with the complexities of large-scale model training. The process reveals not just the technicalities of building a state-of-the-art language model but also illuminates the nuances of machine learning workflows and the importance of computational efficiency. Armed with the insights and methodologies discussed, one is well-equipped to embark on the journey of creating a custom GPT model, tailored to unique tasks and capable of pushing the boundaries of what we expect from AI in language understanding and generation. Whether for academic exploration, industry innovation, or personal projects, the knowledge to build a GPT model from scratch in Python using NumPy is a powerful tool in any developer’s arsenal.

Who is Andrej Karpathy?

Before we delve into the course itself, it’s important to understand who Andrej Karpathy is. Karpathy is a prominent AI researcher, educator, and practitioner. Formerly the Director of AI at Tesla, he played a pivotal role in developing the company’s self-driving technology. Prior to Tesla, he was a founding member of OpenAI, where he contributed significantly to AI research. Karpathy is widely recognized for his expertise in deep learning, and his approachable, clear style of teaching has made complex topics accessible to countless learners worldwide.

What is the “Zero to Hero” Course?

“Zero to Hero” is an open online course designed to teach you neural networks from the ground up. Whether you’re new to the field or have some background in programming and mathematics, this course offers a structured, practical approach to mastering neural networks and deep learning.

The course is structured as a series of lectures, each carefully designed to build upon the previous ones. It focuses on teaching the fundamental concepts of deep learning, such as how neural networks work, how they can be built from scratch, and how to optimize them to achieve high accuracy in various tasks.

Key Features of the “Zero to Hero” Course

1. Practical, Hands-on Learning: Unlike many theoretical courses that bombard you with heavy mathematics, “Zero to Hero” adopts a practical approach. You will learn by coding neural networks from scratch, which helps solidify the underlying concepts and provides a clear understanding of how everything works under the hood.
2. Python and PyTorch Focused: The course utilizes Python, one of the most popular programming languages for AI, along with PyTorch, a leading deep learning framework. Karpathy guides you through every step of coding neural networks in PyTorch, from simple perceptrons to complex deep learning models. This practical experience is invaluable, as it enables you to start building your own AI models almost immediately.
3. Accessible to All: One of the course’s biggest strengths is its accessibility. You don’t need an advanced degree in mathematics or computer science to get started. Karpathy breaks down complex concepts into digestible pieces and gradually introduces more advanced topics. This makes it ideal for anyone, whether you’re a student, a developer transitioning into AI, or just an enthusiast.
4. Free and Open Access: True to the spirit of open education, “Zero to Hero” is completely free and available to anyone with an internet connection. All the materials, including lecture videos, code, and assignments, are available on Karpathy’s website. This accessibility makes high-quality education available to a global audience, allowing people from all walks of life to learn and explore AI.
5. Community and Support: The course also encourages a strong community of learners. Engaging with peers through forums, GitHub repositories, and social media provides support and motivation, which is crucial for mastering complex topics like deep learning.

What Will You Learn?

Here’s a snapshot of what you will learn through the “Zero to Hero” course:

• Foundations of Neural Networks: Understanding how neurons and layers work, the basics of forward and backward propagation, and the mathematics behind neural networks.
• Deep Learning Techniques: Learn various architectures, such as convolutional neural networks (CNNs) for image recognition and recurrent neural networks (RNNs) for sequence data.
• Optimization and Training: Techniques to optimize the learning process, including gradient descent, loss functions, regularization, and more.
• Practical Implementation: Building neural networks from scratch in Python and PyTorch, and understanding every component that makes them work.
• Advanced Topics: Dive into more sophisticated topics like transfer learning, generative models, and reinforcement learning as you progress through the course.

Why Take This Course?

1. Learn from a Thought Leader: Andrej Karpathy is renowned for his deep understanding of AI and his ability to explain complex concepts simply and effectively. Learning directly from an expert in the field provides a unique perspective and a deeper understanding of the subject.
2. Hands-On Approach: The course emphasizes building models from scratch, which helps reinforce your understanding and gives you the skills to tackle real-world problems. By the end of the course, you’ll have a solid foundation in neural networks and the confidence to build your own AI models.
3. No Prerequisites Required: Unlike many other AI courses that require a strong background in mathematics or computer science, “Zero to Hero” is designed for everyone. Whether you’re just starting out or looking to expand your knowledge, this course is a great fit.
4. Flexible and Free: The course is self-paced and available online, allowing you to learn at your own speed and according to your schedule. Plus, it’s completely free, which removes the financial barrier often associated with quality education.

How to Get Started

To get started with the “Zero to Hero” course, simply visit Karpathy’s course page. All you need is a computer with an internet connection, a basic understanding of programming in Python, and a curiosity to learn. Start by watching the introductory videos, setting up your development environment, and following along with the exercises. Engage with the community, share your progress, and don’t hesitate to ask questions.

Final Thoughts

Andrej Karpathy’s “Zero to Hero” course is a fantastic opportunity for anyone interested in neural networks and deep learning. Its practical, hands-on approach, combined with the expertise of a renowned AI leader, makes it an invaluable resource for beginners and seasoned practitioners alike. So, if you’re looking to make your mark in the world of AI, this is the perfect place to start.

Happy learning, and here’s to becoming an AI hero!