LLM Recommendation Systems: Retrieval, Ranking, RAG, and Evaluation

Recommendation systems create a tempting illusion: the newer the model, the more advanced the system.

Anyone who has operated one knows that the model is only the part above water. Below it sit exposure bias, latency budgets, feature freshness, inventory constraints, exploration risk, and counterfactual evaluation. Large language models add a useful semantic layer, but they do not make those older problems disappear. They also introduce new ones: higher cost, variable output, and explanations that can sound persuasive without reflecting the reasons an item was ranked.

So the useful question is not, “Should we rebuild our recommender around an LLM?” It is:

Which decision lacks semantic understanding, which decision should still follow behavioral evidence, and which claim must be proved by an experiment rather than by a language model’s confidence?

This guide lays out a path from a credible baseline to production. It covers retrieval, ranking, embeddings, RAG, exploration, evaluation, and feedback loops. For recent research, it distinguishes public benchmark results from peer-reviewed work and evidence from live industrial systems. A state-of-the-art result is a lead to investigate, not a production permit.

The short answer: an LLM is a component, not the recommender

A dependable recommendation system is still usually a multi-stage funnel:

behavior logs, item content, and business constraints
                         |
      retrieval: CF / two-tower / graph / semantic search
                         |
          scoring: relevance, conversion, and value
                         |
   re-ranking: diversity, novelty, fairness, and inventory
                         |
             delivery, feedback, tests, monitoring

LLMs fit into this funnel better than they replace it:

• Content understanding: extract topics, attributes, style, and constraints for new or poorly labeled items.
• Semantic retrieval: interpret natural-language needs and map queries into the item space.
• Conversation: clarify ambiguous intent, present retrieved candidates, and produce grounded explanations.
• Offline data work: assist labeling, enrich attributes, and distill knowledge into smaller models.

High-throughput retrieval, per-candidate scoring, hard business constraints, and final online validation usually still belong to specialized models and deterministic rules. Google’s published YouTube architecture separated candidate generation from ranking at industrial scale in 2016. LLMs can change the features and interaction layer; they do not repeal capacity or latency limits.

1. Begin with the baseline no new model can bypass

Collaborative filtering still captures collective choice

Item-based collaborative filtering, matrix factorization, two-tower models, and graph recommenders remain useful because they learn directly from the user-item interaction structure. A language model may understand what a book is about. It does not inherently know what people who repeatedly bought that book chose next.

These signals encode different kinds of knowledge:

Keep a simple baseline that the team can explain and reproduce. On sparse data, ItemCF, matrix factorization, or a two-tower model may be easier to train, serve, and roll back than an end-to-end LLM. Add semantic information only when an ablation shows that it closes a real gap—cold start, long-tail discovery, or complex queries—not because embeddings are fashionable.

Attention is not synonymous with Transformer

DIN is often grouped carelessly with Transformer recommenders. That is incorrect.

The Deep Interest Network uses a local activation unit within the embedding-and-MLP family. It weights a user’s historical behaviors according to the candidate ad, producing a candidate-specific interest representation. It uses an attention-like mechanism, but it is not a Transformer.

SASRec is a self-attentive sequential recommender that predicts the next interaction. BERT4Rec uses a bidirectional Transformer with a Cloze-style objective to learn from behavior sequences. Their roles are related, but not interchangeable:

• DIN: selects candidate-relevant interests for click-through-rate prediction.
• SASRec: models a sequence causally to predict the next item.
• BERT4Rec: learns a bidirectional sequence representation by recovering masked items.

Architecture names matter. Confusing them leads directly to bad assumptions about training objectives, serving cost, and what can run inside an online latency budget.

2. Let semantic representations fill gaps, not erase behavior

Item embeddings: begin on the content side

Language or multimodal models can encode titles, descriptions, attributes, images, and transcripts into vectors. This is most valuable when:

1. a new item has rich content but no interactions;
2. a user expresses compound constraints, such as “a quiet film for a rainy evening, without a romantic plot”;
3. the catalog taxonomy is too coarse to describe tone, theme, or use context.

Do not treat raw content similarity as the final ranking score. Use it as one retrieval channel or one feature family alongside ID embeddings, popularity, price, availability, and session context. Semantic similarity means “these things resemble one another.” It does not mean “this user will click, buy, or remain satisfied.”

A practical hybrid retriever can merge several candidate lists, preserve each candidate’s source, remove hard-rule violations, and pass the result to a learned ranker. Source attribution is not bookkeeping trivia: without it, a team cannot tell whether a recall loss came from the collaborative index, the semantic index, a filtering rule, or stale catalog data.

User embeddings: compression is not understanding

Putting an entire history into a prompt is intuitive and usually unsustainable. Histories are long, noisy, privacy-sensitive, and constantly aging. A usable user state normally needs at least two clocks:

• Long-term state: recurring categories, price bands, and themes, refreshed daily or weekly.
• Session state: recent queries, clicks, skips, and the current entry point, refreshed in seconds or minutes.

One accidental click should not rewrite a long-term profile; years of history should not drown out a clear session intent. A lightweight sequential model can maintain the short-term vector, while an offline profile provides a prior. Gating or attention can combine them. An LLM may turn a complex history into a readable summary, but that summary needs timestamps, provenance, deletion support, and downstream evaluation.

UQABench asks the right underlying question: does a compact user embedding retain information that is actually useful for personalization? It evaluates sequence understanding, action prediction, and interest perception rather than trusting a single Recall@K value. Yet it remains a 2025 research benchmark, not an industry-wide production acceptance standard.

Collaborative and semantic spaces must be aligned

SeLLa-Rec studies a genuine mismatch. Collaborative models are good at distinguishing sparse user and item IDs; LLMs are good at language. Their representation spaces do not naturally align. The paper projects collaborative knowledge into special tokens, aligns them with the LLM’s semantic space, and reports gains on MovieLens-1M and Amazon Book.

That makes alignment a reasonable prototype to test. It does not establish production readiness:

• the work is a 2025 preprint;
• its evidence comes from two public offline datasets;
• it does not demonstrate billion-item scale, strict tail latency, or continuous distribution shift.

The disciplined conclusion is modest: aligned fusion can outperform naive prompting on those benchmarks. It is not evidence that a generative model should replace a production ranker.

3. Semantic IDs: useful evidence with a narrow boundary

Conventional ID embeddings struggle with growing catalogs, uneven interaction counts, and the lifecycle of new and retired IDs. Random hashing caps table size, but collisions combine unrelated items and can destabilize representations.

Semantic IDs cluster items hierarchically from content embeddings, allowing related items to share meaningful prefixes. The 2025 Semantic ID prefix n-gram paper turns those prefixes into token parameters, attempting to make collisions semantically coherent rather than random.

Its evidence is stronger than a benchmark-only result: the authors report integration into Meta’s production ads ranking system and gains in live deployment. That is industrial evidence. It still should not be generalized without qualification:

• clustering is only as good as the content embeddings;
• rebuilding IDs as items change creates migration and versioning work;
• shared prefixes can help the long tail while erasing business-critical distinctions;
• an ads distribution and objective do not automatically transfer to music, news, or commerce.

Before rollout, inspect performance by head and tail buckets, measure representation drift during an ID migration, and compare latency and memory. If feature freshness in the ordinary ID model is not observable, a semantic hierarchy will make failures harder—not easier—to locate.

4. RAG and conversational recommendation: generation must follow retrieval

The appeal of conversational recommendation is not that a model can recite ten product names from memory. It is that a person can state a need too nuanced for a row of filters.

A safe architecture looks like this:

natural-language request
          |
extract hard constraints and soft preferences
          |
retrieve live catalog, inventory, prices, and policies
          |
specialized ranker and constraint-aware re-ranker
          |
LLM presents grounded candidates or asks a question

RAG grounds facts: candidates come from the current catalog; availability and prices come from live records; explanations refer to retrieved attributes. RAG does not automatically personalize the ranking, repair incomplete retrieval, or guarantee that a generated explanation is faithful.

Separate three kinds of output:

• Facts such as price, inventory, and release date must come from retrieved records.
• Inferences about why an item may suit the user must name the evidence.
• Clarifying questions should replace confident invention when constraints are underspecified.

An explanation can be fluent and still false. If a promotion weight moved an item upward but the LLM says, “This best matches your long-term taste,” the system has invented a reason. Give the explanation layer the actual ranking features and applied rules, suppress sensitive profile attributes, and permit it to say less.

RAG also needs ordinary retrieval discipline: document versioning, access control, freshness guarantees, recall tests, and a deterministic fallback. A bigger context window does not make stale inventory true.

5. Ranking and re-ranking are different decisions

Retrieval answers, “What might be relevant?” Ranking answers, “In what order should these candidates appear?” Re-ranking asks, “What sequence can we responsibly show under real constraints?”

A ranking model can combine:

• collaborative and semantic similarity;
• user, item, and session features;
• calibrated probabilities for clicks, conversion, or other outcomes;
• freshness, quality, price, and availability;
• uncertainty estimates when the training distribution is thin.

Then a re-ranker can enforce deduplication, diversity, novelty, creator or seller constraints, safety, and inventory. Keeping these stages explicit makes trade-offs visible. If diversity is encoded only in an opaque scalar reward, the system may appear to improve while quietly sacrificing relevance or business value.

LLM re-ranking is most defensible for small, high-value candidate sets where semantic comparison matters and the latency budget permits it. For every-candidate scoring at large scale, distillation or a conventional cross-encoder is usually easier to control. Measure the full cost per request, not just model quality on a static list.

6. Exploration: distinguish recommendation RL from RLHF

Recommendation reinforcement learning is not automatically RLHF

Reinforcement learning in recommendation usually targets cumulative value over time: retention, discovery, diversity, or long-term satisfaction. It must handle delayed rewards, exposure bias, off-policy evaluation, and the risk of exploring on real people.

RLHF is a narrower family of methods that uses human preference feedback to train a reward model or optimize a policy. Its importance in LLM post-training does not imply that recommendation systems broadly use RLHF. Most production recommenders still depend primarily on implicit behavior, supervised learning, rules, bandits, and A/B tests. Pairwise human preferences are expensive and can carry annotator bias into the reward model.

Rec-R1 is not standard RLHF

Rec-R1 optimizes an LLM with feedback from a fixed black-box recommender. The reward comes from the recommendation model or task objective, not from labeled human preferences, so “reinforcement learning from recommender feedback” is more precise than RLHF.

The work reports gains in product search and sequential recommendation against prompting, supervised fine-tuning, and several discriminative baselines. As of its January 2026 revision, the paper is marked as published in TMLR. That strengthens its review status, but its reported evidence is still task-level experimental evidence—not proof of long-term user value in a live recommender. A fixed recommender used as the teacher also transmits its blind spots: rewarding NDCG will not cause fairness, diversity, or satisfaction to appear by themselves.

MA-RLHF is relevant inspiration, not recommender evidence

MA-RLHF treats sequences of tokens or higher-level language units as macro actions to reduce long-horizon credit-assignment difficulty. Its experiments cover summarization, dialogue, question answering, and code generation. They do not include recommendation or a recommender deployment.

The method may inspire post-training for conversational recommenders. It cannot support a claim that RLHF for recommendation is mature. Ideas can travel between domains; conclusions cannot.

Start with a contextual bandit when the action is local

For a banner, channel, notification, or a few re-ranking slots, a contextual bandit is often the more honest tool:

• actions and immediate rewards are easier to define;
• exploration traffic and risk can be capped;
• counterfactual evaluation and rollback are more tractable.

Full RL becomes more defensible when actions materially change later user state, long-term reward truly matters, and the team can support simulation, off-policy evaluation, constrained exploration, and rapid rollback. A complicated policy cannot rescue an ambiguous reward.

7. Evaluation: an offline score is testimony, not a verdict

Cover the entire funnel

One NDCG or CTR number cannot describe a recommendation system.

Use temporal splits so the future does not leak into training. Treat impressions, not just clicks, as first-class events. Segment results by new and established users, head and tail items, activity level, language, platform, and other relevant slices. Compare a new model with a simple strong baseline, not merely with whichever baseline is easiest to beat.

Counterfactual estimators can help when logs include action propensities and the assumptions are plausible, but they are not an excuse to skip online validation. Logged data reflects the old policy’s choices; what it never showed is difficult to evaluate honestly.

A/B tests remain the final production check

Offline metrics ask what might happen under assumptions about historical logs. A controlled online experiment asks what happens after the system changes. This matters because a recommender alters the data it will later learn from.

A serious experiment needs:

1. a predeclared primary metric, guardrails, and minimum detectable effect;
2. stable user-level assignment to prevent cross-group contamination;
3. a fixed duration that covers weekly cycles;
4. short-term engagement, longer-term retention, and negative feedback;
5. automatic stops for latency, cost, complaints, and inventory failures.

Calling A/B testing the gold standard does not save a careless design. Peeking repeatedly at significance, changing several policies at once, or allowing inconsistent bucketing can invalidate an online result.

8. Feedback loops: the recommender trains on its own shadow

Every recommendation changes what can be observed. Items placed high receive more impressions and interactions; unseen items remain data-poor; the next model may then interpret this manufactured popularity as preference. This is the core feedback loop, and LLM-generated interfaces do not escape it.

Instrument the loop deliberately:

policy chooses exposure
        |
user responds—or has no chance to respond
        |
logging records exposure, position, context, and outcome
        |
training data inherits the policy's selection
        |
next policy amplifies or corrects the pattern

Three practices matter:

• Log impressions, rank position, policy version, candidate source, and exploration probability.
• Preserve a controlled amount of exploration so the system can learn beyond its current certainty.
• Monitor distribution shifts, creator or seller concentration, repeated exposure, and negative feedback—not only positive events.

Feedback also includes explicit controls. “Not interested,” “show less like this,” saved preferences, and profile deletion can supply clearer intent than another click. These controls must alter serving and training behavior, not merely decorate the interface.

9. A practical route from zero to production

Stage 0: ask whether recommendation is needed

For a small catalog or explicit intent, search, filters, and a popularity list may be enough. A model’s gain must justify its data pipeline, experiment platform, privacy work, and long-term maintenance.

Stage 1: establish an explainable baseline

• Define impressions, clicks, dwell, purchases, skips, and negative feedback consistently.
• Build popularity, ItemCF, matrix factorization, or two-tower baselines.
• Create temporally split datasets and segment-level reports.
• Make canaries, rollback, experimentation, and monitoring routine.

Stage 2: build the multi-stage funnel

• Combine popularity, collaborative, content, and rule-based retrieval.
• Fuse user, item, and context features in ranking.
• Handle deduplication, diversity, inventory, and compliance in re-ranking.
• Log the source of every retrieved candidate.

Stage 3: introduce LLMs only where semantics are missing

Begin with offline or asynchronous work:

• generate structured attributes and embeddings for new items;
• parse hard constraints and soft preferences from natural-language queries;
• propose timestamped, revocable summaries of long histories;
• distill a large model into a small classifier or re-ranker.

Consider online LLM re-ranking or conversational generation only after these uses show reproducible incremental value.

Stage 4: design failure before serving generation

Answer these questions before an online call:

• On timeout, does the user receive the specialist ranker’s list, a cache, or nothing?
• When the model is unavailable, do hard constraints still hold?
• If an explanation lacks evidence, is it omitted or invented?
• Do prompts, retrieved records, or profiles contain sensitive data?
• What are the token, accelerator, and tail-latency budgets per request?

A fallback is not an emergency patch. It is an admission that probabilistic systems fail, followed by a precise definition of what failure should look like.

Stage 5: add exploration and long-term optimization carefully

Validate the value of exploration with a small, bounded bandit test before attempting offline RL or preference-based training. Rewards should include negative feedback, repeated exposure, and ecosystem constraints—not clicks alone. Any RL system needs a conservative policy, offline evaluation, a traffic cap, and one-step rollback.

10. Production checklist

Data

• Impressions are first-class training events.
• Consent withdrawal, deletion, and retention policies actually execute.
• Training and validation use temporal splits with no future feature leakage.
• New, sparse, highly active, and long-tail groups are evaluated separately.

Models

• A popularity or collaborative baseline exists.
• Semantic and collaborative signals have been ablated.
• DIN, SASRec, and BERT4Rec are compared according to their actual architectures.
• A preprint result is never presented as an industrial fact.

Systems

• Retrieval, ranking, and re-ranking each have latency and error budgets.
• LLM timeout or throttling has a deterministic fallback.
• Embedding models and Semantic IDs have versioning and rollback.
• Cost is observable by scenario, model, user segment, and experiment arm.

Evaluation

• Accuracy, diversity, long-tail exposure, negative feedback, and fairness are monitored together.
• Generated explanations trace back to candidate facts and real decision signals.
• A/B tests predeclare the primary metric, guardrails, duration, and stopping rules.
• Online value exceeds the added latency, inference cost, and operational complexity.

11. How to read the recent evidence

When reading a paper, ask four questions: Does the data resemble ours? Are the baselines strong? Does the reported gain include cost? Is there online or long-term evidence? New terminology does not compound. Reproducible engineering does.

Conclusion: recommendation should clarify choice

The purpose of recommendation should not be to keep a person scrolling forever. It should reduce the cost of finding something worthwhile. Click-through rate is easy to measure, so it is often mistaken for value. Language models are good at explaining, so fluency is often mistaken for understanding.

A reliable system keeps its restraint. Collaborative data tells us what groups of people have done. Semantic models help us understand content and intent. Rules protect the boundaries of reality. Experiments test our judgment. Each technique owns only a portion of the truth.

As models become more articulate, engineers need to remember this more clearly: the responsibility of recommendation is not to decide for the user. It is to make the choice easier to see.

References

The list below uses paper pages, proceedings, or official research pages. Recent work is labeled by its current publication status.

1. Covington, Adams, and Sargin, Deep Neural Networks for YouTube Recommendations , ACM RecSys 2016.
2. Cheng et al., Wide & Deep Learning for Recommender Systems , 2016.
3. Zhou et al., Deep Interest Network for Click-Through Rate Prediction , KDD 2018.
4. Kang and McAuley, Self-Attentive Sequential Recommendation , ICDM 2018.
5. Sun et al., BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer , CIKM 2019.
6. Wang et al., Enhancing LLM-based Recommendation through Semantic-Aligned Collaborative Knowledge , SeLLa-Rec, 2025 preprint.
7. Zheng et al., Enhancing Embedding Representation Stability in Recommendation Systems with Semantic ID , 2025 preprint with reported Meta deployment.
8. Liu et al., UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering , 2025 preprint.
9. Lin, Wang, and Qian, Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning , TMLR, revised 2026.
10. Chai et al., MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions , 2024 preprint, revised 2025.
11. Sanner et al., Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences , ACM RecSys 2023.
12. Qin et al., Attribute-based Propensity for Unbiased Learning in Recommender Systems , KDD 2020.

Further reading

• Context Is Not Prompt: Why Context Engineering Is Becoming AI’s New Foundation
• GEO Mechanics: How AI Retrieves, Re-ranks, and Cites You
• The Super Individual’s Intelligence System
• The Fastest Numbers Are Often Furthest from the Outcome