Daily AI Paper

Retrieval-augmented generation has become one of the standard ways to make language models more useful in the real world. Instead of relying only on what was stored in pretraining, the model first fetches evidence from an external corpus, then uses that evidence to answer. That helps with freshness, grounding, and verifiability. But this paper starts from a simple and important complaint: the retriever part of the system has not improved nearly as much as the generator part. In many systems today, retrieval still means either lexical matching, like BM25, or nearest-neighbor search in a fixed embedding space. Those methods are fast and useful, but they have a hard ceiling. If the retriever never surfaces the right document or passage, the generator never even gets a chance to use it. The authors call this a capability asymmetry: the language model may be strong enough to reason over the right evidence, but the retrieval stage can block it before that happens. There are already ways to push retrieval closer to the capabilities of large language models. One is agentic search, where an LLM navigates a corpus step by step, deciding where to look next. That can be powerful, but it is slow because it requires many generated tokens at each step. Another option is reranking, where a stronger model reorders a candidate list produced by a first-stage retriever. That is cheaper, but it is still constrained by whatever the first-stage system happened to return. So the core problem is: how do we let a pretrained language model make retrieval decisions directly, without paying the full cost of token-by-token search and without depending on a separate retriever to hand it candidates? The key idea in this paper is DecisionRAG. Instead of searching the corpus in the usual flat way, DecisionRAG navigates it like a human browsing a library. The corpus is treated as a hierarchy: broad topics, documents, sections, passages. At each level, a decision model looks at a set of candidates in one shared context and decides which branches are worth exploring. Then the system moves down only those branches, pruning the rest. So rather than scoring every passage independently, it progressively narrows the search space. This matters because the model is not generating free-form text at each step. It is making decisions. The authors use what they call decision models, which are pretrained language models configured to answer questions with probabilities over predefined options, such as yes or no, or a small set of choices. The important point is that they do this in a single forward pass, without autoregressive generation. That makes them much faster than an LLM agent that has to emit tokens for every navigation step. A second important design choice is that candidates are judged jointly in a shared context. That means the model sees the query and multiple candidates together, and assigns each one a relevance probability in the presence of the others. This is different from scoring each candidate separately. The paper argues, and shows empirically, that this shared-context setup makes the scores more comparable, because the model is implicitly contrasting the options against one another. A passage that looks good on its own may look less relevant once a clearly better passage is present alongside it. The method has three main stages. First is document screening. Given the query and the descriptions of candidate documents, the decision model asks in parallel whether each document is worth exploring. Since all of these yes-or-no questions are evaluated in one forward pass, the system can score many documents quickly. It then keeps only the top few documents and prunes the rest. Second is passage scoring. The retained documents are expanded into their passages, and those passages are again judged in parallel in a shared context. The model ranks them by relevance and keeps the top passages across the selected documents. For some datasets, there are extra details to account for structure, such as weighting paragraph scores by page scores when the hierarchy is especially large. Third is optional final reranking. After the binary judgments produce a shortlist, the system can apply a multiple-choice decision over the top few passages. Here the model chooses among the shortlist as competing alternatives within a single decision, which can refine the final order. The authors use this only at the end, because multiple-choice mode has limits on the number of options, while binary judgments can scale to larger candidate sets. A nice aspect of this method is that it does not require a task-specific trained retriever at query time, and it does not require a precomputed dense index in the usual sense. It also does not rely on a first-stage retriever to build the candidate pool. Instead, it uses whatever structure the corpus already has, or constructs a document grouping when needed. The experiments span four benchmarks, including multi-turn dialogue, scenario-based questions, and open-domain search over a very large Wikipedia corpus with 2.68 million paragraphs. The evaluation is designed to test whether this navigation-based retrieval actually finds better evidence, not just whether it is elegant in principle. The paper compares against dense retrieval and reranking baselines, including a strong 8B reranker. It also studies ablations to isolate what matters: shared-context judging, hierarchical traversal, and the optional final rerank. The headline results are strong. Without any task-specific training, DecisionRAG beats dense retrieval with an 8B reranker by 13.8 points in R@1 on MultiDoc2Dial and by 17.4 points in Hit@1 on Natural Questions. Those are large gains, and they matter because top-1 retrieval is often the difference between the generator seeing the right evidence or missing it entirely. The paper also reports that judging candidates together in a shared context improves accuracy by up to 7.7 points over judging them independently. So the benefit is not just that the model is stronger than a traditional retriever; the way the candidates are compared is itself important. Another important finding is efficiency. Retrieval takes about 0.7 to 1.8 seconds per query, with a reported cost of 0.2 to 2.3 cents per query. That is fast enough to be practical in many settings, especially considering that the system is using a language model at each stage. The paper also compares against an LLM that performs the same navigation token by token, and DecisionRAG achieves comparable accuracy with 34 to 145 times lower latency. That is a major point in the paper’s favor: it shows that you can get much of the benefit of LLM-guided search without paying the full autoregressive cost. The ablations give some of the clearest insight into why the method works. Scoring candidates together, instead of in separate batches, improves top-1 performance, while splitting a large candidate pool into smaller requests mainly hurts the final ranking but not top-10 recall. That suggests shared-context comparison is especially valuable when the system needs to make a fine-grained choice among plausible candidates. It is less critical when the goal is just to keep enough candidates around for later screening. The paper also includes a stronger trained version. When the authors train a dedicated 8B decision model on in-domain data, they get an R@1 of 60.7 on MultiDoc2Dial, which surpasses the best compared published task-trained retriever by 5.7 points. That result is important because it suggests the approach is not just a clever inference trick; it may have substantial headroom as decision models improve and as more training is applied. At the same time, there are limitations to keep in mind. DecisionRAG still depends on some notion of corpus structure, whether that structure already exists or is constructed. If the hierarchy is poor, the navigation may prune away useful branches too early. The method also inherits the context-length limits of the decision model, so very large candidate sets may need to be split across requests. And while the approach avoids generating tokens during retrieval, it still relies on a capable decision model, so the quality and cost of that model remain central. Another thing to watch is that the strongest gains are reported on retrieval benchmarks and evidence selection tasks. That is exactly the right place to test the method, but it does not by itself prove that every downstream generation task will improve proportionally. Retrieval quality is necessary, not sufficient. So the next question is how much of these gains carry through end-to-end in full RAG systems, especially in settings where the corpus structure is messy or the answer evidence is distributed across multiple passages. Overall, this paper makes a compelling case that retrieval does not have to be a separate, weaker subsystem. By using decision models to judge candidates jointly and by navigating a hierarchy instead of searching a flat space, DecisionRAG brings more of the language model’s capability directly into retrieval itself. The result is a system that is both faster than token-generating agents and stronger than conventional dense retrieval pipelines, which makes it a promising direction for the next generation of retrieval-augmented systems.