Daily AI Paper

Large language models are very good at producing text, but by themselves they cannot actually do things in the world. That is why modern AI agents connect an LLM to external tools and APIs: databases, workflow systems, search endpoints, and other services that can carry out actions. Once you build that kind of agent, the bottleneck changes. The key question is no longer only, “Can the model understand the request?” It becomes, “Can the agent retrieve the right tool from a large and messy tool catalog, and can it still do so when some tools are broken or unavailable?” That second part matters a lot in practice. The paper points out that existing tool retrieval methods mostly optimize semantic match: they try to rank tools based on how well the tool description matches the user query. That is useful, but it assumes the tool repository is basically static and reliable. Real tool ecosystems are not like that. The authors crawled tool catalogs from LangChain, CrewAI, LlamaIndex, and n8n, and they found that over half of these tools have serious issues such as broken links or missing authentication. So two tools may look equally relevant in text, but only one actually works at runtime. If your retriever ignores that, the agent may keep selecting failing tools. This paper proposes RATED, which stands for Reliability-Aware Tool Retrieval via Execution Dynamics. The core idea is to combine two things that are usually treated separately. First, you want strong semantic retrieval, so the system can find the right candidate tools from text. Second, you want a lightweight mechanism that learns from execution feedback, so the system can quickly stop favoring tools that are failing right now. The authors argue that contextual bandits are a good fit for this because they update online from each individual outcome, rather than waiting for large batches or retraining the whole retriever. To support this, the paper introduces EcoAgent, a benchmark built specifically for tool retrieval under reliability concerns. EcoAgent has two parts. EcoAgent-Syn is a synthetic training set, and EcoAgent-Real is a real-world evaluation set. Altogether the benchmark covers 5,172 real-world tools and more than 210,000 query-tool pairs. The tools come from the four open-source platforms I mentioned earlier, and their metadata are normalized into a common schema with names, functional descriptions, endpoints, and authentication fields. The benchmark also assigns each tool an automated reliability score from catalog metadata, not from manual human ratings. Those signals include name-description consistency, source credibility, and documentation quality. Tools with scores of at least 0.8 are treated as reliable, and that identifies 2,940 tools, or 56.8% of the repository. For training, they generate synthetic queries with Gemma-3-1b. For each tool, the model sees the tool name, description, and metadata, and then produces 10 to 20 task-oriented queries. The authors remove near-duplicates to keep the query set diverse, and they add hard negatives from SBERT retrieval as well as random negatives. That gives the semantic retriever a more realistic contrastive learning signal. For evaluation, they build EcoAgent-Real from ToolBench queries, then semi-automatically map those queries to tools in their repository. They use a first-pass retriever, filter out weak matches, and then have Gemma-3-1b verify the remaining pairs. A human review of 200 sampled queries agreed with the verifier on 94% of cases, which gives some confidence in the labeling pipeline. Methodologically, RATED has two stages. The first stage is a semantic perception module. This is a dual-encoder retriever based on DRAGON, with separate encoders for queries and tools. The model is fine-tuned on EcoAgent-Syn using a contrastive objective, specifically an in-batch InfoNCE loss with hard negatives. After training, the encoders are frozen at deployment. That is important because it means the semantic side stays stable and cheap to use. New tools can be added by encoding their documentation, without retraining the whole system. The second stage is the reliability-aware reranker. This is where the execution dynamics come in. After the semantic retriever produces a top-K candidate set, RATED reranks those candidates using a contextual bandit. The context is the query embedding. The action is the selected tool. The reward is binary: success or failure after execution. Only the chosen tool gets feedback, which is exactly the kind of partial feedback bandits are designed for. The bandit starts with a prior based on tool metadata. That prior is built from the same catalog signals used in benchmark construction: name-description consistency, source credibility, and documentation quality. This helps with cold start, when a tool has little or no execution history. Then, at runtime, the system uses a discounted LinUCB-style update with a separate linear model per tool. The discounting is the key non-stationary element: older observations are down-weighted so that recent failures matter more. In other words, if a tool suddenly starts timing out because of an outage, the model can move away from it quickly instead of averaging that failure away. The final ranking score blends semantic relevance, the bandit’s reliability estimate, and the metadata prior. The weights are tuned on validation data and fixed for testing. Operationally, the system retrieves top-K tools by semantic score, reranks them by the final score, executes the top choice, and then updates the bandit from the observed result. That creates a closed loop between retrieval and execution. The experiments are designed to test both standard retrieval quality and behavior under failure. The authors run everything on a single NVIDIA GeForce RTX 5070 GPU with 12 GB of VRAM. They fine-tune DRAGON for one epoch on EcoAgent-Syn, using a batch size of 8, a learning rate of 1e-5, warmup, and mixed precision. At inference, they retrieve the top 10 tools semantically and rerank them. The bandit uses a relatively small exploration parameter and a discount factor of 0.95, with 2,000 warm-up pairs. For the dynamic tests, they simulate 10,000 sequential selections and inject persistent outages by marking a fraction of tools as failing before the run begins. The failure set stays fixed throughout the experiment, so this tests how well the system adapts once some tools become unavailable. On offline retrieval, they evaluate with MRR, Recall@1, and Recall@5. The headline result is that RATED improves retrieval quality substantially over the ToolLLM API retriever on real-world queries, with a 52.2% relative MRR improvement. That tells us the semantic-plus-reliability design is not just helping in failure scenarios; it is also improving ranking quality in the ordinary retrieval setting. The most striking results come from the dynamic reliability experiments. Under a severe outage scenario with a 30% failure rate, RATED reduces the fault selection rate from 86.2% to 2.9%. That is a huge difference. It means the system is not just slightly better at ranking; it is learning to avoid broken tools almost entirely, even when a large fraction of the repository is compromised. The paper also says RATED keeps fault selection below 3% and recovers under both tested failure rates, which suggests the bandit update is reacting quickly enough to real execution feedback. Another important result is efficiency. The authors report negligible latency overhead: less than 1 ms over encode-only inference, specifically 0.172 ms per step. That matters because a reliability layer is only useful in practice if it does not make the agent sluggish. Here, the additional decision logic is lightweight enough to be compatible with real-time use. So what do these results actually show? First, semantic retrieval alone is not enough when the tool environment is unreliable. Second, a frozen retriever plus an online bandit is a practical way to separate stable language understanding from fast-changing operational reliability. Third, metadata can be useful not just for documentation, but as a real signal for cold-start reliability. And fourth, you can get these gains without expensive retraining or obvious latency penalties. There are still limitations to keep in mind. The paper’s dynamic evaluation relies on simulated failure injection, so while it is realistic in spirit, it is still a controlled simulation rather than a full production deployment. The reliability scores are automatically derived from metadata, which is elegant and scalable, but they are still proxies for actual runtime behavior. The system also only observes feedback for the selected tool, so adaptation depends on enough execution traffic to learn from. And because the semantic encoders are frozen, RATED can adapt quickly to changing reliability, but it will not itself learn deeper changes in tool semantics unless retrained. Even with those caveats, the main contribution is clear. RATED reframes tool retrieval as a problem of both meaning and operational trustworthiness. In a world where tool catalogs are large, noisy, and constantly changing, that is a very practical way to think about agent reliability. The benchmark, EcoAgent, is also a useful contribution on its own, because it gives the community a way to study tool retrieval under realistic failure conditions rather than only clean offline ranking.