Daily AI Paper

Multimodal large language models have become very capable at visual reasoning, but this paper starts from a practical limitation: a lot of real-world visual tasks cannot be solved reliably from the input image alone. Sometimes the model needs to find tiny objects in a crowded scene. Sometimes it needs outside knowledge about a rare entity. Sometimes it needs to measure something numerically from intermediate evidence. In those cases, more internal reasoning is not enough if the needed evidence was never grounded in the first place. That matters because visual agents are increasingly being used as reasoning engines over images, documents, webpages, and videos. In principle, they should be able to inspect evidence, search for missing information, and compute over what they find. In practice, though, tool use is messy. Different tools return different kinds of outputs: masks, depth maps, OCR text, web pages, images, arrays, coordinates. A model may not know how to interpret those outputs, and even if it does, it may lose access to the raw artifact it needs for the next step. The core question here is not just “can the model think harder?” but “can the system give the model the right evidence, in the right form, and let it keep using it?” The paper’s motivation is backed by a controlled scaling study. When the base model is pushed from higher reasoning effort to maximum effort, the token spending rises a lot, but the average score improves only a little. The exact point is that inference-time scaling alone shows diminishing returns. So the authors argue that progress requires better access to multimodal evidence, not just more internal deliberation over the initial prompt. To address this, they introduce OpenVLHarness, a multimodal reasoning harness built around two ideas: a capability layer and persistent multimodal memory. You can think of the harness as an orchestration system sitting between the model and a collection of tools. The capability layer standardizes how tools are called, and the memory system keeps the useful artifacts around so they can be referenced later. A Python coding interface then lets the model compute over stored results, which is especially important when the answer depends on combining outputs from multiple tools. The capability layer solves the first big problem: heterogeneous tools need a common interface, and their outputs need to be rendered in a way the model can actually inspect. Instead of tying the orchestrator to one backend, OpenVLHarness defines task-level interfaces. For example, visual grounding is specified as a task like “find the object described by this text in this image,” and then different backends can implement that task behind the scenes. The paper mentions that visual grounding can use SAM 3 or Grounding DINO, and OCR can use PaddleOCR-VL or DeepSeek-OCR, while the orchestrator keeps the same calling convention. That is useful because backend choice can vary based on accuracy, cost, and latency without changing the agent’s overall workflow. The base harness includes nine capabilities. Five are for perception: Zoom-in, Visual Grounding, OCR, Depth Estimation, and Camera Trajectory. Three are for information seeking: Text Search, Image Search, and Webpage Visit. The ninth is a Python Coding Agent for computation over intermediate results. This is a nice division of labor: perception tools acquire evidence, search tools fill in missing context, and code execution combines everything into precise calculations. But raw tool outputs are often not directly useful to the model. A segmentation mask, for instance, may be precise but unreadable if it is just serialized into context. A depth map can be numerically rich but visually opaque. So each capability has an output processor that renders the result into a model-facing observation. That observation may include an annotated image, a concise description, compact structured data, or references to reusable artifacts. For grounding, the harness can return a labeled overlay with the number of detected objects and normalized bounding boxes. For depth estimation, it can return a colorized map. For camera trajectory, it can return a visualization of the recovered path. The key idea is that the model sees something interpretable, while the underlying data are still retained for later use. That brings us to the second major mechanism: persistent multimodal memory. The paper emphasizes that it is not enough to describe a previous result in words. If a later tool call needs the exact mask, image, array, or URL, the system has to preserve it as an addressable artifact. OpenVLHarness therefore maintains an explicit artifact registry for each query. It stores image artifacts, structured data bindings, and web references under reusable handles. So the model might see names like input_image_3 or webpage_url_1, while the harness keeps the actual payloads behind those handles. This is what allows multi-step reasoning to stay grounded. A grounding tool can identify a set of cabinets in an image, and the result is stored as both a rendered overlay and a structured binding containing the boxes and masks. If the next step asks for the average depth inside each cabinet, the model does not need to restate the detections from scratch. It can refer back to the stored masks and combine them with a depth array. The harness resolves those handles automatically when another tool needs them. The coding interface is especially important here. The orchestrator can express a computational objective in natural language, such as “compute the average depth within each detected region,” and then a code-generation session writes a program that runs over the stored bindings. This makes it possible to combine grounding masks with depth arrays, filter detections, aggregate OCR values, or perform other calculations over intermediate results. In other words, the harness is not just a tool router; it is a memory-grounded computation environment. The experiments are broad. The authors evaluate on 22 benchmarks across four task domains: grounded perception, visual search and deep research, general VQA and hallucination, and spatial reasoning. The benchmark suite includes datasets such as CountQA, FSC-147, OdinW-13-All, FSCD-147, FSCD-LVIS, FVQA, MMSearch, MMSearch-Plus, HLE-VL, BrowseComp-VL, and RealX-Bench, among others. They test eight backbones, including open and proprietary models of different scales. The point is not to optimize one model, but to see whether the harness helps across a range of MLLMs. The reported results are strong and consistent. Across the evaluated settings, OpenVLHarness improves average performance over matched no-tool baselines by 9.7 to 12.6 points. The abstract gives specific examples: plus 11.2 with Qwen3-VL-8B, plus 9.7 with Kimi K3, and plus 9.9 with GPT-6 Astra. That consistency matters because it suggests the gains are not limited to one architecture or one benchmark family. The paper also compares against existing visual tool-use baselines. With GPT-6 Luna at medium reasoning effort, OpenVLHarness beats the base model, Codex, and Visual Sketchpad at maximum effort by 7.7 to 8.5 points, while using at most 37 percent of their model-token cost. So the improvement is not just about accuracy; it is also about efficiency. The system is getting better answers without requiring the model to think much harder in-token. Another important finding is that evidence acquisition and reasoning effort are complementary. The scaling experiments show that OpenVLHarness still benefits from greater reasoning effort, which means the harness does not replace inference-time scaling. Instead, it changes what the model is reasoning over. It gives the model better evidence, and then more reasoning effort can be spent on that evidence rather than on guessing from incomplete context. The ablation studies isolate the contributions of the two main components. The paper reports that both the capability layer and the persistent memory matter, which makes sense: usable tool outputs are helpful, but only if the resulting artifacts remain accessible for later steps. The qualitative examples make this especially concrete. In one trajectory, the agent chains camera poses, depth maps, and grounding masks into exact 3D computation. In another, it passes grounded crops by identifier to zoom-in, OCR, and image search before verifying the answer on the web. In a third, it reconciles masks from two grounding queries in code to fix an undercount. These examples show the harness enabling iterative, evidence-driven reasoning rather than one-shot tool calls. The paper also points toward customization. For users adapting the harness to a new domain, it showcases two coding-agent-driven workflows: harness search, which edits prompts, tool descriptions, and output templates, and tool generation, which composes existing capabilities into new tools. Importantly, these workflows do not require updating the underlying model weights. That makes the system more practical for domain adaptation. As for limitations, the paper is careful to frame OpenVLHarness as a harness rather than a fully solved agent. Its success still depends on the quality of the underlying backends, the usefulness of the rendered observations, and the correctness of the generated code. Memory helps only if the artifacts are registered cleanly and referenced correctly. And while the results are broad, they are still benchmark-based evaluations of a specific orchestration design, so real-world deployment would need to watch for robustness, latency, and tool reliability in more varied settings. The main takeaway is straightforward: for multimodal agents, better reasoning is not only about more tokens or bigger models. It is also about giving the model structured access to the right evidence, preserving that evidence across steps, and letting the model compute over it. OpenVLHarness is a concrete attempt to make that workflow systematic, and the results suggest that this kind of evidence-centered design can deliver sizable gains across a wide range of visual tasks.