Daily AI Paper

Imagine asking a language model a question like: after a vaccine, does one adverse event tend to come before another? Or does one symptom usually appear first? The model can answer in a fluent, confident sentence that sounds like a result. But in medicine, that kind of sentence is not enough. If you want to know whether a temporal claim is actually supported, you need a statistical test over the right records, under the right null hypothesis, with the right notion of which cases even count as evidence. That is the problem this paper tackles. It is about temporal clinical hypotheses: statements about how events unfold across many patient timelines. These are not simple facts you can verify by looking up one record. They are population-level claims, and the answer depends on how you define the population, how you define the denominator, what test you use, and what counts as a witness for the hypothesis. A system can get the final verdict right while making the wrong statistical choices along the way. The paper calls this a verdict-evidence gap, and that gap matters a lot in health, where a right-sounding but poorly grounded answer can be costly. To make that concrete, think of a hypothesis like “after Moderna, Pain typically occurs before Asthenia.” You could test that by looking at reports where both events appear, checking which comes first, and then asking whether the observed split is different from what chance would predict. But even there, small decisions matter. If both events happen at the same stage, does that report count? If too few reports are available, do you call the result inconclusive? If the hypothesis is phrased in the opposite direction, does the verdict flip? These are not cosmetic details. They define the actual statistical question. The broader background here is the intersection of fact verification, statistical reasoning, and temporal reasoning. Fact verification usually asks whether a statement is supported by some reference evidence. Statistical analysis benchmarks often ask a model to choose a test or produce a p-value, but they usually hand over the relevant data and only score the final answer. Temporal reasoning benchmarks often focus on one text, one timeline, or one time series. What has been missing is a benchmark for temporal hypotheses over many clinical timelines where the system must also decide which records belong in the analysis and which null and test apply. The paper’s benchmark is called STRATA, short for Statistical Reasoning Assessment for Temporal Assertions. It is built from clinical timelines derived from vaccine adverse-event reports. The key design choice is that every instance comes with gold labels not just for the final verdict, but for every step needed to reach it: the analysis plan, the evidence sets, the p-values, and the verdict itself. In other words, the benchmark does not just ask, “Did you get the answer right?” It asks, “Did you choose the right null? The right test? The right denominator? The right evidence?” The underlying data are the MedTempo clinical timelines, which come from VAERS adverse-event reports for three COVID-19 vaccines: Janssen, Pfizer, and Moderna. Each report is represented as a leveled timeline, meaning a chain of stages. A stage is not a calendar day; it is an ordering unit in the annotation. Events reported at the same stage are treated as co-occurring, with no order between them. The system sees the vaccine name and the stages, not the raw report text. STRATA includes 8,735 hypotheses built from 26 template groups across seven families. These cover temporal relations like precedes and co-occurs, but also sequence-of-three-events, first/last among candidates, position in the early half of a timeline, proximity of consecutive stages, and threshold-style claims about rates. The hypotheses are rendered in four registers: original, research, media, and clinical. There are three difficulty levels. Easy hypotheses concern one named report and can be read directly from its timeline. Medium hypotheses are tested within one cohort. Hard hypotheses compare two cohorts. The held-out test set contains 1,500 instances. What exactly does a system have to produce? The paper defines three outputs. First is the plan: this names the quantity being compared, the null, the tail, the test family, the number of arms, and the denominator rule. Second is the evidence: the set of reports, or arms, that actually count for the hypothesis. Third is the verdict: supported, refuted, or insufficient. The insufficient class is further split into not-locatable, when an arm is empty, and sparse, when there are too few reports to run the test. There is also a floor of 10 reports per arm; below that, the verdict is insufficient. The important statistical idea is that many temporal hypotheses reduce to a rate comparison. For an order hypothesis like “A precedes B,” the denominator is the set of reports where both A and B appear and are not tied at the same stage. The numerator is the subset where A comes before B. For a co-occurrence hypothesis, the denominator is the reports containing both events, and the numerator is the subset where they occur at the same stage. In one-arm cases, the observed rate is compared against a null rate, often 0.5 for order hypotheses. In two-arm cases, the system tests whether the rates of two arms are equal. The verdict is determined by one-sided p-values in the asserted direction and the opposite direction. If the p-value in the supported direction is below alpha, the result is supported; if the opposite direction is significant, it is refuted; otherwise it is insufficient. A major contribution of STRATA is that the codebook is part of the task. The codebook specifies the stipulations that turn a natural-language temporal statement into a testable population and a test. That matters because some choices are not obvious from wording alone. For example, should a report where two events occur at the same stage count for a precedes hypothesis? Should a tie for “first” leave a report in the denominator? STRATA ships these rules as text, and the benchmark evaluates whether the system can apply them correctly. The paper also introduces a staged prompting and verification paradigm called ART: articulate, retrieve, test. In the first stage, the model articulates the hypothesis as a structured plan. In the second, it retrieves the evidence sets named by that plan. In the third, the prescribed statistical test is run deterministically. Each stage has a clear output schema, and the model stages can be wrapped in a bounded verify-and-repair loop. The framework can also be run with gold filled in at different stages, which lets the authors isolate where errors come from. Another useful option is to give the model the codebook or withhold it, so the evaluation can separate what the model knows from what the architecture enforces. What do the experiments show? The paper evaluates five model configurations. The most important finding is that systems fail mainly at selecting which records count as evidence. In other words, they often stumble not on the final statistical judgment, but on the data selection step that defines the test in the first place. This is exactly the kind of error that ordinary accuracy metrics hide. A model can look good on verdicts while being wrong about the evidence sets that justify them. ART helps close that gap. By forcing the system to make the plan and evidence explicit before the test is run, it improves performance substantially. The paper reports that ART brings a 31-billion-parameter open model to within 0.01 of the frontier models under the same staged setup. And when the frontier models are given the codebook, they do well, which suggests that much of the challenge is not raw reasoning alone, but knowing and applying the benchmark’s stipulations correctly. There is also an important methodological point here: STRATA evaluates every decision on the path to a verdict, not just the verdict itself. That means the benchmark can reveal whether a system got the answer right for the wrong reason. In a medical setting, that distinction matters. A system that seems reliable on end-to-end accuracy may still be making brittle or unjustified evidence choices. As for limitations, the paper is careful to frame these as temporal hypotheses about reported adverse events, not causal effects. The supported verdict means the pattern is present in what was reported to VAERS; it does not prove that one event causes another. The benchmark is also built on a specific source of clinical timelines, so the conclusions are tied to that setting. And because the task depends on a released codebook, performance can change significantly depending on whether the system is given those stipulations. That is not a flaw so much as a feature of the benchmark: it makes explicit how much the answer depends on the analysis rule. What should we watch next? First, whether models can generalize beyond this particular benchmark to other temporal reasoning settings where the evidence selection problem is just as important. Second, whether systems can learn to infer the right statistical framing from wording without needing the codebook spelled out. And third, whether similar gold-for-every-step evaluation can be brought to other scientific reasoning tasks, where the final conclusion is only as good as the analysis path that produced it. The big takeaway is simple: for temporal clinical hypotheses, the question is not just what answer a model gives, but whether it can justify that answer statistically. STRATA makes that whole chain visible, and ART gives models a way to expose each step. Together, they move the field from fluent guesses toward evidence-grounded verification.