Daily AI Paper

When AI systems start making consequential decisions, the question is no longer just whether they are accurate on average. The question is when we can safely let them act on their own, and when we should stop and ask for a human or another trusted verifier to check the output. That matters in settings like safety screening, moderation, data labeling, and any workflow where a mistake can propagate into real harm. But there is a practical constraint: experts are expensive, slow, and often unavailable at the scale modern AI systems operate. So the central problem is not “audit everything,” but “audit just enough to keep the system reliable.” This paper studies that problem in an online setting. The AI produces decisions one by one, and along with each decision it also provides a confidence score, or some other trust signal. That score might come from the same model, but it does not have to. It could also come from a separate verifier model. Based on that score, we decide whether to accept the AI’s decision automatically or send it to an auditor. If the auditor is invoked, we observe the truth and can correct an error. If not, the decision goes through unchecked. The key reliability target here is very specific: the paper wants to control the fraction of AI errors that are left unaudited. In other words, among all the errors the system makes, only a user-chosen fraction, say alpha, should slip through without expert review. This is different from just trying to reduce total error rate. It is a selective auditing problem: minimize the amount of human review while guaranteeing that the errors we miss stay under control. To understand why this is nontrivial, it helps to place it in the broader landscape. A lot of prior work on selective prediction or deferral treats the problem as a classification task: learn a rule that either predicts or rejects, and train that rule with a custom loss. That can work well, but its reliability depends on how well the learned classifier generalizes. If the data shifts, or the model is miscalibrated, the guarantees can weaken. Another line of work focuses on calibration and risk estimation, often in offline i.i.d. settings. But this paper is about an online stream of decisions, where the sequence can evolve over time and the system may adapt based on past audits. That makes the problem much closer to online learning and constrained optimization than to static classification. The paper formalizes the setup with a few simple quantities. At each round, the AI proposes a decision and a confidence score between zero and one. There is a latent error indicator saying whether the decision was actually wrong. There is also an audit indicator saying whether we chose to review that decision. Crucially, the true error is only observed when we audit. So the feedback is partial: if we skip review, we never learn whether that decision was right or wrong. That partial observability is the main technical challenge. The first method the authors develop is called additive thresholding. The idea is intuitive. Maintain a confidence threshold. If the AI’s confidence is below the threshold, audit deterministically. If the confidence is above the threshold, audit only occasionally using a small spot-check probability. Then update the threshold over time based on the observed audited errors. If the system is letting too many errors through, raise the audit rate; if it is being overly cautious, lower it. Because errors are only observed when audited, the update uses inverse probability weighting: observed errors are scaled by the inverse of the chance that they were observed. That is a standard trick in bandit and missing-data settings, and here it makes the update unbiased in expectation. Conceptually, the threshold update is trying to keep a certain balance: the long-run fraction of unaudited errors should track the target alpha. The paper shows that this simple procedure already gives strong control. In particular, the error ratio stays tightly around the target, with a deviation that shrinks like one over T in the regime where the total number of errors grows linearly with time. That is a strong guarantee, and it does not require assumptions about the data sequence. The sequence can even adapt to past decisions. An important point is that this first method does not assume the confidence scores are already well calibrated. It works no matter how those scores were produced. But the authors then ask: if the scores are not calibrated, can we use the auditing feedback to improve them online and reduce auditing further? That leads to their second refinement, additive thresholding with confidence calibration. Here the confidence scores are transformed through a learned monotone map. Instead of trusting the raw score directly, the method partitions the score range into bins and learns weights over those bins. The result is a recalibrated score that estimates the probability of a correct decision. Because the map is monotone and lives on a simplex, it stays in the unit interval and preserves ordering. This is essentially an online isotonic-style recalibration procedure. The learning signal again comes only from audited examples, so the gradients are estimated with inverse probability weights, and the weights are updated using exponentiated gradient descent, which is a mirror-descent method on the simplex. The intuition is that if the confidence scores can be made more faithful to actual correctness, then the thresholding rule can be more selective. In other words, better calibration should let the system identify the truly risky decisions more precisely, and that should reduce unnecessary audits while preserving the same error-control guarantee. The paper’s main theoretical contribution goes beyond just control: it asks for optimal efficiency. Among all policies that satisfy the error constraint, how few audits are actually necessary? This is framed as an online constrained optimization problem. The authors derive a family of algorithms with provable guarantees, and then identify one method that is asymptotically optimal. That means it achieves the best possible long-run auditing rate, both in adversarial sequences and in i.i.d. data streams. This is important because it says the method is not just safe; it is also, in a precise asymptotic sense, as efficient as one can hope for. The paper also emphasizes a technical challenge that distinguishes this work from standard constrained online learning: feedback is only revealed when you audit. That makes the constraint harder to optimize because the algorithm must decide whether to pay the cost of learning the outcome. The analysis builds on prior work in online constrained optimization and primal-dual methods, but adapts those ideas to this selective-feedback setting. On the experimental side, the authors test the methods in three applied settings: screening harmful requests to large language models, qualitative text annotation, and data labeling. These are good examples because they all involve expensive review and decisions of varying confidence. In each case, the goal is to see whether the method can maintain valid error control while reducing the number of audited items relative to baselines. The results support that claim. Across the tasks, the selective auditing methods achieve valid error control in practice and substantially reduce the need for human auditing compared with baseline approaches. The paper also notes that the simpler additive-thresholding method already performs strongly, while the calibration-enhanced version can further improve auditing efficiency by refining the confidence signal online. The comparison to a closely related prior online procedure suggests that the new methods can avoid unnecessary audits that earlier approaches would trigger more often. What should we take away from this? First, the paper reframes AI oversight as a sequential control problem rather than a static classification problem. Second, it shows that you can get rigorous, assumption-free guarantees on the fraction of unaudited errors, even when you only observe outcomes on audited examples. Third, it demonstrates that online calibration can be used not just to make scores more interpretable, but to directly save audit budget. There are also limitations to keep in mind. The strongest guarantees are in expectation, not high probability. The paper explicitly notes that when errors are very rare, these methods may not be as sharp, and it points to extensions for sparse-error regimes. Also, the theory focuses on controlling the fraction of errors left unaudited, which is a very useful metric, but not the only one that might matter in practice. Different applications may care about different tradeoffs, such as false rejections, cost asymmetries, or fairness across groups. Finally, like many online methods, the practical performance depends on how informative the confidence signal is in the first place. The calibration step helps, but it cannot create signal out of nothing. Overall, the paper gives a clean and practical answer to a pressing question: how do we supervise AI systems continuously without reviewing everything? Its answer is selective auditing with provable error control, and, in the best case, optimal audit efficiency.