Daily AI Paper

Today’s pick is Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation. A lot of teams now use large language models to grade other models’ outputs, especially summaries, but we still don’t really know how these judges arrive at a score. This paper tackles that black box directly. The authors build controlled pairs of clean and corrupted summaries, then probe two judge models with causal tracing, logit analysis, and attention ablations to see which internal components matter at each stage. They find a surprisingly structured pipeline: early layers compare local errors and route evidence to the final token, while later MLP layers integrate that evidence and produce the rating. They also show fine-tuning doesn’t create this behavior from scratch; it reshapes an existing base model into a better judge. That matters because LLM judges are increasingly used in benchmarks, training loops, and product evaluation, so understanding when they are reliable, and how they can fail, is essential.