Daily AI Paper

Diffusion models are now one of the main engines behind high-quality image generation, but that success comes with a serious downside: these models are trained on huge, often uncurated datasets that may contain private photos, copyrighted art, or other sensitive images. Because diffusion models can memorize parts of their training data, there’s a real need for auditing methods that can answer a simple but important question: was this image in the training set or not? That is the membership inference problem. In security terms, it’s a privacy attack. In practical terms, it can also be a useful verification tool for creators who want to know whether their content was used in training. The challenge is that diffusion models are harder to audit than earlier generative models, because they don’t produce a single output in one step. They generate images through a denoising trajectory, and the signal that distinguishes members from non-members can be subtle. A common strategy is to score an image by reconstruction loss and say that low-loss images are more likely to be members. But the paper points out a fundamental problem with that idea: some images are just easier to reconstruct than others. A simple image may get a low loss even if it was never seen during training, while a complex training image may still get a relatively high loss. So absolute loss mixes together two things that we really want to separate: how difficult the image is in general, and whether the model memorized it. This paper’s main idea is difficulty calibration across time. Instead of asking only, “How low is the loss?”, it asks, “Low compared with what baseline?” The authors use two different kinds of time. First, training time: compare a fine-tuned model to an earlier checkpoint. Second, diffusion time: compare a model’s behavior at one noise level to its behavior at nearby noise levels along the same denoising trajectory. The first method is called Reference-Model Calibration, or RMC. This is for the setting where you have access to a base checkpoint before fine-tuning. That base model gives a natural reference for the image’s baseline difficulty. If the image was newly memorized during fine-tuning, its loss should drop relative to the earlier checkpoint. If it was never included, the loss should stay relatively stable. In other words, RMC looks at the residual between the fine-tuned model and the reference model, rather than the raw loss alone. The authors show that this can isolate newly memorized data very cleanly, and they report near-perfect auditing in this setting, even without text captions. But there’s an important limitation: often you do not have the earlier checkpoint. And if your goal is to audit the original pretraining corpus, a prior model may not be available at all. So the paper’s more general method is Temporal Self-Calibration, or TSC. This is the core contribution. TSC uses only a single diffusion model. The key observation is that image difficulty tends to persist across nearby noise levels, while memorization is more localized in time. So instead of comparing one image to another image, TSC compares the same image’s loss at a target timestep to its losses at neighboring timesteps. That local trajectory acts as an internal baseline. If an image is intrinsically hard, that difficulty should show up across a range of nearby timesteps. But if the model has memorized the image, there may be a sharper, localized deviation at specific diffusion times. By looking at relative loss differences rather than absolute loss, TSC suppresses the difficulty factor and exposes the membership signal. The method is built around the idea that a persistent difficulty term can be treated like a multiplicative factor shared across time, and then canceled out by taking ratios or log-differences. The paper implements this in the log domain for stability. The exact model may predict different native targets, such as x, v, or epsilon, but the general idea is the same: compute a denoising loss trajectory over timesteps, then use local temporal neighbors as a reference to calibrate that trajectory. A useful background point here is that diffusion models and related flow-matching models all corrupt data with noise over time and learn to reverse that process. The authors align these trajectories using log signal-to-noise ratio, which gives a common time axis across model families. That makes the method more general than attacks that are tied to one particular parameterization. The paper also reviews the broader landscape of membership inference attacks. In the simplest threshold-based attacks, you compute some score like reconstruction error and classify samples below a threshold as members. More advanced methods may use shadow models or query the model along a whole denoising path. Prior diffusion attacks include naive loss-based scoring, SecMI, which uses deterministic DDIM inversion to reduce noise variance, and PIA, which starts from a known noise vector near the image. There are also attacks for text-to-image models, like CLiD, which compare conditional and unconditional losses. The authors argue that all of these still suffer from the same core issue: they do not fully disentangle membership from image difficulty. For experiments, the paper evaluates across multiple model architectures, datasets, and training objectives. The main settings include DDPM and Stable Diffusion. They compare against prior baselines, especially trajectory-based attacks. For the reference-model setting, they look at fine-tuned Stable Diffusion models, where RMC is especially effective. For the single-model setting, they test TSC on standard diffusion auditing tasks and also examine how it behaves under different prediction objectives, including x-, v-, and epsilon-prediction. The results are strong. The authors say RMC provides state-of-the-art auditing for fine-tuned Stable Diffusion models. TSC, meanwhile, consistently outperforms prior baselines across DDPM and Stable Diffusion evaluations. On CIFAR-10, they report a 62.3% true positive rate at 1% false positive rate, which they describe as roughly doubling the performance of existing attacks. That detail matters because membership inference is usually judged at very low false-positive rates: in auditing, you want to avoid falsely accusing non-members, so improvements at strict FPR thresholds are especially meaningful. Another important result is robustness across model parameterizations. Many existing attacks are designed around the standard epsilon-prediction setup and can degrade badly when the model predicts something else. TSC is reported to remain highly accurate across x-, v-, and epsilon-prediction models. That suggests the method is not just exploiting one narrow artifact of a particular training objective, but is capturing a more general property of how memorization appears in the denoising trajectory. The paper also analyzes where along the denoising path the membership signal actually appears. One takeaway is that the effective leakage window depends on the training objective. In other words, the best timesteps to inspect are not universal; they shift with the model’s parameterization. That helps explain why rigid attacks can fail when moved across settings. As for limitations, the paper is fairly clear about them. RMC depends on having access to a prior checkpoint, so it is mainly useful for fine-tuning audits rather than pretraining audits. TSC is more general, but it still assumes that membership shows up as a localized deviation in the temporal loss trajectory, and that neighboring timesteps provide a reliable baseline for difficulty. Like any threshold-based attack, performance also depends on the chosen operating point and the dataset or model family being tested. And while the results are strong, the broader lesson is not that diffusion models are trivial to audit — it’s that the audit score must be calibrated against image difficulty, not used raw. So the big message of the paper is this: if you want to infer membership in diffusion models, you should not compare absolute losses alone. You need a baseline that accounts for how hard the image is to begin with. If you have an earlier checkpoint, compare across training time with RMC. If you do not, compare across diffusion time with TSC. In both cases, the paper shows that calibration is what turns a noisy, confounded signal into a much more reliable audit.