No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models
This paper demonstrates that Contamination Detection via output Distribution (CDD) is largely ineffective for small language models (70M–410M parameters) because it fails to detect verbatim memorization, whereas probability-based methods like perplexity and Min-k% Prob consistently outperform it across various benchmarks.