Condence, Calibration, and Abstention in Small Open-Weight Language Models on Psychiatric Knowledge Questions: A Cross-Domain Replication

This pilot study demonstrates that small, open-weight language models applied to psychiatric knowledge questions exhibit poor confidence calibration, domain-dependent accuracy rankings, and that prompting them to abstain from answering often degrades rather than improves performance when evaluated with rigorous paired statistics.

Original authors: Kunal Dhanda