The Statistical Price of Unknown Geometry in In-Context Regression
This paper establishes that in Gaussian in-context regression with one labeled observation, the statistical cost of estimating unknown task geometry is distinct from and strictly positive compared to the cost of selecting a geometric action without label feedback, revealing that label feedback provides a unique advantage in reducing Bayes approximation regret even as total risk decreases with more reference data.