The purpose of an estimator is what it does: Misspecification, estimands, and over-identification

1. Problem Statement

The paper addresses a fundamental disconnect between econometric theory and empirical practice in economics, specifically regarding over-identified models (models where the number of moment conditions exceeds the number of parameters).

• Theoretical Norm: Standard econometric textbooks recommend testing over-identifying restrictions (via Hansen's $J$-test) and, if the model is not rejected, using the efficient Generalized Method of Moments (GMM) estimator with optimal weighting.
• Empirical Reality: A survey of 36 papers in the American Economic Review (2020–2024) reveals that researchers frequently:
  1. Ignore formal $J$-tests (only 3 of 22 over-identified papers reported them).
  2. Use inefficient estimators (e.g., identity weighting, hand-selected weights, or subsets of moments) rather than the optimal efficient estimator.
  3. Rely on "eyeball tests" (visual inspection of moment gaps) rather than formal specification tests.

The Core Issue: If models are assumed to be correct, these practices are suboptimal. However, if models are misspecified (which the authors argue is the norm), the standard efficiency arguments collapse. Different weighting matrices in GMM do not just yield different variances for the same parameter; they converge to different probability limits (different estimands). The paper argues that researchers implicitly choose which parameter they wish to estimate based on their misspecification concerns, but this choice is often opaque and introduces significant "researcher degrees of freedom."

2. Methodology and Framework

The authors employ a rigorous theoretical framework combining asymptotic analysis, local misspecification, and empirical auditing.

A. Conceptual Distinctions

The paper formalizes the distinction between:

1. Econometric Model ($M$): The set of pairs $(\theta, P)$ consistent with economic theory, where $\theta$ is the parameter of interest and $P$ is the data distribution.
2. Statistical Model ($\mathcal{P}$): The set of data distributions $P$ for which there exists a $\theta$ consistent with $M$.
3. Misspecification Types:
   • Statistical Misspecification: The true data distribution $P \notin \mathcal{P}$ (the model cannot fit the data).
   • Econometric Misspecification: The true pair $(\theta, P) \notin M$ (the mapping from data to the economic parameter is incorrect).

B. The "Purpose of an Estimator"

The authors adopt the perspective that "the purpose of an estimator is what it does." Under misspecification, an estimator $\hat{\theta}$ converges to a pseudo-true value $\theta^*(P)$, which depends on the weighting matrix used.

• Key Insight: In over-identified models, changing the weighting matrix $\Omega$ changes the limit $\text{plim}(\hat{\theta}_\Omega)$. Therefore, choosing a weighting matrix is effectively choosing the estimand.

C. Local Misspecification Framework

To analyze the trade-offs between bias and variance, the authors utilize the local misspecification framework (Newey, 1985). They assume the true data generating process $P_n$ satisfies:
$$E_{P_n}[g(X_i, \psi)] = \frac{\eta}{\sqrt{n}}$$
This ensures that the magnitude of misspecification is of the same order as sampling uncertainty ($\sqrt{n}$), preventing the misspecification from dominating the asymptotic distribution while allowing for non-zero bias terms.

D. Empirical Audit

The authors conducted a systematic review of AER papers using GMM, calculating $J$-statistics for papers that did not report them (where data/code were available) to quantify the extent of over-identification violations in practice.

3. Key Contributions and Theoretical Results

A. Characterization of Estimands under Misspecification

The paper synthesizes recent literature showing that under misspecification:

• OLS vs. WLS: In linear causal models, Ordinary Least Squares (OLS) and Weighted Least Squares (WLS) estimate different weighted averages of causal effects. Neither is "more efficient" in a universal sense; they simply target different parameters.
• GMM Weighting: Different weighting matrices in GMM target different pseudo-true values. The "efficient" GMM estimator is only efficient if the model is correctly specified. Under misspecification, it is just one of many possible estimands.

B. Misspecification-Robust Inference

The authors reiterate the necessity of using misspecification-robust standard errors (the "sandwich" estimator) for GMM.

• Conventional GMM standard errors are invalid under misspecification because they ignore the variance contribution from the non-zero mean of the moment conditions.
• They recommend the non-parametric bootstrap (without recentering) as a robust alternative, noting that recentering moments imposes the null hypothesis of correct specification, leading to invalid inference under misspecification.

C. The New Theoretical Result: $J$-Statistics and the Range of Estimates

The paper's most significant theoretical contribution is a new result relating Hansen's $J$-statistic to the range of achievable estimates under local misspecification.

• The Result: The set of GMM estimates attainable by varying the weighting matrix (subject to a bound on the standard error relative to the efficient estimator) forms an interval centered at the efficient GMM estimate. The width of this interval is proportional to $\sqrt{J}$.
• Corollary (The "Weight-Hacking" Bound): A researcher can manipulate the weighting matrix to achieve a $t$-statistic of at least $\sqrt{J}$ for any null hypothesis.
  • If $\sqrt{J} > 1.96$, a "nefarious" researcher can reject any null hypothesis at the 5% level simply by choosing an appropriate (inefficient) weighting matrix.
  • Therefore, the $J$-statistic measures the scope for researcher degrees of freedom in over-identified models.

4. Empirical Results

• Survey Findings: In the sample of 22 over-identified papers, 14 did not use efficient weighting. Only 3 reported $J$-statistics.
• Re-calculated $J$-Statistics: The authors computed $J$-statistics for papers that did not report them.
  • While few models were rejected at the 5% level by the $J$-test (suggesting the models aren't "wildly" wrong), many had $\sqrt{J} > 1.96$.
  • Implication: Even when models are not formally rejected, the magnitude of the $J$-statistic often indicates that researchers could have easily "hacked" their results to achieve statistical significance by changing weights.

5. Significance and Recommendations

The paper fundamentally shifts the interpretation of over-identified models from a hypothesis testing problem ("Is the model true?") to an estimation problem ("What parameter are we estimating?").

Key Recommendations for Researchers:

1. Distinguish Specification Types: Explicitly separate concerns about econometric specification (does the model describe the mechanism?) from statistical specification (does the model fit the data?).
2. Justify Estimands: If using an inefficient estimator (e.g., hand-selected weights), researchers must explicitly state which pseudo-true value they are targeting and why it is preferable to the efficient one.
3. Use Robust Standard Errors: Always use misspecification-robust standard errors (or bootstraps) rather than conventional GMM standard errors.
4. Report $J$-Statistics: Researchers should report $J$-statistics even if they do not perform formal $J$-tests. The $J$-statistic serves as a diagnostic for the sensitivity of results to weighting choices. A large $J$-statistic implies that the results are highly sensitive to the choice of estimator, limiting the credibility of any single point estimate.

Conclusion:
The paper argues that in the presence of inevitable model misspecification, the "best" estimator is not necessarily the most efficient one. Instead, transparency regarding the estimand (what is being estimated) and the sensitivity of results to weighting choices (measured by the $J$-statistic) is crucial for robust empirical research. The $J$-statistic is re-framed not merely as a test of model validity, but as a measure of the "researcher degrees of freedom" available to manipulate findings.