CFOs Meet LLMs

Technical Summary: "CFOs Meet LLMs"

Problem Statement
Business sentiment, particularly managerial expectations, is a critical economic signal. However, traditional measurement via surveys (e.g., the Duke–Federal Reserve CFO Survey) faces significant limitations: they are costly, periodic (quarterly), suffer from low cross-sectional coverage (hundreds of firms), and are subject to selection bias. Furthermore, while these surveys provide valuable forward-looking data, they are slow to compile and limited in frequency. The authors ask whether it is possible to generate credible, scalable, high-frequency CFO-style expectations without the costs, delays, or biases of human surveys.

Methodology
The authors propose a method to generate "synthetic CFO expectations" using Large Language Models (LLMs), specifically GPT-5.4 (OpenAI, 2026). The core approach involves "role-playing" the LLM as the specific CFO of a public company at a specific point in time.

1. Prompt Construction:

   • System Prompt: Establishes the persona of a financial executive and imposes a strict information cutoff matching the date of the actual survey. The model is instructed to use only information available up to that date, including web-searched public data (earnings reports, analyst coverage) prior to the cutoff, to prevent look-ahead bias.
   • User Prompt: Provides a detailed firm-executive profile (industry, revenue, headcount, geography, credit rating, etc.) and, crucially, the respondent's own history. This includes the CFO's prior optimism scores from previous survey waves (strictly before the cutoff) and the model's own prior forecasts for that same individual.
   • Task: The model is asked to rate its optimism about the overall U.S. economy on a 0–100 scale, mimicking the canonical Duke CFO Survey question.

2. Data and Validation:

   • The study covers 6,075 survey responses from public companies between 2002 and 2025.
   • For each human response, the authors generate three independent LLM forecasts and average them to mitigate stochastic noise.
   • Stability: The authors verify that the variance in LLM outputs is driven by differences across survey observations rather than stochastic noise within repeated calls (pairwise correlations ~0.96).
   • Out-of-Sample Nature: The specific human survey responses were never part of the model's training corpus (due to institutional agreements excluding them from training), ensuring the predictions are truly out-of-sample regarding the target variable.

Key Findings

1. Individual-Level Predictive Power:

   • The LLM-generated optimism score is a significant predictor of the actual CFO's response. In a regression of human optimism on the matched LLM score, the coefficient is highly significant (t-stat > 17 in baseline).
   • This relationship survives the inclusion of firm fixed effects (tracking within-firm, over-time variation) and year-quarter fixed effects (tracking cross-sectional variation within a quarter).
   • Crucially, the LLM score remains significant even when controlling for the respondent's most recent prior answer, demonstrating that the model is not merely echoing the last response but capturing additional predictive information.

2. The Role of Conditioning Information (Dose-Response):

   • Predictive accuracy scales positively with the amount of information supplied.
   • Respondent History: When no prior responses are available, the model's $R^2$ is ~0.10. With a few prior data points, it rises to ~0.33, and with more than three prior responses, it approaches ~0.50.
   • Firm Characteristics: Prompts containing a full set of firm profile fields (10–12 characteristics) yield better predictions ($R^2 \approx 0.32$) than sparse prompts ($R^2 \approx 0.21$).
   • Conditioning on history also restores cross-sectional heterogeneity; without it, instruction-tuned models tend to collapse toward modal opinions, compressing the distribution of synthetic scores.

3. Aggregation and Robustness:

   • At the quarterly aggregate level, mean LLM optimism remains a strong predictor of mean CFO optimism.
   • In a "horse race" regression including the Michigan Consumer Sentiment Index and the Survey of Professional Forecasters (SPF) GDP forecast, the LLM score retains significant explanatory power, while the human benchmarks lose significance. This suggests the LLM captures variation in executive sentiment missed by standard macro gauges.

Significance and Claims
The paper claims to provide the first individual-level, within-firm validation of LLM-simulated survey responses. Unlike prior work that focuses on matching the distribution of synthetic populations to real ones, this study demonstrates that an LLM can approximate the specific response of an identified human executive given their public footprint and personal track record.

The authors argue that with appropriate conditioning, LLMs can serve as credible "digital twins" of executives. This offers a new tool for financial research and policy by providing:

• Scalability: The ability to generate sentiment data for any public firm, not just survey participants.
• Frequency: The potential for daily or weekly sentiment updates, circumventing the quarterly constraints of traditional surveys.
• Counterfactuals: The ability to run "synthetic survey experiments" by varying company characteristics (e.g., foreign revenue exposure) while holding other factors constant.

The authors remain modest, acknowledging that LLMs cannot recover private information (e.g., pending orders, internal cost pressures) that human CFOs possess. Therefore, they position the LLM as a complement to, rather than a replacement for, human surveys. They also note that predictive accuracy is lower for first-time respondents lacking historical data, and that the model's "beliefs" are still bound by the regularities of its training data.