Effects of interviewers on response to income and wealth items

This study analyzes data from the Survey of Health, Ageing and Retirement in Europe to demonstrate that interviewers' expectations regarding respondents' willingness to disclose financial information significantly predict actual item nonresponse, suggesting that these expectations offer valuable insights for improving survey design and interviewer training.

The Gist

Imagine you are trying to get a group of neighbors to fill out a very personal questionnaire about their bank accounts, savings, and how much money they make. This is exactly what researchers do in large surveys like SHARE (Survey of Health, Ageing and Retirement in Europe).

But there's a problem: Many people refuse to answer these money questions. They might say, "I don't want to talk about that," or "I don't remember." This is called item nonresponse, and it leaves big holes in the data, making the survey less useful.

This paper asks a fascinating question: Does the person asking the questions (the interviewer) change whether the neighbor answers?

Here is the breakdown of the study using simple analogies:

1. The "Optimistic Coach" vs. The "Pessimistic Coach"

The researchers looked at 41,934 people across 12 European countries. Before the survey even started, they asked the interviewers a simple question: "How many of the people you are about to interview do you think will be willing to tell you their income?"

• The Pessimist: An interviewer who thinks, "Oh, nobody likes talking about money. Most people will refuse."
• The Optimist: An interviewer who thinks, "People are generally open and honest; most will share their info."

The Discovery: The study found that the Optimists got better results. When an interviewer believed people would cooperate, those people were actually more likely to cooperate. It's like a sports coach: if the coach believes the team can win, the team plays better. The interviewer's confidence acts like a subtle signal to the respondent, making them feel more comfortable sharing sensitive details.

2. The "Missing Puzzle Pieces" Problem

In any big survey, data is messy. Sometimes the interviewer didn't fill out their own background form, or the respondent skipped a health question. This creates "missing puzzle pieces."

The researchers wanted to see if the "Optimist" effect was real, but they had to deal with these missing pieces. They tried three different ways to fix the puzzle:

• Method A: The "Throw It Away" Approach (Complete-Case Analysis)

  • Analogy: Imagine you have a 1,000-piece puzzle, but 500 pieces are missing. You decide to only look at the 500 pieces you have and ignore the rest.
  • Result: It's easy to do, but you lose half your picture. You might miss important patterns because you threw away so much data.

• Method B: The "Guess and Fill" Approach (Multiple Imputation)

  • Analogy: You look at the pieces you have and use a smart algorithm to guess what the missing pieces look like, then fill them in. You do this many times to see if the picture changes.
  • Result: You keep the whole picture, but your guesses might be slightly off, adding a little bit of "fog" to the image.

• Method C: The "Smart Average" Approach (Model Averaging)

  • Analogy: You try every possible way to fill in the missing pieces, weigh them by how likely they are to be right, and take an average of all the resulting pictures.
  • Result: This is the most sophisticated method, but the researchers found it didn't give them a much clearer picture than the simpler "Guess and Fill" method.

3. The Big Takeaway

After running the numbers, the researchers found that the "Optimist" effect was real and powerful, regardless of which method they used to fix the missing data.

• The Impact: In some countries, having an optimistic interviewer reduced the number of people refusing to answer money questions by up to 26%. That is a huge difference!
• The Lesson: It's not just about the questions asked; it's about who asks them and how they feel about asking them.

Why Does This Matter?

Think of a survey like a dinner party. If the host (the interviewer) seems nervous, doubtful, or thinks, "No one is going to like this food," the guests (respondents) will feel awkward and leave early. But if the host is confident, warm, and expects everyone to enjoy the meal, the guests relax and stay longer.

The Conclusion for Survey Makers:
If you want better data, don't just train interviewers on how to ask questions. Train them on mindset. Help them believe that people are willing to share. A confident, optimistic interviewer can turn a "no" into a "yes," saving the survey from having holes in its data.

In short: The interviewer's attitude is a secret superpower that can unlock the truth about people's wallets.

Technical Summary

1. Problem Statement

The paper addresses the persistent issue of item nonresponse in interviewer-administered surveys, specifically regarding sensitive financial questions (income and wealth). While non-sampling errors like coverage error and measurement error are well-documented, the specific role of interviewer characteristics in driving item nonresponse remains a critical area of investigation.

• Context: The study focuses on the Survey of Health, Ageing and Retirement in Europe (SHARE), Wave 6. Financial variables in this survey exhibit high nonresponse rates (e.g., ~33% missing for bank accounts, ~24% for total household income), raising concerns about selection bias and data quality.
• Research Gap: Previous literature has established that interviewer sociodemographics (age, gender, education) affect response rates. However, this paper investigates interviewer expectations (specifically, their anticipated response rates for income questions) as a predictor of actual respondent behavior.
• Methodological Challenge: The analysis is complicated by missing covariate data arising from two sources:
  1. Nonresponse in the interviewer survey (IWS).
  2. Item nonresponse in the main respondent survey (CAPI).
     The paper seeks to determine the most robust statistical approach for handling these missing covariates when modeling the probability of response.

2. Data and Sample

• Source: SHARE Wave 6 (2015) combined with the SHARE Interviewer Survey (IWS) and the Interviewer Roster (IWR) database.
• Scope: 12 European countries (Austria, Belgium, Estonia, Germany, Greece, Italy, Luxembourg, Poland, Portugal, Slovenia, Spain, Sweden).
• Sample Size: 41,934 eligible respondents linked to 808 interviewers (out of 1,172 total).
• Key Variables:
  • Dependent Variables: Binary indicators for responding to four financial items: Total Household Income, Pensions, Bank Accounts, and Value of Main Residence.
  • Key Independent Variable: A binary indicator for Interviewer Expected Response Rate (ERR). Interviewers were asked to estimate the percentage of respondents who would provide income data. A binary variable was created (1 = expectation > country median, 0 = otherwise) to mitigate measurement error.
  • Controls:
    • Interviewer level: Sociodemographics, workload, health, own income reporting, and behavioral traits (e.g., speaking speed, clarifying questions).
    • Respondent level: Sociodemographics, cognitive ability (numeracy, fluency), health status, and depression.

3. Methodology

The paper compares three distinct statistical approaches to handle missing covariate values in a logistic regression framework modeling the probability of response ($Y=1$ if response, $0$ otherwise):

A. Complete-Case Analysis (CCA)

• Approach: Deletes all observations with missing covariate values.
• Pros: Simple implementation; consistent estimates if data is Missing Completely at Random (MCAR) or Missing at Random (MAR) conditional on observed covariates.
• Cons: Discards significant data (reducing sample size from 41,934 to 22,609), leading to a loss of precision and potential efficiency.

B. Fill-In Approach (Multiple Imputation - MI)

• Approach: Replaces missing values with estimated ones using Fully Conditional Specification (FCS).
• Implementation: Uses two sequential Gibbs samplers (one for respondent variables, one for interviewer variables) to ensure consistency for interviewers across their assigned respondents.
• Imputation Count: 100 imputations ($M=100$) were used, based on the rule $M \geq 100 \times \text{FMI}$ (Fraction of Missing Information), as the FMI for interviewer variables was high (~0.85).
• Pros: Utilizes all available data; accounts for uncertainty via between-imputation variance.
• Cons: Requires the imputation model to be "congenial" (more general) than the analysis model; estimates may be biased if the imputation model is misspecified.

C. Generalized Missing-Indicator (GMI) with Model Averaging (MA)

• Approach: Based on the framework by Dardanoni et al. (2011, 2012, 2015). It creates a "Grand Model" that includes both the observed regressors and auxiliary parameters for missingness indicators.
• Model Averaging: Instead of selecting a single "best" model, it averages estimates across a model space ($2^{H_j}$ possible models) using Bayesian Model Averaging (BMA) weights.
• Weights: Calculated using information criteria:
  • BIC weights: Approximates posterior model probabilities using Schwarz's criterion.
  • AIC weights: Uses Akaike's Information Criterion.
• Goal: To address model uncertainty arising from the choice of how to handle missing data.

4. Key Results

Effect of Interviewer Expectations

• Positive Association: Across almost all specifications and countries, respondents interviewed by interviewers with higher expected response rates (optimistic interviewers) are significantly more likely to provide financial information.
• Magnitude:
  • Optimistic interviewers are associated with up to a 14% reduction in nonresponse for income questions.
  • Up to a 26% reduction in nonresponse for asset questions in certain countries.
• Exceptions: Significant negative effects (where high expectations correlate with lower response) were rare and limited to specific country-item combinations (e.g., main residence value in Greece under specific models).

Comparison of Statistical Methods

• Precision vs. Bias:
  • CCA produced smaller standard errors within the subsample but discarded nearly half the data.
  • MI (Fill-In) produced larger pooled standard errors than CCA because the between-imputation variance (uncertainty from imputation) outweighed the gain from the larger sample size.
• Model Averaging Performance:
  • The BMA estimates did not yield clear gains over simpler approaches.
  • BMA-AIC estimates converged closely to CCA estimates.
  • BMA-BIC estimates converged closely to MI estimates.
  • This suggests that the information criterion weights effectively selected either the complete-case or the imputed model, rather than finding a superior "middle ground."

5. Key Contributions

1. Empirical Evidence on Expectations: Provides robust evidence that interviewer expectations are a significant determinant of item nonresponse, independent of sociodemographic characteristics. This supports the hypothesis that an interviewer's mindset influences respondent cooperation.
2. Methodological Comparison: Offers a rigorous comparison of CCA, MI, and GMI/Model Averaging in the context of complex survey data with dual sources of missingness (interviewer and respondent). It highlights that while MI uses more data, the uncertainty introduced by imputation can inflate standard errors.
3. Policy Implications: Demonstrates that the "missing at random" assumption can be better addressed by including interviewer characteristics, which serve as valid exclusion restrictions for identifying missing data mechanisms.

6. Significance and Limitations

Significance:

• Survey Design: The findings suggest that interviewer selection and training are critical levers for improving data quality in sensitive financial modules. Training interviewers to be more confident and optimistic about respondents' willingness to answer may directly reduce nonresponse.
• Data Correction: Interviewer expectations can be used in post-stratification weighting or imputation models to correct for selection bias in financial data.

Limitations:

• Computational Cost: The authors note that Bayesian Model Averaging with conjugate priors (using MCMC) was computationally prohibitive due to the large number of imputed datasets (100).
• Standard Errors: The study uses standard logit models. Since respondents are nested within interviewers, the lack of a multilevel (hierarchical) structure in the final estimation may lead to underestimated standard errors and potential over-rejection of the null hypothesis.
• Causality: While interviewer characteristics are collected prior to the main survey (mitigating reverse causality), the assignment of interviewers to respondents is not random, leaving room for unobserved confounding.

Conclusion

The paper concludes that interviewer expectations are a powerful predictor of financial item nonresponse. Optimistic interviewers significantly improve response rates. While advanced statistical techniques like Model Averaging offer a theoretically sound way to handle missing data, in this specific application, they did not outperform standard Complete-Case or Multiple Imputation approaches. The primary takeaway is the operational importance of managing interviewer psychology and expectations in survey fieldwork.