Assessing Science for Citizenship on High School Science Tests: An Exploratory Study in Seven States

Technical Summary: Assessing Science for Citizenship on High School Science Tests

Problem Statement
The study addresses a critical misalignment within the U.S. science education system regarding the goals of K-12 science learning. A Framework for K-12 Science Education (the basis for the Next Generation Science Standards, or NGSS) establishes five "overarching goals," three of which focus on "learning science for citizenship." These goals require students to possess sufficient knowledge to engage in public discussions, act as careful consumers of scientific information in daily life, and continue learning about science outside school. Despite these goals being embedded in national and state policy documents, there is a lack of empirical evidence regarding whether high-stakes state assessments actually measure these competencies. The authors posit that if assessments do not reflect these goals, the theory of systemic school reform—which relies on the alignment of goals, standards, curriculum, instruction, and assessment—is failing in the domain of science education.

Methodology
This exploratory study utilized a content analysis approach to examine high school science test items from seven states: California, Florida, Illinois, Massachusetts, New York, North Carolina, and Texas. These states collectively enroll approximately 44% of all public school students in the United States. The sample included a mix of full-length released assessments (Massachusetts, New York, North Carolina, Texas) and full-length practice assessments or sample items (California, Florida, Illinois), totaling 365 test items.

The researchers developed a coding scheme based on the five overarching goals of A Framework for K-12 Science Education, excluding Goal 1 (appreciation of beauty/wonder) as it is not assessed. The coding categories were:

• Goal 2: Public policy issues (e.g., laws, regulations).
• Goal 3: Application of science in personal life.
• Goal 4: Competencies to evaluate scientific claims outside school (media literacy).
• Goal 5: Preparation for college and careers (disciplinary core ideas).
• Emergent Categories: "Data visualization/interpretation" (D) and "Reading text" (R), where items required interpreting graphs or reading comprehension without specific prior science knowledge.

Three experienced science educators independently coded the items. Initial inter-rater agreement averaged 81.2%, which was refined to 99% through consensus discussions. The study focused specifically on biology tests where available, as this is the most widely taken high school science course, though some states included earth and physical sciences.

Key Results
The analysis revealed a significant under-representation of "science for citizenship" goals in state assessments:

• Overall Prevalence: Only 2.7% of the 365 items (10 items) were coded as primarily focusing on learning science for citizenship (Goals 2, 3, or 4).
• State Variation: Three states (California, New York, and North Carolina) included a small number of citizenship-focused items (ranging from 1 to 5 items per test). The other four states (Florida, Illinois, Massachusetts, and Texas) included zero items primarily focused on learning science for citizenship.
• Goal Specifics:
  • Goal 2 (Public Policy): 7 items (1.9% of total).
  • Goal 3 (Personal Life): 3 items (0.8% of total).
  • Goal 4 (Evaluating Claims): 0 items.
• Dominance of Goal 5: 82.5% of all items were coded as Goal 5 (college and career readiness), focusing on specialized disciplinary knowledge.
• Emergent Categories: 13.7% of items fell into the "Data" category, and 0.3% into the "Reading" category. In some states, such as Illinois, a majority of items (60%) fell into the "Data" category, requiring interpretation of provided information rather than recall of scientific facts.

Key Contributions
The study provides the first empirical evidence quantifying the extent to which high-stakes state science assessments align with the "citizenship" goals of the NGSS and A Framework for K-12 Science Education. By analyzing a sample representing nearly half of the U.S. student population, the authors demonstrate that the systemic alignment between stated educational goals and actual assessment practices is broken. The research highlights that while policy documents emphasize civic engagement and media literacy, the operational reality of testing prioritizes disciplinary knowledge for college and career readiness.

Significance and Claims
The authors claim that their findings validate skepticism among educators regarding the alignment of state assessments with Framework goals. They argue that the lack of alignment suggests that the "theory of action" for systemic school reform is not functioning as intended in science education. Specifically, the paper claims:

1. Assessment Drives Instruction: Because teachers and policymakers pay attention to what is tested, the absence of citizenship-focused items likely leads to the under-emphasis of these competencies in classrooms.
2. Standard Misalignment: The misalignment is not merely a testing issue but stems from the standards themselves. The authors note that NGSS and most state standards heavily emphasize "Vision I" (science as scientists see it) and under-emphasize "Vision II" (science-technology-society links), despite the Framework's broader vision.
3. Policy Implications: The study suggests that for the goal of learning science for citizenship to be realized, state assessments must be adjusted to include items that evaluate public policy engagement, personal application, and the ability to evaluate scientific claims. The authors note that such items are feasible, citing examples from PISA and specific items found in California and New York tests.

The paper concludes modestly, stating that while preparation for college and careers is important, the current over-emphasis on this goal at the expense of citizenship goals creates an imbalance that fails to prepare students for the complex scientific and societal decisions they will face as citizens.