Technical Summary: Evaluating Nova 2.0 Lite under Amazon's Frontier Model Safety Framework

Problem Statement

The rapid advancement of frontier-scale multimodal models, such as Amazon Nova 2.0 Lite, introduces significant safety obligations. While these models offer enhanced reasoning capabilities over extended contexts (up to 1M tokens) and multimodal inputs, they also lower the barrier for misuse in high-consequence domains. The core problem addressed is determining whether Nova 2.0 Lite crosses the "Critical Capability Threshold" defined by Amazon's Frontier Model Safety Framework (FMSF). This threshold is breached if a model can provide expert-level, interactive instruction that enables a non-expert to reliably produce and deploy harmful capabilities beyond what is achievable with public tools. The evaluation focuses on three specific high-risk domains:

1. Chemical, Biological, Radiological, and Nuclear (CBRN) Weapons Proliferation.
2. Offensive Cyber Operations.
3. Automated AI R&D (the ability to autonomously design, run, and assess frontier machine learning experiments).

Methodology

The evaluation employs a dual-methodology approach combining reproducible automated benchmarks with human-centric risk evaluations. To ensure rigor and independence, external auditors reviewed test sets, scoring rubrics, and safety-judge notes.

1. Automated Benchmarks

• CBRN Domain:
  • Knowledge Recall: Utilized the Weapons of Mass Destruction Proxy (WMDP) benchmarks for Biology (WMDP-BIO) and Chemistry (WMDP-CHEM) to test hazardous knowledge.
  • Procedural Understanding: Evaluated via PROTOCOLQA (troubleshooting wet-lab procedures) and BIOLP-BENCH (long-answer plan generation for real-world protocols).
  • Expert-Level Troubleshooting: Used the Virology Capabilities Test (VCT), a difficult benchmark where even expert virologists score low.
  • Multimodal Safety: Tested via a custom "Chem-Vision" dataset where the model must identify chemicals from structural images and assess hazards, balancing accuracy with safety refusals.
• Offensive Cyber Operations:
  • Theoretical Knowledge: Assessed using CyberMetric, SECURE-CWET, and CTIBench.
  • Practical Application: Utilized the CyBench framework, a Capture-The-Flag (CTF) environment with 40 challenges across vulnerability identification, web exploitation, cryptography, and reverse engineering.
• Automated AI R&D:
  • Evaluated using RE-Bench, a suite of open-ended ML development tasks where an agent must autonomously modify codebases and iterate within sandboxed containers to solve engineering challenges (e.g., embedding repair, training pipeline optimization).

2. Human-Centric Risk Evaluation

• CBRN: Conducted a large-scale "uplift study" with Nemesys Insights involving nearly 800 participants in red-teaming exercises to assess if the model could provide material uplift for weaponization.
• Offensive Cyber: Performed extensive manual testing in Hack The Box (HTB) environments. This included fully autonomous modes, scenario-based testing with context supplements, and human-in-the-loop (HITL) interactions to test capabilities in realistic offensive scenarios, including malware development and lateral movement.
• Automated AI R&D: External auditors from Model Evaluation and Threat Research (METR) reviewed transcripts from RE-Bench runs and internal scores to verify if the model could replace human researchers in high-risk workflows.

Key Results

CBRN Weapons Proliferation

• Knowledge & Procedure: Nova 2.0 Lite demonstrated measurable capability gains over predecessors (Nova 1.0 Pro and Nova Premier). It achieved higher scores on WMDP-CHEM (0.71) and PROTOCOLQA (0.49) compared to previous versions.
• Multimodal Safety: The model balanced chemical image recognition accuracy with safety refusals effectively, showing a stronger safety posture on dangerous queries than Nova Premier.
• Threshold Assessment: Despite improved factual accuracy, the independent uplift study concluded the model remains below the CBRN critical threshold. While a performance uplift was noted in the radiological domain regarding attack planning, chemical, biological, and nuclear domains remained below the risk threshold.

Offensive Cyber Operations

• Theoretical Knowledge: The model achieved >85% accuracy on general knowledge benchmarks, performing on par with or slightly surpassing previous baselines.
• Practical Application (CyBench): Nova 2.0 Lite showed a 7.5% performance uplift over predecessors, primarily in "easy" and "very easy" challenges.
• Qualitative Analysis: Detailed trace analysis revealed that while the model excels at theoretical analysis, static code review, and educational scenarios (e.g., identifying RSA flaws, analyzing packet structures), it struggles with active exploitation. Successes often required structured human guidance. In advanced tasks like binary exploitation and privilege escalation, the model identified attack vectors but failed to operationalize them into working exploits.
• Threshold Assessment: Human-centric testing in HTB environments confirmed the model did not cross the critical threshold. It did not provide material capability uplift enabling moderately skilled actors to discover novel high-value vulnerabilities or bypass modern defenses autonomously.

Automated AI R&D

• Performance: The model demonstrated strong foundational skills in code inspection, debugging, and iterative problem-solving (e.g., correctly identifying state dictionary mismatches in embedding repair).
• Limitations: The model often treated complex tasks as pure architecture search problems, overlooking necessary training steps or failing to debug dimensional mismatches without external intervention.
• Threshold Assessment: Both internal analysis and the independent METR assessment concluded that Nova 2.0 Lite does not cross the Automated AI R&D critical threshold. It cannot autonomously drive the high-risk research workflows required to accelerate the development of enhanced CBRN weapons or offensive cyber methods.

Significance and Claims

The paper presents Nova 2.0 Lite as a system-card-style disclosure, aiming to provide a transparent template for future cross-organizational safety audits of frontier models. The authors claim that:

1. Safety Compliance: Nova 2.0 Lite meets the safety criteria for public release under the FMSF. Despite gaining substantial knowledge and theoretical proficiency, it does not enable material uplift for weaponization or offensive operations.
2. Mitigation Effectiveness: The model's safety is maintained through a layered mitigation stack, including policy-tuned refusal behaviors, dynamic content filters, and continuous safeguard monitoring.
3. Iterative Safety: The evaluation process highlights the importance of independent auditing and responsive mitigation. When a specific risk (radiological uplift) was identified, additional safeguards were deployed proactively.
4. Transparency: By formalizing evaluation protocols and disclosing empirical risk signals, the report seeks to establish a standard for evaluating the safety of future models in the Nova family and beyond.

The authors emphasize that while the model shows capability gains, these gains remain within safe operational boundaries, supporting the responsible deployment of Nova 2.0 Lite without posing severe risks to public safety.