Prompt-to-prescription: towards generative design of diffraction-limited refractive optics

This paper introduces an end-to-end generative framework that integrates Large Language Models with differentiable ray-tracing to autonomously translate semantic requirements into valid, high-performance diffraction-limited optical designs across diverse applications, thereby democratizing optical engineering.

The Gist

Imagine you want to build a custom camera lens, but you don't know the first thing about glass, curves, or light physics. Traditionally, you'd have to hire a highly paid expert engineer who spends weeks drawing blueprints, doing math, and tweaking designs until they work.

This paper introduces a new "magic tool" that changes the game. It's a system that lets you simply type a request in plain English (like "I need a lens to take close-up pictures of tiny electronic parts"), and it automatically designs a high-performance, scientifically perfect lens for you.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Blank Page" Panic

Designing a lens is like trying to write a symphony without knowing music theory. You have to start from scratch. Even experts struggle with the "blank page problem"—figuring out where to begin. Usually, they rely on years of experience and intuition to pick a starting shape, then spend months tweaking it.

2. The Solution: The "Smart Architect" + The "Physics Engine"

The authors built a two-part team to solve this:

• Part A: The "Smart Architect" (The AI Brain)
  Think of this as a super-intelligent librarian who has read every lens design manual ever written. When you type your request, this AI doesn't just guess; it looks through its library of 1,700 real, working lenses. It finds the ones that are most similar to what you asked for and says, "Okay, for a close-up camera, we usually start with a shape called a 'Double Gauss.' Let's use that as our blueprint."

  • The Analogy: It's like asking a master chef, "I want a spicy pasta dish." The chef doesn't invent a recipe from thin air; they recall their best pasta recipes, pick the one that fits "spicy," and give you a solid starting recipe.

• Part B: The "Physics Engine" (The Digital Workbench)
  Once the AI gives you the rough blueprint, the second part takes over. This is a computer program that simulates how light actually travels through glass. It's like a video game physics engine, but for light.

  • The Analogy: If the AI is the architect drawing the house, this engine is the construction crew that actually builds it, checks if the walls are straight, and fixes any leaks. It tweaks the curves of the glass millions of times per second until the light focuses perfectly.

3. How They Work Together: The "Prompt-to-Prescription" Pipeline

The system works in a seamless loop:

1. You speak: "I need a lens for a smartphone that is tiny but takes sharp photos."
2. The AI translates: It turns your words into numbers (focal length, size, etc.) and picks a "starter shape" from its library.
3. The Physics Engine refines: It runs a high-speed simulation, bending the light virtually to see where the image is blurry. It then automatically adjusts the glass shapes to fix the blur.
4. Result: You get a complete, ready-to-manufacture lens design.

4. What Did They Actually Build? (The Proof)

To prove this isn't just a toy, they tested it on three very different, difficult challenges:

• The "Microscope" (Industrial Inspection): They asked for a lens to look at tiny computer chips. The system designed a lens that could see details as small as a human hair, perfect for factory robots.
• The "Infrared Eye" (Night Vision): They asked for lenses that see heat (infrared) or light we can't see (like the kind used in night-vision goggles). The system figured out which special glass to use (like Germanium) to make these lenses work, even though it had never seen a specific "heat lens" in its training data before.
• The "Smartphone Lens" (The Hard Mode): They asked for a lens for a 200-megapixel phone camera that is incredibly thin. This is the hardest challenge because the lens is so small that the light has to bend in crazy ways. The system struggled at first (the "blueprint" had overlapping parts), but it used a "staged" approach: first, it fixed the physical shape so the light could pass through, and then it tweaked the curves to make the image sharp. It succeeded!

5. Why This Matters

• Democratization: You don't need a PhD in optics to design a lens anymore. If you can describe what you need, the machine can build it.
• Speed: What used to take weeks of human work now happens in minutes.
• Innovation: It can combine ideas in ways humans might not think of, potentially leading to new types of cameras and sensors for AR glasses, medical devices, and space telescopes.

The Catch (What's Still Hard)

The system is amazing, but it's not perfect yet.

• Material Limits: Sometimes it picks glass that is hard to buy or manufacture.
• Complex Shapes: If you ask for a lens with mirrors or weird angles, the system gets confused because it's mostly trained on standard curved glass.
• Color Issues: Sometimes the lens focuses red light perfectly but blue light a little off, requiring a human to do a final polish.

The Bottom Line

This paper presents a "Copilot" for optical engineers. It doesn't replace the engineer; it handles the boring, math-heavy starting phase so the human expert can focus on the creative, high-level problems. It turns the dream of "talking to a machine to build a camera" into a reality.

Technical Summary

1. Problem Statement

The design of high-performance optical systems is currently a bottleneck in the imaging hardware industry due to its reliance on specialized human expertise. Traditional workflows are iterative, manual, and dependent on expert intuition to conceptualize valid optical architectures from functional requirements. While deep learning has improved parameter optimization, existing methods fail to address the fundamental challenge of generating valid optical topologies and initial prescriptions directly from high-level natural language descriptions. Current AI approaches often suffer from "hallucinations" (non-physical designs), lack of physical grounding, or are restricted to optimizing isolated components rather than complete systems.

2. Methodology

The authors propose an end-to-end generative framework that bridges semantic intent and physical realization by coupling Large Language Models (LLMs) with a differentiable ray-tracing engine. The pipeline operates in three distinct stages:

• Semantic Controller (LLM):

  • Architecture: Instead of fine-tuning an LLM (which risks catastrophic forgetting), the system uses a Reasoning-First Retrieval-Augmented Generation (RAG) approach powered by Claude Sonnet 4.5.
  • Process: The LLM interprets user prompts to extract critical constraints (Effective Focal Length, F-number, sensor diagonal, magnification). It then queries a curated library of ~1,700 validated optical designs (from patents and literature) to retrieve the top $N$ most similar "expert demonstrations."
  • Synthesis: Using analogical reasoning, the LLM interpolates between these retrieved examples to generate a physically plausible "Intuitive Design Seed" (initial prescription), effectively solving the "cold start" problem of traditional optimization.

• Physics-Aware Optimization (Differentiable Ray Tracing):

  • Engine: The generated seeds are passed to DiffOptics, an open-source differentiable ray-tracing library built on PyTorch.
  • Optimization: The system employs the Levenberg-Marquardt (LM) algorithm to minimize a multi-objective loss function:
    $$L_{total} = \omega_{RMS}L_{RMS} + \omega_{phys}L_{phys} + \omega_{spec}L_{spec}$$
    • $L_{RMS}$: Minimizes root-mean-square ray intercepts (focus quality).
    • $L_{phys}$: Penalizes non-physical geometries (e.g., negative thickness, overlapping lenses).
    • $L_{spec}$: Enforces adherence to user constraints (focal length, F-number, etc.).
  • Curriculum Strategy: For complex aspheric systems, a staged optimization is used: first stabilizing spherical geometry, then progressively releasing aspheric coefficients (conic $\to$ 4th order $\to$ 8th order) to navigate the non-linear optimization landscape.

• Validation:

  • Final prescriptions are exported to standard ZMX format and rigorously validated using Ansys Zemax OpticStudio to ensure results are not artifacts of the differentiable approximation.

3. Key Contributions

• First End-to-End Prompt-to-Prescription Pipeline: Demonstrates the direct translation of natural language requests into optimized, diffraction-limited optical prescriptions compatible with industry-standard software.
• Hybrid AI Architecture: Introduces a novel "Semantic Controller + Differentiable Physics" paradigm that combines the creative flexibility of generative AI with the deterministic reliability of classical engineering databases.
• Solving the "Blank Page" Problem: Automates the most experience-dependent phase of optical design: the selection of the initial topological starting point.
• Staged Optimization for Aspheres: Develops a curriculum-based learning strategy to successfully optimize high-dimensional aspheric mobile lenses, preventing solver divergence.

4. Key Results

The framework was validated across three distinct optical regimes, demonstrating versatility and high performance:

• Finite-Conjugate Industrial Metrology:

  • Macro Inspection: Synthesized a symmetric Double-Gauss lens for electronic inspection (0.1mm features) from a non-expert prompt. Achieved diffraction-limited performance at the center and sufficient resolution for automated optical inspection (AOI).
  • Telecentric Lens: Designed a double-sided telecentric lens for mechanical metrology. The system correctly inferred the need for telecentricity to eliminate perspective error, achieving near-diffraction-limited performance ($RMS \approx 3.29 \mu m$) with sub-pixel edge detection capabilities.

• Infrared (IR) Objectives:

  • NIR & SWIR: Successfully synthesized telephoto and wide-aperture objectives for Near-Infrared (700–900 nm) and Short-Wave Infrared (0.9–1.7 $\mu m$).
  • LWIR: Designed a thermal imaging lens using Germanium substrates. The system leveraged the material's inherent low dispersion in the LWIR band to achieve fully diffraction-limited performance ($RMS < Airy$ disk) without complex chromatic correction.
  • Note: Polychromatic performance in NIR/SWIR was limited by simplified refractive index models in the optimizer, highlighting a current limitation.

• Complex Aspheric Mobile Lenses:

  • Challenge: Designed a flagship mobile lens (1 Glass + 6 Plastic, F/1.7, 7.5mm TTL) for a 200MP sensor.
  • Outcome: The raw LLM output contained geometric overlaps. The two-phase optimization (Geometric Stabilization $\to$ Aspheric Refinement) resolved physical conflicts and corrected wavefront errors, achieving an on-axis RMS spot radius of $3.52 \mu m$. Chromatic aberration remained the dominant residual error due to material selection constraints.

5. Significance and Future Outlook

• Democratization of Optical Engineering: Lowers the barrier to entry, allowing non-specialists to generate high-quality optical designs from natural language, potentially accelerating innovation in AR/VR, consumer electronics, and scientific instrumentation.
• New Paradigm for Automation: Establishes a workflow where AI acts as an active participant in the engineering loop rather than just a catalog or a parameter tuner.
• Scalability: The framework is modular. Future work aims to expand the dataset to include more diverse patents, integrate full dispersion curves (Sellmeier coefficients) for better polychromatic performance, and extend the physics engine to support reflective elements, coordinate breaks, and diffractive/metasurface optics.

In conclusion, this work validates that the fusion of semantic reasoning (LLMs) and differentiable physics creates a robust path toward autonomous, diffraction-limited optical design.