DMCpy: A powder and single crystal neutron diffraction software for DMC

The paper introduces DMCPy, a modular Python-based software package designed to streamline data reduction, visualization, and analysis for both powder and single-crystal neutron diffraction experiments conducted on the upgraded DMC diffractometer at SINQ.

The Gist

Imagine you are a detective trying to solve a mystery inside a tiny, invisible world made of atoms. To do this, you use a special flashlight made of neutrons (tiny particles with no electric charge) that can pass through solid objects and bounce off the atoms inside.

The DMC is a high-tech "neutron camera" located at a massive research lab in Switzerland. Recently, they upgraded this camera with a giant, super-sensitive 2D detector (think of it as a massive digital billboard instead of a single lens). This new camera is amazing because it can take pictures of two very different things:

1. Powders: Like taking a photo of a bucket of sand to see the average shape of every grain.
2. Single Crystals: Like taking a photo of a single, perfect diamond to see its internal structure in 3D.

However, this new camera is so powerful that it produces too much data. It's like trying to drink from a firehose; the raw data is a chaotic, overwhelming flood of numbers that standard tools can't handle.

Enter DMCpy.

What is DMCpy?

Think of DMCpy as a specialized kitchen for neutron data. Just as a chef needs specific knives and pans to turn raw ingredients into a delicious meal, scientists need specific software to turn raw neutron data into a clear picture of the atomic world.

Here is how DMCpy works, using some everyday analogies:

1. The "Noise Cancelling" Headphones (Normalization)

The giant detector is actually made of 9 smaller panels glued together. Sometimes, one panel is slightly more sensitive than its neighbor, or a few wires are broken. If you just looked at the raw photo, it would look like a patchwork quilt with bright and dark spots that aren't real.

• The Analogy: DMCpy takes a "test photo" of a perfect, boring metal (Vanadium) first. It calculates exactly how "loud" or "quiet" each part of the detector is. Then, it acts like noise-cancelling headphones, adjusting the volume of every pixel so the final image is perfectly even and free of detector glitches.

2. The "Map Translator" (Coordinate Conversion)

When the camera takes a picture, the data is in "Camera Coordinates" (pixels on a screen). But scientists want to see the data in "Crystal Coordinates" (where the atoms actually are in 3D space).

• The Analogy: Imagine you are looking at a map of a city, but the map is rotated, upside down, and stretched. DMCpy is the GPS translator that instantly rotates, shrinks, and aligns the map so that "North" on the screen matches "North" in the crystal. It turns a confusing jumble of pixels into a clear, 3D grid of atoms.

3. The "3D Slicer" (Visualization)

The data is a massive 3D block of information. Scientists often need to slice through it to see what's happening inside.

• The Analogy: Imagine a giant loaf of bread (the data). DMCpy lets you slice the loaf in any direction you want.
  • Viewer3D: This is like a virtual reality window where you can fly through the loaf, looking at different layers of atoms.
  • 2D/1D Cuts: This is like taking a specific slice of bread to look at the texture, or cutting a thin strip to measure the density of a single line.

4. The "Smart Filter" (Masking)

Sometimes, the camera sees things that aren't part of the experiment, like a stray reflection from the sample holder or a broken wire.

• The Analogy: DMCpy acts like a Photoshop eraser tool. It lets the scientist draw a circle around the "bad spots" and tell the software, "Ignore this part." This ensures that the final analysis only looks at the real atoms, not the camera's mistakes.

Why does this matter?

Before DMCpy, analyzing data from this new camera was like trying to build a house with a hammer and a spoon. It was slow, difficult, and prone to errors.

With DMCpy, scientists can:

• Speed up the process: It handles millions of data points automatically.
• See the invisible: It reveals magnetic structures and atomic arrangements that were previously hidden in the noise.
• Be flexible: It works for both the "bucket of sand" (powders) and the "perfect diamond" (single crystals).

The Bottom Line

DMCpy is the essential software that turns the raw, chaotic output of a super-advanced neutron camera into clear, understandable scientific discoveries. It allows researchers to unlock the secrets of materials, from new batteries to superconductors, by making the invisible world of atoms visible and manageable.

Technical Summary

1. Problem Statement

The DMC diffractometer at the SINQ facility (Paul Scherrer Institute, Switzerland) was recently upgraded with a state-of-the-art large-area 2D $^3$He detector. While this upgrade significantly enhanced the instrument's capabilities for both powder and single-crystal neutron diffraction (particularly for magnetic structures), it introduced two major challenges:

• Data Volume and Complexity: The 2D detector generates massive datasets (e.g., a standard scan can exceed 88 million data points), making traditional analysis workflows inefficient or unfeasible on standard hardware.
• Workflow Disparity: Existing software solutions were often designed for either powder or single-crystal instruments, not both. The DMC's unique constant-wavelength operation and large 2D detector require a specialized data reduction workflow that differs significantly from traditional single-crystal instruments. There was a lack of a unified, Python-based tool tailored specifically to the DMC's geometry and data structure.

2. Methodology

The authors developed DMCpy, a modular Python-based software package designed to handle the specific data reduction, visualization, and analysis needs of the DMC diffractometer.

• Architecture & Core Objects:

  • DataFile: Handles the conversion of raw detector pixel coordinates to reciprocal space ($Q$-space), performs normalization, and masks instrumental artifacts.
  • Sample: Converts data into sample-specific reciprocal lattice units (HKL) using the SICS neutron scattering library.
  • DataSet: An advanced list structure managing multiple scans simultaneously.
  • Visualization Tools: Includes Viewer3D for navigating 3D reciprocal space and InteractiveViewer for inspecting raw detector data scan-by-scan.
  • RLUAxes: A plotting tool ensuring distortion-free visualization across different space groups.

• Key Technical Implementations:

  • Coordinate Transformation: Utilizes the Lumsden coordinate system to transform pixel positions to scattering vectors ($\vec{Q} = \vec{k}_i - \vec{k}_f$). It accounts for vertical sample displacements and monochromator angles.
  • UB Matrix Calculation: Converts instrument coordinates to sample coordinates (HKL) using a unitary transformation ($U$) and a unit cell matrix ($B$). It supports both manual peak indexing and an algorithmic approach to suggest peaks.
  • Vanadium Normalization: Implements a gain correction using a Vanadium standard measured for ~2 hours. This creates a normalization table to correct sensitivity discrepancies between the detector's 9 modules and 128x128 wire grids. Invalid pixels (wires/modules) are masked using NaN values.
  • Memory Optimization (Batched Computation): To handle datasets too large for RAM, DMCpy employs "lazy calculation" (Just-In-Time) and batched processing. Algorithms process data in chunks, updating bin grids iteratively rather than loading the entire file into memory at once. This allows for operations like subtraction and integration without requiring massive RAM.
  • Integration Methods:
    • Powder: Supports $|Q|$-integration (physically accurate) and vertical integration (historical method), with options for angular masking to improve resolution.
    • Single Crystal: Features "box integration" for Bragg peaks, 2D cuts in scattering planes, and optimized 1D cuts using bounding boxes to speed up processing.

• User Interface: A separate GUI script, DMCS.py, was created for on-the-fly use during experiments to provide quick data overviews and assist in UB matrix generation.

3. Key Contributions

• Unified Software Suite: DMCpy is the first software specifically designed to seamlessly handle both powder and single-crystal data reduction for the DMC instrument, bridging the gap between distinct experimental modes.
• Scalable Data Processing: The implementation of batched computation and "lazy" evaluation allows researchers to analyze datasets exceeding 88 million points on standard laptops, a feat previously unattainable with standard Python libraries.
• Advanced Visualization: The introduction of Viewer3D and RLUAxes provides researchers with intuitive tools to map reciprocal space, inspect diffuse scattering, and visualize 3D data structures interactively.
• Robust Normalization: The integration of a specific Vanadium normalization workflow ensures high-fidelity data by correcting module-to-module sensitivity variations inherent in the large 2D detector.
• Open Source & Accessibility: Distributed via PyPI and licensed under MPL-2.0, the software is built on standard scientific libraries (NumPy, SciPy, Matplotlib, H5py), ensuring broad compatibility and ease of use.

4. Results

• Validation:
  • Powder Data: The powder reduction capabilities have been validated through Rietveld refinements published in scientific journals and specific calibration studies using NAC ($Na_2Ca_3Al_2F_{14}$). Comparisons showed that $|Q|$-integration produces more symmetrical peaks compared to vertical integration, especially at low and high angles.
  • Single Crystal Data: The software has been successfully used in multiple publications to analyze magnetic ordering vectors and diffuse scattering.
• Performance: The software successfully processes large-scale scans (e.g., 60-degree scans with multiple detector positions) by managing memory constraints through chunked processing.
• User Adoption: The tool is fully utilized in the user program at SINQ. The DMCS.py GUI has proven effective for real-time data curation and initial alignment during experiments.

5. Significance

DMCpy represents a critical enabler for the scientific community using the DMC diffractometer. By addressing the specific data volume and complexity challenges of modern 2D neutron detectors, it:

• Unlocks Instrument Potential: Allows researchers to fully exploit the high flux and resolution of the DMC for probing complex nuclear and magnetic structures in condensed matter.
• Standardizes Workflows: Provides a consistent, reproducible framework for data reduction that minimizes user error and streamlines the path from raw data to scientific insight.
• Future-Proofing: The modular architecture allows for future extensions, such as support for goniometers/four-circle setups and deeper integration with instrument control software, ensuring the tool evolves alongside the instrument.

In summary, DMCpy transforms the raw, high-volume output of the DMC diffractometer into actionable scientific data, significantly enhancing the efficiency and depth of neutron diffraction research.