Test Management and Coordination During the Vera C. Rubin Observatory Commissioning and Early Operations Using Zephyr Scale

Imagine the Vera C. Rubin Observatory as a massive, incredibly complex spaceship parked on a mountain in Chile. Its job is to take millions of photos of the universe to create the ultimate map of the cosmos. But before it could start its main mission, the team had to "commission" it—essentially, they had to test every single button, dial, and camera lens to make sure it worked perfectly.

This paper tells the story of how a team of over 50 experts managed hundreds of these tests without losing their minds, using a digital tool called Zephyr Scale (which was originally built for testing software, not giant telescopes).

Here is how they did it, broken down into simple concepts:

1. The Problem: Too Many Cooks, Too Many Recipes

Imagine trying to run a dinner service where 50 different chefs are trying to decide what to cook, when to cook it, and how to cook it, all while the kitchen is constantly changing.

• The Reality: The observatory team had to coordinate daily changes, technical glitches, and new scientific goals. They needed a way to tell the "kitchen staff" (the Observing Specialists) exactly what to do every single night, and they needed a way to record if the food actually tasted right.
• The Solution: They adopted Zephyr Scale. Think of this as a digital recipe book that lives inside a project management app (Jira). It wasn't designed for telescopes, but the team realized it was the perfect tool to organize their chaos.

2. The Three Main Ingredients

The paper describes three key parts of their system, which we can think of as the Recipe, the Daily Menu, and the Cook's Log.

• The Test Case (The Recipe):
  This is a single, reusable instruction set. It's like a recipe for "How to bake a cake." It has a title, a list of steps, and what the result should look like.

  • Analogy: If the test is "Point the telescope at a star and take a picture," the Test Case is the written recipe that says: "Step 1: Turn on the motor. Step 2: Wait 5 seconds. Step 3: Snap photo."
  • Complexity: Some recipes are simple. Others are so complex (involving hundreds of steps) that they are stored as JSON BLOCKs—think of these as pre-packaged, automated meal kits that the computer can just "plug in" and run without a human reading every single word.

• The Test Cycle (The Daily Menu):
  Every day, the team creates a new "Test Cycle." This is the menu for the night. It groups together all the specific recipes (Test Cases) they plan to try that evening.

  • Analogy: Just as a restaurant has a "Tuesday Specials" menu, the observatory has a "Tuesday Night Test Plan."
  • The Twist: They often plan more than they can actually do. Sometimes they add 30 recipes to the menu but only have time to cook 23. The system tracks exactly what was cooked and what was left on the shelf.

• The Test Execution (The Cook's Log):
  When a human actually performs a step from the recipe, they mark it as done in the system. This creates a "Test Execution."

  • Analogy: This is the chef writing in a notebook: "I baked the cake. It rose perfectly. I added extra sugar."
  • Why it matters: Even if the recipe changes later, this log stays frozen in time. It proves exactly what happened on that specific night, which is crucial for scientists looking back years later to understand why a photo turned out a certain way.

3. How the Team Worked Together

The paper describes a very specific rhythm to their day, like a well-choreographed dance:

1. The Idea: Someone has a new idea for a test (e.g., "Let's check if the camera handles heat well").
2. The Chat: They discuss it on Slack (a group chat app).
3. The Formalization: They turn that chat idea into a formal Test Case (a recipe) in Zephyr.
4. The Review: A senior scientist reviews the recipe to make sure it's safe and clear.
5. The Planning: The next morning, a "Test Planner" looks at the daily menu (Test Cycle) and adds the approved recipes to it.
6. The Execution: The Observing Specialists (the people actually at the telescope) follow the menu, checking off steps as they go.
7. The Weekly Check: Once a week, the whole team meets to decide what the "flavor of the week" should be. Do we focus on fixing a broken part? Or do we focus on taking more pictures? They prioritize based on what will teach them the most.

4. The Good, The Bad, and The Ugly

The authors are honest about the tool's strengths and weaknesses.

The Good:

• It creates a permanent record: Unlike a piece of paper or a whiteboard, the system remembers every change. If a recipe was updated, the system knows exactly which version was used on which night.
• It connects the dots: Because it lives inside Jira, it links the test directly to the bug reports or engineering tickets, so everyone knows why a test is happening.

The Bad:

• It's a bit clunky: The tool wasn't built for telescopes, so some features are annoying. For example, it's hard to compare two versions of a recipe side-by-side.
• Stale information: Sometimes, notes written for one night get accidentally copied to the next night's menu, confusing the staff. The team has to manually clean these up every day.
• Link rot: If a recipe changes versions, the link to it breaks, making it hard to find old instructions later.

5. The Big Takeaway

The paper concludes that while Zephyr Scale isn't a perfect, custom-built tool for a giant telescope, it worked because the team was disciplined.

They treated the software like a strict set of rules:

• If a recipe isn't marked "Ready," it doesn't go on the menu.
• If a step isn't checked off, it didn't happen.

The authors admit they expected to stop using this tool once the telescope was "commissioned" and running smoothly. But they realized they still need it. Why? Because a simple checklist on a website doesn't scale. When you have to check off hundreds of steps every night, you need a system that automatically creates a new, clean log for every single night, so you never lose track of what happened.

In short: They took a tool designed for software bugs and used it to run a giant scientific machine, proving that with enough discipline and a good workflow, you can make a "square peg" fit into a "round hole" very successfully.