Correctness, confidence, and context: Framing software assurance in the AI age

The Great Software Puzzle: Why "Perfect" Isn't the Goal

Imagine you are building a giant, invisible castle out of logic and instructions. This is what software engineers do every day. They write code to make computers do things, from ordering pizza to flying airplanes. For a long time, the dream was to build these castles so perfectly that they could never, ever make a mistake. Engineers wanted to prove mathematically that every single brick was in the right place. But here's the catch: the real world is messy. It's full of unspoken rules, cultural habits, and "common sense" that you can't write down in a list of instructions.

Now, imagine a new kind of helper has arrived: Artificial Intelligence (AI). This AI is incredibly smart at guessing what comes next, like a super-fast guesser who has read almost every book ever written. But unlike the old-school engineers who wanted to prove things were right, this AI works on probabilities. It says, "I'm 90% sure this is the right answer," rather than "I know for a fact this is the right answer." This creates a big problem: How do we trust a system that is built on guesses when we need it to be safe? This is the puzzle Mary Shaw is tackling. She wants to change how we think about "correctness" in software, moving away from the impossible dream of perfection and toward a more practical, engineer-like approach of making sure things are "good enough" for the job they need to do.

---

The Paper's Big Idea: From "Perfect" to "Good Enough"

In this keynote presentation for software engineers, Mary Shaw argues that we need to stop trying to prove our software is 100% perfect and start focusing on whether it is fit for its purpose. She suggests that the old way of thinking—trying to write a perfect set of rules to prove a program is correct—is broken, especially now that we are using AI.

Here is the story of her argument, broken down into simple parts:

1. The Shadow Problem: What the AI Can't See

Imagine you are trying to describe a complex object, like a folding paper cup, to a friend who has never seen one. You can only show them the shadows it casts on a wall. From the front, the shadow looks like a square. From the side, it looks like a triangle. From the top, it looks like a circle. If you only had the shadows, you might guess it's a weird block, but you'd miss the fact that it's a cup that can hold water.

Shaw says that AI models are like that friend looking at shadows. They are trained on "digital shadows"—the text, code, and data that exist online. But the real world is full of tacit knowledge: things we know but don't write down. This includes the "vibe" of a workplace, the unspoken rules of a culture, or the gut feeling an experienced mechanic has when they hear a strange noise in an engine. Because this knowledge isn't written down, the AI can't see it. It only sees the shadow, not the real object. When AI tries to build software, it might miss these crucial, invisible details, leading to mistakes that a human expert would have caught immediately.

2. The "Cargo Cult" of Coding

Shaw tells a story about people on islands who saw airplanes landing with food during a war. When the war ended and the planes stopped coming, the locals built bamboo control towers and marched in formation, hoping to summon the planes back. They copied the shadows of the activity (the marching, the towers) but missed the real cause (the logistics and scheduling of the military).

She warns that using AI to write code can sometimes become a "cargo cult." We might ask the AI to write code that looks right (the shadow) without understanding the deep reasons why the code needs to be that way. If we just ask for "vibe coding" (writing code based on a feeling or a simple prompt) without understanding the hidden rules, we might build systems that look good but fall apart when real-world problems happen.

3. The "Credentials" Instead of "Specs"

Traditionally, engineers try to write a perfect "specification"—a giant, rigid rulebook that says exactly what the software must do. Shaw suggests this is impossible because the world changes, and we can't predict everything.

Instead, she proposes we treat software like a person applying for a job. We don't need their entire life story; we just need their credentials.

• A scuba diver shows a card saying, "I can dive to 30 meters."
• A pilot shows a license saying, "I can fly this plane."
• A software system should show a "credential" that says, "I am 90% confident I can handle 1,000 users," or "I have been tested to be safe for this specific task."

These credentials aren't perfect proofs; they are honest statements about how confident we are, where that confidence came from, and what the limits are. It's a way of saying, "We checked this, and here is how sure we are."

4. The Engineer's Choice: Satisficing

The paper suggests that we should stop trying to be perfect and start satisficing. This is a fancy word for "finding a solution that is good enough."

• If you are building a game, you might not need 100% perfection. A few bugs are okay if the game is fun and cheap to make.
• If you are building a medical device, you need a much higher level of confidence.

Shaw argues that engineers should act like real engineers: they should look at the cost, the risk, and the purpose, and then choose the right mix of tools to get the job done. Sometimes, it's cheaper to fix a problem after it happens than to try to prevent it from ever happening. The goal isn't to eliminate all errors (which is impossible); the goal is to manage the risk so the system is safe and useful for its specific job.

What This Means for the Future

Mary Shaw isn't saying AI is bad. She is saying that we can't treat AI like a magic wand that solves everything. We need to be smarter about how we use it. We need to accept that AI works on guesses (statistics) and that it can't see the invisible "shadows" of human culture and experience.

Her main message is a call to action for the software community: Let's stop pretending we can prove everything is perfect. Instead, let's be honest engineers. Let's build systems where we know exactly what we are confident about, what we aren't, and why. By doing this, we can use AI to help us build better software, without falling into the trap of thinking we have solved the unsolvable.

In short, the paper suggests that the future of software isn't about being "correct" in a math textbook sense. It's about being fit for purpose, knowing our limits, and making smart, cost-effective choices to keep our digital world running safely.