Measuring Operational Reliability in Banking Disclosures: Development and Single-Coder Methodological Validation of the ORRDI Index
This paper introduces and validates the Operational Risk Reliability Disclosure Index (ORRDI), an 18-item formative composite that systematically measures the specificity of operational risk disclosures across six dimensions in European, UK, and Swiss banks, demonstrating its calculability and robustness while explicitly limiting its scope to public disclosure quality rather than actual risk capability or inter-coder reliability.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the banking world as a massive, high-stakes video game where the goal isn't just to have a lot of gold (money), but to keep the game running without crashing when a dragon (a cyber-attack, a system failure, or a fraud) shows up. For a long time, players only looked at the "Game Over" screen—the actual losses—to see who was good at the game. But what if you could peek at the players' strategy guides before the game even started?
That's exactly what this paper does. It introduces a new tool called the ORRDI Index (Operational Risk Reliability Disclosure Index). Think of ORRDI not as a scorecard for how well a bank actually survives a disaster, but as a magnifying glass that checks how clearly a bank tells the story of its survival plan in its public reports.
The Great Disclosure Scavenger Hunt
The researchers went on a scavenger hunt through the annual reports and regulatory filings of 14 bank-years (a mix of banks from Europe, the UK, and Switzerland). They didn't just count how many words about "risk" were in the documents; that would be like counting how many times a player says "I'm ready" without checking if they actually have a shield.
Instead, they used a strict 18-item checklist (the ORRDI) to see if the banks were specific. Did they just say, "We have a plan"? That gets a low score. Did they say, "Here is the exact name of the committee, the specific test we ran last Tuesday, and the exact number of minutes it took to recover"? That gets a high score.
They had to find exact quotes and page numbers to give a bank points. If they couldn't find the proof in the document, the score was zero. It was a single coder (one person) doing the work, acting like a super-detailed detective who wrote down every clue.
The Results: Who's Talking the Talk?
After checking 252 specific items across all those reports, the average score came out to 59.39 (on a scale where the highest possible is 100). The scores ranged from a low of 40.74 to a high of 77.78.
Here is the big surprise the paper found:
- The Strongest Area: Banks were great at talking about their "Governance" and "Prevention." They loved to explain who was in charge and what rules they had. It's like a player showing off their character stats and inventory list.
- The Weakest Area: The weakest link was "Continuity of Critical Services." This is the part about what happens when the power goes out or the internet breaks. The average score here was only 42.06. The paper suggests that while banks might have these plans, they are either keeping them secret, not testing them in public reports, or just not writing them down clearly enough for the scavenger hunt to find.
What This Tool Is (and What It Is NOT)
This is the most important part to get right. The paper is very clear about what the ORRDI index does not do.
- It is NOT a crystal ball. A high score does not mean a bank is actually safer or won't lose money. It just means they wrote a very detailed story about how they might be safe. A bank could have a perfect score and still crash if their plan is just a fancy story.
- It is NOT a test of reality. The paper explicitly rules out the idea that this measures "actual capability." It measures "public disclosure." If a bank has a brilliant plan but doesn't write it down, ORRDI gives them a zero.
- It is NOT a guarantee of perfection. The researchers admit they used only one coder. They didn't have a second person to double-check the work to see if they agreed. They built a system of "audit trails" (leaving a paper trail of exactly where they found the quote) to make sure the work could be checked later, but they don't claim the results are perfectly objective in the way a machine would be.
The "Stability" Check
To make sure their game wasn't rigged by a weird rule, the researchers tried changing the rules. What if they ignored the "maybe" answers? What if they removed one of the six categories entirely?
- The results stayed almost the same. The rankings of the banks didn't flip-flop. The correlation between the main score and these "what-if" scenarios was between 0.933 and 0.995. This suggests the tool is robust; it's not just a fluke of one specific rule.
The Bottom Line
The paper concludes that the ORRDI index is a calculable, traceable, and robust way to measure what banks say about their safety plans. It successfully turned messy, long reports into a clear, six-part score.
However, the authors are careful to say this is just the beginning. They haven't proven that a high score stops a bank from failing, and they haven't tested this on thousands of banks yet. They've built a really good flashlight to look at the "strategy guides" of 14 banks, but they haven't yet proven that the guides match the actual gameplay.
In short: The paper gives us a new, very specific ruler to measure how well banks talk about their safety. It tells us that while banks are good at talking about their rules, they are surprisingly quiet about how they keep the lights on when things go wrong. But remember, talking about a plan is not the same as having a working plan.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.