Governing Agentic AI in FinTech
This paper argues that the primary governance constraint for agentic AI in FinTech is not capability but verifiability, introducing the "Verifiability Gap" and demonstrating through empirical studies that current systems lack the reproducibility and explainability required for defensible delegation of consequential financial decisions.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are the captain of a spaceship, but instead of steering the ship yourself, you've hired a brilliant, super-fast robot crew to handle the navigation. You tell them, "Get us to the red planet," and they zoom off, checking maps, calculating fuel, and firing thrusters without asking you for permission at every turn. This is the world of Agentic AI: artificial intelligence that doesn't just chat or write poems, but actually does things, like approving loans, buying stocks, or managing money. For a long time, scientists and bankers have worried about the "black box" problem—meaning, the robot makes a decision, but no one knows why or how it did it. The usual fix was to just run the robot's brain over and over again to see if it gave the same answer. But here is the twist: what if the robot's brain changes every time you look at it? What if the robot forgets its own steps, or the tools it uses to navigate get swapped out by the company that built it? This paper asks a scary but necessary question: If a robot makes a huge financial decision today, can we prove tomorrow exactly how it happened, or will the evidence have vanished into thin air?
The paper, titled "Governing Agentic AI in FinTech" by Henry Han, dives deep into this mystery. The author argues that the biggest problem isn't whether these AI agents are smart enough to make decisions, but whether we can verify them. He introduces a concept called the Verifiability Gap. Think of it like this: You give a robot a million dollars to invest. It makes a trade. Later, an auditor asks, "Show me the receipts." If the robot can't show you the exact digital trail of how it made that trade, or if the "receipts" it shows you are different from the ones it actually used, there is a gap between what you asked it to do and what you can prove it did. The paper suggests that as we let AI agents take more control, this gap is getting wider, and we might be flying blind.
To figure this out, the researcher ran three different "lab experiments" using financial scenarios like loan approvals and stock trading. Here is what he found, and it's a bit of a plot twist.
First, he tested what happens when the AI's "brain" gets an update. He used a powerful AI model and ran the same 32 financial cases over a period of 185 days. Even though the instructions were identical, the AI's decisions started to drift. In the later updates, the AI became more conservative, changing its mind on 5 out of the 32 cases. But the real kicker? The company that built the AI also took away the "dials" that let users control how the AI thinks. They removed the ability to set a "random seed" (a number that makes the AI's choices repeatable). It's like the robot captain suddenly changed the ship's engine and then locked the door to the control room, saying, "You can't touch the knobs anymore." The study found that even with the tightest controls possible, a local version of the AI could repeat its own actions perfectly (320 out of 320 times), but a commercial, hosted version could not (319 out of 320, and 959 out of 960). The AI was making defensible decisions, but the bank couldn't prove exactly how it did it because the tools to replay the decision were gone.
Second, the paper looked at what happens when you add more robots to the team. Imagine one robot doing the work versus a team of 50 robots passing notes to each other. The researcher found that adding more robots didn't just make the system more complex; it actually changed the rules of the game without anyone noticing. It's like a relay race where the baton gets passed so many times that the runner at the finish line has no idea who started the race. In these multi-agent setups, the final decision (the verdict) might look stable, but the path to get there was a mess. In fact, the study found that as the team got bigger, the AI started making the same decision for every case, just to be safe. It was like a student who, when confused, just writes "I don't know" on every test question. The AI looked super reliable because it was giving the same answer over and over, but it had stopped actually thinking about the differences between the cases. The paper calls this "degenerate outcome reproducibility"—it looks stable, but it's actually broken because it's stopped distinguishing between different situations.
Finally, the researcher tested a simple, transparent credit model (one that isn't a "black box" at all) to see if the problem was just about complex AI. He used two versions of the same model, one old and one new. Even though the new model was almost identical to the old one, it changed the decision for 23 out of 1,937 loan applications. One applicant was denied by the old model but referred for human review by the new one, just because the score moved by a tiny fraction. The scary part? If you try to replay the old decision using the new model, you can't. The new model is perfectly stable on its own, but it can't recreate the history of the old one. It's like if you rewrote a recipe with a tiny change in the amount of salt, and suddenly the cake tastes different, but you can't go back and taste the original cake anymore because you threw away the old recipe.
So, what does this all mean? The paper suggests that we can't just trust AI to do its job and hope for the best. We can't assume that because an AI is smart, it's also accountable. The author proposes a new way of thinking called evidence-contingent delegation. This means we should only let AI agents make big decisions as long as we can keep a perfect, unbroken "evidence bundle" that proves exactly how they did it. If the AI changes its brain, or if the tools it uses change, or if the team of robots gets too big and messy, we have to stop and say, "Wait, we can't prove this anymore." We might need to slow down, keep more detailed logs, or have a human step in to double-check the work. The paper doesn't say AI is bad; it says that for AI to be truly useful in finance, we need to make sure we can always look back and say, "Yes, this is exactly how it happened," even if the robot has moved on to the next task. Without that proof, the robot might be flying the ship, but we're the ones who will be held responsible if it crashes.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.