Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis

This randomized study of 164 law students demonstrates that targeted training significantly boosts both the adoption and productive performance of generative AI in legal analysis, whereas untrained access fails to improve outcomes and may even reduce answer quality.

The Gist

Imagine you are a chef in a busy kitchen, and someone hands you a brand-new, super-smart robot assistant. This robot can chop vegetables, mix sauces, and even suggest recipes faster than you can blink. Sounds like a dream, right?

But here's the catch: The robot is great at following instructions, but it's terrible at guessing what you actually want. If you just say, "Make a salad," it might give you a bowl of raw lettuce and a bottle of motor oil because it doesn't know the difference between "dressing" and "engine lubricant."

This is exactly what happened in a recent study by researchers at the University of Hong Kong. They wanted to see if giving professionals (in this case, law students) a powerful AI tool would automatically make them better at their jobs, or if they needed a user manual to get the most out of it.

Here is the story of their experiment, broken down simply:

The Setup: Three Kitchens

The researchers gathered 164 law students and gave them a tough legal exam (a "spot the issue" test). They split the students into three different groups:

1. The "No Robot" Group: These students had to do the exam the old-fashioned way, using only traditional legal databases. No AI allowed.
2. The "Just Give Me the Robot" Group: These students were allowed to use a powerful AI (called DeepSeek), but they got zero instructions. They were just told, "Here's a robot, go ahead and use it if you want."
3. The "Robot with a Manual" Group: These students also had access to the AI, but before the exam, they watched a short 10-minute video. The video taught them how to talk to the robot. It explained things like: "Don't just ask for the answer; ask the robot for ideas, then check its work yourself because it sometimes lies."

The Results: What Happened?

1. The "Just Give Me the Robot" Group (No Training)
This group was a bit of a disaster.

• Did they use the robot? Only about 26% of them did. Most were too scared or didn't know how to start.
• Did they do better? No. In fact, they scored slightly lower than the group with no robot at all.
• Why? It's like giving a chef a fancy robot but no instructions. They spent time fumbling with it, asking it the wrong questions, and getting confused answers. They ended up writing shorter, less detailed answers because they wasted time fighting the technology.

2. The "Robot with a Manual" Group (Training)
This group was the clear winner.

• Did they use the robot? Yes! The usage rate jumped to 41%. The training gave them the confidence to try it.
• Did they do better? Yes, significantly. They scored about one-third of a letter grade higher than the group with just the robot.
• Why? The training taught them how to be the "pilot" and the AI the "co-pilot." They learned to ask the right questions, break the problem down, and double-check the robot's work. They didn't just let the robot drive; they used it to help them drive faster and safer.

The Big Lesson: It's Not About the Tool, It's About the Driver

The study found two main things:

1. Training unlocks the door: Without training, smart people are afraid to use powerful new tools because they worry about making mistakes (like the robot giving fake legal facts). The training removed that fear.
2. Training changes how you use it: The training didn't just make the students use the AI more; it made them use it smarter. They learned that the AI is a brainstorming partner, not an oracle that knows everything.

The "Principal Stratification" (The Fancy Math Part)

The researchers used some complex math to figure out why the trained group did better. They asked: "Did the training make the AI work better for everyone, or did it just convince more people to try it?"

The answer is a bit of both, but mostly the latter. The training convinced the "skeptics" (people who wouldn't have used the AI otherwise) to give it a try. These new users did great! The people who were already going to use the AI anyway also did a little better, but the biggest boost came from getting more people to jump on board.

The Takeaway for Everyone

Whether you are a lawyer, a writer, a doctor, or a student, this study has a simple message:

Giving people a powerful new tool isn't enough. If you just hand someone a Ferrari without teaching them how to drive, they might crash it. But if you give them a quick driving lesson, they can speed past everyone else.

In the world of AI, the "training" is just as important as the "technology." To get the most out of these smart machines, we need to invest time in teaching people how to collaborate with them, not just how to turn them on.

Technical Summary

Here is a detailed technical summary of the paper "Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis" by Benjamin M. Chen and Bao Hong.

1. Problem Statement

The paper addresses a critical gap in the deployment of Generative Artificial Intelligence (GenAI) in high-stakes, knowledge-intensive professions like law. While GenAI (specifically Large Language Models or LLMs) promises to augment human productivity, its adoption in professional settings remains uneven.

• The Tension: Legal professionals face a trade-off between the efficiency gains of GenAI and the high risks of "hallucinations" (fabricated facts/citations) and errors, which can lead to severe professional liability.
• The Research Question: Can targeted user training unlock the productive potential of GenAI by increasing its adoption rate (the scope of use) and/or improving the effectiveness of its use (the quality of output conditional on adoption)?
• Context: Previous studies often show GenAI benefits lower-skilled workers or simple tasks, but evidence on its impact on complex, judgment-intensive tasks (like legal issue-spotting) without training is mixed. The authors hypothesize that without training, high-skill professionals may abstain from using GenAI due to perceived risks, while those who do use it without guidance may use it ineffectively.

2. Methodology

The authors conducted a randomized controlled trial (RCT) involving 164 law students (LLB and JD) at the University of Hong Kong.

• Task: Participants completed a mock "issue-spotting" examination in contract law. This task required identifying legal issues in a complex fact pattern and analyzing them to reach a conclusion. It tests black-letter law knowledge, sensitivity to facts, and legal judgment.
• Experimental Design: Participants were randomly assigned to one of three groups:
  1. Control Group (No GenAI): Allowed access only to a traditional legal database (Westlaw).
  2. Treatment Group A (Untrained Access): Allowed access to an LLM (DeepSeek) but received no guidance on how to use it.
  3. Treatment Group B (Trained Access): Allowed access to DeepSeek and received a 9.5-minute training intervention.
     • Training Content: A video and a quiz teaching specific strategies: breaking tasks into steps (chain-of-thought), prompting for multiple perspectives, providing positive/negative feedback, and critically verifying all LLM outputs (facts, rules, cases). It emphasized that LLMs are probabilistic and prone to hallucinations.
• Data Collection:
  • Primary Outcomes: Self-reported GenAI usage (binary) and examination performance (Grade Point and Number of Issues Missed).
  • Secondary Outcomes: Answer length (word count), readability (Flesch-Kincaid score), and participant attitudes.
  • Sample: 164 valid submissions (49 Control, 57 Untrained, 58 Trained).

3. Key Contributions

This study makes several distinct contributions to the literature on AI and labor economics:

1. Causal Evidence on Training: It is one of the first studies to isolate the marginal effect of user training on GenAI productivity in a professional context, distinguishing between "access" and "competence."
2. Decomposition of Effects: The authors move beyond simple average treatment effects. They use Principal Stratification to decompose the overall impact of training into two channels:
   • Adoption Effect: Expanding the scope of use (inducing non-users to adopt).
   • Effectiveness Effect: Improving the quality of use among those who would have adopted anyway.
3. High-Stakes Context: Unlike many studies focusing on routine tasks, this experiment focuses on "boundary applications" (complex legal reasoning) where error costs are high, providing evidence relevant to professional practice.
4. Theoretical Framework: The paper proposes a stylized model where training lowers the "cost of adoption" (risk premium/psychological barriers) for high-ability individuals who would otherwise abstain, thereby shifting the adoption curve.

4. Results

A. Adoption Rates

• Significant Increase: Training significantly increased the likelihood of using GenAI.
  • Untrained Access: 26.3% adoption rate.
  • Trained Access: 41.4% adoption rate.
  • Difference: +15.06 percentage points ($p = 0.044$).
• Interpretation: Training successfully overcame the hesitation of high-skill users who were wary of using unverified AI tools.

B. Examination Performance

• Untrained Access vs. Control: Access to GenAI without training did not improve performance. In fact, the untrained group scored slightly lower (though not statistically significant) and wrote shorter answers than the control group. This suggests untrained users may have wasted time interacting unfruitfully with the LLM or relied on it poorly.
• Trained Access vs. Untrained Access: The training intervention significantly improved performance.
  • Grade Point: Trained users scored 0.27 grade points higher than untrained users ($p = 0.027$), equivalent to roughly one-third of a letter grade.
  • Issues Missed: Trained users missed fewer issues than the control group, though this specific metric was not statistically significant.
• Answer Characteristics: Contrary to the expectation that AI would make answers longer, untrained users wrote significantly shorter answers than the control group. Trained users' answer lengths were statistically indistinguishable from the control group, suggesting they used the tool efficiently rather than just generating bulk text.

C. Mechanism Decomposition (Principal Stratification)

Using principal stratification, the authors attempted to determine if the gains came from more people using the tool (Adoption) or existing users using it better (Effectiveness).

• Findings: Point estimates suggest the Adoption Effect is the dominant driver. The training primarily induced high-ability individuals (who would otherwise abstain) to use the tool, and their usage was productive.
• Uncertainty: Confidence intervals are wide, meaning the data cannot definitively rule out that training also improved the effectiveness of "always users" (those who would use AI regardless of training). However, the pattern suggests the extensive margin (adoption) is the primary channel.

5. Significance and Implications

• For Legal Education and Practice: The findings argue against a "hands-off" approach to AI integration. Simply providing access to GenAI tools is insufficient and may even be counterproductive (reducing output length and quality). Complementary investment in human capital (training) is critical. Legal education must incorporate "AI literacy" focused on prompt engineering, verification, and risk management.
• For Labor Economics: The study challenges the notion that AI simply substitutes for low-skill workers or augments high-skill workers automatically. It suggests that training acts as a gatekeeper: without it, high-skill professionals may abstain from using AI in complex tasks due to risk aversion. With training, AI can augment high-skill judgment, potentially altering the traditional skill-biased technological change models.
• For Diffusion of General-Purpose Technologies: The results highlight that the productivity gains of GenAI are not technologically predetermined but depend on task-specific learning. The "jagged frontier" of AI capability requires tailored training protocols for specific domains (e.g., legal issue-spotting) to realize benefits.
• Policy Recommendation: Organizations should not view AI adoption as a binary choice (buy tool vs. don't). Instead, they must view Tool + Training as a single unit of investment. Failure to train staff may lead to low adoption rates among the most capable workers and poor performance among those who do use the tools.

Conclusion

The paper concludes that while GenAI holds transformative potential for the legal profession, its current reliability issues create a barrier to adoption. Targeted, brief training interventions can effectively lower this barrier, inducing high-skill professionals to adopt the technology and utilize it in a way that measurably improves the quality of their work. The productivity gains of GenAI are therefore contingent on the complementary investment in user training.