Source: Fortune
Summary
A study led by David Autor, a MIT economist, found that AI improved patent drafting for lawyers, but only experienced professionals saw long-term skill gains. Junior lawyers did not show improvement when tested without AI. The research, conducted with Google, suggests AI may act as a performance equalizer but not a skill equalizer. Autor warned that relying too much on AI could hinder the development of foundational expertise, especially for early-career workers. The study, published as a working paper, involved 133 lawyers and was funded by Google.
Our Reading
The numbers tell one story.
AI boosted drafting quality across the board, but only experienced lawyers improved their independent judgment.
Juniors showed no average gain when tested without the tool.
The study highlights a divide between skill development and performance.
AI may help, but it doesn’t replace the need for deep, unassisted practice.
Author: Evan Null
From the China shock to AI
David Autor, head of MIT’s economics department, is known for his work on the China shock, which showed how Chinese import competition hurt American manufacturing. He now warns that AI could have a different impact, boosting productivity rather than causing mass job loss. His latest research, published with Google, suggests AI may not be a skill equalizer for all workers.
The study involved 133 patent lawyers at U.S. firms with Google. Researchers tested how AI affected their drafting skills and independent judgment. While AI improved quality, only experienced lawyers saw long-term skill gains. Junior lawyers did not.
Google funded the study, and its researchers included Google employees. The paper, not peer-reviewed, found that AI helped seniors more than juniors. It also warned that AI could create new jobs while eliminating others, but workers in losing roles may not benefit.
Autor said the danger is not mass unemployment, but the mismatch between lost jobs and new opportunities. He urged workers to avoid the “illusion of competence” and to practice without AI to build real skills.
The findings are limited by a small sample and a short study period. They also don’t prove that AI prevents juniors from becoming experts over time. Autor stressed that human expertise, built through practice, remains essential.
Better drafts, uneven learning
The study tested how AI affected patent drafting. Lawyers used a custom AI tool for 90 days. Their work was graded on enforceability, accuracy, and clarity. AI users improved by 0.38 standard deviations, an 11-percentile-point gain. But the real test came when they had to mark up a flawed patent without AI.
Senior lawyers who used AI outperformed their peers by 0.45 standard deviations. Junior lawyers showed no average gain. The paper noted that only 91 of 133 lawyers finished the final test, and some may have used AI illegally. Even after excluding those, the senior advantage remained, though weaker.
The study found that AI helped some junior lawyers but not all. Their scores split: more poor scores, fewer mediocre ones, more good ones, and no top scores. The authors called this a “springboard” for some and a “cushion” for others.
Juniors liked the tool and reported higher task satisfaction. But they didn’t improve their unassisted work. Autor warned that relying on AI could create an “illusion of competence” without real skill development.
The paper suggests that AI can help, but only if used as a critic, not a crutch. It also highlights the need for guided learning and mentorship to ensure juniors develop real expertise.
The ‘illusion of competence’
The study found that junior lawyers using AI scored more poorly when tested without the tool. They had more poor scores and fewer mediocre ones, but no improvement at the top. The authors called this a “springboard” for some and a “cushion” for others.
Juniors tended to polish introductory text before addressing main claims. Some noticed flaws but left them uncorrected. This pattern, called a “baseline junior deficit,” persisted even after three months of AI use.
Juniors reported higher task satisfaction with AI, as it helped them avoid the “blank page problem.” They felt they could step into a reviewer role more easily. But Autor warned that this could create a false sense of skill without real mastery.
The paper suggests that AI can help, but only if used as a critic, not a crutch. It also highlights the need for guided learning and mentorship to ensure juniors develop real expertise.
Autor said that young professionals should avoid the “illusion of competence” and practice without AI to build real skills. He warned that relying too much on AI could hinder long-term development.
Why the older lawyers gained
Experienced lawyers used AI as a “logic auditor,” not a finished product. They spotted flaws in their own work and improved their strategic thinking. This allowed them to focus on the bigger picture, like legal doctrine and scope, rather than just drafting.
On the unassisted test, senior lawyers skipped low-stakes prose, rebuilt claims from scratch, and made edits tied to legal doctrine. They were more strategic and focused on the “forest, not the trees.”
Autor said AI helped seniors by freeing them from baseline drafting, allowing them to focus on higher-level tasks. But he warned that juniors need to practice without AI to build real expertise.
The paper suggests that AI can help, but only if used as a critic, not a crutch. It also highlights the need for guided learning and mentorship to ensure juniors develop real expertise.
Autor emphasized that human expertise, built through practice, remains essential. He warned that relying too much on AI could hinder long-term development and weaken the apprenticeship pipeline.
Don’t cut the apprenticeship
Autor warned that firms may be tempted to hire fewer juniors if AI improves output. But he called this short-sighted. Without formative practice, firms risk severing the apprenticeship pipeline that produces future leaders.
He suggested firms decide which tasks can be done with AI and which require unassisted mastery. They could also pair AI with regular skill checks and hold partners accountable for mentoring juniors through the tool’s mistakes.
The paper lists other limits, including a small sample and a short study period. It also notes that newer AI models and wider use could change the results. The study doesn’t prove that AI prevents juniors from becoming experts over time.
Autor tied the issue back to his China work, emphasizing that technological advantage comes from human know-how, not just equipment. He warned that relying on AI to replace expertise could lead to disappointment.
Human and machine intelligence, he said, will be complements for the long term. But real expertise still requires slow, laborious mastery, not just tool use.
