Artificial intelligence tools are now a routine part of today’s software development, promising faster output and reduced workloads. But new research from Anthropic suggests that when developers rely on AI while learning new skills, those productivity gains may come at the expense of understanding.
In a randomized controlled study involving 52 mostly junior software engineers, researchers found that participants who used AI assistance while learning a new Python library performed significantly worse on a follow-up test than those who worked without AI. The findings raise questions about how aggressively AI tools should be integrated into learning and early-career dev workflows.
The experiment focused on Trio, a Python library for asynchronous programming that none of the participants had used before. All participants had at least a year of regular Python experience and prior exposure to AI coding tools.
They were split into two groups: one with access to an AI assistant during the task, and a control group limited to documentation and web search.
Both groups were asked to complete two coding tasks as quickly as possible, followed by a quiz designed to assess understanding. The test measured debugging ability, code reading, code writing, and conceptual understanding. No AI assistance was allowed during the follow-up quiz.
AI Help Not Good for Learning Skills
Developers who used AI scored an average of 50 percent on the quiz, compared with 67 percent for those who coded without AI assistance, a gap equivalent to nearly two letter grades. The largest difference appeared in debugging questions.
AI users completed the tasks slightly faster, by about two minutes on average, but the difference was not statistically significant. Researchers attribute the limited time savings to the effort required to formulate prompts and interpret responses. Remarkably, some participants spent nearly a third of their allotted time interacting with the AI assistant.
The study suggests that productivity gains from AI are not guaranteed, especially when developers are learning something new. We know that AI can dramatically speed up work, though those benefits are observed in tasks where users already have the relevant skills. Yet this study examined skill learning, not mastery.
A Learning Aid, Not a Substitute
How developers used AI turned out to be as important as whether they used it at all. Participants who used AI for most or all of their coding tended to finish fastest but retained little understanding. Others gradually leaned more heavily on AI as tasks became difficult, with similarly poor learning outcomes.
In contrast, developers who treated AI as a learning aid rather than a substitute performed far better. Those who asked questions or wrote code and then checked how it worked scored at or above the control group’s average. These participants often encountered more errors, but resolving those mistakes appeared to strengthen their understanding.
Working Through Problems is Essential for Learning
Researchers argue that this kind of cognitive effort, like getting stuck and working through errors, is essential for developing durable skills. AI assistance can remove friction from the process, but removing friction also reduces important learning moments.
The findings carry implications beyond individual productivity. As companies increase the share of AI-written code, human oversight becomes more important, not less. Debugging, code review, and conceptual understanding are critical for catching errors in high-stakes systems. If those skills weaken during training, organizations, and developers, will likely face long-term risks.
The researchers caution that their study is limited in scope, measuring comprehension shortly after a single learning session. Whether these effects persist over time remains an open question. Still, the results suggest that AI-enhanced productivity is not a shortcut to expertise.
