🔍 Read the full analysis: ChatGPT Raised Consultants’ Data Science Performance, But The Gains Faded on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on coding, statistics and prediction tasks than a control group using web resources. After ChatGPT was removed, researchers found no evidence that the users answered technical questions better than the control group.
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT substantially improved their performance on data science tasks, but the advantage was not evident when the tool was removed. The study, published in Nature Human Behaviour on October 26, 2026, suggests that using generative AI can help workers complete tasks beyond their usual skills without necessarily teaching them to perform those tasks independently.
In the trial, consultants who received 15 to 20 minutes of ChatGPT training used GPT-4 to work on technical tasks. A control group received training in using Google and websites including Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: Python coding, statistics, and prediction. Of 986 consultants allocated to the study, 487 submitted work on both assigned tasks and were included in the main analysis, according to the source account of the paper.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The tasks were designed with BCG data scientists and OpenAI’s Economic Impacts Research team so that ChatGPT could offer useful guidance but could not independently answer every question correctly. A separate group of 44 BCG data scientists completed the tasks without AI to provide an expert benchmark.
The AI-assisted consultants still scored below that expert benchmark on average across all three tasks. After the work, participants answered five technical questions on related topics without ChatGPT. The authors report no evidence that the ChatGPT group did better than the control group on that follow-up test. They also report that the ChatGPT group was more confident that ChatGPT could solve problems it could not, while describing that finding as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Better Task Scores, Not Independent Skill
The findings distinguish performance with AI from demonstrated ability without it. For employers, that difference affects how to interpret productivity gains: an employee may complete a technical assignment more effectively with a chatbot without having acquired the underlying knowledge needed to repeat the work unaided.
That distinction matters for training, hiring and quality control. If a company treats AI-assisted output as proof of an employee’s independent expertise, it may overestimate what that employee can do when the tool is unavailable or when a problem falls outside the tool’s reliable capabilities. The trial does not establish how every workplace or AI system will perform, but it provides controlled evidence that task gains and lasting learning are not interchangeable in this setting.
The result is not that ChatGPT offered no benefit. Participants performed better on the assigned tasks while using it, and the study reports faster completion for coding and prediction. The narrower conclusion is that the measured advantage did not carry over to the researchers’ unaided technical questions.
Popular right now
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream
De'Longhi Classic Signature Espresso Machine with Adjustable Milk Frother
Hisense 75-Inch U7 Mini-LED ULED 4K UHD Smart Google TV (75U7SG)
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Run
The experiment took place at BCG in late March and early April 2024 and was preregistered before data collection, on March 13, 2024. Participants had at least a bachelor’s degree; the authors said most reported no coding experience. Each participant worked on two tasks, selected from coding, statistics and prediction.
The comparison was between short training in ChatGPT prompting and short training in web-based research, not between AI use and no access to information. The data scientists’ scores served as an expert reference, rather than as another randomized participant group. The paper’s title describes the measured benefit as “temporary upskilling,” and its authors compare AI to an exoskeleton: a tool that can extend a person’s capability while in use.
The timing also matters: the experiment used GPT-4 in 2024. The study reports results from those tasks and that setup; it does not by itself establish whether later models, longer training or different kinds of workplace work produce the same pattern.
““Once the technology is removed, the enhanced capabilities disappear.””
— Wiles, Krayer and co-authors, as quoted in the paper
Limits of the Follow-Up Test
The unaided assessment consisted of five technical questions after the task sessions. The reported result shows no evidence of a group advantage on that test, but the available source account does not establish whether a longer or different assessment would find other forms of learning. Nor does it show how durable task gains might be after extended practice with AI.
The trial involved one employer, a particular set of tasks, a short training period and GPT-4 as used in 2024. The results should not be taken as a direct measurement of all occupations, AI tools or training programs. The source account also does not provide enough detail to assess the effects of attrition beyond the reported participant counts, or to determine whether the exploratory confidence finding holds in other settings.
Testing Longer-Term Learning
The paper reports an experiment completed in 2024 and published in Nature Human Behaviour in October 2026; the source material does not identify a scheduled follow-up study or a next research milestone. Further research would need to test whether longer training, repeated use or different teaching methods help workers retain skills after AI assistance ends.
For organizations applying the findings now, the study supports keeping AI-assisted performance distinct from unaided competence when evaluating technical work. Whether those gains become durable with different tools or training remains an open question, rather than a result established by this trial.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group using web resources on coding, statistics and prediction tasks. The reported differences were 49, 20 and 17 percentage points, respectively.
Did the ChatGPT users outperform BCG data scientists?
Not on average. The consultants using ChatGPT remained below the scores of 44 BCG data scientists who completed the tasks without AI.
Did the task gains remain after ChatGPT was taken away?
The study authors report no evidence of an advantage for the ChatGPT group on five technical questions answered without the tool. That result applies to the follow-up assessment used in this trial.
Does the study show that ChatGPT cannot help people learn?
No. It found that the measured task advantage did not appear on the unaided follow-up questions. The trial does not settle whether different training, longer practice or other assessments could produce lasting learning.
Which AI model did the consultants use?
The treatment group used GPT-4 after a short prompting session. The experiment took place in March and April 2024, so its results are tied to that model and study setup.
Source: ThorstenMeyerAI.com
