GPT-4 helped BCG's lower-scoring consultants most

On tasks GPT-4 could actually do, Boston Consulting Group consultants who scored below average on a baseline exercise improved 43% against their own scores, and those above that line improved 17%. On a task the tool handled badly, the people using it were 19 percentage points less likely to reach the right recommendation.

Fabrizio Dell'Acqua and his co-authors ran the experiment with 758 consultants, about 7% of the individual-contributor consulting staff at BCG. The write-up is their working paper, Navigating the Jagged Technological Frontier. After a baseline task with no AI, people were randomly assigned to no AI, to GPT-4, or to GPT-4 plus a short briefing on how to instruct it.

One group, 385 people, worked on product development. The main exercise asked them to take a shoe into an underserved market, from the idea through a launch plan. An executive at a global footwear company checked that the steps matched the path that company uses. There were 18 questions: new ideas, a market split, a press release, a memo to staff. The other group, 373 people, worked a business problem with financial data, interviews, and a recommendation to a CEO.

On those 18 questions, people using GPT-4 completed 12.2% more tasks and finished 25.1% faster, at more than 40% higher quality than the control group. The control group got through about 82% of the questions. The briefing group got through about 93%, and the GPT-4-only group about 91%. Human graders scored the work. Controls for spelling and grammar left the scores almost unchanged, so the jump was in the substance.

Because the lower scorers gained more, drafts from the two halves of the room now land closer together. The authors are clear that this split is relative: in most other settings, everyone in the study would count as highly skilled. Staff the client work from the finish of the page and you will have a hard time seeing who was already strong.

The researchers also built a task they expected GPT-4 to miss. They adapted the kind of case BCG uses in job interviews: a spreadsheet plus interviews with company insiders. The spreadsheet was designed to look complete. The interviews held details that pointed to a different recommendation. The question was which brand had the most room to grow, and what to do about it.

Consultants with no AI got that call right about 84.5% of the time. The GPT-4 groups got it right 60% and 70% of the time, a combined drop of 19 percentage points. The briefing group fell by 24 percentage points, and the GPT-4-only group by 13. They were faster while they were wrong. The briefing group spent about 30% less time, more than 11 minutes, and the GPT-4-only group about 18% less, more than 6 minutes.

The shoe ideas showed a second cost on the work the tool does well. GPT-4 ideas scored higher, and they resembled each other more than the ideas from people working alone.

1. Keep one piece of work this week that starts with the tool closed, so you still have a baseline for the person.
2. On a draft GPT-4 touched, ask which source changed the recommendation. In the hard case the spreadsheet looked sufficient, and the interviews held the detail that flipped the answer.
3. Ask what the speed cost. The groups that finished the hard case sooner missed it more often, and the group with a briefing on the tool missed it most.
4. When you choose who presents to the client, ask who can defend the recommendation with the laptop shut.

A critical thinking session is a useful place to practise that check on a live memo. Bring the draft and the source it rests on.