Woodpecker distillation uses weak models to correct reasoning bugs in strong models

Large language models often struggle with complex reasoning tasks even when they theoretically possess the capability to solve them. Rather than stemming from global incompetence, many of these failures originate from localized reasoning bugs occurring in intermediate steps. Researchers have found that these errors are frequently repairable by inserting a short patch generated by a weak probe model after the reasoning prefix of a strong model.

The challenge of internalization

While inserting patches can redirect a flawed reasoning trajectory toward a correct solution, this corrective effect is rarely internalized through straightforward fine-tuning on the repair text itself. This limitation suggests that the genuinely useful signal resides not within the intervention text directly, but rather in the way that intervention reshapes the future reasoning distribution of the model.

Introducing Woodpecker Distillation

To address this limitation, researchers proposed Woodpecker Distillation, a weak-to-strong training framework that learns directly from contrastive local interventions. The methodology operates through a specific sequence of steps:

• Contrasting successful and unsuccessful weak-model patches at the exact same reasoning prefix.
• Constructing a corrective teacher distribution derived from their induced future token predictions.
• Distilling this refined signal back into the strong model.

Experimental results

Evaluations conducted across various mathematical reasoning benchmarks indicate that Woodpecker Distillation consistently enhances the performance of strong models. Furthermore, the approach outperforms traditional direct imitation baselines, offering a reliable path forward for improving multi-step model reliability without requiring exhaustive manual oversight.