RL post-training in diffusion transformers is activation steering

Reinforcement learning post-training significantly improves image and video generation models.

2 min read

RL post-training in diffusion transformers is activation steering

Reinforcement learning, a training method that uses rewards and punishments to guide model behavior, is widely used to improve the visual quality, text rendering, and compositional accuracy of diffusion transformers — the neural network architectures commonly used for generating images and videos from text.

Because capabilities like rendering requested text or composing multiple objects depend heavily on the prompt, researchers previously assumed that enhancing these skills required learning intricate, input-dependent transformations. A new study tests this assumption by applying a deliberately simple approach: a fixed set of steering vectors added to the block outputs of the frozen base model.

Key facts

Base Model
SD3.5-M
Steering Vectors Parameters
0.29M
RL-tuned LoRA Parameters
18.8M
Parameter Ratio
65x fewer parameters
Reward Recovery (Image Models)
90.0-96.3%
arXiv ID
arXiv:2611.26627

Source page video

Product demonstration

Demonstration of diffusion transformer generation improvements via constant activation steering.

How activation steering works

The researchers fit these steering vectors to match the velocity predictions of public reinforcement learning-tuned checkpoints without including any reward in the objective function. Although these vectors are identical for every prompt, token, and sampling seed, they recover 90.0 to 96.3 percent of the reward gains across six different image models. On two video models, they recover 89.8 and 111.9 percent of the gains.

On the SD3.5-M model, these steering vectors use 65 times fewer parameters than the reinforcement learning-tuned LoRA, which stands for low-rank adaptation, a technique for efficiently fine-tuning large models by adjusting a small subset of parameters. Specifically, the steering vectors use 0.29 million parameters compared to 18.8 million for the LoRA.

Implications for model builders

When initialized at zero and trained with online reinforcement learning, the steering vectors alone exceed the reward gains of weight-based reinforcement learning on text rendering, evaluated using optical character recognition, and human preference scores on SD3.5-M.

These findings suggest that a constant activation shift is enough to capture most of the benefits of policy-gradient reinforcement learning post-training. Instead of learning brand new input-dependent transformations, the training process primarily steers the already pretrained generator toward outputs it was already capable of computing.

Source: Original announcement

Worth a read?

Report

Comments