RL post-training in diffusion transformers is activation steering
Reinforcement learning post-training significantly improves image and video generation models.
2 min read, Sunday, 22 November

Reinforcement learning, a training method that uses rewards and punishments to guide model behavior, is widely used to improve the visual quality, text rendering, and compositional accuracy of diffusion transformers — the neural network architectures commonly used for generating images and videos from text.
Because capabilities like rendering requested text or composing multiple objects depend heavily on the prompt, researchers previously assumed that enhancing these skills required learning intricate, input-dependent transformations. A new study tests this assumption by applying a deliberately simple approach: a fixed set of steering vectors added to the block outputs of the frozen base model.
Key facts
- Base Model
- SD3.5-M
- Steering Vectors Parameters
- 0.29M
- RL-tuned LoRA Parameters
- 18.8M
- Parameter Ratio
- 65x fewer parameters
- Reward Recovery (Image Models)
- 90.0-96.3%
- arXiv ID
- arXiv:2611.26627
Source page video
Demonstration of diffusion transformer generation improvements via constant activation steering.
How activation steering works
The researchers fit these steering vectors to match the velocity predictions of public reinforcement learning-tuned checkpoints without including any reward in the objective function. Although these vectors are identical for every prompt, token, and sampling seed, they recover 90.0 to 96.3 percent of the reward gains across six different image models. On two video models, they recover 89.8 and 111.9 percent of the gains.
On the SD3.5-M model, these steering vectors use 65 times fewer parameters than the reinforcement learning-tuned LoRA, which stands for low-rank adaptation, a technique for efficiently fine-tuning large models by adjusting a small subset of parameters. Specifically, the steering vectors use 0.29 million parameters compared to 18.8 million for the LoRA.
Implications for model builders
When initialized at zero and trained with online reinforcement learning, the steering vectors alone exceed the reward gains of weight-based reinforcement learning on text rendering, evaluated using optical character recognition, and human preference scores on SD3.5-M.
These findings suggest that a constant activation shift is enough to capture most of the benefits of policy-gradient reinforcement learning post-training. Instead of learning brand new input-dependent transformations, the training process primarily steers the already pretrained generator toward outputs it was already capable of computing.
Source: Original announcement
Worth a read?



Comments