How Can Post-Trained Models Continually Learn with Off-Policy Training?

1.  Off-policy = forgetting?

A long-standing goal of AI is a model that continually learns and improves itself. Today's models undergo extensive post-training with RL. Further supervised finetuning (SFT) on new data can degrade their existing capabilities and generalize poorly. This can happen even within the same domain: as we will show, learning harder math problems can make a model worse at the comparatively easier math problems it could previously solve. On-policy training, where the model learns from its own current behavior, such as RL, has shown better retention (Chen et al., 2025). This motivates the view that continual learning needs to be on-policy.

However, the data that carry new knowledge or capabilities are usually off-policy: new facts after the pretraining cutoff, expert demonstrations, and synthetic data for self-improvement. These are precisely the data we want the model to learn from.

To bridge this gap, many works try to convert off-policy data into on-policy signals, such as on-policy self-distillation (OPSD), Pedagogical RL, and SFT with sampling.

But converting off-policy data into on-policy training signals while keeping its original meaning and behavior is itself a challenge. This can cause artifacts like suppressed uncertainty and reduced exploration. If you have tried these approaches, you may have found it tricky to balance on-policyness and data quality.

These failures motivate us to seek better ways to learn from off-policy data. We begin by questioning a key assumption: Is on-policy training fundamental to continual learning?

2.  Model merging alone mitigates forgetting

Direct SFT on off-policy data can cause substantial forgetting. But could a simple change to the learned update help? We start with the simplest operation possible: simply merging the SFT model back toward the original post-trained model, \(\theta_{\mathrm{merge}} = (1 - \lambda)\theta_{\mathrm{post}} + \lambda\theta'_{\mathrm{post}}\). This simple operation makes SFT a strong baseline for continual learning. In the figure below, we see that without merging, gains on the new task come at the cost of performance on old tasks. With merging, SFT generalizes much better, and OPSD loses its edge over SFT despite its expensive on-policy sampling.

3.  From model merging to grafting

We see that model merging largely mitigates forgetting of existing capabilities, but why does forgetting happen in the first place? Can we make model merging even better?

Key intuition: Training on narrow distributions (e.g., post-training, or even late pretraining) skews which task a model infers its inputs to come from (Kotha et al., 2024). So finetuning later checkpoints routes more of the new learning signal to the parameters that support existing capabilities. This suggests learning the update on an earlier checkpoint instead. But how?

Task arithmetic (Ilharco et al., 2023) shows that an update need not be derived from the model it is applied to. Based on this, we change where the update is learned and how it is applied. We call the method grafting:

1. learn the update on an earlier donor model \(\theta_{\mathrm{donor}}\);
2. scale the weight update by \(\lambda\) and apply it to the post-trained model;
3. optionally, mask the most sensitive weight directions (estimated with the Fisher information on old-task data) when the new data distribution is far from the post-trained model.

\[ \theta(\lambda, \rho) = \theta_{\mathrm{post}} + \lambda \cdot M_\rho \odot (\theta_{\text{donor-ft}} - \theta_{\mathrm{donor}}) \]

Here \(\theta_{\text{donor-ft}}\) is the donor after finetuning on the new data, and the mask \(M_\rho\) zeroes the top-\(\rho\) fraction of Fisher-weighted directions of the update. In our experiments, masking at most 1% of directions is enough.

Note: Scaling the graft is equivalent to merging the fully grafted model back toward the original post-trained model. Let \(\Delta_{\mathrm{graft}} = M_\rho \odot (\theta_{\text{donor-ft}} - \theta_{\mathrm{donor}})\), and define the fully grafted model as \(\theta_{\mathrm{graft}} = \theta_{\mathrm{post}} + \Delta_{\mathrm{graft}}\). Then

\[ (1 - \lambda)\theta_{\mathrm{post}} + \lambda\theta_{\mathrm{graft}} = \theta_{\mathrm{post}} + \lambda\Delta_{\mathrm{graft}} \]

With the post-trained model itself as the donor and no masking, this reduces to the model-merging baseline in Section 2.

Which donor model?

We want a donor that is early enough to cause less forgetting, but late enough to stay close to the final model. OLMo-3 releases many intermediate checkpoints, which lets us study this. Each post-training stage (SFT, DPO, RL) makes the model harder to finetune, and the best donor sits before the end of pretraining.

4.  Results

We study settings where off-policy data arises naturally.

General settings: In all scenarios, we assume access only to the data, not to the weights or logits of a stronger teacher, so direct on-policy distillation from a teacher is not possible. New-task performance is measured on held-out data from the new task; old-task performance is the mean over AIME 2024, AIME 2025 and HMMT 2025. All methods receive the same merging sweep in the comparisons below.

4.1  Math

For math, we want the model to learn to solve hard problems while preserving its existing reasoning capabilities. Here, retaining old capabilities means maintaining performance on easier math problems while learning harder ones. We study two sources of thinking traces: (1) expert traces from a stronger model, which may differ substantially from the student's behavior, and (2) synthetic traces generated by the model itself based on the reference solution without thinking traces.

Distillation from expert traces. On hard problems the model rarely solves, we can learn from a stronger model by doing SFT on its traces. We use traces written by DeepSeek-V4-Pro. Their reasoning may differ substantially from the student's, making forgetting a concern.

Grafting improves the joint old/new-task frontier over SFT and OPSD. The gains persist when allowing multiple attempts and in non-thinking mode. OPSD offers little improvement in old-task generalization over the original thinking models.

Self-improvement with Pedagogical RL. Solutions from humans or expert systems rarely come with thinking traces. Pedagogical RL addresses this challenge by teaching the model to write thinking traces by only looking at the solution. Specifically, it trains a copy of the model that sees the reference solution to write traces the student can learn from, then uses SFT. On our hard math problems, however, direct SFT on these pedagogical traces still causes substantial forgetting. We therefore test whether grafting on the same traces can learn the new task while better preserving existing capabilities.

Learning the same generated traces with grafting instead of SFT improves both new-task performance and generalization to old tasks. Grafting also outperforms OPSD.

4.2  Knowledge injection

Models at deployment need to learn new facts as the world or environment changes while retaining their existing reasoning abilities. To study this, we try to inject new facts that emerge after the pretraining cutoff into the model. We use invented entities so that no fact can leak from pretraining, and the question-answer form is far from how a reasoning model writes. We therefore expect this to be a particularly demanding test of retaining its math capabilities.

SFT learns the new facts but forgets existing math capabilities. OPSD largely retains those capabilities but learns fewer facts. Grafting learns nearly all the facts while keeping old-task performance at or above that of the original model.

5.  Continual learning through better optimization

Continual learning requires models to acquire new knowledge and skills while preserving existing capabilities. Converting available data into on-policy signals is one approach, but our results show that changing the weight update can make off-policy learning substantially more effective. Even traces designed to be easier to learn from, such as those produced by Pedagogical RL, benefit from a better update rule.

These results motivate more work on how new information is incorporated into a model's parameters. Alongside efforts to make data easier to learn from, we should develop optimization and update rules that let models learn from the data available to them without causing forgetting.