Next-token Preference: Scaling Unsupervised Reward Modeling

Revised October 5, 2026

Jingxuan Fan ¹, Yueying Li ², Zhenting Qi ¹, Dinghuai Zhang ³,
Kianté Brantley ^1,4, Sham M. Kakade ^1,4, Hanlin Zhang ¹

¹Harvard University ²Cornell University ³Microsoft Research ⁴Kempner Institute

Paper arXiv Code

Can raw web text teach a model what makes a response better? Next-token preference (NTP) turns the continuation structure of existing text into reward-model supervision, without generating candidates or collecting human or model-generated preference labels.

Transferable rewards: 11M tokens per corpus yield peak overall RewardBench 2 gains of 7.6–8.3 percentage points across mathematical and general-domain web text.
Downstream utility: the learned rewards improve response selection in math, code, safety, and instruction following, and support math policy optimization.
Supervised training: NTP provides a useful initialization for reward learning from curated preference data.

Motivation

Reward models guide response selection and policy optimization in reinforcement learning from human feedback (RLHF) [1]. Conventional preference pipelines require candidate generation and judgments from people or models [2], [3], [4]. Automating the judging step still leaves generation and scoring costs.

Human and model preferences can also be noisy or inconsistent [5], [6], [7], [8]. This motivates a question: how much useful reward supervision is already present in the structure of web text? NTP uses the relationship between a prefix and its original continuation as an implicit preference signal. All comparisons come from existing text, so constructing them requires neither new responses nor explicit preference annotations.

Method: Next-token Preference

We concatenate documents into a continuous token stream and divide it into fixed-length blocks. Each block becomes a prefix \(x\) and its original continuation \(y\). Grouping \(B\) nearby chunks into a batch gives one chosen continuation per prefix and \(B-1\) rejected continuations from the other chunks.

Nearby negatives matter. They can share topic, vocabulary, and style with the true continuation while breaking the semantic or logical dependencies across the split. Ranking these alternatives encourages the reward model to learn continuation consistency. Each batch supplies \(B(B-1)\) preference comparisons assembled online.

NTP workflow: web text is split into prefixes and suffixes, with same-sequence pairs chosen and cross-sequence pairs rejected — All-to-all scoring constructs a preference-learning signal directly from web text. Select the figure to view it at full size.

For a scalar reward \(s_\theta(x,y)\), the Bradley–Terry objective ranks each original continuation above the in-batch alternatives:

\[ \mathcal{L}_{\mathrm{BT}} = -\frac{1}{B(B-1)} \sum_{i=1}^{B}\sum_{j\ne i} \log\sigma\!\left(s_\theta(x_i,y_i)-s_\theta(x_i,y_j)\right). \]

We add a quadratic score-centering penalty to limit reward-scale drift and overly large margins:

\[ \mathcal{L}_{\mathrm{center}} = \mathbb{E}\!\left[ s_\theta(x_i,y_i)^2 + \frac{1}{B-1}\sum_{j\ne i}s_\theta(x_i,y_j)^2 \right]. \]

The final objective is \(\mathcal{L}=\mathcal{L}_{\mathrm{BT}}+c\,\mathcal{L}_{\mathrm{center}}\), with \(c=0.01\) in the reported recipe. The split can occur mid-sentence; preserving sentence boundaries is an ablation, not a requirement.

Experimental Protocol

Training data. Our main corpus is FineMath-4plus. We also test MegaMath-Web-Pro-Max (mathematical web text) and FineWeb-Edu (general educational web text), using 11M tokens per corpus. The default split uses a 512-token prefix and a 1,024-token continuation.

Backbones. We train base Llama-3.2-1B and Llama-3.2-3B models, plus instruction-tuned Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct models. Llama-3.2-3B is the controlled-ablation backbone. The default batch size is 32; Qwen2.5-7B-Instruct uses 8.

Preference alignment. We report overall RewardBench and RewardBench 2 accuracy, alongside:

In-domain (ID): RewardBench Reasoning (math and code) and RewardBench 2 Math.
OOD Safety: the Safety subset of each benchmark.
OOD Others: the remaining subsets, including conversational quality in RewardBench and factuality, ties, focus, and precise instruction following in RewardBench 2.

Downstream evaluation. Best-of-N selection tests math, code, safety, and instruction following with 1B, 3B, and 8B actors. GRPO tests the reward signal for math policy training. Finally, curated-preference training tests NTP as an initialization for supervised reward learning.

Data Scaling and Transfer

Reward learning improves as more web text is used. With Llama-3.2-3B, the overall trend across the 11M-token budget is upward on both benchmarks, with gains extending beyond mathematical reasoning. At 11M tokens, RewardBench Safety improves by 16.0 percentage points over initialization, compared with 13.1 points for Reasoning and 7.0 points for Chat Hard.

RewardBench and RewardBench 2 learning curves across the 11M-token FineMath budget — Continuation-based reward learning transfers to in-domain reasoning and out-of-domain preference judgments. Select the figure to view it at full size.

The recipe transfers across model families and sizes. All four trained models rank in the top 10 on RewardBench 2 within parameter-matched cohorts (0.5–1.5B, 3–4B, and 7B).

Scores (%) and recorded leaderboard ranks within parameter-matched cohorts.
Backbone	RewardBench		RewardBench 2
Backbone	Score	Rank	Score	Rank
Llama-3.2-1B	60.0	3	33.2	3
Llama-3.2-3B	65.6	4	36.2	9
Qwen2.5-3B-Instruct	70.0	3	46.2	8
Qwen2.5-7B-Instruct	73.8	18	57.0	5

Comparison with Curated Preferences

At the same 11M-token budget, NTP produces larger RewardBench 2 gains than training on the tested curated preference datasets. FineMath NTP improves overall accuracy by 7.7 percentage points, compared with 1.8 for Skywork-80K-v0.2 and 0.8 for RLHFMix1. Training on the full curated datasets remains stronger, but requires substantially more labeled data.

Overall gains over Llama-3.2-3B initialization. The first three rows share a token budget; the final two use the full labeled datasets.
Training data / method	Budget	RewardBench 2 gain (pp)
FineMath NTP	11M tokens	+7.7
Skywork-80K-v0.2	≈9K pairs (11M tokens)	+1.8
RLHFMix1	≈22K pairs (11M tokens)	+0.8
Skywork-80K-v0.2	77K pairs (full dataset)	+9.8
RLHFMix1	244K pairs (full dataset)	+8.3

Recipe Ablations

Batch size

Increasing batch size expands the number of in-batch comparisons and improves peak overall RewardBench 2 gains. ID gains continue to grow with batch size, while OOD gains show little benefit beyond \(B=16\). This also increases scoring work because the number of comparisons grows quadratically.

Peak RewardBench 2 gains over initialization, in percentage points. Subset peaks need not occur at the same checkpoint.
Batch size	ID	OOD Safety	OOD Others	Overall
8	+9.6	+3.4	+0.6	+1.0
16	+10.7	+8.3	+7.7	+6.7
32	+16.1	+5.4	+7.4	+7.7

RewardBench 2 learning curves for batch sizes 8, 16, and 32 — Larger batches improve the training trajectory across the 11M-token budget. Select the figure to view it at full size.

Splitting format

Allowing a split within a sentence produces stronger gains. Enforcing sentence boundaries discards candidate spans during packing and leaves less contextually similar negatives. Contiguous chunks preserve more shared context, making mismatched continuations harder to reject through topic or style alone.

Peak RewardBench 2 gains over initialization, in percentage points.
Splitting format	ID	OOD Safety	OOD Others	Overall
Preserve sentence	+4.6	+1.7	+3.5	+0.8
Break sentence	+16.1	+5.4	+7.4	+7.7

RewardBench 2 learning curves comparing sentence-preserving and sentence-breaking splits — Breaking sentences retains more contextually confusable in-batch negatives. Select the figure to view it at full size.

Score centering

Centering improves peak overall RewardBench 2 gain from 6.4 to 7.7 points. Its larger benefit is in downstream selection: on GSM8K with the Llama-3.1-8B-Instruct actor, the gain in best-of-N Maximum Achieved Performance (ΔMAP) increases from 2.8 to 5.4 percentage points. Without centering, accuracy peaks at small N and eventually falls below random selection.

Peak RewardBench 2 subset/overall gains and GSM8K BoN ΔMAP, in percentage points.
Centering	ID	OOD Safety	OOD Others	Overall	BoN ΔMAP
Without	+16.9	+5.9	+8.6	+6.4	+2.8
With	+16.4	+5.7	+7.4	+7.7	+5.4

RewardBench 2 learning curves with and without score centering — Score centering improves overall preference-alignment performance. Select the figure to view it at full size.

Does More Next-token Training Explain the Gains?

We continue pretraining Llama-3.2-3B on the same 11M FineMath tokens in the same order, using next-token prediction. To read out reward quality, we fit a frozen-backbone linear probe or jointly train a scalar reward head and backbone on 512 auxiliary preference pairs from held-out FineMath blocks. These pairs compare original continuations with perturbed copies.

Continued pretraining (CPT) does not improve RewardBench 2 performance over the untouched base model under either readout. In contrast, replacing NTP’s learned head with the same frozen-backbone probe preserves its reward performance. This supports a contribution from the preference-learning objective beyond additional exposure to the corpus.

Absolute RewardBench 2 scores (%). CPT and NTP each use 11M FineMath tokens. A dash indicates an unreported configuration.
Readout	Untrained backbone	CPT backbone	NTP backbone
Frozen-backbone probe	30.0	28.9	36.4
Full RM conversion (512 pairs)	32.3	31.2	—
Original trained RM head	—	—	36.2

Generalization to Other Web Corpora

NTP also works beyond FineMath. Under the same Llama-3.2-3B initialization, training settings, and 11M-token budget per corpus, MegaMath-Web-Pro-Max gives the largest overall gain. General-domain FineWeb-Edu comes within 0.1 percentage points of FineMath, showing that useful reward supervision also exists in educational text outside a math-specific corpus.

Peak overall RewardBench 2 gains over the shared initialization, with 11M tokens per corpus.
Training corpus	Domain	RewardBench 2 gain (pp)
FineMath-4plus	Mathematical web text	+7.7
MegaMath-Web-Pro-Max	Mathematical web text	+8.3
FineWeb-Edu	General educational web text	+7.6

Best-of-N Selection

We sample \(N\in\{1,2,4,8,16,32\}\) responses per prompt and select the one with the highest reward. MAP is Maximum Achieved Performance, the best task accuracy across the tested values of N. Actors are Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct.

We evaluate MATH500 and GSM8K (math), HumanEval+ (code), ToxiGen (safety), and IFEval (instruction following). The two NTP models are FineMath-RM-Llama-3.2-3B and FineMath-RM-Qwen-2.5-7B. Baselines include the Skywork 3B and 8B reward models, the supervised safety-oriented beaver-7b-unified-reward, and random selection.

Selection gains become more consistent with stronger actors. The Qwen-based NTP model exceeds Beaver’s MAP on every evaluated task and actor size. It also exceeds Skywork’s 3B RM on ToxiGen at every actor size and matches Skywork’s 8B RM on MATH500 with the 8B actor. The strongest curated-data RMs remain ahead overall, and improvements are not monotonic in every setting.

Best-of-N results for three actor sizes across GSM8K, MATH500, HumanEval+, ToxiGen, and IFEval, including Skywork and Beaver baselines — Five tasks across three actors; each column uses a different actor size. Metrics follow each task, including pass@1 with base and extra tests for HumanEval+ and prompt-level strict accuracy for IFEval. Select the figure to view it at full size.

Policy Optimization with GRPO

We train instruction-tuned 3B and 8B Llama actors on MATH and GSM8K for five GRPO epochs, using FineMath-RM-Qwen-2.5-7B as the reward signal. We report peak mean@1 test accuracy within that budget.

Our RM improves both actors on both tasks and achieves the best or second-best result in three of four settings, including the best GSM8K result for the 8B actor. It exceeds Beaver by 0.4–2.0 percentage points and outperforms its own untrained initialization in all four settings.

Peak mean@1 accuracy (0–1). Actors: Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct. Bold and underlined values mark the best and second-best results.
Reward model	3B actor		8B actor
Reward model	MATH	GSM8K	MATH	GSM8K
Before training	0.268	0.789	0.423	0.876
Skywork-Reward-V2-Llama-3.1-8B	0.419	0.833	0.447	0.882
Skywork-Reward-V2-Llama-3.2-3B	0.439	0.819	0.465	0.884
beaver-7b-unified-reward	0.416	0.803	0.430	0.881
FineMath-RM-Qwen-2.5-7B	0.420	0.823	0.437	0.886
FineMath-RM-Qwen-2.5-7B-Init	0.399	0.812	0.428	0.879

NTP as an Initialization for Supervised Training

Unlabeled web text can make subsequent preference labels more useful. We compare base, instruction-tuned, and NTP-trained Llama-3.2-3B initializations, then train each on RLHFMix1 [9], a mixture of UltraFeedback, HH-RLHF, SHP, and Summarize-from-Feedback.

The NTP checkpoint achieves a reported overall RewardBench 2 gain of 10.4 points after curated training, compared with 8.3 for base and 13.1 for instruction-tuned initialization. NTP gives the strongest reported ID-Math and OOD-Safety gains, while instruction-tuned initialization remains stronger overall and on OOD Others.

RewardBench 2 gains (percentage points) reported at the end of RLHFMix1 training for the three initializations.
Llama-3.2-3B initialization	ID Math	OOD Safety	OOD Others	Overall
Base	+14.5	+22.3	+1.6	+8.3
Instruction-tuned	+19.4	+22.6	+7.8	+13.1
NTP (FineMath RM)	+21.4	+24.4	+2.0	+10.4

RewardBench 2 learning curves during supervised RLHFMix1 training from base, instruction-tuned, and NTP initializations — Training trajectories over one epoch of curated preference learning. Select the figure to view it at full size.

References

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., & Sun, M. (2024). UltraFeedback: Boosting Language Models with Scaled AI Feedback. arXiv:2310.01377. https://arxiv.org/abs/2310.01377
Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., & Sun, M. (2024). Advancing LLM Reasoning Generalists with Preference Trees. arXiv:2404.02078. https://arxiv.org/abs/2404.02078
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., & others. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. https://arxiv.org/abs/2204.05862
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., & others. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35. https://proceedings.neurips.cc/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., & others. (2023). Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv:2307.15217. https://arxiv.org/abs/2307.15217
Zhang, M. J. Q., Wang, Z., Hwang, J. D., Dong, Y., Delalleau, O., Choi, Y., Choi, E., Ren, X., & Pyatkin, V. (2024). Diverging Preferences: When do Annotators Disagree and do Models Know? arXiv:2410.14632. https://arxiv.org/abs/2410.14632
Gao, Y., Alon, D., & Metzler, D. (2024). Impact of preference noise on the alignment performance of generative language models. arXiv:2404.09824. https://arxiv.org/abs/2404.09824
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., & Zhang, T. (2024). RLHF Workflow: From Reward Modeling to Online RLHF. arXiv:2405.07863. https://arxiv.org/abs/2405.07863

BibTeX

@misc{fan2026nexttokenpreference,
  title         = {Next-token Preference: Scaling Unsupervised Reward Modeling},
  author        = {Jingxuan Fan and Yueying Li and Zhenting Qi and Dinghuai Zhang and
                   Kianté Brantley and Sham M. Kakade and Hanlin Zhang},
  year          = {2026},
  eprint        = {2603.02225},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2603.02225}
}