Continual Pre-training GPT‑OSS‑120B

GPT‑OSS‑120B has about 117B total parameters and roughly 5.1B active parameters per token. MXFP4 can place its inference weights near the storage budget of a single 80GB GPU, but that says nothing about full-parameter training: optimizer states, master weights, gradients, and activations are substantially larger. More importantly, GPT‑OSS prompt, reasoning, and tool-use behavior depend on the Harmony response format; its training contract is not ordinary `system/user/assistant` ChatML.

1. Separate the Four Training Regimes

“Mid-training” is often used to blur settings whose learning rates, data budgets, and batch sizes differ by orders of magnitude. CPT/domain-adaptive pre-training uses raw text, code, and domain documents to change the model distribution. SFT teaches behavior, format, tool use, and task mappings. DPO/preference optimization learns from chosen/rejected comparisons. Online RL learns from rollouts, rewards, and policy versions. Their hyperparameters are not interchangeable. Agent Factory 3 demonstrates agentic RL with GPT‑OSS‑120B, but does not disclose a directly reusable LR or global policy batch. Its useful engineering ideas are attention LoRA, frozen MoE/router weights, asynchronous rollouts, routing replay, and staleness filtering. Missing fields should remain “undisclosed,” rather than inferred from model size.

2. Starting Points: Sweep, Do Not Search for a Magic LR

| Objective | GPT‑OSS‑120B initial / peak LR | Schedule and warmup | Global batch | | -------------------------- | -----------------------------: | ------------------------------------------ | ------------------------------ | | Narrow CPT, <5B tokens | 5e‑6–2e‑5; begin at 1e‑5 | cosine; 1–2% warmup for a reset optimizer | 0.25–1M tokens/update | | Domain CPT, 20–100B tokens | 1e‑5–4e‑5; begin at 2e‑5 | cosine; 0.5–1% warmup | 1–4M tokens/update | | Full SFT | 2e‑6–2e‑5 | linear/cosine or constant+warmup; about 3% | 16–128 sequences/update | | LoRA SFT | 5e‑5–2e‑4; begin at 1e‑4 | cosine; 3% warmup | 8–32 sequences/update | | DPO | 1e‑7–5e‑7; begin at 2e‑7 | linear; 10% warmup | 64–128 preference pairs/update | Direct anchors include Axolotl’s GPT‑OSS‑120B full-parameter recipe at `2e-5` with constant-with-warmup, the GPT‑OSS‑20B Cookbook LoRA recipe at `2e-4` and global batch 16, and Tülu‑3’s 70B SFT/DPO runs at `2e-6`/`2e-7`. For domain CPT, make the first sweep **1e‑5 → 2e‑5 → 4e‑5**. A released GPT‑OSS checkpoint does not include the original optimizer and scheduler state, so a short warmup is safer than assuming a continuous pre-training schedule.

3. Data Scale: From 1k SFT Examples to 100B CPT Tokens

Track unique examples/pairs, consumed tokens, and disk bytes separately. The following GB values are only `tokens × 4 bytes` for int32 token IDs—not JSON, Parquet, or raw-corpus sizes. | Training tokens | int32 token IDs | Practical meaning | | --------------: | --------------: | ---------------------------------- | | 1M | 0.004 GB | small behavioral SFT | | 1B | 4 GB | large SFT / very small CPT | | 20B | 80 GB | meaningful narrow-domain CPT | | 100B | 400 GB | OLMo-scale mid-training ingredient | | 1T | 4 TB | large specialization | For behavior, format, and tool use, 1k–100k high-quality Harmony examples can be enough. Broad instruction tuning is commonly 0.1B–5B tokens or 100k to about 1M conversations. A 0.1–5B-token CPT pilot is a sensible feasibility test; 20–100B tokens is the credible main range for 70B/120B domain adaptation. Use 100B–1T only when the goal is a major capability or distribution shift.

4. Mixture and Retention Matter More Than Feeding Domain Data Forever

Plan replay, staged mixtures, or checkpoint merging. Code Llama retained rehearsal data; Reuse, Don’t Retrain studies general and capability-related blends; Arcee’s SEC adaptation shows domain gains can coincide with general-score loss. A robust sequence is: pilot on 0.5–1B tokens; choose LR and replay ratio using domain and general evaluations; scale to 20–100B tokens; perform a small high-quality Harmony recovery SFT; only then consider DPO or RL.

5. Batch: Use Data-Parallel Degree, Not Physical GPU Count

$$B_{seq}=N_{DP}\times B_{micro}\times N_{accum},\qquad B_{token}=\sum_{batch}\text{non-padding tokens}.$$ With tensor or expert parallelism, `N_DP` is not total accelerators. For CPT, record non-padding tokens/update. Begin around 0.5–4M tokens/update for CPT, 16–128 sequences for full SFT, 8–32 for LoRA, and 64–128 pairs for DPO. On 8×A100‑80GB, validate LoRA/SFT and data/loss correctness first. On 32×A100‑80GB, FSDP2/ZeRO‑3 full FFT and CPT become more realistic; increased hardware should reduce accumulation and wall-clock time, not mechanically scale LR.

6. The GPT‑OSS Prompt and Loss Contract

Strict CPT should remain mostly raw documents and code with a causal-LM objective. Harmony is the native semantic layer for recovery SFT, tool use, and agents. Store structured messages plus `reasoning_effort`, then render them with the official Harmony renderer or a GPT‑OSS-aware template. In multi-turn training retain historical finals but remove historical reasoning traces; only the current turn’s reasoning is a target. Use low, medium, and high effort across the same task families: `high` permits more thought, rather than forcing verbosity. Mask system, developer, user, and tool outputs; train the current assistant analysis/commentary/tool-call/final channels. For pure knowledge injection, down-weight analysis loss or use only verified reasoning traces.

7. Engineering Checklist

- Use BF16, activation checkpointing, and sharded state; do not equate MXFP4 inference footprint with AdamW training memory. - Start CPT with AdamW $\beta=(0.9,0.95)$, weight decay near 0.1, and grad clip near 1; do not blindly transfer that decay to SFT/DPO. - Preserve optimizer, scheduler, RNG, topology, and both best-domain and best-retention checkpoints. - Unit-test Harmony serialization, labels/masks, attention kernels, and logits/loss parity before scaling.

8. Further Reading and Evidence Boundaries

Official sources, open recipes, and comparison studies (click to expand)

Direct GPT‑OSS evidence: OpenAI’s GPT‑OSS announcement, Harmony renderer, fine-tuning notebook, and Axolotl guide.

Comparison anchors: Tülu 3, OLMo 3, Code Llama, Reuse, Don’t Retrain, and Agent Factory 3.

These sources support the direct facts and comparison anchors. GPT‑OSS‑120B’s original CPT optimizer state, terminal LR, total token count, TPUv4 recipe, and 120B RL policy batch are not publicly specified; where they are missing, this post labels its advice as engineering extrapolation.

**Bottom line:** run a 0.5–1B-token CPT pilot at 1e‑5/2e‑5/4e‑5, select the best retention mixture, scale to 20–100B tokens, restore behavior with small high-quality Harmony SFT, and only then move to DPO/RL. If the goal is only formatting, tool calling, or a few tasks, skip tens-of-billions-token CPT and begin with 1k–100k high-quality Harmony SFT/LoRA examples.

GPT‑OSS‑120B 约有 117B 总参数、每 token 约 5.1B active parameters。MXFP4 权重可以落在约一张 80GB GPU 的推理存储级别，但这不能推导出全参数训练能在单卡完成：optimizer state、master weights、gradients 和 activations 远大于推理权重。更关键的是，GPT‑OSS 的 prompt、reasoning、tool use 都依赖 Harmony response format；训练 contract 不是普通 ChatML 的 `system/user/assistant` 三元组。

1. 先区分四种完全不同的训练 regime

“mid-training” 很容易把相差几个数量级的学习率、数据规模和 batch 混在一起。 | 阶段 | 训练信号 | 真正目标 | 不应照抄的超参 | | --------------------------------- | --------------------------------------------------- | ------------------------ | --------------------------- | | CPT / domain-adaptive pretraining | raw text、code、domain documents 的 next-token loss | 改变知识、语言或领域分布 | SFT/DPO 的小 sequence batch | | SFT | instruction、格式、工具使用、任务映射 | 恢复/改变行为 | CPT 的百万 token batch | | DPO / preference | chosen/rejected pair | 对齐偏好与质量判断 | CPT 的 1e‑5–1e‑4 LR | | online RL / RLVR | rollout、reward、policy version | agent policy 和长期交互 | 静态 dataset 的 token 规模 | Agent Factory 3 已展示 GPT‑OSS‑120B 的 agentic RL，但公开材料没有给可直接照抄的 LR 或 global policy batch：它的核心是 attention LoRA、冻结 MoE/router、异步 rollout、routing replay 和 staleness filtering。这里应当写“未说明”，而不是从模型大小倒推一个数。

2. 推荐的起点：不是魔法 LR，而是短程 sweep

| 训练目标 | GPT‑OSS‑120B 初始 / peak LR | 调度与 warmup | global batch | 置信度 | | ------------------------------ | ---------------------------------------------: | -------------------------------------------------------- | ------------------------------- | ------ | | 窄域 CPT，<5B token | 5e‑6–2e‑5；首跑 1e‑5 | cosine；新 optimizer warmup 1–2%；末端 peak/30～peak/100 | 0.25–1M tokens/update | 中 | | 正式 domain CPT，20–100B token | 1e‑5–4e‑5；首跑 2e‑5 | cosine；短 warmup 0.5–1% | 1–4M tokens/update | 中高 | | 全参数 SFT | 2e‑6–2e‑5；广泛 SFT 首跑 5e‑6–1e‑5 | linear/cosine 或 constant+warmup；约 3% | 16–64 sequences，大集群可到 128 | 高 | | LoRA SFT | 5e‑5–2e‑4；首跑 1e‑4 | cosine；3% warmup；final LR 可留 10% | 8–32 sequences | 中高 | | DPO | 1e‑7–5e‑7；首跑 2e‑7 | linear；10% warmup | 64–128 preference pairs | 中高 | | 120B agentic RL/GRPO | 全参公开 run 未说明；LoRA 可从 1e‑6–5e‑6 sweep | 依算法 | rollout tokens / policy batch | 低 | 直接锚点包括：Axolotl 的 GPT‑OSS‑120B full-parameter recipe 使用 `learning_rate=2e-5`、constant-with-warmup、约 3% warmup、BF16、gradient checkpointing 和 CPU parameter offload；OpenAI Cookbook 的 GPT‑OSS‑20B LoRA SFT 使用 2e‑4、3% warmup、global batch 16；Tülu‑3 的 70B SFT 使用 2e‑6/3% warmup/batch 128，70B DPO 使用 2e‑7/10% warmup/batch 128。 对 20–100B token 的 domain CPT，建议把第一轮固定成 **1e‑5 → 2e‑5 → 4e‑5**。若自己拥有真正连续的 optimizer/scheduler state，Reuse, Don’t Retrain 的“无 warmup、沿原 schedule 延续”经验更有参考价值；但 released GPT‑OSS checkpoint 不含这些 state，所以从头建 AdamW 时短 warmup 更保守。CPT 后仍应同时看 domain validation、general validation、instruction following 与下游任务，而不是只看 train loss。

3. 数据规模：从 1k SFT 到 100B CPT

数据必须同时记录 examples/pairs、实际消费 tokens 与磁盘字节数；它们不能互换。下表的 GB 是仅把离线 int32 token IDs 以 `tokens × 4 bytes` 归一化，并不等于 JSON、Parquet 或 raw corpus 的体积。 | 训练 token | GB@int32-token | 对 GPT‑OSS‑120B 的含义 | | ---------: | -------------: | ---------------------------- | | 1M | 0.004 GB | 小型行为 SFT | | 100M | 0.4 GB | 中等 post-training | | 1B | 4 GB | 大型 SFT / 很小 CPT | | 20B | 80 GB | 窄域 CPT 开始有意义的量级 | | 70B | 280 GB | Arcee SEC 规划量级 | | 100B | 400 GB | OLMo mid-training ingredient | | 500B | 2 TB | Code Llama specialization | | 1T | 4 TB | Code Llama 70B / 大型 CPT | | 目标 | 推荐规划范围 | 公开锚点 | | ---------------------------- | ---------------------------------------------------- | ------------------------------------------------------------ | | 格式、语气、固定行为 SFT | 约 1M–100M token；常为 1k–50k 高质量例 | GPT‑OSS Cookbook 的特定任务使用 1,000 examples | | 广泛 instruction tuning | 0.1B–5B token，或 100k–约 1M conversations | Tülu‑3 mixture 约 938k rows；Code Llama Instruct 约 5B token | | preference / DPO preparation | 50k–500k pairs；优先记 pair 数和有效 response tokens | Tülu‑3 70B preference mixture 约 337k pairs | | CPT feasibility pilot | 0.1–5B token | 这是工程 pilot，不是“社区中位数” | | 正式 domain adaptation | 20–100B token | Arcee SEC 约 70B domain token；OLMo mid-training 100B | | 大幅 capability shift | 100B–1T token | Reuse, Don’t Retrain 与 Code Llama | 一个算术检查很有用：`100B / 4M ≈ 25,000` updates，`20B / 1M ≈ 20,000` updates，但 `1B / 1M` 只有 1,000 updates。小预算 CPT 若仍坚持 4M token batch，常常只是因为更新次数太少而失败。对小规模 CPT 或 SFT，宁可降到 0.25–1M tokens/update，也不要机械模仿 foundation-training batch。

4. 数据 mixture 比“纯领域数据灌到底”更重要

真正的 domain CPT 应有 replay / staged mixture / merge 计划。Code Llama 在后续 instruction 阶段保留 rehearsal；Reuse, Don’t Retrain 系统比较 general blend 与 capability-related blend；Arcee 的 SEC adaptation 观察到 domain gain 与 general score 下滑并存，model merging 能部分恢复遗忘。最实用的工作流是： 1. 以 0.5–1B token pilot 验证 1e‑5/2e‑5/4e‑5 和 domain/general mixture； 2. 根据 retention 选择 LR、replay ratio 和切换点； 3. 扩到 20–100B token，通常以 1–4M global tokens/update； 4. 用小而高质量的 Harmony SFT 恢复 instruction、reasoning-effort control 与 tool behavior； 5. 最后才做 DPO/RL。 这也解释了为何“120B 模型是否要 50B token SFT”是错误问题：行为、格式、tool use 可能只需 1k–100k 条好样本；只有要明显改变知识、语言或 code distribution 时，几十 B token CPT 才是合理路线。

5. Batch：使用 DP degree，不要使用物理 GPU 数

仅 data parallel/FSDP 时： $$B_{seq}=N_{DP}\times B_{micro}\times N_{accum},\qquad B_{token}=\sum_{batch}\text{non-padding tokens}.$$ 有 tensor parallel 或 expert parallel 时，`N_DP` 不是总 accelerator 数。packing、sequence length 与 padding 会让同一个 `batch=32` 相差数倍 token；对 CPT 应优先记录 non-padding tokens/update。 | Stage | 合理起点 | 公开锚点 | 原因 | | --------- | ----------------------------- | -------------------------------------- | -------------------------------------- | | CPT | 0.5–4M tokens/update | Code Llama、OLMo、Reuse 均在约 4M 附近 | 高吞吐、低梯度噪声 | | Full SFT | 16–128 sequences/update | Tülu‑70B=128 | 小数据需要足够 optimizer updates | | LoRA SFT | 8–32 sequences/update | OpenAI Cookbook=16 | LoRA 的有效 batch regime 更小 | | DPO | 64–128 pairs/update | Tülu‑70B=128 | preference gradient 的噪声结构不同 | | online RL | rollout/group/accepted tokens | Agent Factory 3 未公开统一 batch | 异步系统无法由静态 sequence batch 描述 | 8×A100‑80GB 的优先级是 LoRA/SFT 与 data/loss correctness；Axolotl 的 120B full FFT 公开验证是 8×H100‑80GB + FSDP2 CPU offload，且需要约 3TB disk，不能因为同为 80GB 而把配置直接迁到 A100。32×A100‑80GB 才更适合把 FSDP2/ZeRO‑3 full FFT 与 CPT 当作主路线：4k context、microbatch 1、32 DP ranks 时，一个 microstep 已约 131k tokens，accumulation 8/16/32 即约 1/2/4M tokens。

6. GPT‑OSS 特有的 prompt 与 loss contract

严格 CPT 的大头仍应是 raw document、raw code 与专业 text 的 causal LM objective，**不需要**全部 Harmony chat 化。Harmony 是后续 recovery SFT、tool use 与 agent 数据的原生语义层。 推荐的结构化数据 schema 是保存 messages 和 `reasoning_effort`，交给官方 Harmony renderer 或 GPT‑OSS/TRL 的 patched chat template 渲染，不要手写 special tokens： ```json { "reasoning_effort": "medium", "messages": [ { "role": "system", "content": "Reasoning: medium" }, { "role": "developer", "content": "Answer as a precise domain expert." }, { "role": "user", "content": "..." }, { "role": "assistant", "channel": "analysis", "content": "..." }, { "role": "assistant", "channel": "final", "content": "..." } ] } ``` 实用 SFT mixture 可从 normal instruction/QA 35%、medium reasoning 20%、high reasoning 15%、low-effort concise QA 10%、tool trajectory 10%、multi-turn 5%、instruction-hierarchy conflict 5% 开始。低/中/高 effort 应在相同 task family 中交叉出现：`high` 表示允许充分思考，而不是强制冗长输出。工具轨迹则应覆盖 no-tool、single-tool、multi-tool、tool-error recovery 与 tool-not-needed；只训练一次成功调用会把 agent 退化成 function-call classifier。 loss mask 的初始规则可为：mask system/developer/user/tool output，训练当前 assistant 的 analysis/commentary/tool call/final。若目标只是 domain knowledge 注入，analysis loss 可低于 final loss，或只使用 verifier 筛过的 reasoning trace；错误 CoT 往往比错误措辞更容易污染 policy。Axolotl 还指出 Harmony 会隐藏非最后一轮 thinking，可能与 chat-template loss masking 冲突：应只训练最后一轮，或确保 thinking 只出现在最后一轮。

7. 工程 checklist

- **精度与状态：**BF16、activation checkpointing、sharded state dict；MXFP4 inference footprint 不能外推到 AdamW training footprint。 - **Optimizer：**CPT 可从 AdamW $\beta=(0.9,0.95)$、WD≈0.1、grad clip≈1 起步；SFT/DPO 不应机械沿用 WD=0.1（Tülu 的 reproduction 使用 WD=0）。 - **Checkpoint：**Axolotl 报告 full FFT checkpoint 可约 720GB；保留 best-domain 与 best-retention checkpoint，保存 optimizer/scheduler/RNG/topology。 - **Correctness first：**固定 model/tokenizer revision；逐 token 单测 Harmony serialization、labels/mask、attention implementation 与 logits/loss parity；尤其在 H100→A100 或 kernel 更换时。 - **TPU：**目前缺少可验证的 GPT‑OSS‑120B TPUv4-specific mid-training recipe。先做 conversion/dequantization、forward/loss parity 与 100–500 update pilot，再承诺 20B/100B token run。

8. 扩展阅读与证据边界

官方资料、开源 recipe 与对照研究（点击展开）

GPT‑OSS 直接证据：OpenAI GPT‑OSS announcement、Harmony renderer、GPT‑OSS fine-tuning notebook、Axolotl GPT‑OSS guide。

CPT/SFT/DPO 对照锚点：Tülu 3、OLMo 3、Code Llama paper、Reuse, Don’t Retrain、Agent Factory 3。

这些资料支持本文的直接事实与比较锚点；凡 GPT‑OSS‑120B 没有公开的 CPT optimizer state、terminal LR、完整 token count、TPUv4 recipe 或 120B RL policy batch，本文均标为“未说明”或“工程外推”，而不把邻近模型的设置伪装成官方数值。

最后的建议可以浓缩为一句话：**先做 0.5–1B token、1e‑5/2e‑5/4e‑5 的 CPT pilot；选择 retention 最好的 mixture，再扩到 20–100B；随后做小而高质量的 Harmony SFT，最后才进入 DPO/RL。**如果目标只是格式、工具调用或少数任务，跳过几十 B token CPT，直接用 1k–100k 条高质量 Harmony SFT/LoRA 往往更有效。