How Qwen-AgentWorld Trains a Language World Model

Most agent training work focuses on the policy:

where the model reads a history \(h_t\) and chooses an action \(a_t\). Qwen-AgentWorld trains the complementary object:

Here \(c\) is the task and environment instruction, \(o_{\le t}\) are prior observations, and \(a_{\le t}\) includes the current action. In plain terms: if the agent does this, what will the environment say next?

大多数 Agent 训练工作关注的是 policy：

也就是模型读取历史 \(h_t\) 后选择动作 \(a_t\)。Qwen-AgentWorld 训练的是互补对象：

其中 \(c\) 是任务和环境说明，\(o_{\le t}\) 是此前的 observations，\(a_{\le t}\) 包含当前 action。直白地说：如果 Agent 做了这一步，环境接下来会说什么？

The word “world” needs a narrow reading. This is not a pixel-level physical world model. It is a language world model for digital environments. The seven domains are MCP, Search, Terminal, SWE, Android, Web, and desktop OS. Even the GUI domains are converted into language: accessibility trees, UI hierarchies, or renderable HTML, not raw screenshots. That design choice is the whole point. If the relevant state transition can be expressed as structured text, then a language model can be trained directly on environment dynamics.

这里的 “world” 需要窄读。它不是 pixel-level 的物理世界模型，而是面向数字环境的 language world model。七个领域分别是 MCP、Search、Terminal、SWE、Android、Web、desktop OS。即使是 GUI 领域，也被转成语言形式：accessibility tree、UI hierarchy 或 renderable HTML，而不是原始 screenshot。这个选择正是论文的关键：如果环境状态转移能表示成结构化文本，就可以直接用语言模型学习环境动力学。

The paper’s own slogan can be compressed as:

1. CPT injects environment dynamics and domain knowledge.
2. SFT activates explicit next-state reasoning.
3. RL sharpens the generated observation using realism and correctness rewards.

This is a useful framing because the base next-token objective never disappears. What changes is the presentation of the same transition prediction problem.

论文自己的训练逻辑可以压缩成三句话：

1. CPT 注入环境动力学和领域知识。
2. SFT 激活显式的下一状态推理。
3. RL 打磨生成 observation 的真实感与正确性。

这个 framing 很有用，因为底层 next-token objective 一直没有消失。变化的是同一个 transition prediction 问题被如何呈现。

The disclosed post-training mixture is small enough to inspect, but important enough not to hand-wave.

论文公开的 post-training mixture 规模不大，但足够关键，不能只用一句“用了很多轨迹”带过。

The full training story is larger than this table: the paper reports more than 10 million real-environment interaction trajectories overall. But CPT counts by source and domain are not disclosed, so the table above should be read as the public SFT/RL part of the recipe, not as the full corpus.

完整训练故事比这张表大得多：论文称整体使用超过 1,000 万 条真实环境交互 trajectory。但 CPT 的逐来源、逐领域数量没有公开，所以这张表应理解为公开的 SFT/RL 部分，而不是完整 corpus。

CPT is the least glamorous stage and probably the most important one. The model is exposed to real action-observation traces from sandboxes, MCP servers, stateful terminals, software repositories, Android/browser/desktop emulators, and open public traces. The paper also adds specialized knowledge corpora covering industrial control, cybersecurity, law, medicine, finance, current affairs, and encyclopedic knowledge.

Why mix factual corpora into a world model? Because many digital environments are not just state machines. They also contain domain facts. A finance tool, a medical interface, or a legal search system can return observations whose correctness depends on knowledge outside the local UI state. CPT tries to make those facts available before asking the model to simulate the environment explicitly.

CPT 是最不炫的阶段，但很可能是最关键的阶段。模型在这个阶段接触真实 action-observation traces，来源包括 sandbox、MCP server、stateful terminal、软件仓库、Android/browser/desktop emulator，以及公开 trace。论文还加入了专业知识语料，覆盖工业控制、网络安全、法律、医疗、金融、时事和百科知识。

为什么 world model 需要混入事实语料？因为很多数字环境不只是状态机。它们还包含领域事实。金融工具、医疗界面、法律检索系统返回的 observation 是否正确，常常依赖局部 UI 状态之外的知识。CPT 的目标是在要求模型显式模拟环境之前，先让这些事实进入模型。

The key technical detail is turn-level loss masking. Not every observation token teaches the same amount. Boilerplate, echoes, or unchanged GUI trees can dominate the loss while teaching little about dynamics. Qwen-AgentWorld classifies turns using overlap, novelty, Jaccard similarity, and length-ratio features, then downweights low-information categories.

关键技术细节是 turn-level loss masking。不是每个 observation token 都同样有用。boilerplate、echo、几乎没变化的 GUI tree 会占据大量 loss，却几乎不教环境动力学。Qwen-AgentWorld 用 overlap、novelty、Jaccard similarity、length ratio 等特征分类 turn，然后降低低信息类别的 loss 权重。

Notice the subtlety: masked turns are still kept in context. The model still sees them when predicting later observations; they just do not dominate the loss. That is the right bias for a simulator. State history matters even when a particular state description is not itself a good training target.

注意这里的细节：被 mask 的 turn 仍然保留在 context 里。模型在预测后续 observation 时仍能看到它们，只是这些 token 不主导 loss。这对 simulator 是合理偏置：状态历史很重要，即便某个具体状态描述本身不是好的训练目标。

After CPT, the model may have absorbed many transition regularities, but that does not mean it knows how to present them as a controlled next-observation prediction. SFT changes the interface. The model is shown demonstrations that reason about the prior state, the action semantics, the expected output format, and the likely next observation.

The SFT data construction is also a quality filter. The authors use ten prompt variants, generate three candidate reasoning traces per query from a stronger reasoning model, then use an independent judge to select the best candidate and discard low-quality examples. The final retained set is 7,094 examples from 10,250 candidates, a 69.2% retention rate.

CPT 之后，模型可能已经吸收了很多转移规律，但这不等于它知道如何把这些规律稳定地表达成“下一条 observation 预测”。SFT 改变的是接口。模型看到的 demonstration 会显式分析既有状态、action 语义、输出格式，以及最可能的下一条 observation。

SFT 数据构造本身也是质量过滤。作者使用十种 prompt variant，让更强的 reasoning model 为每个 query 生成三条候选 reasoning trace，再由独立 judge 选择最佳候选并丢弃低质量样本。最终从 10,250 个候选中保留 7,094 条，保留率 69.2%。

This stage matters because next-observation prediction is not just “continue the transcript.” The output must respect environment-specific format constraints. A terminal should return stdout, stderr, exit-code-like behavior, and a plausible prompt. A browser observation should preserve UI structure. An MCP server should follow the tool protocol. SFT teaches the model to put the latent dynamics from CPT into the right response shape.

这一阶段重要，是因为 next-observation prediction 不只是“续写 transcript”。输出必须遵守环境特定的格式约束。Terminal 应返回 stdout、stderr、类似 exit code 的行为和合理 prompt。Browser observation 应保持 UI 结构。MCP server 应遵守工具协议。SFT 教模型把 CPT 中学到的隐式动力学放进正确的 response shape。

RL is where the paper has the most useful engineering details. The setup is awkward for standard language-model RL because the prompt can be tens of thousands of tokens, while the generated next observation is often only a few hundred or a few thousand tokens. Qwen-AgentWorld uses GSPO, caps prompts at 128K tokens, and samples one target turn per trajectory.

The reward is a hybrid:

The rubric is a five-axis LLM judge: format, factuality, consistency, realism, and quality. Each axis is scored from 1 to 5, and the mean is scaled to the 5-25 range. The rule reward is an executable verifier where possible, scaled to the same range; invalid generations receive 0.

RL 是论文中工程细节最有价值的部分。这个设置对标准 language-model RL 很别扭：prompt 可能有几万 token，而模型生成的下一条 observation 往往只有几百到几千 token。Qwen-AgentWorld 使用 GSPO，把 prompt cap 到 128K token，并且每条 trajectory 只采样 一个目标 turn。

reward 是混合形式：

rubric 是五维 LLM judge：format、factuality、consistency、realism、quality。每维 1 到 5 分，取均值后缩放到 5-25。rule reward 在可执行验证可用时加入，同样缩放到这个范围；无效生成给 0。

The one-turn sampling choice is not cosmetic. The authors report that expanding every trajectory into many training turns creates a shared-prefix failure mode: the model sees many highly correlated prompts and learns to echo or overfit the prefix instead of modeling the next transition. They call this the Echo Trap. Sampling exactly one target turn per trajectory reduces that correlation.

每条 trajectory 只采一个 turn 不是细枝末节。作者报告说，把一条 trajectory 展开成许多训练 turn 会制造共享长前缀的失败模式：模型看到大量高度相关的 prompt，容易学会 echo 或过拟合 prefix，而不是学习下一步转移。论文把这称为 Echo Trap。每条 trajectory 只采一个目标 turn，可以降低这种相关性。

The other reward-design lesson is that an LLM rubric alone is too easy to game. The paper reports self-praise and content-type failures in early experiments, then adds strict tag extraction, content classification, and rule anchors. This is the same pattern that shows up across agent RL: open-ended judges help with dense feedback, but executable checks are needed to keep the judge attached to the environment.

另一个 reward-design 教训是：只靠 LLM rubric 太容易被 reward hacking。论文报告早期实验中出现过 self-praise 和 content-type 错误，于是加入严格 tag extraction、content classification 和 rule anchor。这和很多 Agent RL 工作中的模式一致：开放式 judge 提供 dense feedback，但可执行检查用于把 judge 固定在环境事实上。

AgentWorldBench contains 2,170 turn-level samples from seven domains and nine source benchmarks. The input is a real prior trajectory prefix plus the current action; the target is the real next observation. This is the right benchmark for the paper’s objective, but it is important to keep the protocol in mind: it is teacher-forced one-step prediction, not long free-running rollout. It tests whether the model can predict the next observation given real history, not whether it remains stable after simulating 30 self-generated steps.

AgentWorldBench 包含 2,170 个 turn-level 样本，覆盖七个领域和九个来源 benchmark。输入是真实 trajectory prefix 加当前 action；目标是真实下一条 observation。这很适合论文的目标，但必须记住协议：它是 teacher-forced one-step prediction，不是长程自由 rollout。它测试的是模型在真实历史条件下预测下一条 observation 的能力，而不是模型连续自我模拟 30 步后是否仍稳定。

The headline rubric results are:

主表中的 rubric 结果如下：

The open 35B model gains +8.66 over its base. That is strong evidence that the recipe works as a whole. The closed 397B model gains +3.97 over its base. But the paper does not publish a clean CPT-only, CPT+SFT, and CPT+SFT+RL ablation on AgentWorldBench, so we should not over-attribute the final gain to any single stage.

开源 35B 模型相比 base 提升 +8.66。这强烈支持完整 recipe 有效。闭源 397B 相比 base 提升 +3.97。但是论文没有公开 AgentWorldBench 上的 CPT-only、CPT+SFT、CPT+SFT+RL 干净 ablation，所以不应把最终提升过度归因到某个单独阶段。

The domain pattern is also revealing. Qwen-AgentWorld-397B scores high on MCP, Terminal, SWE, and OS, but is weaker in GUI-heavy settings and Search remains hard.

domain pattern 也很有信息量。Qwen-AgentWorld-397B 在 MCP、Terminal、SWE、OS 上很强，但 GUI-heavy 场景相对弱，Search 仍然困难。

This pattern matches the training interface. Text-structured environments are easier for a language world model. GUI environments are represented as text trees rather than pixels, which helps with language modeling but loses visual information. Search is especially hard because the observation depends on retrieval, freshness, ranking, and external knowledge, not just local transition mechanics.

这个 pattern 和训练接口是匹配的。文本结构化环境对 language world model 更友好。GUI 环境被表示为文本 tree 而不是 pixel，这有利于语言建模，但会丢失视觉信息。Search 尤其困难，因为 observation 取决于 retrieval、freshness、ranking 和外部知识，而不只是局部转移机制。

The downstream experiments are not just demos. They explain why one would train this model in the first place.

Decoupled simulation. On synthetic OpenClaw environments, replacing a weaker simulator with Qwen-AgentWorld-397B improves the downstream agent from 65.4 to 69.7 on the base benchmark and from 47.9 to 55.0 on QwenClaw. This is the cleanest use case: train the policy against a learned environment model when the real environment is expensive.

Controlled MCP simulation. In MCP tasks, uncontrolled simulation does not reliably help. Controlled simulation, where the world model is constrained by the real server state, improves Tool Decathlon from 32.4 to 36.1 and MCPMark from 21.5 to 33.8. This is the main lesson: a language world model becomes much more useful when anchored to executable state.

Fictional search. For generated databases, simulation-based RL improves WideSearch substantially for the 35B model. The interesting behavior is not just score increase: the simulator-trained agent learns to call the extractor more often, while the real-environment-trained agent calls it less. The world model is shaping exploration style, not merely providing cheaper labels.

下游实验不只是 demo。它们解释了为什么要训练这个模型。

解耦模拟。 在 synthetic OpenClaw 环境中，把较弱 simulator 换成 Qwen-AgentWorld-397B 后，下游 Agent 在 base benchmark 上从 65.4 提升到 69.7，在 QwenClaw 上从 47.9 提升到 55.0。这是最直接的用法：真实环境昂贵时，用 learned environment model 训练 policy。

受控 MCP 模拟。 在 MCP 任务中，uncontrolled simulation 不稳定。controlled simulation，也就是让 world model 受真实 server state 约束，可以把 Tool Decathlon 从 32.4 提升到 36.1，把 MCPMark 从 21.5 提升到 33.8。这里的核心教训是：language world model 被 executable state 锚住时才更有用。

虚构搜索。 在生成数据库上，simulation-based RL 显著提升 35B 模型的 WideSearch 结果。最有意思的不只是分数，而是行为变化：simulator-trained agent 学会更频繁调用 extractor，而 real-environment-trained agent 调用次数下降。world model 改变了探索风格，不只是提供便宜 label。

I would place Qwen-AgentWorld between three environment strategies:

The likely future is hybrid. Use the learned simulator for breadth, curriculum, and counterfactual rollouts; use real environments and executable verifiers as anchors. That is also why the controlled MCP result matters more than the uncontrolled one. Pure imagination is brittle. Anchored imagination is a training tool.

我会把 Qwen-AgentWorld 放在三类环境策略之间：

更可能的未来是 hybrid：用 learned simulator 提供广度、curriculum 和 counterfactual rollout；用真实环境和 executable verifier 作为锚点。这也是为什么 controlled MCP 结果比 uncontrolled 更重要。纯想象很脆弱；有锚点的想象才是训练工具。

The most important idea in Qwen-AgentWorld is that environment modeling can be made native to language-model training. The model is not merely prompted to role-play a terminal, browser, or MCP server. It is continually pretrained on real transitions, supervised to reason about next states, and reinforced against reference-grounded realism signals.

The most important limitation is attribution. The paper gives a convincing full-recipe result and useful RL diagnostics, but not a clean public decomposition of CPT vs SFT vs RL on the main benchmark. So the right conclusion is not “RL alone creates a world model.” It is: a long-context language model can become a useful digital-environment simulator when its entire post-training pipeline is organized around next-observation prediction.

Qwen-AgentWorld 最重要的想法是：环境建模可以变成语言模型训练的原生目标。模型不是靠 prompt 扮演 terminal、browser 或 MCP server，而是先在真实转移上持续预训练，再用 SFT 学会显式下一状态推理，最后用 reference-grounded 的真实感信号做 RL。

最重要的限制是归因。论文给出了有说服力的完整 recipe 结果，也给出了有用的 RL diagnostics，但没有在主 benchmark 上公开干净的 CPT vs SFT vs RL 分解。因此正确结论不是“RL 单独创造了 world model”，而是：当整个 post-training pipeline 围绕 next-observation prediction 组织起来时，长上下文语言模型可以成为有用的数字环境 simulator。

• Qwen Team. Qwen-AgentWorld: Training Agents with Real-world Environment Simulation.
• Qwen. Official Qwen-AgentWorld release post.
• QwenLM. Qwen-AgentWorld GitHub repository.
• Qwen. Qwen-AgentWorld-35B-A3B model card.

• Qwen Team. Qwen-AgentWorld: Training Agents with Real-world Environment Simulation.
• Qwen. Qwen-AgentWorld 官方发布文章.
• QwenLM. Qwen-AgentWorld GitHub 仓库.
• Qwen. Qwen-AgentWorld-35B-A3B model card.