Token-level MDP view of autoregressive generation. Each token is an action, the prefix is the state, and the transition is deterministic (append). The sequence probability decomposes as a product of per-token probabilities.