Continual Learning: From Attention to Neural Memory
Suppose a document says that a particular animal is called Mira. Several paragraphs later, a question refers to that animal. The model must carry useful information across those paragraphs. Where is it stored, how does the question find it, and what happens if a later sentence corrects the name?
These questions lead to four papers: Linear Transformers (2020), Fast Weight Programmers (2021), TTT Layers (2024), and Titans (2025). Rather than start with their terminology, we will build a small memory, discover its limitations, and introduce each method when it answers the next question.
The discussion concerns learning and remembering while processing a sequence. Whether a system also preserves knowledge permanently across tasks is a separate question, addressed at the end. All numerical vectors below are constructed examples, not measurements of named semantic features in a real LM.
1. What Must Be Carried From One Token to the Next?
1. 从一个 Token 到下一个 Token，需要传递什么？
1.1 First Separate Tokens, Representations, and Parameters
1.1 先区分 Token、表示与模型参数
A language model receives a sequence of tokens: pieces of text represented by integer IDs. A word may occupy more than one token. We write the IDs as $a_1,a_2,\ldots$; the subscript is a position, not a feature coordinate.
An embedding lookup replaces each ID with a vector of numbers. Later layers transform that vector using the visible context. At the particular layer we are studying, call the vector at position $t$ $x_t$. It has $d_{\mathrm{model}}$ entries:
\[x_t=(x_{t,1},\ldots,x_{t,d_{\mathrm{model}}}).\]One entry is a coordinate, and the number of entries is the dimension or width. Thus $t$ selects a token position, while the second subscript selects a coordinate within that position’s vector. The vector can change between layers even though the token ID does not.
The rules that compute these vectors contain learned numbers: the model’s parameters, collected under the name $\Theta$. During an ordinary inference run, $\Theta$ is fixed, but $x_t$ changes with the input. A changing representation does not imply that training has occurred.
For next-token prediction, the model uses the observed prefix $a_{\le t}=(a_1,\ldots,a_t)$ to produce a distribution over $a_{t+1}$. It must not use the unknown future token to construct that prediction. This restriction is called causality. In a causal network, $x_t$ can contain information from the prefix, not later positions.
1.2 Every Memory Needs a State, a Write, and a Read
1.2 每种记忆都需要 State、写入与读取
Imagine processing the document one piece at a time without repeatedly rereading everything. When the next piece arrives, some information from the past must still be available. Call that carried information the state $s_t$. “State” describes its role; it does not specify whether the data structure is a vector, a list, a matrix, or a small model.
There are two different operations. Writing changes what later steps will have available. Reading uses the current state to produce an output. In abstract notation,
\[s_t=U_\Theta(s_{t-1},x_t),\qquad z_t=R_\Theta(s_t,x_t).\]$U$ names the update function and $R$ the read function. $s_{t-1}$ means state before the current input; $s_t$ means state after the write. $z_t$ is a feature passed to subsequent layers, not necessarily a token or a probability. The superscript-free $\Theta$ indicates that the same parameterized rules can be reused at every position.
A conventional vector-state RNN is one example: take $s_t=h_t\in\mathbb R^D$. Its learned recurrence converts the old vector and current input into a new vector. A KV cache is another example: the state is a growing list of key–value pairs, and a write appends a pair. Later we will make the state a matrix whose entries are updated by learning.
This gives us three questions to ask of every paper: what is carried, how is it changed, and how is it queried? We will use the same questions even when the carried numbers are called “weights.”
2. Attention: Read Content Through a Matching Address
2. Attention：通过匹配地址来读取内容
2.1 Why Do We Need Both a Key and a Value?
2.1 为什么需要 Key 和 Value 两种东西？
A database lookup makes the distinction concrete. “Employee 1024” can locate a record, but the desired answer might be the employee’s name or department. The lookup criterion and returned record serve different purposes. A library catalog is similar: its topic tags already have meaning, but are not the book’s contents.
Attention uses a learned, soft version of this organization. At the receiving position $i$, a query $q_i$ expresses a request. Each candidate position $j$ offers a key $k_j$ for matching and a value $v_j$ for transmission. These are not hand-assigned database fields; they are computed from hidden representations.
We use row vectors throughout. A learned matrix transforms an input by mixing its coordinates; this is a linear projection. One attention head computes
\[q_i=x_iW_Q,\qquad k_j=x_jW_K,\qquad v_j=x_jW_V.\]For example, one key coordinate is $k_{j,b}=\sum_{a=1}^{d_{\mathrm{model}}}x_{j,a}(W_K)_{ab}$. Thus “projecting” does not mean choosing a literal column labeled “animal”; it means learning a weighted combination of available features. The dimensions are
\[W_Q,W_K:\ [d_{\mathrm{model}},d_k],\qquad W_V:\ [d_{\mathrm{model}},d_v].\]The outputs $q_i,k_j$ have $d_k$ coordinates; $v_j$ has $d_v$. $d_k$ is the matching width, $d_v$ the transmitted-content width. A head is one such set of projections and its associated read operation; multiple heads can learn different matches and contents.
A key therefore can contain semantic information: its meaning is expressed through which queries match it. But it need not be unique, human-interpretable, or a complete encoding of the original vector. Two candidates can match similar requests while carrying different values. “Key = address” is a computational analogy, not a claim that keys are meaningless integers or guaranteed unique identifiers.
2.2 From a Match Score to an Actual Read
2.2 从匹配分数到真正的读取
How can two vectors match? The simplest score multiplies corresponding coordinates and sums them:
\[q_i k_j^\top=\sum_{b=1}^{d_k}q_{i,b}k_{j,b}.\]The transpose symbol $\top$ turns the key row into a column, allowing row-times-column multiplication. The result is one number. Q and K must have equal width for this calculation; the value width has no role yet.
Raw scores are not mixing weights: they can be negative and need not sum to one. Softmax exponentiates each score, making it positive, then divides by the sum over the allowed candidates. Write the scaled score as $s_{ij}=q_i k_j^\top/\sqrt{d_k}$. The read is built in two steps:
\[a_{ij}=\frac{\exp(s_{ij})}{\sum_{r\le i}\exp(s_{ir})}, \qquad z_i=\sum_{j\le i}a_{ij}v_j.\]The denominator runs over positions, not vocabulary entries. Causality limits candidates to $j\le i$. A large $a_{ij}$ means that position $j$ contributes more of its value to receiving position $i$. The coefficients add to one, so this read is a weighted average of values, not usually an exact lookup.
Why divide by $\sqrt{d_k}$? A wider dot product adds more terms. Under a simple model of independent, zero-mean, unit-variance Q/K coordinates, its variance grows as $d_k$ and its typical scale, measured by standard deviation, grows as $\sqrt{d_k}$. The divisor compensates for that width effect. Actual learned coordinates need not satisfy those assumptions; this is not measured unit-variance normalization. Normalizations develops that qualification.
Now consider “The animal didn’t cross the street because it was tired.” At “it,” a head might retrieve an animate entity. As a deliberately tiny calculation, retain only two candidates and set
\[q=(1,0),\quad k_{\mathrm{animal}}=(1,0),\quad k_{\mathrm{street}}=(0,1).\]The dot products are 1 and 0. With $d_k=2$, the scaled scores are approximately $(0.707,0)$. Exponentiation gives approximately $(2.028,1)$; dividing by their sum gives $(0.670,0.330)$. Both candidates remain active.
If $v_{\mathrm{animal}}=(2,0)$ and $v_{\mathrm{street}}=(0,3)$, then
\[z=0.670(2,0)+0.330(0,3)\approx(1.340,0.991).\]Matching decides the coefficients; the values supply the coordinates being mixed. The output would approach the animal value only if the score gap were much larger.
There is also a timing constraint in this example. The causal vector at “animal” cannot already contain the later claim “didn’t cross the street.” At “it,” earlier layers may have gathered that preceding clause, but the future word “tired” is still unavailable. The linguistic story is an analogy, not evidence that real heads encode the named attributes. The full matrix view is in Attention and Language Models.
Could we use K or X directly as the values? (Click to expand)
Yes. Setting $V=K$ is valid when dimensions fit, but ties the features used for matching to those transmitted afterward. Setting $V=X$ transmits the unprojected hidden vectors. Separate $W_V$ lets each head choose its own content projection and width.
For example, a head could match by a subject-related feature while transmitting information about the matched entity. This need not be a literal disentangled representation. The benefit is freedom to optimize the two roles separately, not a theorem that tied values always fail. Some projections can be absorbed into subsequent learned linear maps, depending on the architecture.
2.3 What Does a Growing KV Cache Cost?
2.3 不断增长的 KV Cache 有什么代价？
For one head, retaining $t$ pairs uses $t(d_k+d_v)$ numbers. To produce the next read in full attention, the query must compare with those keys and combine their values. At fixed widths, the work of that read grows with $t$.
Across $T$ generated positions, the number of candidate comparisons is proportional to $1+2+\cdots+T=T(T+1)/2$. This is the source of quadratic sequence work for full attention. It does not mean every implementation stores a full $T\times T$ attention matrix: memory-efficient kernels can avoid materializing it, while retaining the pairwise computation.
The attraction of explicit storage is equally important: individual retained K/V pairs remain available for later matching. A fixed-size state must instead combine history. Can we summarize the pairs before knowing which query will arrive, and still read useful associations afterward? The next subsection constructs such a summary. The summary is of projected representations, not a guarantee of lossless storage of the original text.
2.4 Linear Transformers: A Summary We Can Update
2.4 Linear Transformers：构造可以递推的汇总
The obstacle is that softmax’s exponential score depends jointly on the future query and each individual key. The 2020 Linear Transformers paper chooses a different similarity, one that can be separated into features of the query and features of the key. This changes the attention rule; it is not merely deleting a costly line of code. Katharopoulos et al.
We will first use simple positive vectors, then name the general transformation. Suppose two candidate keys are $k_1=(1,1)$ and $k_2=(1,2)$, their values are $v_1=(2,0)$ and $v_2=(0,3)$, and the query is $q=(1,1)$. Use the dot product itself, rather than its exponential, as the nonnegative similarity. The similarities are 2 and 3, so the normalized read should be
\[z=\frac{2(2,0)+3(0,3)}{2+3}=(0.8,1.8).\]Now compute the same answer without keeping those two pairs separately. Turn each key row into a column and multiply it by its value row:
\[k_1^\top v_1= \begin{bmatrix}2&0\\2&0\end{bmatrix}, \qquad k_2^\top v_2= \begin{bmatrix}0&3\\0&6\end{bmatrix}.\]This is an outer product, a matrix rather than the scalar produced by a dot product. Entry $(a,b)$ multiplies key coordinate $a$ by value coordinate $b$. Add the two matrices and, separately, add the keys:
\[S_2=\begin{bmatrix}2&3\\2&6\end{bmatrix}, \qquad c_2=(2,3).\]The matrix stores the numerator information; the vector stores the normalization information. Reading gives $qS_2=(4,9)$ and $qc_2^\top=5$, hence the same $(4,9)/5=(0.8,1.8)$.
Why did that work? Matrix multiplication distributes over addition: $q(k_j^\top v_j)=(qk_j^\top)v_j$. Therefore the query can be applied after summing all the outer products. Adding a third pair requires only adding one more outer product to $S_2$ and one more key to $c_2$.
Real projected Q/K coordinates may be negative. To obtain suitable nonnegative similarities, introduce a feature map $\phi$: a function that transforms each query or key separately into an $r$-coordinate feature vector. Here we choose positive features so nonempty sums have positive denominators. A kernel, in this discussion, is simply the resulting similarity function $\kappa(q,k)=\phi(q)\phi(k)^\top$. These names do not introduce another learning loop.
With $\widetilde q_t=\phi(q_t)$ and $\widetilde k_t=\phi(k_t)$, the same calculation becomes
\[S_t=S_{t-1}+\widetilde k_t^\top v_t,\qquad c_t=c_{t-1}+\widetilde k_t,\] \[\boxed{z_t=\frac{\widetilde q_tS_t}{\widetilde q_tc_t^\top}},\qquad S_0=0,\quad c_0=0.\]$S_t$ has shape $[r,d_v]$ and $c_t$ has shape $[r]$, independent of sequence length. At fixed widths, a write and read use a fixed amount of work; processing $T$ tokens takes work proportional to $T$ for this mixer. This is what “linear” refers to here, not a claim that the entire network is a linear function.
The rearrangement is exact for the selected feature-map similarity. A fixed finite feature map does not generally reproduce exact exponential softmax on arbitrary inputs. Also, dropping $c_t$ changes normalized averaging into an unnormalized sum. We will deliberately examine that unnormalized memory next, but will not call it the same function. Derivation in §§3.2–3.4
3. Fast Weight Programmers: How Do We Correct a Memory?
3. Fast Weight Programmers：怎样修正一条记忆？
3.1 A Matrix Can Store More Than One Answer
3.1 一个矩阵怎样保存多个答案？
The previous summary matrix is not merely an array we inspect: we multiply it by a query to obtain an output. That suggests treating a matrix as a small function. A function is a rule taking an input to an output. Let our rule be $f(q;W)=qW$, where $W$ has shape $[d_k,d_v]$.
Here we deliberately study a bare, unnormalized memory. There is no feature map or denominator in this example. Suppose $d_k=d_v=2$, and we want two associations:
\[k_1=(1,0)\mapsto v_1=(1,0),\qquad k_2=(0,1)\mapsto v_2=(0,1).\]Start from a zero matrix $W_0$. The additive writes produce
\[W_1=W_0+k_1^\top v_1= \begin{bmatrix}1&0\\0&0\end{bmatrix},\qquad W_2=W_1+k_2^\top v_2= \begin{bmatrix}1&0\\0&1\end{bmatrix}.\]A query $(1,0)$ selects the first row, returning $(1,0)$. A query $(0,1)$ selects the second row, returning $(0,1)$. The same matrix supplies two different answers. A query $(0.5,0.5)$ mixes the rows and returns $(0.5,0.5)$; no discrete dictionary lookup is being performed.
Because $W_t$ changes while we process the sequence, its entries are called fast weights. The projections that generate keys and values are called slow weights when they are learned across training sequences and reused within one sequence. Fast versus slow describes the update timescale, not necessarily the learning-rate magnitude or permanent versus temporary storage.
The 2021 Fast Weight Programmers paper makes this connection between linear attention and dynamically written weight matrices explicit. Its next question is more important than the name: what should happen when a key’s association needs correction? Schlag, Irie & Schmidhuber
3.2 First Measure What the Memory Predicts Incorrectly
3.2 先测量 Memory 现在预测错了什么
Suppose the first association changes: $(1,0)$ should now return $(0,1)$. Adding the new value to the old matrix would make its first row $(1,1)$, not $(0,1)$. Repeating an unchanged association has a related problem: an unnormalized additive memory keeps increasing its amplitude. The normalized averaging method does not have that amplitude problem, but repeated contradictory values are averaged rather than explicitly replaced.
The remedy is to ask the old memory what it already predicts before writing. For the incoming pair $(k_t,v_t)$, define
\[\widehat v_t=k_tW_{t-1},\qquad e_t=\widehat v_t-v_t.\]The hat marks a prediction. $e_t$ is the error vector, with one predicted-minus-target difference per value coordinate. In the correction example, $\widehat v_t=(1,0)$, the target is $(0,1)$, and $e_t=(1,-1)$. We want to reduce the first output coordinate and increase the second.
To judge a proposed memory matrix $W$ with one number, square the coordinate errors and add them:
\[\ell_t(W)=\frac12\sum_{b=1}^{d_v}\big[(k_tW)_b-v_{t,b}\big]^2 =\frac12\lVert k_tW-v_t\rVert_2^2.\]This number is the local loss. Squaring prevents positive and negative errors from cancelling. Summing requires the whole output, not just one coordinate, to match. The notation $\lVert u\rVert_2^2$ means the sum of squares of the entries of $u$. The factor $1/2$ simplifies the derivative without changing which $W$ minimizes the loss.
Here $W$ is the candidate variable being judged; $W_{t-1}$ is its current value. That distinction matters: a formula for the loss describes many possible matrices, while an update chooses a new matrix from the current one.
3.3 Turn the Error Into a Change of Weights
3.3 怎样把输出误差变成 Weight 的修改？
The error says which output coordinates are wrong. It does not yet say which matrix entries to change. Consider entry $W_{ab}$. Output coordinate $b$ is
\[(k_tW)_b=\sum_{a=1}^{d_k}k_{t,a}W_{ab}.\]A small change in $W_{ab}$ changes that output in proportion to $k_{t,a}$. If this key coordinate is zero, changing $W_{ab}$ does not affect the current prediction at all. If it is large, the same weight change has a larger effect.
The partial derivative measures how the loss changes when we vary one weight entry while holding the other entries fixed. Differentiating the half-square therefore gives, at the old matrix,
\[\frac{\partial\ell_t}{\partial W_{ab}}=k_{t,a}e_{t,b}.\]Collect every partial derivative in a matrix of the same shape as $W$. This is the gradient, written $\nabla_W\ell_t$. Our entry formula is exactly an outer product:
\[g_t=\nabla_W\ell_t(W_{t-1})=k_t^\top e_t.\]The small positive learning rate $\eta_t$ sets how strongly to act on this gradient. Subtracting the gradient moves against the locally increasing direction of the loss:
\[W_t=W_{t-1}-\eta_tg_t =W_{t-1}+\eta_tk_t^\top(v_t-\widehat v_t).\]This is the delta rule: write a correction based on the difference between the desired and predicted values. If the prediction is already right, the error and this write are zero. This is not “add the whole target again.” The full fast-weight paper has additional feature and normalization choices; the equation here isolates its error-correcting idea. Fast Weight Programmers, §4.2
For the two-row example, $k_t=(1,0)$ and $e_t=(1,-1)$ give
\[g_t=\begin{bmatrix}1&-1\\0&0\end{bmatrix}.\]With $\eta_t=1$,
\[W_{\mathrm{new}} =\begin{bmatrix}1&0\\0&1\end{bmatrix} -\begin{bmatrix}1&-1\\0&0\end{bmatrix} =\begin{bmatrix}0&1\\0&1\end{bmatrix}.\]The corrected key $(1,0)$ now reads $(0,1)$. The other key $(0,1)$ still reads $(0,1)$. This particular correction changed only the first row.
Why does subtracting the gradient help, and is the step always safe? (Click to expand)
A derivative is a local slope. In one variable, a positive slope means a small positive change raises the loss, so a small negative change lowers it. The gradient collects those slopes across all weight entries. This is the local argument behind gradient descent, developed in Basics of Optimizers.
It is not a guarantee for arbitrarily large steps. For this linear, single-example squared loss, multiplying the update by the training key gives
\[e_{\mathrm{new}}=(1-\eta_t\lVert k_t\rVert_2^2)e_{\mathrm{old}}.\]The current error shrinks when $0<\eta_t\lVert k_t\rVert_2^2<2$. It becomes exactly zero when the product equals 1. A unit-length key with $\eta_t=1$ is the example above; the same learning rate can overshoot for a longer key. These conditions concern this one local quadratic loss, not global LM convergence.
3.4 Why Can a Correct Write Damage an Older Read?
3.4 为什么写对新信息，也可能影响旧记忆？
The two keys above were orthogonal: their dot product is zero. They addressed separate rows. Real learned keys need not be so neatly separated.
Hold an old query $q$ fixed. The new write changes its output by
\[\Delta z=q(W_t-W_{t-1}) =\eta_t(qk_t^\top)(v_t-\widehat v_t).\]Read the three factors separately. The last vector is the correction desired for the new key. The scalar $qk_t^\top$ measures how much the old query overlaps that key. The learning rate controls the overall strength. A zero overlap protects this old read from this particular delta update. A nonzero overlap allows the correction to spill into it.
For example, take the new key $(0.8,0.6)$. Its dot products with the old probes $(1,0)$ and $(0,1)$ are $0.8$ and $0.6$, so neither probe is protected. This is interference: multiple associations use overlapping directions in the same memory. It occurs even if the projections that created the keys remain frozen.
In the lab, leave $\eta=1$ and select Delta correction. The first two writes build the two rows. The third replaces the first target. The fourth introduces the overlapping key. Compare the probe bars before and after that fourth write. Switching to additive storage shows why correction differs from accumulation. The momentum-and-decay option will be explained in Section 5; it is a small update-rule demonstration, not a full Titans implementation.
4. TTT Layers: Put a Learning Procedure Inside the Forward Pass
4. TTT Layers：把学习过程放进 Forward Pass
4.1 Have We Replaced the Hidden State, or Renamed It?
4.1 这里是替换了 Hidden State，还是重新组织了它？
Recall the definition from Section 1: a state carries information from earlier inputs to later computation. The matrix $W_t$ does exactly that. It contains the accumulated effect of previous writes, and later queries use it. Therefore $W_t$ is a hidden state whose entries happen to be used as model weights.
The same numbers have two roles at two levels. To the small function $f(q;W)=qW$, they are parameters that define its mapping. To the surrounding sequence model, they are temporary state passed from one position to the next. There is no contradiction.
A conventional RNN might carry a vector $h_t$ and use a fixed readout $G_\Theta(h_t,q)$ to answer a query. Our matrix carries a linear mapping and answers with $qW_t$. Both can return different answers for different queries. The advantage under discussion is an explicit way to organize and update associations, not a capability that vector states are forbidden to have.
For example, flatten a matrix by listing its entries:
\[W=\begin{bmatrix}w_{11}&w_{12}\\w_{21}&w_{22}\end{bmatrix}, \qquad \operatorname{vec}(W)=(w_{11},w_{12},w_{21},w_{22}).\]This operation, called $\operatorname{vec}$, loses nothing. An update function could reshape that vector, perform the gradient write, and flatten it again. The computation is still a recurrent update. With momentum, its carried update history must be included in the full state too.
Consequently, “vector stores a representation, weights store a function” is a helpful organization metaphor, not an absolute mathematical boundary. A sufficiently expressive recurrence can implement the same update. Choosing the learner explicitly supplies an inductive bias: a preferred family of computations that training does not have to discover from scratch.
Does matrix state automatically have more capacity or higher cost? (Click to expand)
A vector in $\mathbb R^D$ has $D$ entries. A matrix of shape $[d_k,d_v]$ has $d_kd_v$ entries, or $d^2$ in the square case. Those counts describe storage at a chosen precision; they are not counts of guaranteed retrievable facts. A vector could also have $D=d_kd_v$, and recurrent models need not all use vector states.
A bare matrix read and delta write require work proportional to $d_kd_v$. A conventional dense vector recurrence can itself have quadratic cost in its width. A nonlinear memory may require substantially more work. Shape alone does not establish that one family is always faster, larger, or better at remembering. Interference and the structure of the queries matter as well.
4.2 Replace the Linear Mapping With a Small Learner
4.2 把 Linear Mapping 换成一个小学习器
Our matrix memory can only implement a linear rule: the response to a sum of queries is the sum of their responses. If useful key–value relationships require interactions that a linear map cannot express, we can choose a more flexible function.
Let $f(u;W)$ be a small model taking an input vector $u$ with $d_k$ coordinates and returning $d_v$ coordinates. It could be a matrix multiplication or a multilayer perceptron (MLP): several learned linear transformations separated by nonlinear activation functions. The nonlinearities allow the combined function to do more than a single linear transformation.
Now $W$ means the entire collection of fast parameters, possibly multiple matrices and biases. We still know how to write: ask the model to predict the value from the key, measure the error, and take a gradient step. We still know how to read: run the model on a query.
\[\ell_t(W)=\frac12\lVert f(k_t;W)-v_t\rVert_2^2,\] \[W_t=W_{t-1}-\eta_t\nabla_W\ell_t(W_{t-1}), \qquad z_t=f(q_t;W_t).\]The subscript on $\nabla_W$ specifies the variables being changed. If $W$ contains multiple tensors, the gradient contains one same-shaped tensor for each, and the update is applied to each tensor. It does not instruct us to update every weight in the surrounding LM.
This is the core viewpoint of TTT Layers: the recurrent state is a learner, and its transition is a self-supervised learning step. The sequence layer’s forward pass—the computation from input to output—now contains a small training operation. That operation still runs on a test sequence, hence “Test-Time Training.” Sun et al., §§2.1–2.3
Why write with $k_t$ but read with $q_t$? Writing asks, “what association should this input teach the memory?” Reading asks, “what information is useful for the current output?” A diary entry and a question about the diary need not have the same form. Separate learned projections let the two roles differ; the read query must nevertheless be learned to work with the mapping trained on keys.
In our token-wise schedule, the current token is written before it is read. This is causal when the resulting feature helps predict the next token: the current input is already known. It would be leakage to use the unknown next token in that write and then claim to predict it. Other architectures can arrange reads and writes differently; timing is part of the algorithm.
4.3 Who Supplies the Correct Keys and Values?
4.3 谁提供“正确的”Keys 和 Values？
So far, numerical keys and values were supplied by us to make the mechanics visible. A real TTT layer does not receive hand-labeled memory records. It constructs both from the current hidden vector:
\[k_t=x_tW_K,\qquad v_t=x_tW_V,\qquad q_t=x_tW_Q.\]These are three views of the same input: three transformations emphasizing potentially different information. The training view $k_t$ is the small learner’s input. The label view $v_t$ is the target it tries to predict. The test/read view $q_t$ is used to obtain the output. In this terminology “test view” does not mean it is unavailable during training.
There is no external ground-truth key. The value is a generated continuous target, sometimes loosely called a pseudo-label, not an externally certified answer. Self-supervised means the target is constructed from observed data itself. We can make the key and value now, without waiting for a future token or asking a human to label the document.
But why should the generated target be useful? The local reconstruction loss cannot answer that. For a sufficiently simple model, setting every target to zero and making the learner output zero gives zero reconstruction loss, yet remembers nothing useful. This is a collapsed solution: a trivial representation solves the local objective while discarding the intended information.
We need a second criterion that evaluates the whole LM. Let $\Theta$ collect its slow parameters: the projections, the remaining layers, and, when learned, the initial fast state and step-size rule. After the memory produces $z_t$, the remaining network produces vocabulary logits and a next-token distribution. Denote that probability model by $p_\Theta$. The observed next token $a_{t+1}$ supplies the target for
\[\mathcal L_t=-\log p_\Theta(a_{t+1}\mid a_{\le t};W_t(\Theta)).\]The probability is evaluated at the actual next-token ID. Taking its negative logarithm penalizes assigning little probability to the observed continuation. For example, probability $0.8$ gives a smaller loss than $0.2$. Summing or averaging these terms gives the LM cross-entropy objective; background on this conditional prediction loss is in Use of Information Theory in Learning Theory, §2.1.
Writing $W_t(\Theta)$ emphasizes that the memory state depends on the slow parameters used to construct its learning process. It is not extra ground truth supplied to the model.
| Question | Inner reconstruction | Outer LM prediction |
|---|---|---|
| What is predicted? | Value view from key view | Next token from observed prefix |
| What supplies the target? | $v_t=x_tW_V$, generated from current input | $a_{t+1}$, observed in the training text |
| What is optimized? | Sequence-specific fast state $W$ | Slow parameters $\Theta$ |
| What does success mean? | Better fit to this local association | Better prediction by the complete LM |
Both tasks are self-supervised, but at different levels. The next-token “ground truth” is an observed token, not a claim that every statement in the corpus is factually true.
Outer training can favor targets that preserve information useful later—for instance, entity identity when it helps predict a later reference. No human needs to name that feature. However, the outer objective is not a proof that collapse is impossible: other network paths may bypass memory, gradients may be unhelpful, or optimization may stall. It provides a usefulness criterion, not guaranteed success.
4.4 Why Are There Two Loops Rather Than Two Updates Per Token?
4.4 为什么叫两层 Loop，而不是每个 Token 更新两次？
There are three different actions that a short diagram can accidentally conflate: compute a loss, compute its gradient, and change parameter values. Producing $\mathcal L_t$ at each token does not mean taking an outer optimizer step at each token.
Use $t$ for positions within a sequence and $n$ for outer optimizer steps. For this teaching schedule, take $T+1$ token IDs. The first $T$ are inputs; the extra token supplies the last next-token target. During outer step $n$, all slow parameters stay at $\Theta_n$ while each independent sequence builds its own trajectory:
\[W_0^{(n)}\longrightarrow W_1^{(n)} \longrightarrow\cdots\longrightarrow W_T^{(n)}.\]The superscript $(n)$ is a step label, not exponentiation. At each arrow, one inner update modifies $W$. At each position, the model also reads the state and computes a next-token loss contribution. After processing the batch—a collection of sequences used together for one training update—we average those contributions and then change $\Theta$.
For example, a batch of two sequences, each containing four IDs, has three inputs and three targets per sequence. It produces two independent three-step memory trajectories, six next-token loss terms, and one outer optimizer step. It does not run one six-step memory across both examples.
An illustrative SGD outer update is
\[\Theta_{n+1}=\Theta_n-\eta_{\mathrm{outer},n} \left.\nabla_\Theta\mathcal L_{\mathrm{batch}}(\Theta)\right|_{\Theta=\Theta_n}.\]$\eta_{\mathrm{outer},n}$ is the outer learning rate, distinct from the inner rate $\eta_t$. The vertical bar means evaluate the derivative at the current slow parameters. The next batch starts with the updated $\Theta_{n+1}$, including an updated $W_0$ if it is learned.
“Inner” and “outer” therefore refer to different variables and objectives, and to a dependency: the outer objective evaluates the outcome of the inner learner. Even one inner step could form a nested problem. A long token-wise trajectory just makes that relationship more visible. Gradient accumulation can combine several batches before an outer optimizer step; chunking can alter the inner schedule. Neither variation changes the need to distinguish the two levels.
4.5 How Can a Loss Train the Rule That Produced an Update?
4.5 Loss 怎样训练“产生 Update 的规则”？
It may sound as if a gradient step is an opaque instruction that blocks further differentiation. In the linear example, it is visibly ordinary arithmetic:
\[W_t=W_{t-1}-\eta_tk_t^\top(k_tW_{t-1}-v_t).\]Change $v_t$ slightly and the new $W_t$ changes. Change $W_t$ and a subsequent read changes. Change the read and the downstream prediction loss can change. That chain is how the LM objective can train the value projection.
A one-number example makes the dependence explicit. Set the old memory to zero, use $k=q=1$, let the generated target be $v=\beta$, and choose $\eta=1/2$. Here $\beta$ is a scalar standing in for a target produced by slow parameters. The write and read become
\[W_1=0-\tfrac12(0-\beta)=\tfrac12\beta,\qquad z=qW_1=\tfrac12\beta.\]Increasing $\beta$ by $0.1$ increases the read by $0.05$. Therefore any differentiable downstream loss satisfies
\[\frac{d\mathcal L}{d\beta} =\frac{d\mathcal L}{dz}\,\frac12.\]The first factor tells us whether changing this output helps the real prediction; the second tells us how changing the generated target changes that output. We did not optimize $\beta$ to make the reconstruction target easy. We evaluated how its effect on the write changes the outer loss.
The same logic applies to $W_K$, $W_Q$, the step-size rule, and the initialization. During the inner partial derivative, $k_t$ and $v_t$ are held fixed while differentiating with respect to $W$. During outer differentiation, the computed update still depends on the projections that produced $k_t$ and $v_t$. Holding a target fixed for one partial derivative does not justify deleting its outer gradient path.
Following the dependence through many writes (Click to expand)
Let $w_t=\operatorname{vec}(W_t)$ flatten the complete fast state. Write an update as $w_t=U_\Theta(w_{t-1},x_t)$ and let $J_t=dw_t/d\Theta$. The entries of $J_t$ record how each state entry changes with each slow parameter. By the chain rule,
\[J_t=A_tJ_{t-1}+B_t.\]$A_t$ is the derivative of the update with respect to the previous state. It propagates effects that arrived through older writes. $B_t$ collects dependence on $\Theta$ and on the current $x_t$ while the previous state is held fixed. If $W_0$ is learned, $J_0$ contains that initial dependence; if other adaptive states produce $x_t$, include those states in the full recurrence.
Backpropagation through time follows this expanded sequence of state transitions backward. Unrolled optimization describes the fact that some transitions are learning steps. Meta-learning describes the fact that their outcomes train the learning procedure. These are related views of the same computation, not three extra algorithms that must be run.
In code, detaching each state cuts the dependence on earlier writes. That can be an intentional approximation, but is not the full unrolled gradient. For a general inner model differentiated with an automatic-differentiation tool, outer training needs the gradient computation itself to remain differentiable. The analytic linear update in our snippet already expresses it using ordinary tensor operations.
4.6 What Stops Updating at Inference, and What Does Not?
4.6 Inference 时，哪些更新停止，哪些继续？
After outer training, consider inference with a fixed checkpoint $\Theta^\ast$. We stop the outer optimizer. The Q/K/V projection matrices, surrounding LM, and learned initialization no longer receive persistent optimizer updates.
We do not stop forming new $q_t,k_t,v_t$, because each new input produces new views. We also do not stop the inner learning procedure: start from $W_0$ and run $W_0\to W_1\to\cdots$ as the sequence arrives. The distinction is between fixed projection parameters and changing projected activations.
Why design this separation? It lets a reusable learning procedure adapt a sequence-specific state without directly rewriting the base checkpoint for every document. It localizes the update and can limit adaptation cost. Yet nonlinear inner gradients and outer training through them may still be expensive; a constant-size inference state does not make training free.
The separation has limits. Frozen projections do not eliminate interference inside memory. They do not make every address invariant, because the hidden input $x_t$ can change with context and earlier adaptive layers. Resetting $W$ protects independence between new sequences, but discards their transient memories; carrying it across documents is a separate policy, not automatically lifelong learning or appropriate cross-user sharing.
“No next-token label” also requires precise timing. At the instant we predict an unknown future token, its external label is unavailable. The current-token reconstruction target is available immediately. But an observed prefix already supplies next-token targets for earlier positions, so test-time LM fine-tuning is possible. The lack of an unknown future label does not prove QKV must be frozen; it explains why this particular local target is convenient.
A final implementation detail matters: “frozen slow parameters” does not mean “no derivatives anywhere.” A nonlinear learner may need automatic differentiation to compute its local $W$ gradient at inference. Our linear snippet can explicitly calculate that gradient, so disabling graph recording does not stop its arithmetic write.
4.7 What Does Published TTT Add to the Teaching Example?
4.7 论文 TTT 在教学例子上还加了什么？
We now have enough vocabulary to read the implementation choices without treating them as unexplained names. TTT-Linear and TTT-MLP differ in the inner model. The practical learner also uses a residual path and LayerNorm: roughly, it adds a normalized learned transformation to its input. The bare $qW$ example does not include those operations, so it is not the complete published TTT-Linear.
The initialization $W_0$ can be learned in the outer loop. Every independent sequence begins with the same learned starting point but then develops different fast weights. The inner rate can also depend on the current input through a learned rule. For example, a sigmoid converts a scalar score into a number between zero and one, which can multiply a positive base rate. The rule’s parameters are slow; its token-specific rate is not. The normalization and residual concepts are developed in Normalizations. TTT implementation choices, §2.7
A remaining practical obstacle is sequential dependence. If every gradient uses the immediately preceding updated state, computing token 2’s gradient must wait for token 1’s update. GPUs favor doing many similar operations together. Mini-batch TTT groups tokens into chunks and evaluates gradients at a shared chunk-start state, then accumulates the appropriate prefix of writes. It is a different schedule from fully online gradient descent.
A two-token example: why chunking changes the update schedule (Click to expand)
Take scalar memory $w_0=0$, scalar keys both 1, targets $v_1=1,v_2=3$, and rate $\eta=0.5$. The local gradient is $w-v$.
Fully online updates give $g_1=-1$, $w_1=0.5$, then $g_2=0.5-3=-2.5$ and $w_2=1.75$. In a shared-start two-token chunk, both gradients use $w_0$: $g_1=-1,g_2=-3$. Accumulating them gives $w_2=0-0.5(-1-3)=2$. The two procedures are not identical.
For causality, the first token’s read must use only its prefix of updates, giving $w_1=0.5$, not the chunk-final $w_2=2$. Computing gradients together does not authorize future writes to affect earlier predictions.
The paper’s dual form is an algebraic organization of the chosen chunked computation using matrix operations, avoiding explicitly constructing every intermediate parameter matrix. It accelerates that schedule; it does not magically make all possible update schedules equivalent. Mini-batch TTT and dual form, §§2.4–2.5
4.8 Read the Complete Training and Inference Code
4.8 连起来看完整 Training 与 Inference 代码
The first tab implements our bare linear memory for one sequence of hidden vectors. Follow the prediction, error, gradient, write, and read in that order. The second adds momentum and decay, discussed next.
The “Outer training” tab constructs a minimal embedding–memory–vocabulary-head model and shows the batch, sequence, and token loops explicitly. The matrix $W$ is reset separately for each sequence. The target $v$ appears inside the memory write; the observed next token appears inside cross-entropy. There is one outer optimizer step after the batch’s losses have been averaged.
The “Inference” tab reuses the same linear learner on an observed prefix but does not update slow parameters. It returns next-token logits and the final fast state. This helper replays the prefix from its starting state; efficient streaming would carry the state forward explicitly. None of these small examples includes a published paper’s full backbone or optimized chunked kernels.
Every marked expression has a nearby hover/focus annotation explaining its role and shape. Click to keep the annotation open; press Escape to close it.
5. Titans: How Long Should a Write Keep Affecting Memory?
5. Titans：一次写入应该影响记忆多久？
5.1 From the Current Error to a History of Updates
5.1 从当前误差到一段更新历史
A plain gradient write reacts to the current pair. After computing $g_t$, it adds only $-\eta_tg_t$ to memory. This raises another question: should a recent trend in the writes continue to influence the next update, instead of being discarded immediately?
Introduce a second fast state $U_t$, with the same shape or parameter structure as $W_t$. It stores the update, not the key, value, or memory prediction. Combine a fraction of the previous update with the new gradient contribution:
\[U_t=\mu_tU_{t-1}-\eta_tg_t,\qquad U_0=0.\]$\mu_t$ is the retention factor for the previous update; $\eta_t$ scales the current gradient. This is a momentum-style rule. With $\mu_t=0$, only the new gradient contributes. With $\mu_t=0.9$ and a zero current gradient, $0.9U_{t-1}$ still remains. Thus influence can continue across nearby sequence positions.
Repeated expansion shows what is carried. For constant $\mu$ and $\eta$, $U_t=-\eta(g_t+\mu g_{t-1}+\mu^2g_{t-2}+\cdots)$ when starting from zero. Older gradients have undergone more decay; gradients pointing against each other can cancel. With input-dependent coefficients the weights become products of the intervening retention factors. This is a weighted history of update evidence, not a second exact store of old token records.
Titans uses an associative-memory objective together with this kind of gradient-history mechanism. The paper calls its gradient signal surprise. In our notation the differentiated variable is the memory’s parameters. Titans, §3.1
Why can prediction error be a useful writing signal? A memory that already predicts a target well needs little correction. A mismatch can call for change. But large gradient does not mean “semantically important fact”: it also depends on input scale and the function’s sensitivity. Noise can produce a large gradient; a large loss in a flat region can produce a small one. This is a model-dependent signal, not a relevance oracle.
Nor is it automatically Shannon surprisal, $-\log p(a_t\mid a_{<t})$. That quantity comes from a token probability; this gradient comes from a memory reconstruction objective. The shared word does not make them the same measurement. Basics of Optimizers discusses momentum as optimization; here its running index is the incoming sequence.
5.2 Keeping Update Momentum and Keeping Old Memory Are Different
5.2 保留更新趋势，与保留旧记忆，不是一回事
Momentum says how much of the previous update to retain. It does not directly specify how much of the existing memory to retain. To control the latter, introduce a forgetting coefficient $\alpha_t$ between zero and one:
\[W_t=(1-\alpha_t)W_{t-1}+U_t.\]Before adding the new update, this multiplies the old parameter values by a retention factor $1-\alpha_t$. With $\alpha_t=0$, no explicit parameter decay is applied. With $\alpha_t=0.1$, old values retain 90% of their magnitude before the update. With $\alpha_t=1$, the old $W$ contribution disappears, but $U_t$ may still be nonzero.
Together the teaching recurrence is
\(g_t=\nabla_W\ell_t(W_{t-1}),\) \(U_t=\mu_tU_{t-1}-\eta_tg_t,\qquad W_t=(1-\alpha_t)W_{t-1}+U_t.\)
The gradient is evaluated at the old memory, before the decay shown here. Our $\mu_t$ names the momentum coefficient and $\eta_t$ the gradient step size; these correspond to different symbols in the Titans paper. The distinction avoids confusing a paper-specific letter with a universal meaning. Titans, Eqs. 12–14
Use a scalar example to separate their effects. Suppose old memory is $W_{t-1}=2$, the previous update is $U_{t-1}=0.2$, and the new gradient is $g_t=-0.4$. Take $\mu_t=0.5$, $\eta_t=0.1$, and $\alpha_t=0.1$. Then $U_t=0.5(0.2)-0.1(-0.4)=0.14$, and $W_t=0.9(2)+0.14=1.94$. The gradient and momentum contribution is positive, yet the final memory value decreases because decay is stronger. The three coefficients do different jobs.
Why permit forgetting at all? A finite memory may receive corrected, obsolete, or unrelated information. Always preserving all accumulated influence can make new associations hard to represent. A learned gate can regulate retention as context changes. “Gate” here means a computed multiplier, often constrained to a range such as $[0,1]$; it is not a literal database delete operation.
Consequently, multiplying weights by a scalar does not selectively erase exactly one named fact, and $\alpha_t=0$ does not guarantee no forgetting: new writes can still interfere. Input-dependent gates can learn useful behavior, but the equations do not prove semantic selectivity or perfect long-term recall.
5.3 How Does This Memory Fit Into a Language Model?
5.3 这种 Memory 怎样放进 Language Model？
A writing rule is not a complete architecture. After defining the adaptive memory, we must decide what feeds it and how its output is combined with other information. Titans distinguishes three roles:
- Recent-context attention: explicitly match against representations in the currently retained local window. A local window bounds how many recent positions are considered.
- Adaptive neural memory: store older associations in a parameterized function that continues to receive writes. It can use a deeper MLP rather than only a bare linear matrix.
- Persistent learned memory: input-independent learned entries shared across sequences. These are trained slow parameters, not a continuously updated record of the current user’s conversations.
“Long-term” in the second role is relative to the local attention window: its state can carry effects from earlier segments. It does not itself promise persistence across independent runs. A deeper function expands the family of key–value mappings, but not to unlimited, error-free capacity.
To combine the components, the paper studies Memory as a Context (MAC), Memory as a Gate (MAG), and Memory as a Layer (MAL). These answer a wiring question: present retrieved memory as additional context for attention, combine memory and attention through gating, or compose them as layers. They are architecture choices, not three names for the same recurrence. Titans, §§3.3–4
Walk through MAC at the level of one segment, a consecutive block of tokens. First, the current segment supplies queries to the memory left by previous segments. The retrieved vectors represent relevant historical information. Next, attention receives those vectors, the persistent entries, and the current segment. The attention-produced representations are then used in the adaptive memory update and in constructing the output.
That order differs from the simple “write this token, then read it” cell in Section 4. The old memory first helps interpret the current segment; the resulting computation then helps determine what to write. Exact masks and state versions still matter: an earlier output cannot be allowed to consult a chunk-final state containing future-token writes. A segment-level diagram does not waive token-level causality.
The design balances two forms of access: explicit recent records and compressed historical associations. How well that balance works depends on the learned representations, available state and update costs. The formulas alone do not settle it.
5.4 What Has Improved—and What Has Not Been Guaranteed?
5.4 我们解决了什么，又没有保证什么？
Return to the original question: where does the document’s information go? Full attention can retain individual K/V entries; a recurrent summary combines them into fixed-size state; a delta learner corrects a stored mapping; TTT makes that mapping a trainable model; Titans additionally studies how update history, forgetting and attention cooperate.
| Paper | Problem reached in our derivation | Main mechanism |
|---|---|---|
| Linear Transformers, 2020 | Avoid scanning a growing list with a suitable similarity | Recurrent numerator matrix and normalization vector |
| Fast Weight Programmers, 2021 | Correct an existing association instead of only adding | Error-based write into fast weights |
| TTT Layers, 2024 | Choose a richer state and learn what its local task should be | Inner learner, learned views, outer LM objective |
| Titans, 2025 | Manage continuing writes and combine history with local context | Momentum, decay, neural-memory/attention integration |
These are conceptual connections, not a proof that each later method uniformly dominates the earlier ones. They also describe different meanings of “learning”:
In ordinary in-context learning, outputs change with the input context without gradient updates to the model parameters. Its KV cache can still change. In test-time memory learning, selected fast weights also change during the sequence. In continual pre-training, optimization continues on new data and changes persistent model parameters. Continuing pre-training is not the same mechanism as the per-sequence fast-weight updates studied here; sharing the word “continual” does not establish that connection.
To establish lifelong retention, we would additionally need to state what persists across tasks, what old capabilities must be retained, and how they are measured after new learning. Transfer asks whether learning one task helps another. A replay method revisits stored or regenerated old examples; a fair comparison must account for that storage and computation. Those are further design and evaluation questions, not consequences of a local reconstruction loss decreasing.
The core insight is therefore precise: a model’s weights can serve as its sequence state, and a learning algorithm can serve as its state-update rule. The outer loop learns a reusable way to create addresses, targets and queries; the inner loop executes that procedure on the current sequence. This explains the different QKV update roles without assuming all continual-learning methods freeze the same parameters—or that a temporary memory has become permanent knowledge.