the transformer
is not excused

1. Which thing is compact?

A state is a point. An operation is a map. A matrix of attention weights is a third object: it depends on the state and participates in the operation. An operator on observables is a fourth. Moving a theorem from one object to another without its hypotheses is an excellent way to make nonsense look educated.

Fix sequence length n, residual width d, finite weights, fixed positional information and masks, and a finite computational graph. Write a state as a row matrix X with rows x_i. Unless otherwise stated, vector norms are Euclidean, matrix weight norms are induced Euclidean operator norms, and state distance is the maximum row distance:

For a nonempty compact metric state space K, write C(K,K) for its continuous self-maps. This chapter uses the uniform metric, with the distance of the underlying state space denoted by d_K:

Composition means f o g, with g acting first. A semigroup is composition-closed; a monoid also contains the identity. The generated monoid contains every finite word in the selected generators, including the empty word. Its closure below always means closure in this uniform metric. Matrix families instead use their usual finite-dimensional topology and matrix multiplication.

A compact family need not be composition-closed. A finite family is compact automatically, while its words can fail even to be equicontinuous. A stochastic monoid is not a real algebra under unrestricted addition and scaling. Keep the nouns attached.

2. The finite transformer has compact attained states

Let E = {e₁, ..., e_M} be a finite set of embedding vectors in R^d, and let p_i be the fixed positional row at position i. Relax each discrete token to a convex combination of embeddings:

Proposition 1. K₀ is compact and convex. At every continuous node of a fixed finite transformer graph, the set attained from K₀ is compact.

Proof. Each coefficient vector belongs to the closed probability simplex in R^M, which is bounded and closed, hence compact by Heine-Borel. A finite product of these simplexes is compact. The displayed affine map is continuous, so its image K₀ is compact; affine maps preserve convexity. Each graph node is a continuous function of the original input, by induction in a topological ordering of the graph. Its image of K₀ is therefore compact. For a node with multiple inputs, apply this to their joint map from K₀, preserving their correlations. This proves the claim without pretending that independently attained intermediate values can be combined arbitrarily.

Linear maps, finite sums and products, concatenations, continuous activations and softmax on a fixed nonempty support are continuous. LayerNorm and RMSNorm with a strictly positive stabilizer are continuous everywhere. Disable dropout or fix its realization. Every attention row must permit at least one key; all permitted biases must be finite. Hard token selection, input-dependent mask changes and other discontinuous choices require their own analysis.

Consequently residual states, logits and attention matrices have compact attained sets at every fixed node. Finite unions of such state sets at a common width remain compact. This does not prove that a block sends that union into itself when reapplied, nor that depths unbounded in advance have a common bound. Even the map x to x + 1 has compact images at each finite depth and an unbounded orbit.

This proof uses an infinite continuum of relaxed inputs. Counting token strings or floating-point states is unnecessary. Fixed context is substantive: a union over all sequence lengths lives in changing dimensions, and arbitrary-depth claims are not finite-depth claims wearing a longer coat.

3. Put numbers on the fence

The previous proof establishes existence of bounds. The following derivations make them explicit. They apply to the scaled dot-product architecture introduced by Vaswani and colleagues; they do not estimate any particular checkpoint.

3a. One head

For a head of key width d_k, let q_i = x_iW_Q + b_Q, k_j = x_jW_K + b_K, and v_j = x_jW_V + b_V. Scores on permitted positions are s_ij = q_ik_j^T/sqrt(d_k) + b_ij. Let A(X) be their row softmax, with zero on forbidden positions, and H(X) = A(X)V(X).

On the row ball ||X||_row <= R, define

Then |s_ij| <= S. If row i permits m_i keys, every permitted entry obeys

Proof. Divide the softmax denominator by exp(s_ij). Each remaining ratio exp(s_ik - s_ij) is at most exp(2S). Forbidden entries stay zero; the floor is only on the permitted support.

For two states in that ball, the bilinear product difference gives a common score Lipschitz bound

Indeed, subtract q_i(X)k_j(X)^T - q_i(Y)k_j(Y)^T by changing one factor at a time. Fixed biases cancel from the difference.

Softmax lemma. Softmax is 1-Lipschitz from the maximum norm on logits to the sum norm on probabilities, independently of the number of permitted keys.

Proof. At a probability vector p, its derivative in direction h has coordinate p_j(h_j - sum_kp_kh_k). Its sum norm is E|H-EH| for a random variable taking h_j with probability p_j. Cauchy-Schwarz bounds this by sqrt(Var H). If |H| <= ||h||_infty, then Var H <= E(H²) <= ||h||_infty². Integrate the derivative along the line segment between the logits. The same argument applies on each fixed mask support.

Thus the score constant just defined also controls weights:

Since each output row is a convex combination of value rows, ||H(X)||_row <= v_R. Splitting the difference into A(X)[V(X)-V(Y)] + [A(X)-A(Y)]V(Y) gives

The later zero-sum lemma improves v_R here to half the diameter of the value rows at Y. That is the hinge on which the all-depth theorem turns.

3b. Several heads, an MLP, a residual

If head h has radius bound v_h and Lipschitz bound L_h, their concatenation has radius at most sqrt(sum_h v_h²) and Lipschitz constant at most sqrt(sum_h L_h²). A following output projection multiplies these by ||W_O||; its bias adds ||b_O|| to the radius only.

For a row MLP M(x) = phi(xW₁ + b₁)W₂ + b₂, let its hidden width be m and its scalar activation have Lipschitz constant l on an interval containing zero and all attained preactivation coordinates. Then

A residual map x + H(x) has radius at most R + R_H and Lipschitz constant at most 1 + L_H. A finite stack multiplies its layer Lipschitz bounds and propagates its radius bounds. Every resulting finite number is a valid certificate. Its growth with depth is exactly why it does not certify an infinite family of compositions.

3c. The exact normalization image

Let U = {z : sum_j z_j = 0}, and let P be orthogonal projection onto U. For epsilon > 0, diagonal gains D_gamma and bias beta, define

This is the centered normalization mechanism associated with Ba, Kiros and Hinton's Layer Normalization, with the positive stabilizer explicit. If B_R is the closed Euclidean input ball and B_U(r) the closed radius-r ball in U, its image is exactly

Proof. P(B_R) = B_U(R). On U the map is radial, preserves direction, and sends radius r to r/sqrt(r²/d + epsilon), a continuous strictly increasing function. Its derivative is epsilon/(r²/d + epsilon)^3/2, so the interval of radii [0,R] maps exactly onto the displayed interval.

On the entire unbounded space, N attains the relative open ball in U of radius sqrt(d). Its affine image is generally bounded but not compact; it is compact only in the degenerate constant-image case D_gammaP = 0. Compactness of a restricted attained image does not follow merely from an asymptotic radius fence.

The radial derivative just computed and the tangential derivative 1/sqrt(r²/d + epsilon) show

Proof of sharpness. The radial and tangential factors are at most 1/sqrt(epsilon), so composition with D_gamma restricted to U proves the upper bound. At z = 0 the derivative is exactly D_gammaP/sqrt(epsilon); a global Lipschitz constant cannot be smaller than its operator norm.

LayerNorm forgets the row mean, since LN(z + c1) = LN(z). Positive-epsilon radial normalization itself remains injective on U when all gains are nonzero. Boundedness and information loss are separate questions; the inverse chapter derives the inverse and its boundary conditioning.

4. The matrix envelope keeps its appointments

Proposition 2. Let S_n be all nonnegative n-by-n matrices whose rows sum to one. Then S_n is a compact topological monoid under multiplication.

Proof. Nonnegativity and row-sum equations define a closed set. Every entry belongs to [0,1], so the set is bounded and therefore compact. For A,B in S_n, AB is nonnegative and AB1 = A1 = 1. The identity belongs to S_n. Multiplication is polynomial in matrix entries, hence continuous.

For 0 < alpha <= 1/n, the smaller family S_n,alpha = {A in S_n : A_ij >= alpha for every i,j} is a compact semigroup, because

For n > 1 it excludes the ambient matrix identity. If alpha < 1/n it contains the invertible matrix (1-n alpha)I + alpha 11^T, so any identity for this semigroup would have to be I; in that case it is not a monoid. At alpha = 1/n the family is the singleton {J}, where every entry of J is 1/n, and J is its own internal identity. Adjoining the ambient identity always makes a compact monoid. In contrast, the family with merely A_ij > 0 is not closed: positive matrices can converge to matrices with zeros.

Masks have an algebra too

Let E be the allowed directed pairs (i,j), with every row nonempty. The family S_E of stochastic matrices supported within E is compact. It is multiplication-closed exactly when E is transitive: (i,k) and (k,j) in E imply (i,j) in E. It contains the identity exactly when all diagonal pairs belong to E.

Proof. An entry (AB)_ij can be nonzero only through an allowed two-step path i to k to j, proving sufficiency. For necessity, if such a path ends at a forbidden pair, choose row i of A concentrated on k and row k of B concentrated on j; fill other rows using any permitted keys. Then (AB)_ij = 1 at a forbidden entry. The identity criterion follows directly from its support.

Causal lower-triangular support is transitive and includes the diagonal. A uniform positive floor on every allowed entry need not survive products: for

A masked positivity claim must therefore name the support and the property preserved under multiplication. The full-matrix floor proof cannot be pasted onto masked zeros.

5. Two tokens commit a small algebraic offence

The stochastic envelope is larger than the matrices actually attained by one fixed head. Consider scalar rows x₁, x₂, identity queries and keys, no mask or bias, and A(x) = row-softmax(xx^T). Let p₁, p₂ be the first column of A; the second is (1-p₁, 1-p₂). With sigma(t) = 1/(1+exp(-t)),

Proposition 3: the exact unrestricted attained family. As x ranges over R², the first-column pairs are precisely

Proof. Write logit(p) = log(p/(1-p)). The difference of the two logits is (x₁-x₂)². It is strictly positive unless x₁ = x₂, in which case both logits vanish and both probabilities are 1/2. Conversely, choose any 0 < p₂ < p₁ < 1. Set a = logit(p₁), b = logit(p₂), delta = sqrt(a-b), x₁ = a/delta, x₂ = b/delta. Then x₁-x₂ = delta, and the head attains exactly the requested logits. This proves both inclusions.

Now take two actual inputs:

JQ has equal first-column entries other than 1/2, so Proposition 3 proves it unattainable. Thus T is not composition-closed. The same witnesses work on a compact bounded input square whenever its radius is at least sqrt(log 3). The resulting attained set is compact and still not composition-closed.

The correction that must stay attached: the closure of the unrestricted family is the whole triangle

and this closure is a compact monoid. If A has first column (p₁,p₂) and B has first column (q₁,q₂), then the first column of AB is

Both r_i are convex combinations of q₁ and q₂, so remain in [0,1]. The identity is (1,0). Density of the open triangle proves the stated closure. Failure of attainment is not failure of closure.

For a fixed square [-R,R]², the attained set is already compact and hence closed; it is not this full triangle. The exact inverse construction above shows that a strict pair is attained in that square exactly when max(|a|,|b|)/sqrt(a-b) <= R. Both simultaneous sign choices for x give the same matrix. J is attained for every R >= 0.

Finally, A(X)A(Y) is a product of two matrices. Composing the nonlinear state map F(X) = A(X)X instead gives F(F(X)) = A(F(X))A(X)X. The left factor is evaluated at the intermediate state. Matrix closure alone has not controlled this dependence.

6. A theorem strong enough to compose

Here is a positive result with the depth quantifier where it belongs: outside the bound. Attention may depend nonlinearly on the state. The price is a specific mechanism, uniform overlap, and uniform control of how weights change.

Let C be a nonempty compact convex subset of a finite-dimensional normed space, with diameter D. Set K = Cⁿ with the maximum row norm. For every generator label s, take a stochastic matrix-valued function A_s and define pure identity-value mixing:

Assume there are constants eta > 0 and L_A < infinity, independent of both state and generator, such that

The overlap is common probability mass between any pair of rows. Its complement is the classical Dobrushin contraction coefficient; see Gaubert and Qu for the Markov and consensus framework. The proofs needed here are supplied below. This application and its displayed bounds are not a claim of mathematical novelty.

Lemma 1: disagreement shrinks

Proof. For rows i,k, subtract the common masses c_j = min(A_ij,A_kj). The two remaining nonnegative coefficient lists have the same sum q = 1 - sum_jc_j. If q = 0 the output rows agree. Otherwise divide both lists by q. They define two convex combinations of the input rows, whose distance is at most Delta(X). To verify that last assertion in any norm, write the difference as sum_j,l a_jb_l(x_j-x_l) and use the triangle inequality. Restoring the factor q and maximizing over row pairs proves the lemma.

Lemma 2: a zero sum pays only for disagreement

Proof. The positive and negative parts of c have the same total mass q = (1/2)sum_j|c_j|. For q > 0, divide them by q to obtain two probability vectors; their mixtures are at distance at most Delta(Y), by the preceding convex-hull argument. Multiply by q. For q = 0 every coefficient vanishes.

Theorem 1: compact uniform monoid closure

Under (M1) and (M2), every finite word in the F_s preserves K and is Lipschitz with one common bound. Put r = 1-eta. For a word of length m,

The generated monoid has compact closure in C(K,K) under d_infty, and that closure is a compact topological monoid.

Proof: invariance and decay. Every output row is a convex combination of input rows, so lies in C. Along any word, Lemma 1 gives Delta(X_t) <= D r^t. For r = 0 use r⁰ = 1; after the first step all rows agree.

Proof: sensitivity. For the same generator at X and Y, split

The first term has row norm at most ||X-Y||_row. Each row of the coefficient matrix in the second term sums to zero. Lemma 2 and (M2) therefore give

Apply one fixed word to both initial states. At time t, the second trajectory has diameter at most D r^t. Multiply the one-step factors. The inequality 1+u <= exp(u) and sum_t=0,...,m-1r^t <= 1/(1-r) give the claimed bound, independent of word length.

Proof: compactness and composition. The common Lipschitz bound makes all words equicontinuous, including the identity since M >= 1. Their values lie in compact K. Arzela-Ascoli gives compact uniform closure, with limits still mapping into K. If f_j and g_j converge uniformly to f and g,

where omega_f is a modulus of uniform continuity of f on K. Thus composition is jointly continuous. Approximate any two limit maps by words; their products are words and converge to the product of the limits. The closure is composition-closed and contains the identity.

The proof does not require contraction in the entire state metric. Consensus states X = 1c are fixed exactly, so two distinct consensus states never get closer. What shrinks is the disagreement that multiplies the weight-sensitivity error. The sum of these errors is finite. That is the mechanism, not a blessing bestowed by the word compact.

Softmax supplies the hypotheses on this domain

Take unmasked fixed-parameter softmax generators with identity values on the same compact convex C. The head bounds give a positive entry floor alpha and finite L_A. Then eta(A) >= n alpha, so Theorem 1 applies with M = exp[L_AD/(2n alpha)]. For a finite generator family choose the smallest floor and largest Lipschitz constant; for an infinite family these constants must be uniform by hypothesis.

A causal mask can work too: if every row assigns at least eta to key 1, that shared column alone proves (M1). Other masks require actual overlap analysis. A connected union of support graphs is not by itself the stated one-step overlap condition.

On the antisymmetric orbit (-a,a), direct substitution gives the next state (-a tanh(a²), a tanh(a²)). For a > 0 both magnitudes stay strictly positive at every finite step over the reals. Machine arithmetic can round them to zero. A rounded collision is not a proof of a finite-step real collision.

7. The operation survives. The disagreement does not.

Corollary 1. Fix an infinite word of pure-mixing generators satisfying (M1). There is a row c(X) in C such that

Under (M2), the map P(X) = 1c(X) is M-Lipschitz, belongs to the compact monoid closure, and is a retraction onto the consensus diagonal D_c = {1c : c in C}.

Proof. Let H_t be the convex hull of the rows at time t. Mixing gives H_t+1 contained in H_t. These are nonempty compact sets, and their diameters are at most D r^t. Their intersection is therefore a single point c(X). Every row at time t and c(X) belong to H_t, proving the bound. It is uniform in X, so prefix words converge uniformly to P. Pass their common Lipschitz bound to the limit. Every generator fixes every consensus state; hence P does too, and P² = P.

Every uniform limit of words whose lengths tend to infinity likewise maps into D_c and fixes it. If P and Q are two such limits, P o Q = Q: Q lands where P does nothing. Under our composition convention these limits form a right-zero semigroup. For one repeated generator F, the power-monoid closure is {identity, F, F², ... , P}, and F o P = P o F = P.

For pure causal mixing with the first row fixed and the shared-column condition above, c(X) is exactly the original first row. The theorem then certifies convergence to that row. Whether erasing the others is a successful computation depends on the task.

Consensus is a warning against conflating good forward behavior with recoverability. A limit can collapse distinctions even when finite-time maps in a chosen example remain invertible. See the finite mixing inverse and its growing condition number. The target of reconstruction must be named before contraction is called beneficial.

8. Which numerical errors can be paid for at every depth?

Theorem 2. Let exact generators satisfy Theorem 1. Let approximate generators have the form F~_s(X) = A~_s(X)X, where A~ is stochastic, its overlap is uniformly at least eta~ > 0, and its row-sum weight error is uniformly at most delta_A:

For exact and approximate compositions G_m, G~_m of the same word,

Proof. Telescope by replacing one operation at a time with an approximate prefix and an exact suffix. At step t the approximate prefix state has diameter at most D(1-eta~)^t. Lemma 2 bounds that step's error by delta_AD(1-eta~)^t/2. Every exact suffix has Lipschitz constant at most M. Sum the resulting geometric series.

No Lipschitz assumption on A~ is needed for this inequality, but discontinuous approximate maps do not thereby become members of C(K,K). Nor does an arbitrary additive numerical error automatically shrink with the token diameter. If each step instead has a uniform arbitrary error delta and exact suffixes have Lipschitz bound M, telescoping only gives mMdelta. For a single map with Lip(f) <= L, the usual bound is delta sum_t=0,...,m-1L^t.

If a scalar readout h is L_h-Lipschitz, multiply these state-error bounds by L_h. For vector logits measured in maximum norm, an exact winning-logit margin greater than twice their error bound preserves the argmax. At a tie no such certificate follows. This is a conditional robustness result, not a universal generalization theorem.

9. A compact interval refuses to behave

LayerNorm supplies an exact counterexample to the inference from a compact invariant set to compact composition closure. Restrict width-two normalization to mean-zero rows (t,-t), take unit gains, zero bias, and epsilon = 1/4. The induced scalar map is

The interval [-1,1] is invariant because |g(t)| <= 1/sqrt(5/4) < 1 there. Every iterate is continuous and has compact image. The generator family {g} is compact. Nevertheless, its iterates are not equicontinuous.

Proposition 4. For t nonzero and integer m >= 0,

with the sign of g^m(t) equal to the sign of t.

Proof. If u_m = 1/(g^m(t))², direct substitution gives u_m+1 = 1 + u_m/4. Solve this affine recurrence by a geometric series. The formula also holds at m = 0. Zero remains zero.

Take t_m = 2^-m. These inputs approach zero, but

This directly contradicts equicontinuity at zero. The derivative (g^m)'(0) = 2^m is a useful symptom, but the separated-input calculation proves the claim. Also, for each fixed positive t the iterates tend to sqrt(3)/2; for negative t they tend to -sqrt(3)/2; zero stays zero. Any subsequence with depths tending to infinity has this same discontinuous pointwise limit, so cannot converge uniformly to a continuous map. The powers have no compact uniform closure.

Changing epsilon changes this example. At epsilon = 1, h(t) = t/sqrt(1+t²) is nonexpansive, h^m(t) = t/sqrt(1+mt²), and sup_|t|<=1|h^m(t)| = 1/sqrt(1+m). The powers now converge uniformly to zero. A positive stabilizer assures continuity; its positivity alone does not assure a common all-depth Lipschitz bound.

10. What a complete transformer still owes us

The pure-mixing theorem is about F_s(X) = A_s(X)X. Arbitrary value and output projections, residual additions, normalization and MLPs are extra operations. Their effects must preserve an invariant domain and the sensitivity mechanism, or satisfy another compactness criterion. A theorem about their component is not a theorem about their composition by courtesy.

PreNorm counterexample. A residual sublayer x + H(LN(x)) can have H constantly equal to a nonzero b, implemented by zero weights and a constant output bias. Its iteration is x + mb. No nonempty compact invariant set exists, because every orbit is unbounded. Normalizing the input to a residual branch does not bound the accumulated residual stream.

PostNorm positive statement. If every selected complete block ends in a LayerNorm with finite gains and bias, then its output row norm is at most ||gamma||_inftysqrt(d) + ||beta||. A finite family of such blocks has a common finite radius R; the product of closed row balls of radius R is a compact invariant domain. This proves invariance. Proposition 4 shows why it does not prove equicontinuity of arbitrary words.

A sufficient replacement test. If a family of continuous complete blocks preserves a compact K and is nonexpansive in one fixed compatible metric on K, every word is nonexpansive. Arzela-Ascoli and joint continuity of composition give compact monoid closure. More generally, prove any common modulus of continuity for all words. Per-layer constants L > 1 multiplied into L^m do not supply such a modulus.

Common nonexpansive row maps can extend the mixing theorem: apply the same nonexpansive v_s: C to C at every position before or after mixing. Diameter contraction and the zero-sum sensitivity estimate survive. Nested-hull consensus need not survive: a common rotation can keep equal rows moving forever. Once again, preserve the conclusion actually proved, rather than every conclusion on the neighboring page.

Depth, context length, weight family and permitted masks are part of the quantifiers. A theorem uniform over one of them must exhibit constants uniform over that variable. This chapter has proved a strong all-word result for a precise class; it has not certified all trained transformers.

11. A compact monoid is not a finite algebra in disguise

There are useful linear objects here, but they require another change of nouns. For a continuous self-map f of compact K, define its Koopman operator on real continuous observables:

It is linear in phi and has operator norm 1 in the supremum norm. Also U_fU_g = U_{g o f}: composition order reverses. Nonlinear finite-dimensional states have become linear operations on a generally infinite-dimensional function space, not a free finite-dimensional linearization.

Topology matters again. If f_j tends uniformly to f, then for each fixed continuous phi, ||U_{f_j}phi-U_fphi||_infty tends to zero by uniform continuity of phi. This is strong operator convergence. It is usually not operator-norm convergence. In fact, for distinct f,g on a compact metric K,

Proof. The upper bound follows from ||U_f|| = ||U_g|| = 1. Choose x with a = f(x) distinct from b = g(x). The function phi(y) = [d_K(y,b)-d_K(y,a)]/[d_K(y,b)+d_K(y,a)] is continuous, has supremum norm at most one, and takes values 1 at a and -1 at b. Evaluation at x attains the lower bound 2.

A compact state-map closure therefore has a compact Koopman image in the strong operator topology, because f to U_f is continuous for that topology. An infinite such image is a distance-2 separated set in operator norm and cannot be operator-norm compact. The topology is part of the assertion, not a footnote.

One scalar coordinate already produces infinitely many directions

Return to h(t) = t/sqrt(1+t²) on [-1,1]. Its powers have compact uniform closure {identity,h,h²,...,0}. Nevertheless, the functions h^m, m >= 0, are linearly independent.

Proof. Suppose sum_j=1,...,N c_jh^m_j(t) = 0 for distinct nonnegative integers m_j. Divide by t near zero and expand (1+m_jt²)^-1/2. For k = 0,...,N-1 its coefficient is a nonzero scalar depending only on k times m_j^k. Thus sum_j c_jm_j^k = 0 for those N values of k. The Vandermonde matrix on the distinct m_j has determinant product_i<j(m_j-m_i), which is nonzero. Every c_j vanishes.

Applying U_h^m to the coordinate observable phi(t) = t gives h^m. Hence the generated Koopman operator algebra is infinite-dimensional too. Compactness of the nonlinear power family and finite dimension of the state space have not prevented this.

A genuine finite-dimensional certificate. Choose observables phi₁,...,phi_q, let Phi be their vector, and prove for every generator s an exact identity Phi(F_s(x)) = M_sPhi(x) on K. Their span is then invariant under every Koopman generator, with a matrix representation. If Phi is injective, it separates states and transports the dynamics faithfully to Phi(K). If it is not injective, this representation recovers only the observed quotient. Exact invariance, separation and stability are three separate tests; a numerical fit to finitely many samples supplies none of them automatically.

For genuinely linear state maps on R^q, the algebra they generate lies in all q-by-q matrices and has dimension at most q². For affine maps, homogeneous coordinates give an analogous (q+1)-dimensional linear representation. These are legitimate finite algebra claims. A nonzero real linear algebra cannot itself be compact: it contains unbounded scalar multiples of any nonzero element. Finite dimension and topological closure do not change that. A nonlinear attention matrix depending on the current state is not a fixed linear state map.

12. A finite description. At which resolution?

Compactness gives finite nets at every positive resolution. It does not turn exact real coordinates into a finite set. To make that distinction quantitative, suppose K lies in a Euclidean ball of radius R in a p-dimensional affine subspace, with p >= 1.

Proposition 5 (packing and covering). For every delta > 0, K has a delta-net C contained in K with at most (1+2R/delta)^p points.

Proof. Open balls of radius delta/2 about points separated by more than delta are disjoint. They lie in a ball of radius R+delta/2. Volume comparison in the affine subspace bounds their number by the displayed expression. Greedily add points more than delta from the previous ones. The bound forces this process to stop, and maximality says that the resulting set is a delta-net.

For a row-radius-R state domain in R^{n x d}, the Frobenius radius is at most sqrt(n)R. A sufficient net bound is therefore (1+2sqrt(n)R/delta)^nd. This may be enormous. Its logarithm bounds the bits needed to index this particular approximation, not to encode every exact real state.

Choose a nearest-net-point function Q with deterministic tie-breaking and Q(c)=c on C. For f:K to K, the finite table f_C(c)=Q(f(c)) obeys

Proof. Insert f(Q(x)) between the two outputs. Quantizing the output costs at most delta. Changing the input by at most delta costs at most its continuity modulus omega_f(delta). The triangle inequality adds the two.

For typed maps f_j:K_j to K_j+1 with Lipschitz bounds L_j and nets of radius delta_j, use xhat₀=Q₀(x₀) and xhat_j+1=Q_j+1(f_j(xhat_j)). The error then satisfies

The one-step inequality follows by the same insertion, and induction gives the expanded expression; an empty product is one. Repeated rounding does not inherit the all-depth stochastic-weight bound unless the rounding itself meets that theorem's hypotheses. Existence of a finite net also does not provide an efficient algorithm to construct it: effective domain descriptions, moduli and resource budgets still matter.

13. The cache has left the finite room

A fixed finite neural call and an indefinitely growing external process have different state spaces. Recomputing from at most N tokens with positions from a fixed finite family and fixed depth stays within finitely many compact input relaxations. A bounded-size cache adds finitely many coordinates; uniform bounds on those coordinates give a compact envelope. Neither observation covers arbitrary recurrent memory updates.

A full-prefix cache grows in dimension. With zero-padded Euclidean encoding, m identical nonzero rows v have norm sqrt(m)||v|| and are unbounded. Merely declaring every row bounded does not give compactness in that metric. The visited points along even a bounded trajectory need not themselves be closed; their closure is a different set.

An actual termination theorem needs a probability floor

Proposition 6. Suppose generation samples conditionally at each step, stops at the first EOS, and gives EOS probability at least a > 0 after every surviving history. For the first EOS step T >= 1,

Proof. Conditional on survival through t, survival at the next step has probability at most 1-a. Conditional expectation and induction give the tail bound without independence. Its limit is zero. The tail-sum formula E[T]=sum_t>=0Pr(T>t) and the geometric series give 1/a.

For M output tokens, all logits bounded in absolute value by B, full-vocabulary softmax at temperature tau > 0 and always-eligible EOS, a valid floor is a=1/[1+(M-1)exp(2B/tau)]. This requires a common envelope over every reached history. Greedy decoding, truncated support or EOS masking changes the premise. A bound can be mathematically finite and practically useless.

Pointwise positive EOS probability is insufficient. Taking p_t(EOS)=1/(t+1)² for t >= 1 gives

The product telescopes by writing each factor as t(t+2)/(t+1)². Every step can be a finite-logit softmax, but half the probability never terminates. There is no common logit envelope.

With a uniform EOS floor and a finite alphabet, the induced finite-string laws have compact closure in total variation: truncate at a length with uniformly small tail, move that tail to a fixed short string, and use a finite net in the finite-dimensional probability simplex of short strings. Completeness of the probability simplex in the l₁ norm closes the argument. This is compactness of laws, not a fixed upper bound on every generated length.

These are conditional mathematical constructions. No deployed model's stopping behavior, cache bound or inference trace is certified by this page.

14. Where the irreverence enters

The finite transformer cannot stop being a computation by moving to a bounded latent space. Nor does boundedness appoint one representation as the correct one. A latent regime specifies which distinctions the encoder retains, which operations count as admissible, what target counts as recovery, and what information and budget a repair may use.

The mixing theorem shows a regime in which arbitrarily long compositions stay uniformly controlled while distinctions between token rows vanish. The normalization counterexample shows a regime in which every state stays inside a compact fence while repeated operations amplify arbitrarily small differences. The attained-family example shows that two individually authorized matrices can multiply into something the one-head representation cannot express, even though its unrestricted closure can express the limit. These are different departures from different specifications.

None establishes intelligence, creativity, generalization or model equivalence. The operational-profile hypothesis asks whether adequate learned models differ in the cost of realizing core algebra, geometry, analysis and their compositions. The results here identify objects and obstructions that a serious experiment must distinguish. They do not turn that hypothesis into a theorem.

Next: name the information that is missing before attempting to invert it. Or give the defect a budget and make the repair show its work.