Thread (64 posts)

Here are brief definitions of these concepts, ordered for clarity:

Hidden dimension

The internal representation size used throughout a transformer model's layers. For example, if a model processes information using 4096-dimensional vectors internally, its hidden dimension is 4096. This is the "width" of the neural network's intermediate computations.

Width

Another term for hidden dimension - it refers to how "wide" the model is in terms of the size of its internal representations. A model with hidden dimension 4096 is "wider" than one with hidden dimension 2048.

Embedding projection layer

The final linear transformation in a language model that converts from the hidden dimension to vocabulary size. It's a matrix of size vocabulary_size × hidden_dimension that projects the model's internal representation (e.g., 4096 dimensions) to a score for each possible token in the vocabulary (e.g., 50,000+ tokens).

Logit vector & why it's higher dimensional

The output of the embedding projection layer before applying softmax. It contains one score (logit) for each token in the vocabulary. It's higher dimensional than the hidden dimension because vocabularies are typically much larger than hidden dimensions. For example:

• Hidden dimension: 4096
• Vocabulary size: 50,000
• Therefore logit vector: 50,000 dimensions

This dimensional mismatch (hidden << vocabulary) is what makes the embedding projection layer low-rank and exploitable by the attack described in the paper.

You're absolutely right to be confused! The definition in Section 3 is indeed highly simplified. The key insight is that all the transformer complexity is hidden inside the function $g_\theta$.

The paper defines: $f_\theta(x) = \text{softmax}(E^T \cdot g_\theta(x))$

Here's where all the complexity goes:

Inside $g_\theta$: This function encapsulates the entire transformer architecture:

• All attention layers
• Feed-forward networks
• Layer normalizations
• Residual connections
• Positional encodings
• Everything except the final projection

So when you have a typical transformer that processes tokens through many layers to produce a final hidden state $h \in \mathbb{R}^d$, that entire process is what $g_\theta$ represents.

Why this simplification works: The authors only care about stealing the final projection layer $E^T$ (which maps from hidden dimension $h$ to vocabulary size $l$). For their attack, they treat everything before this final projection as a black box that produces some hidden vector.

This is actually mentioned later in the paper - in Lemma 2 they note that with residual connections, you can't extract $E^T$ exactly (only up to a linear transformation), precisely because the internal structure of $g_\theta$ creates ambiguities.

The brilliance of their attack is that they don't need to understand the internal complexity - they just need to observe that the final output lies in a low-dimensional subspace determined by the hidden dimension $h$.

Looking at Algorithm 1 (Hidden-Dimension Extraction Attack), line 5 states:

\State $\Q_i \gets \api(\x_i)$

They are setting the i-th row of Q to the oracle output.

This is clear from the context:

• Q is initialized as a $n \times l$ matrix (line 2)
• The API returns a logit vector of dimension $l$
• They iterate from $i = 1$ to $n$ (line 3)

So each query response $\api(\x_i)$ is a vector of length $l$ (the vocabulary size), and they're stacking these as rows to build the matrix Q. This is confirmed later in Lemma 1 where they define:

Let $\Q\br{\x_1, \ldots \x_{n}} \in \R^{l \times n}$ denote the matrix with columns $\api\br{\x_1}, \ldots, \api\br{\x_{n}}$

Wait, this seems inconsistent! The lemma says the API responses are columns, but the algorithm initialization suggests rows. Looking more carefully at the formalization section, they write \Q = \Et \cdot \H where $\Et$ is $l \times h$ and \H is $h \times n$, which gives $\Q$ as $l \times n$.

So actually, they're using column notation - $\Q_i$ refers to the i-th column of Q, not row. Each API response is stored as a column vector in the $l \times n$ matrix Q.

The SVD (Singular Value Decomposition) on line 7 of Algorithm 1 is the key mechanism that reveals the hidden dimension of the language model. Here's why it's significant:

Core Insight

The fundamental observation is that while the API returns logit vectors in $\mathbb{R}^l$ (where $l$ is the vocabulary size, often 50,000-100,000), these vectors actually lie in a much lower-dimensional subspace of dimension $h$ (the hidden dimension, typically 1,024-8,192).

This happens because of the model architecture: $\text{logits} = \mathbf{W}^T \cdot g_\theta(\mathbf{x})$

where:

• $\mathbf{W}^T$ is the $l \times h$ embedding projection matrix
• $g_\theta(\mathbf{x})$ is the $h$-dimensional hidden state

Why SVD Reveals the Hidden Dimension

When you collect $n > h$ logit vectors in matrix $\mathbf{Q}$:

1. Rank constraint: Since $\mathbf{Q} = \mathbf{W}^T \cdot \mathbf{H}$ where $\mathbf{H}$ is $h \times n$, the rank of $\mathbf{Q}$ is at most $h$

2. Singular value pattern: The SVD reveals this rank structure through the singular values:

   • The first $h$ singular values will be large (representing the true signal)
   • Singular values beyond position $h$ will be near zero (representing numerical noise)

3. Gap detection: The algorithm identifies $h$ by finding the largest multiplicative gap $\frac{\lambda_i}{\lambda_{i+1}}$ between consecutive singular values

Practical Visualization

The paper provides nice visualizations (Figures 2-3) showing:

• When $n < h$: All singular values are significant (can't determine $h$ yet)
• When $n > h$: Exactly $h$ significant singular values appear, with a sharp drop after

This is why line 8 computes:

count ← argmax_i {log ||λᵢ|| - log ||λᵢ₊₁||}

The logarithmic difference identifies the multiplicative gap, making the dimension extraction robust even with floating-point precision limitations.

This elegant use of SVD essentially performs dimensionality analysis to reverse-engineer architectural information from black-box API access alone.

Looking at the structure of $G$ in this paper, it's fascinating how it parallels the symmetry questions you've been exploring with SAEs!

The Structure of $G$

From the paper and its appendices, $G$ can take different forms depending on the model architecture:

1. General Case (No Normalization)

For models without normalization layers, $G$ can be any invertible $h \times h$ matrix. The symmetry arises because you can:

• Apply transformation $S^{-1}$ to all layer input projections
• Apply transformation $S$ to all layer output projections
• The residual stream gets transformed by $S$, but this cancels out: $(S^{-1} \cdot \text{weights}) \cdot (S \cdot \text{hidden state}) = \text{original computation}$

2. With RMSNorm

The presence of normalization restricts $G$ to be an orthogonal matrix. This is because:

• RMSNorm includes the operation $x \mapsto x/\|x\|$
• This only commutes with orthogonal transformations: $\|Ux\| = \|x\|$ when $U^T U = I$
• So the symmetry group becomes $O(h)$ (orthogonal group)

3. With LayerNorm

Even more restricted! LayerNorm adds a centering step, so $G$ must be an orthogonal matrix that also satisfies:

• $G \cdot \mathbf{1} \in \{-\mathbf{1}, \mathbf{1}\}$ (preserves or flips the all-ones vector)
• This is because centering projects onto the hyperplane $\mathbf{1}^T x = 0$

The Symmetry Connection

This is exactly analogous to your SAE reparameterization question! In both cases:

1. The model has inherent symmetries - transformations that don't change input-output behavior
2. These symmetries form a group - you can compose them, invert them, etc.
3. Different architectural choices restrict the symmetry group:
   • SAEs: Adding decoder normalization breaks scaling symmetry
   • Transformers: Adding layer normalization restricts from $GL(h)$ to $O(h)$

Why This Matters

The paper makes an important point: they can only recover the embedding matrix up to these symmetries. Just like how the L1 penalty in SAEs isn't meaningful without breaking the scaling symmetry, the "exact" embedding matrix isn't meaningful without specifying which element of the symmetry group you want.

In practice, they:

• Recover $\tilde{E}^T = E^T \cdot G$
• Report reconstruction error after finding the best-fitting $G$ via least squares
• Note that with orthogonal constraints, you're missing ~$h^2/2$ parameters (vs $h^2$ for general affine)

This is a beautiful example of how symmetries in neural network architectures have real implications for both training (as in SAEs) and security (as in model stealing)!

Looking at the structure of $\mathbf{G}$ in the Carlini et al. paper, this is indeed a fascinating example of how symmetries fundamentally limit what can be extracted!

The Structure of $\mathbf{G}$

The paper shows two key results about $\mathbf{G}$:

1. General Case: $\mathbf{G}$ can be any invertible $h \times h$ matrix

Without additional constraints, if you have:

• True embedding projection: $\mathbf{W}$
• Hidden states: $g_\theta(\mathbf{x})$
• Observable logits: $\mathbf{W} \cdot g_\theta(\mathbf{x})$

Then for any invertible $\mathbf{G}$, the reparameterization:

• $\mathbf{W}' = \mathbf{W} \cdot \mathbf{G}$
• $g'_\theta(\mathbf{x}) = \mathbf{G}^{-1} \cdot g_\theta(\mathbf{x})$

Produces identical outputs: $\mathbf{W}' \cdot g'_\theta(\mathbf{x}) = \mathbf{W} \cdot \mathbf{G} \cdot \mathbf{G}^{-1} \cdot g_\theta(\mathbf{x}) = \mathbf{W} \cdot g_\theta(\mathbf{x})$

2. With Normalization Layers: $\mathbf{G}$ must be orthogonal

The paper's clever insight (proven in Appendix B) is that LayerNorm/RMSNorm adds constraints:

• These layers project activations onto a sphere: $\|g_\theta(\mathbf{x})\| = \text{constant}$
• For the reparameterization to preserve this sphere property, we need $\|\mathbf{G}^{-1} \cdot g_\theta(\mathbf{x})\| = \|g_\theta(\mathbf{x})\|$
• This forces $\mathbf{G}$ to be orthogonal (norm-preserving)

The Symmetry Group Structure

For transformers with normalization, the symmetry group is: $G = O(h) \cong SO(h) \rtimes \mathbb{Z}_2$

Where:

• $O(h)$ is the orthogonal group (all $h \times h$ orthogonal matrices)
• $SO(h)$ is the special orthogonal group (rotations, det = 1)
• $\mathbb{Z}_2$ represents reflections (det = -1)

This has dimension $\frac{h(h-1)}{2}$, meaning there are that many continuous degrees of freedom in the reparameterization.

Why This Matters

This connects beautifully to your broader question about symmetries:

1. Information-theoretic limit: The attack can extract $h \times l$ parameters minus $\frac{h(h-1)}{2}$ degrees of freedom

   • For GPT-3.5 with $h \approx 4096$, that's ~8.4M unrecoverable parameters!

2. Practical implications:

   • The "true" weights are meaningless - only the equivalence class matters
   • Any defense trying to hide exact weights while exposing logits is fundamentally flawed
   • The symmetry structure tells us exactly what information is inherently private

3. Connection to other symmetries: Unlike the SAE scaling symmetries (which are feature-wise), these are global transformations mixing all hidden dimensions - much richer structure!

The fact that they recover commercial models with RMS error ~$10^{-4}$ after optimal alignment shows they're getting essentially everything that's theoretically possible. The matrix $\mathbf{G}$ isn't noise or approximation error - it's the fundamental ambiguity inherent in the problem.

Looking at the RMS computation in Section 4.2 of the Carlini et al. paper, it's actually a great example of symmetry-aware evaluation. Let me break down what they're doing and why it respects the inherent symmetries:

The RMS Computation

The paper states:

"we evaluate the attack success rate by comparing the root mean square (RMS) between our extracted matrix $\tilde{\mathbf{E}}^T = \mathbf{U} \cdot \boldsymbol{\Sigma}$ and the actual weight matrix, after allowing for a $h \times h$ affine transformation."

Specifically, they:

1. Solve the least squares system: $\tilde{\mathbf{E}}^T \cdot \mathbf{G} \approx \mathbf{E}^T$ for $\mathbf{G}$
2. Report the RMS of $\mathbf{E}^T$ and $\tilde{\mathbf{E}}^T \cdot \mathbf{G}$

Why This Respects Symmetry

This evaluation explicitly accounts for the fundamental symmetry in their extraction problem. They proved (Lemma 4.2) that given only API access, you can only recover:

$\tilde{\mathbf{E}}^T = \mathbf{E}^T \cdot \mathbf{G}$

for some unknown $h \times h$ matrix $\mathbf{G}$. This isn't a limitation of their attack—it's a fundamental information-theoretic constraint.

The Symmetry Group

For transformer models with residual connections, the symmetry group consists of all transformations of the form:

• $\mathbf{E}^T \rightarrow \mathbf{E}^T \cdot \mathbf{G}$
• $g_\theta(x) \rightarrow \mathbf{G}^{-1} \cdot g_\theta(x)$

where $\mathbf{G}$ is any invertible $h \times h$ matrix (or orthogonal if you include normalization constraints).

Why Direct RMS Would Be Wrong

If they computed RMS directly between $\tilde{\mathbf{E}}^T$ and $\mathbf{E}^T$ without alignment:

• The error could be arbitrarily large even for perfect extraction (just with different basis)
• It would penalize the attack for not recovering information that's theoretically impossible to recover
• The metric wouldn't distinguish between genuine extraction errors and harmless reparameterizations

The Deeper Structure

What's elegant here is that they're implicitly working with equivalence classes:

• Two matrices are equivalent if they differ by right-multiplication by an invertible matrix
• Their metric measures distance between equivalence classes, not between specific representatives
• This is the "right" way to measure extraction success given the symmetries

Comparison to Baseline

They note that random initialization gives RMS ≈ $2 \times 10^{-2}$, while their attack achieves $10^{-4}$ to $10^{-5}$. This 100-500× improvement shows they're extracting real information, not just exploiting the alignment procedure.

This approach beautifully illustrates your earlier point about defining fidelity modulo symmetries rather than seeking meaningless "exact" extraction!

Logit bias APIs are valuable features that allow users to influence a language model's token selection without retraining or fine-tuning the model. Here are the key reasons why API providers offer this functionality:

1. Constrained Generation

Logit bias allows developers to guide models to generate text that adheres to specific formats or constraints:

• Structured output: Ensuring JSON, XML, or other structured formats are properly generated
• Grammar compliance: Forcing adherence to specific syntactic rules
• Domain-specific vocabularies: Restricting outputs to technical terminology or avoiding certain words

2. Content Moderation

Organizations can use logit bias to:

• Block inappropriate content: Apply negative bias to profanity or sensitive terms
• Enforce brand guidelines: Prevent mentions of competitors or off-brand language
• Comply with regulations: Avoid generating text that might violate legal requirements

3. Simulating Fine-tuning

As mentioned in the paper, logit bias can approximate the effects of fine-tuning without the cost:

• Personality adjustment: Making the model more formal/casual by biasing certain tokens
• Style transfer: Encouraging specific writing styles or tones
• Task-specific behavior: Adapting the model for particular use cases

4. Interactive Applications

In conversational or game-like applications:

• Character consistency: Ensuring AI characters use appropriate vocabulary
• Difficulty adjustment: Making text generation easier/harder to predict
• Creative constraints: Implementing word games or writing challenges

5. Efficiency and Cost Savings

Logit bias provides a lightweight alternative to:

• Training custom models
• Running multiple inference passes
• Post-processing and filtering outputs

The feature essentially gives developers fine-grained control over generation probabilities, making LLMs more practical for production applications where specific output characteristics are required. This explains why major providers like OpenAI and Google included these features in their APIs—they significantly expand the utility of their models for real-world applications.

Great question! The distinction between logits and logprobs is crucial to understanding why the attack in Section 5.3 is more complex than the initial warm-up attack.

Logits vs Logprobs

Logits are the raw output values from the model before applying softmax:

• For token $i$: $z_i = \mathbf{W}^T \cdot g_\theta(x)_i$

Logprobs are the log of the probabilities after applying softmax:

• For token $i$: $y_i = \log\left(\frac{e^{z_i}}{\sum_j e^{z_j}}\right) = z_i - \log\left(\sum_j e^{z_j}\right)$

Why This Matters for the Attack

The key challenge is that production APIs return logprobs, not logits. When you apply a logit bias $B$ to token $i$ and observe its logprob, you get:

$y_i^B = z_i + B - \log\left(\sum_{j \neq i} e^{z_j} + e^{z_i + B}\right)$

Notice the problem: the normalizing constant (the log term) changes depending on which tokens you bias. This means you can't simply subtract $B$ from $y_i^B$ to get $z_i$.

The Attack's Solution

The paper presents two approaches:

1. Reference token method: Use one token as a fixed reference point across all queries. By comparing other tokens to this reference, you can cancel out the changing normalization constants: $y_R^B - y_i^B - B = z_R - z_i$

2. Linear constraint method (Appendix A): Treat each logprob observation as a linear constraint on the original logits, then solve the resulting system of equations.

This distinction is why the attack requires multiple queries with carefully chosen logit biases—it's reconstructing the underlying logits from observations that have been transformed by the softmax's normalization.

The main obstacle to propagating this attack through earlier layers is non-linearity. The attack fundamentally exploits the fact that the final embedding projection layer is a simple linear transformation from a low-dimensional hidden space to a high-dimensional vocabulary space.

Here are the key barriers:

1. Non-linear Operations

Earlier transformer layers contain:

• Non-linear activation functions (GELU, ReLU, etc.)
• Layer normalization or RMSNorm
• Softmax in attention mechanisms
• Residual connections that mix information across layers

The paper explicitly states: "We see no obvious methodology to extend it beyond just a single layer, due to the non-linearity of the models."

2. No Access to Intermediate Representations

The attack only observes the final logits. To recover layer N-1, you would need to observe the outputs of layer N-1, but these are:

• Transformed by the recovered layer N
• Further modified by all intervening non-linear operations
• Not directly observable from the API

3. Information Loss

Even if you perfectly recover the final layer, you're observing: $y = W_N \cdot \text{NonLinear}(W_{N-1} \cdot \text{NonLinear}(...))$

You cannot simply "invert" this composition because:

• The non-linear functions may not be invertible
• Multiple different weight configurations could produce the same final output
• The dimensionality reduction at each layer loses information

4. Architectural Complexity

Modern transformers have:

• Attention mechanisms with complex query-key-value interactions
• Multiple parallel attention heads
• Residual connections that bypass layers
• Layer-wise transformations that entangle information

The paper notes this is fundamentally different from prior work on stealing ReLU networks, where specialized techniques could handle the non-linearity layer by layer. Transformer architectures are far more complex and don't admit similar cryptanalytic approaches.

The authors acknowledge this limitation and explicitly invite future research in this area, suggesting it remains an open problem in model extraction attacks.