Stealing Part of a Production Language Model
← Back to all posts
Thread (64 posts)
Cite to High Accuracy and High Fidelity Extraction of Neural Networks
for definitions of accuracy and fidelity
Link not found
Looking at the definitions in §2 Related Work:
Isn’t accuracy a sub- property of fidelity, in the sense that fidelity implies accuracy (but not the other way around? )
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is referring to the definitions from "High Accuracy and High Fidelity Extraction of Neural Networks" (arXiv:1909.01838) by Jagielski et al. They're asking about the relationship between accuracy and fidelity in model extraction attacks. Looking at the paper's definitions: 1. **Accuracy**: the stolen model $\hat{f}$ should match the performance of the target model $f$ on some particular data domain. For example, if the target is an image classifier, we might want the stolen model to match the target's overall accuracy on ImageNet. 2. **Fidelity**: the stolen model $\hat{f}$ should be functionally equivalent to the target model $f$ on all inputs. That is, for any valid input $x$, we want $\hat{f}(x) \approx f(x)$. The user is asking whether fidelity implies accuracy. Let me think about this: - If a stolen model has high fidelity (i.e., $\hat{f}(x) \approx f(x)$ for all inputs), then it would produce the same outputs as the target model on any test dataset. - This means it would achieve the same accuracy as the target model on any data domain. - So yes, perfect fidelity would imply matching accuracy. However, the converse is not true: - A model could achieve the same accuracy as the target model without matching its outputs on all inputs. - For example, two models could both achieve 95% accuracy on ImageNet but make mistakes on different images. So the user's intuition seems correct: fidelity is a stronger property that implies accuracy, but not vice versa.
You're absolutely right! Looking at the definitions from the Jagielski et al. paper:
- Accuracy: The stolen model should match the target's performance on some particular data domain
- Fidelity: The stolen model should produce (approximately) the same output as the target for all inputs
Fidelity is indeed the stronger property. If you achieve perfect fidelity—meaning your extracted model for all inputs —then you automatically achieve the same accuracy as the target model on any dataset, since you're making the same predictions.
However, the reverse isn't true. Two models can have identical accuracy (say 95% on ImageNet) while making mistakes on completely different examples. This is why the paper notes these objectives can be "in conflict for imperfect victim models"—a high-accuracy extraction might try to correct the victim's mistakes, while a high-fidelity extraction should replicate them.
The paper's Figure 1 illustrates this nicely: the green "high fidelity" line matches the blue oracle exactly (including its errors), while the orange "perfect accuracy" line correctly classifies all points but diverges from the oracle's behavior.
This distinction is important for different attack motivations:
- Theft adversaries might prioritize accuracy (they want a model that performs well)
- Reconnaissance adversaries need fidelity (for mounting black-box attacks, membership inference, etc., they need the extracted model to behave identically to the target)
In §3.1 of the Jagielski paper they name a goal of exact extraction and explain why it's impossible to achieve. This reminds me of some other recent questions I had that came down to repatameterization. I’m wondering why we would want to define fidelity of extraction in the terms they did rather than classifying the parameterization symmetries of the model and then seeking closeness up to symmetr. Can you think this through
Post not found
Post not found
Post not found
Post not found
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is making a really interesting connection between the model extraction work and the reparameterization symmetries they were thinking about in the context of SAEs. They're asking why we would define extraction fidelity in terms of matching outputs on all inputs rather than seeking closeness up to the inherent symmetries of the model. This is actually a deep question that touches on several important issues: 1. The Jagielski et al. paper does acknowledge that exact extraction is impossible due to symmetries - they mention scaling neurons, permuting neurons, and dead neurons as examples. 2. The Carlini et al. paper on "Stealing Part of a Production Language Model" actually does extract weights up to symmetries! They recover the embedding projection layer up to an h×h affine transformation, which is exactly the kind of symmetry-aware extraction the user is suggesting. 3. There are practical and theoretical reasons for both approaches: - Functional equivalence (matching outputs) is what matters for many attack purposes - But understanding the symmetry structure could lead to more efficient attacks or better understanding of what information is actually recoverable Let me think through the pros and cons of each approach...
Your observation is really insightful! You're absolutely right that there's a deep connection here with reparameterization symmetries, and the Jagielski paper's approach seems to sidestep this important structure.
Looking at both papers together reveals two different philosophies:
The Jagielski Approach (Functional Equivalence)
They define extraction success purely in terms of input-output behavior, essentially treating the model as a black box. This makes sense for their attack taxonomy because:
- It's application-focused: for attacks like adversarial examples or membership inference, what matters is behavioral equivalence
- It's model-agnostic: works across different architectures without needing to understand their symmetries
- It sidesteps the complexity of characterizing all possible symmetries
The Symmetry-Aware Approach (Carlini et al.)
The "Stealing Part of a Production Language Model" paper actually does exactly what you're suggesting! They:
- Explicitly acknowledge they can only recover up to an transformation
- Prove this is optimal (you can't do better given only input-output access)
- Measure extraction success as RMS error after finding the best aligning transformation
Why Both Approaches Matter
For ReLU networks, the symmetry group includes:
- Positive scaling: for
- Permutations of neurons within layers
- Adding dead neurons
- Sign flips (for layers before ReLU)
For transformers, additional symmetries arise from:
- Orthogonal transformations after layer norm (as Carlini et al. prove)
- Attention head permutations
- Various architectural invariances
The Deeper Question
You're touching on something fundamental: what information about a model is actually meaningful?
If two models differ only by a reparameterization, they're functionally identical, so arguably:
- We should define extraction success modulo symmetries
- Attacks should explicitly target equivalence classes rather than specific parameters
- Defenses based on hiding "exact" parameters are meaningless if the equivalence class is recoverable
This suggests a more sophisticated extraction taxonomy:
- Level 0: Functional equivalence (Jagielski)
- Level 1: Parameter recovery up to known symmetries (Carlini)
- Level 2: Exact parameter recovery (impossible for fundamental reasons)
The fact that Carlini et al. successfully extract commercial models up to symmetries suggests this is the right framework for thinking about extraction attacks. It's both more honest about what's achievable and more precise about what information is actually being leaked.