Scaling and evaluating sparse autoencoders
← Back to all posts
Thread (60 posts)
However, latents are the important artifact of training (not reconstruction predictions), whereas for language models we typically care only about token predictions. Comparing MSE across different n is thus not a fair comparison — the latents have a looser information bottleneck with larger n , so lower MSE is more easily achieved. Thus, this approach is arguably unprincipled for autoencoder training.
I thought I understood the distinction between latents in traing and next token prediction during inference but I don't understand the "unfair comparison" or the reason for it being unprincipled so I must be missing something.
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is asking about a specific passage from the "Scaling and evaluating sparse autoencoders" paper. They're confused about why comparing MSE across different n (number of latents) is considered an "unfair comparison" and why this approach is "arguably unprincipled for autoencoder training." Let me break down what's happening here: 1. The paper is discussing two approaches to choosing autoencoder size and token budget 2. The first approach (L(C)) trains autoencoders to optimal MSE given available compute 3. The authors criticize this approach The key insight is about what we're actually trying to optimize for: - In language models: We care about the final output (token predictions). The internal representations are just a means to an end. - In sparse autoencoders: We care about the latents themselves - they are the interpretable features we're trying to extract. The reconstruction is just a way to ensure the latents capture meaningful information. When you increase n (number of latents), you're giving the autoencoder more capacity. With more latents, it's easier to achieve lower MSE because you have a "looser information bottleneck" - more degrees of freedom to represent the input. So comparing MSE across different n values is like comparing the performance of two students on exams where one student gets to use more notes than the other. The one with more notes will likely do better, but that doesn't mean they understand the material better. The "unprincipled" part comes from the fact that if you're optimizing for MSE across different n values, you're not really optimizing for what you care about (quality of individual latents), you're just optimizing for reconstruction quality, which naturally improves with more latents.
The key insight here is about what we're actually optimizing for in sparse autoencoders versus language models.
In language models, the internal representations are just a means to an end - we ultimately care about the quality of token predictions. But in sparse autoencoders, the latents themselves are the primary output - they're the interpretable features we're trying to discover.
The "unfair comparison" arises because:
-
Capacity increases with n: When you have more latents (larger n), you have more degrees of freedom to represent the input. It's like having a larger vocabulary to describe something - naturally you can be more precise.
-
Information bottleneck loosens: With more latents, the autoencoder doesn't need to compress information as tightly. Each latent can be more specialized and capture narrower features, making reconstruction easier.
-
MSE naturally decreases: This means that a larger autoencoder will almost always achieve lower MSE than a smaller one, but this doesn't necessarily mean the individual latents are better or more interpretable.
The comparison is "unprincipled" because optimizing for MSE across different n values incentivizes simply making n larger, rather than finding the best interpretable features at a given scale. It's like judging compression algorithms by how well they preserve data while ignoring the compression ratio - you'd always choose the one that barely compresses at all.
This is why the authors prefer training to convergence at fixed n (their L(N) approach) - it gives a fairer comparison of what quality of features you can extract at different scales, rather than just rewarding models for having more capacity.