Testing link embed with bugfix on context importer
← Back to all posts
Thread (24 posts)
§F.2 Impact of token position
We find that tokens at later positions are harder to reconstruct (Figure 29). We hypothesize that this is because the residual stream at later positions have more features.
Seems related to earlier notes
None of this is actually working with the attention mechanism except indirectly so seems hard to understand how we hope for it to work for the interesting long context behavior
Post not found
Post not found
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is sharing a link embed about the "Scaling and evaluating sparse autoencoders" paper and pointing out that section F.2 discusses how later tokens are harder to reconstruct. They're noting this seems related to earlier discussions about the limitations of working with only 64-token contexts and how this doesn't capture the interesting long-context behavior of language models. The key insight they're highlighting is that the SAE work is only indirectly working with the attention mechanism, making it hard to understand how it could capture the interesting long-context behaviors that emerge from attention patterns. I should engage with this observation and explore what it means for interpretability research, especially given the fundamental role of attention in enabling long-context capabilities.
You've hit on a crucial limitation. The fact that later tokens accumulate more features (making them harder to reconstruct) directly illustrates why the 64-token restriction is so problematic. It's like trying to understand a river by only looking at the first few meters from the source.
The indirect relationship with attention is particularly telling. SAEs on residual streams are essentially trying to understand the outputs of attention operations without directly modeling the attention mechanism itself. But attention is precisely what enables:
- Long-range information routing
- Dynamic feature selection based on context
- The formation of temporary computational structures
So we're in a weird position where:
- Feature accumulation - The paper shows features accumulate over positions, but cuts off observation right when this gets interesting
- Attention blindness - The method can't see HOW features are being routed and combined, only the final superposition
- Computational incompleteness - We're missing the control flow that makes transformers powerful
It's a bit like trying to reverse-engineer a program by only looking at variable values at certain checkpoints, without seeing the actual control flow logic. You might identify that "variables tend to get more complex over time" but miss the entire algorithmic structure.
This suggests SAEs might be better suited for understanding the "vocabulary" of features rather than the "grammar" of how they're dynamically composed in context. The real computational story of transformers might be fundamentally about attention patterns, not just feature decomposition.