Scaling and evaluating sparse autoencoders
← Back to all posts
Thread (60 posts)
In this literature is it correct that “parameter” generally means learned parameter and hyperparameter generally means selected (aside from the other implications of the terms)
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is asking about the terminology in the context of the "Scaling and evaluating sparse autoencoders" paper, specifically about the distinction between "parameter" and "hyperparameter". In machine learning: - Parameters are typically the learned values during training (weights, biases, etc.) - Hyperparameters are typically set before training and control the learning process (learning rate, architecture choices, etc.) Looking at the paper, I can see: - Parameters mentioned: $W_{enc}$, $W_{dec}$, $b_{enc}$, $b_{pre}$ - these are learned during training - Hyperparameters mentioned: $\lambda$ (L1 penalty coefficient), $k$ (sparsity level in TopK), learning rate, etc. So yes, the user's understanding is correct in this context.
Yes, that's correct! In this paper (and in machine learning literature generally):
Parameters = learned values that are optimized during training:
- (encoder weights)
- (decoder weights)
- (encoder bias)
- (pre-encoder bias)
Hyperparameters = values selected before training that control the learning process:
- (L1 penalty coefficient for ReLU autoencoders)
- (number of active latents in TopK)
- (total number of latents)
- Learning rate
- Batch size
- Architecture choices
The paper explicitly mentions this distinction, for example:
- "λ is a hyperparameter that needs to be tuned" (Section 2.2)
- "We sweep learning rates at small scales and extrapolate the trend" (Section 2.1)
The key difference is that parameters are optimized by gradient descent during training, while hyperparameters are typically selected through experimentation, grid search, or based on scaling laws (as this paper does). The TopK approach is praised partly because it removes the need to tune the λ hyperparameter - instead directly setting the sparsity level k.