Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
← Back to all posts
Thread (53 posts)
In equation 11, why do they use an "eth" symbol in place of a partial derivative? Is this a known convention?
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is asking about the notation in equation 11 of the paper, where they use ð (eth) instead of the standard partial derivative symbol ∂. Let me look at the context. Looking at equation 11: ``` ððθJumpReLUθ(z):=−θεK(z−θε) ``` And in the footnote 4, they explain: "We use the notation ð/ðz to denote pseudo-derivatives, to avoid conflating them with actual partial derivatives for these functions." So they're using ð (the lowercase eth letter from Old English and Icelandic) to denote "pseudo-derivatives" rather than true partial derivatives. This is because the JumpReLU function and Heaviside step function are not differentiable in the classical sense at the threshold point - they have discontinuities. The authors are defining custom gradients (straight-through estimators) for these non-differentiable functions. This doesn't seem to be a widely established convention - it appears to be a notation choice made by the authors of this specific paper to clearly distinguish between: - True partial derivatives (∂) - Their custom-defined pseudo-derivatives for the backward pass (ð) The eth symbol (ð) is sometimes used in mathematics, but not typically for this purpose. It's more commonly seen in: - Icelandic and Old English text - Some areas of mathematics like differential geometry (though not commonly) - Phonetics (to represent the voiced dental fricative) The authors are being careful to make it clear that these aren't true derivatives but rather custom gradient definitions used for training through non-differentiable functions.
Great observation! The ð (eth) symbol is not a standard mathematical convention - it's a notation choice specific to this paper.
The authors explain in footnote 4:
"We use the notation ð/ðz to denote pseudo-derivatives, to avoid conflating them with actual partial derivatives for these functions."
They're using ð instead of the standard ∂ (partial derivative) because the JumpReLU and Heaviside functions aren't differentiable in the classical sense - they have discontinuities at the threshold. The "pseudo-derivatives" are custom gradient definitions they create for the backward pass during training.
This is actually quite thoughtful notation design! Since they're defining straight-through estimators (STEs) that approximate gradients where true gradients don't exist or are zero, using a visually similar but distinct symbol helps avoid confusion. It makes it immediately clear when they're talking about:
- ∂: True partial derivatives (for differentiable functions)
- ð: Their custom-defined gradients for training (for non-differentiable points)
The eth symbol (ð) comes from Old English and Icelandic alphabets. While it's occasionally used in some mathematical contexts, this specific usage as "pseudo-derivative" appears to be novel to this paper. It's a clever way to maintain mathematical clarity while introducing new gradient approximations for discontinuous functions.
Maybe jumping ahead but where does the parameter epsilon used to define the pseudo derivative come from?
Post not found
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is asking about the epsilon parameter used in the pseudo-derivatives defined in equations 11 and 12. Let me look at what the paper says about this. From the paper: 1. In equations 11 and 12, epsilon appears as a bandwidth parameter in the kernel function K. 2. The paper mentions that epsilon plays the role of a KDE (kernel density estimation) bandwidth. 3. In footnote 5, they mention: "For the experiments in this paper, we swept this parameter and found ε=0.001 (assuming a dataset normalised such that 𝔼𝐱[𝐱²]=1) works well across different models, layers and sites. However, we suspect there are more principled ways to determine this parameter, borrowing from the literature on KDE bandwidth selection." 4. In Section 4, they explain: "As we show in Section 4, the hyperparameter ε plays the role of a KDE bandwidth, and needs to be selected accordingly: too low and gradient estimates become too noisy, too high and estimates become too biased." 5. The paper shows that the STEs are actually computing kernel density estimates of the true gradient of the expected loss. So epsilon is a bandwidth parameter that controls the width of the window around the threshold where gradients are non-zero. It's a hyperparameter they had to tune.
Excellent question! The epsilon (ε) parameter is essentially a hyperparameter that controls the "bandwidth" or width of the window where the pseudo-derivatives are non-zero.
Here's what the paper tells us about it:
Where it comes from
-
It's borrowed from kernel density estimation (KDE): The authors show that their STEs are actually computing kernel density estimates of the true gradient. In KDE, you always need a bandwidth parameter that controls the smoothing.
-
They found it empirically: From footnote 5:
"For the experiments in this paper, we swept this parameter and found ε=0.001 (assuming a dataset normalised such that 𝔼[𝐱²]=1) works well across different models, layers and sites."
What it does
Looking at the pseudo-derivatives:
The epsilon controls:
- Width of the gradient window: Only when (for rectangular kernel) do we get non-zero gradients
- Magnitude of gradients: Smaller ε means larger gradient magnitudes (note the 1/ε factor)
The tradeoff
As they explain in Section 4:
- ε too small: Gradient estimates become too noisy (few samples fall in the narrow window)
- ε too large: Estimates become too biased (you're averaging over too wide a region)
Visual intuition
From Figure 4, you can see that epsilon defines a small window around the threshold where the pseudo-derivative is non-zero. Outside this window, the gradient is zero (just like the true derivative).
Why not something more principled?
The authors acknowledge this is somewhat ad-hoc:
"However, we suspect there are more principled ways to determine this parameter, borrowing from the literature on KDE bandwidth selection."
So epsilon is essentially a "gradient window width" hyperparameter they tuned empirically. It's small enough to be local (ε=0.001) but large enough to catch some activations and provide gradient signal. The fact that the same value works across different models and layers suggests it's reasonably robust, though more principled selection methods could likely improve it.