Thread (60 posts)

When replacing the activation function at test time (as described in Section 5.3.2), here's what happens:

What stays the same:

• All trained weights: encoder weights $W_{\text{enc}}$, decoder weights $W_{\text{dec}}$
• All biases: $b_{\text{enc}}$, $b_{\text{pre}}$
• The overall architecture: encoder → activation → decoder

What changes:

Only the activation function itself is swapped out. For example:

• If trained with TopK(k=32), you might test with TopK(k'=64) or JumpReLU(θ)
• If trained with ReLU, you might test with TopK(k') or JumpReLU(θ)

Where JumpReLU parameters come from:

The threshold parameter $\theta$ for JumpReLU is chosen at test time as a hyperparameter. The authors sweep across different values of $\theta$ to generate the curves in Figure 10. Specifically:

For JumpReLU: $J_\theta(x) = x \cdot \mathbf{1}_{(x>\theta)}$

The process is:

1. Train autoencoder with original