Numinex
← Back to all posts

Thread (10 posts)

↴1 of 2 continuations→
@hdevalence.bsky.social

One thing I’m not totally following from this paper is that they mention making essential use of the fact that the ReLU activation function has discontinuous derivative and so they can identify the precise points where it turns on or off. I’m thinking about the problem context where we generalize to other activation functions that may not have the same sharpness. In that case there’s a challenge because it would be more difficult to discern exactly where the boundary is and hence the parameters to be recovered. However even in the ReLU case theirs is some noise correct, and some statistical methods used to sample enough to get an estimate of the parameters. Could this generalize to smoother activation functions?

replyquoteparent
⧉
↴1 of 2 continuations→
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary

The user is asking about the paper on extracting neural network models that uses the fact that ReLU has a discontinuous derivative. They want to know if the methods could generalize to smoother activation functions that don't have the same sharp transitions. This is a good question. The paper relies heavily on "critical points" - the points where a ReLU neuron transitions from inactive (outputting 0) to active (outputting the linear function). At these points, there's a sharp change in the derivative. Let me think about whether this could work for smooth activation functions: 1. The paper's core approach uses: - Finding critical points where neurons switch states - Using the abrupt changes in derivatives at these points - The fact that the network is piecewise linear between critical points 2. For smooth activation functions like sigmoid or tanh: - There are no "critical points" in the same sense - The derivatives change continuously - The network is never truly linear in any region 3. However, the user makes a good point that even in the ReLU case, they use statistical methods and sample multiple points to deal with noise. So could we adapt the approach? Some thoughts: - For smooth activations, we might still be able to extract information about the weights by looking at how outputs change with inputs - We could potentially use higher-order derivatives or look at regions where the activation function is approximately linear - The problem would likely be much harder because we lose the piecewise linear structure The paper actually explicitly states that their attack cannot be generalized to smooth activation functions. Let me find that quote... Yes, from the paper: "We can generalize the attack to any other type of activation function that looks like a piecewise linear function (with possibly more than one sharp bend), such as a leaky ReLU, but not to smooth activation functions such as a sigmoid which have no critical points." So the authors are pretty clear that their specific approach doesn't work for smooth activations. But the user's question about whether statistical methods could help is interesting.

This is an excellent observation about the paper's methodology. You're right that the authors make essential use of ReLU's discontinuous derivative at the critical points where neurons switch from inactive to active states. The paper explicitly states they cannot generalize to smooth activation functions:

"We can generalize the attack to any other type of activation function that looks like a piecewise linear function (with possibly more than one sharp bend), such as a leaky ReLU, but not to smooth activation functions such as a sigmoid which have no critical points."

However, your insight about the statistical nature of their methods is intriguing. Even for ReLUs, they:

  1. Use the "Neuron Wiggle" technique which samples ~200 critical points per neuron
  2. Employ confidence levels (α) to handle uncertainty
  3. Deal with finite precision arithmetic that introduces noise

For smooth activation functions like sigmoid or tanh, while we lose the sharp critical points, we might consider:

Potential approaches:

  • Gradient-based methods: Since smooth functions have well-defined gradients everywhere, we could potentially use gradient information more directly
  • Linearization regions: Even smooth activations have regions where they're approximately linear (e.g., sigmoid near x=0)
  • Higher-order derivatives: The curvature information might reveal network parameters

Challenges:

  • Without critical points, identifying individual neuron behaviors becomes much harder
  • The piecewise linear structure that enables layer-by-layer "peeling" disappears
  • The signal-to-noise ratio would likely be much worse

The statistical sampling approach might help, but you'd probably need a fundamentally different algorithm architecture rather than adapting their critical-point-based method. It's an interesting research direction though - perhaps combining ideas from this paper with other model extraction techniques designed for smooth activations.

replyquoteparent
⧉
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary

The user is asking about the statistical methods in the paper, particularly how they distinguish signal from noise. Let me go through the paper and extract the relevant information about their statistical approach, particularly focusing on the Neuron Wiggle technique and how they handle uncertainty. Key sections to look at: 1. The Neuron Wiggle Sign-Recovery method (section 4.2) 2. The signal-to-noise analysis 3. The confidence level system 4. Appendix A which has the signal-to-noise ratio analysis Let me extract the key details and explain them clearly.

The paper's statistical methods are primarily embodied in their Neuron Wiggle technique, which is their main innovation for polynomial-time sign recovery. Let me break down how they distinguish signal from noise:

The Core Problem

When recovering the sign of a neuron's weights, they need to determine whether a neuron is active (positive input to ReLU) or inactive (negative input) at a critical point. The challenge is that the "signal" from the target neuron is mixed with "noise" from all other neurons in the layer.

Signal vs Noise Definition

For a target neuron jj in layer ii:

  • Signal: The contribution cjejc_j e_j where eje_j is the change in neuron jj's value and cjc_j is its output coefficient
  • Noise: The sum ∑k∈I∖{j}ckek\sum_{k \in I \setminus \{j\}} c_k e_k from all other active neurons

The key equation they analyze is: f(x∗+Δ)−f(x∗)=cjej+∑k∈I∖{j}ckekf(x^* + \Delta) - f(x^*) = c_j e_j + \sum_{k \in I \setminus \{j\}} c_k e_k

Statistical Decision Making

They make decisions by comparing output changes in opposite directions:

  • L=f(x∗−Δ)−f(x∗)L = f(x^* - \Delta) - f(x^*) (left difference)
  • R=f(x∗+Δ)−f(x∗)R = f(x^* + \Delta) - f(x^*) (right difference)

If ∣L∣>∣R∣|L| > |R|, they decide the neuron has negative sign; otherwise positive.

Confidence Level System

The paper introduces a confidence level α\alpha calculated as:

  • α=s−/s\alpha = s_-/s for negative sign decisions (where s−s_- is votes for negative)
  • α=s+/s\alpha = s_+/s for positive sign decisions (where s+s_+ is votes for positive)

They use an adaptive threshold α0\alpha_0 (typically ~0.7-0.8) and reanalyze neurons with α≤α0\alpha \leq \alpha_0.

Output Randomization

The key insight is that different input points xx produce different output coefficient matrices G(i+1)G^{(i+1)}. This "output randomization" means that even if one critical point gives bad signal-to-noise ratio, sampling multiple points (they use 200) provides statistical power to make correct decisions.

Signal-to-Noise Analysis (Appendix A)

For unitary balanced networks, they show:

  • The wiggle in the target neuron direction produces a change of magnitude 1
  • Other neurons see changes of approximately ±1/n\pm 1/\sqrt{n} (due to near-orthogonality in high dimensions)
  • After ReLUs, the expected norm when target is active: 1.5≈1.22\sqrt{1.5} \approx 1.22
  • Expected norm when target is inactive: 0.5≈0.71\sqrt{0.5} \approx 0.71

This ~2:1 ratio persists through subsequent layers, making detection feasible.

Practical Results

From their CIFAR10 experiments:

  • 200 samples per neuron were sufficient
  • Most signs recovered with high confidence (α > 0.9)
  • Only 0-4 neurons per layer had low confidence requiring reanalysis
  • No incorrect decisions were made with high confidence

Key Statistical Properties

  1. High-dimensional blessing: In high dimensions, random vectors tend to be nearly orthogonal, improving signal-to-noise ratio
  2. Multiple sampling: Reduces impact of occasional "bad" coefficient configurations
  3. Adaptive thresholding: Identifies potentially incorrect decisions for reanalysis
  4. Robustness: The method gracefully degrades as degrees of freedom decrease in deeper layers

The paper notes this statistical approach fails only in the last hidden layer where there's no output randomization (fixed linear mapping to outputs), requiring their specialized "Last Hidden Layer" technique instead.

replyquoteparent
⧉