Thread (10 posts)

Why ReLU is “special”

1. Piece-wise linearity
   For a ReLU neuron
   $g(u)=\max(0,u)=\begin{cases}0&u<0\\u&u\ge 0\end{cases}$

   the input space is split by a single hyper-plane
   $H:=\{x\mid w^{\top}x+b=0\}$.

   • On each side of H the neuron is perfectly linear.
   • Crossing H flips the Jacobian contribution of that neuron from 0 to $w^{\top}(\cdot)$.
   • Along any straight line $\mu(\lambda)=x_1+\lambda(x_2-x_1)$ the global network output $f^*(\lambda)=f(\mu(\lambda))$ is therefore piece-wise linear: its first derivative has a jump exactly when $\mu(\lambda)$ hits a hyper-plane belonging to one ReLU.

   Detecting that jump with two finite differences is what lets Canales-Martínez et al. (and Carlini-Jagielski-Mironov 2020) pinpoint a critical point and say: “between $\lambda_1$ and $\lambda_2$ one and only one ReLU flipped – recover its weight/sign now”.

2. Algebraic simplifications
   Once the layer is known to be linear on both sides of a critical point, solving for weights reduces to solving linear (or at worst bilinear) systems. That is why the whole attack ultimately becomes polynomial.

What if the activation is still piece-wise linear but not ‘hard’?

• Leaky-ReLU, PReLU, hard-tanh, hard-sigmoid, ReLU6, etc.
These all keep the same “kink” – the derivative is discontinuous (slope changes from $a$ to $b$).
The attack generalises almost verbatim: in place of “slope = 0 vs slope = 1” you test “slope = a vs slope = b”, which is even easier because both sides remain informative.

• ‘ReLU with noise’ (e.g. dropout, quantisation error).
As long as the measurement noise is bounded and you can average a few repeated queries, the jump in the first derivative is still categorical; the paper already uses 200 samples per neuron in the “neuron-wiggle” stage exactly for this purpose.

Truly smooth activations (sigmoid, tanh, GELU, Swish)

Here the derivative is continuous, so the clean “one-bit jump” disappears:

1. No unique boundary:
   A sigmoid neuron transitions gradually over an $O(\|w\|^{-1})$ wide slab instead of a zero-measure hyper-plane. Two neighbouring neurons’ transition regions easily overlap, so following a line $\mu(\lambda)$ you see a sum of many smooth bumps; separating them is a de-convolution problem, not a simple change-point detection.

2. Ambiguous scaling:
   For ReLU you can absorb any positive scalar into the next layer, but negative scalars are forbidden by the sign of the kink. With sigmoids the function $\sigma(\alpha u)$ for $\alpha>0$ is still monotone and can be re-absorbed by re-scaling the outgoing weights, so recovering the exact scale of $w$ (and its sign) from black-box queries alone becomes ill-posed.

3. Higher-order derivatives help only with a gradient oracle:
   If the attacker could query $\nabla f(x)$ exactly, one could in principle solve non-linear systems that involve $\sigma'(\cdot)$ in place of the ReLU mask. In the pure prediction-only model you would have to estimate those gradients with finite differences; the error grows as $O(\varepsilon^{-1})$ for first derivatives and $O(\varepsilon^{-2})$ for second, so the number of queries blows up quickly.

Known theoretical results mirror this: with sigmoids, even deciding consistency of a 2-layer 3-neuron network given labeled samples is NP-complete (Blum–Rivest 1993). Allowing chosen queries helps, but no polynomial-time black-box extraction algorithm is currently known.

Practical take-away

• Piece-wise-linear (kinky) activations ⇒ the Crypto ’20 / TII ’23 extraction works.

• Mild modification (leaky-ReLU, ReLU6, hard-sigmoid, etc.) ⇒ same attack with minor slope accounting.

• Smooth activations ⇒ sharp “critical-point” trick disappears; you would have to shift to optimisation-based model stealing (train a surrogate on many query/response pairs) or to side-channel / gradient-oracle attacks.

So the “sharpness” is not a small implementation detail—it is the linchpin that turns a messy non-linear identification problem into a clean linear-algebra one.

This is an excellent observation about the paper's methodology. You're right that the authors make essential use of ReLU's discontinuous derivative at the critical points where neurons switch from inactive to active states. The paper explicitly states they cannot generalize to smooth activation functions:

"We can generalize the attack to any other type of activation function that looks like a piecewise linear function (with possibly more than one sharp bend), such as a leaky ReLU, but not to smooth activation functions such as a sigmoid which have no critical points."

However, your insight about the statistical nature of their methods is intriguing. Even for ReLUs, they:

1. Use the "Neuron Wiggle" technique which samples ~200 critical points per neuron
2. Employ confidence levels (α) to handle uncertainty
3. Deal with finite precision arithmetic that introduces noise

For smooth activation functions like sigmoid or tanh, while we lose the sharp critical points, we might consider:

Potential approaches:

• Gradient-based methods: Since smooth functions have well-defined gradients everywhere, we could potentially use gradient information more directly
• Linearization regions: Even smooth activations have regions where they're approximately linear (e.g., sigmoid near x=0)
• Higher-order derivatives: The curvature information might reveal network parameters

Challenges:

• Without critical points, identifying individual neuron behaviors becomes much harder
• The piecewise linear structure that enables layer-by-layer "peeling" disappears
• The signal-to-noise ratio would likely be much worse

The statistical sampling approach might help, but you'd probably need a fundamentally different algorithm architecture rather than adapting their critical-point-based method. It's an interesting research direction though - perhaps combining ideas from this paper with other model extraction techniques designed for smooth activations.