Thread (10 posts)

Why ReLU is “special”

1. Piece-wise linearity
   For a ReLU neuron
   $g(u)=\max(0,u)=\begin{cases}0&u<0\\u&u\ge 0\end{cases}$

   the input space is split by a single hyper-plane
   $H:=\{x\mid w^{\top}x+b=0\}$.

   • On each side of H the neuron is perfectly linear.
   • Crossing H flips the Jacobian contribution of that neuron from 0 to $w^{\top}(\cdot)$.
   • Along any straight line $\mu(\lambda)=x_1+\lambda(x_2-x_1)$ the global network output $f^*(\lambda)=f(\mu(\lambda))$ is therefore piece-wise linear: its first derivative has a jump exactly when $\mu(\lambda)$ hits a hyper-plane belonging to one ReLU.

   Detecting that jump with two finite differences is what lets Canales-Martínez et al. (and Carlini-Jagielski-Mironov 2020) pinpoint a critical point and say: “between $\lambda_1$ and $\lambda_2$ one and only one ReLU flipped – recover its weight/sign now”.

2. Algebraic simplifications
   Once the layer is known to be linear on both sides of a critical point, solving for weights reduces to solving linear (or at worst bilinear) systems. That is why the whole attack ultimately becomes polynomial.

What if the activation is still piece-wise linear but not ‘hard’?

• Leaky-ReLU, PReLU, hard-tanh, hard-sigmoid, ReLU6, etc.
These all keep the same “kink” – the derivative is discontinuous (slope changes from $a$ to $b$).
The attack generalises almost verbatim: in place of “slope = 0 vs slope = 1” you test “slope = a vs slope = b”, which is even easier because both sides remain informative.

• ‘ReLU with noise’ (e.g. dropout, quantisation error).
As long as the measurement noise is bounded and you can average a few repeated queries, the jump in the first derivative is still categorical; the paper already uses 200 samples per neuron in the “neuron-wiggle” stage exactly for this purpose.

Truly smooth activations (sigmoid, tanh, GELU, Swish)

Here the derivative is continuous, so the clean “one-bit jump” disappears:

1. No unique boundary:
   A sigmoid neuron transitions gradually over an $O(\|w\|^{-1})$ wide slab instead of a zero-measure hyper-plane. Two neighbouring neurons’ transition regions easily overlap, so following a line $\mu(\lambda)$ you see a sum of many smooth bumps; separating them is a de-convolution problem, not a simple change-point detection.

2. Ambiguous scaling:
   For ReLU you can absorb any positive scalar into the next layer, but negative scalars are forbidden by the sign of the kink. With sigmoids the function $\sigma(\alpha u)$ for $\alpha>0$ is still monotone and can be re-absorbed by re-scaling the outgoing weights, so recovering the exact scale of $w$ (and its sign) from black-box queries alone becomes ill-posed.

3. Higher-order derivatives help only with a gradient oracle:
   If the attacker could query $\nabla f(x)$ exactly, one could in principle solve non-linear systems that involve $\sigma'(\cdot)$ in place of the ReLU mask. In the pure prediction-only model you would have to estimate those gradients with finite differences; the error grows as $O(\varepsilon^{-1})$ for first derivatives and $O(\varepsilon^{-2})$ for second, so the number of queries blows up quickly.

Known theoretical results mirror this: with sigmoids, even deciding consistency of a 2-layer 3-neuron network given labeled samples is NP-complete (Blum–Rivest 1993). Allowing chosen queries helps, but no polynomial-time black-box extraction algorithm is currently known.

Practical take-away

• Piece-wise-linear (kinky) activations ⇒ the Crypto ’20 / TII ’23 extraction works.

• Mild modification (leaky-ReLU, ReLU6, hard-sigmoid, etc.) ⇒ same attack with minor slope accounting.

• Smooth activations ⇒ sharp “critical-point” trick disappears; you would have to shift to optimisation-based model stealing (train a surrogate on many query/response pairs) or to side-channel / gradient-oracle attacks.

So the “sharpness” is not a small implementation detail—it is the linchpin that turns a messy non-linear identification problem into a clean linear-algebra one.