Numinex
← Back to all posts

Thread (10 posts)

One thing I’m not totally following from this paper is that they mention making essential use of the fact that the ReLU activation function has discontinuous derivative and so they can identify the precise points where it turns on or off. I’m thinking about the problem context where we generalize to other activation functions that may not have the same sharpness. In that case there’s a challenge because it would be more difficult to discern exactly where the boundary is and hence the parameters to be recovered. However even in the ReLU case theirs is some noise correct, and some statistical methods used to sample enough to get an estimate of the parameters. Could this generalize to smoother activation functions?

replyquoteparent
Thinking Summary

**Exploring activation functions** I need to address the point about the ReLU activation function’s discontinuous derivative, which allows for identifying specific activation points. If we generalize to other activation functions that lack this sharpness, it complicates discerning boundaries and parameters. There’s some noise in the ReLU case, and they use statistical methods for estimation. I'm curious if these approaches could apply to smoother functions and how piecewise linear functions like leaky ReLU or hard sigmoid might fit into this discussion. **Analyzing activation functions** When considering smooth activation functions, approximating second derivatives and inflection points can be quite challenging. For piecewise-linear functions like ReLU and leaky ReLU, critical points remain clear. However, for smooth activations like sigmoid and tanh, parameters become harder to extract due to continuous slope changes. I’d like to explore the implications of querying approximated derivatives and how the logistic function's properties could help recover weight vectors, but we face stability issues and noise. Let's highlight the distinction in detecting activation boundaries between these function types. **Exploring methods for activation functions** There are potential methods to explore, like treating the network as a polynomial composition or using symbolic regression to approximate it. Adding a small dither to the input can help approximate the Jacobian, though this could be costly. For ReLU, boundaries behave like measure-zero hyperplanes detectable through derivative sign changes. With smooth functions, identifying specific boundaries becomes difficult. However, approximating regions with small outputs is possible, but it complicates disentangling weight signs. I need to consider how piecewise-linear approximations might help, or if a gradient oracle is necessary.

Why ReLU is “special”

  1. Piece-wise linearity
    For a ReLU neuron
    g(u)=max(0,u)={0u<0uu0g(u)=\max(0,u)=\begin{cases}0&u<0\\u&u\ge 0\end{cases}

    the input space is split by a single hyper-plane
    H:={xwx+b=0}H:=\{x\mid w^{\top}x+b=0\}.

    • On each side of H the neuron is perfectly linear.
    • Crossing H flips the Jacobian contribution of that neuron from 0 to w()w^{\top}(\cdot).
    • Along any straight line μ(λ)=x1+λ(x2x1)\mu(\lambda)=x_1+\lambda(x_2-x_1) the global network output f(λ)=f(μ(λ))f^*(\lambda)=f(\mu(\lambda)) is therefore piece-wise linear: its first derivative has a jump exactly when μ(λ)\mu(\lambda) hits a hyper-plane belonging to one ReLU.

    Detecting that jump with two finite differences is what lets Canales-Martínez et al. (and Carlini-Jagielski-Mironov 2020) pinpoint a critical point and say: “between λ1\lambda_1 and λ2\lambda_2 one and only one ReLU flipped – recover its weight/sign now”.

  2. Algebraic simplifications
    Once the layer is known to be linear on both sides of a critical point, solving for weights reduces to solving linear (or at worst bilinear) systems. That is why the whole attack ultimately becomes polynomial.

What if the activation is still piece-wise linear but not ‘hard’?

• Leaky-ReLU, PReLU, hard-tanh, hard-sigmoid, ReLU6, etc.
These all keep the same “kink” – the derivative is discontinuous (slope changes from aa to bb).
The attack generalises almost verbatim: in place of “slope = 0 vs slope = 1” you test “slope = a vs slope = b”, which is even easier because both sides remain informative.

• ‘ReLU with noise’ (e.g. dropout, quantisation error).
As long as the measurement noise is bounded and you can average a few repeated queries, the jump in the first derivative is still categorical; the paper already uses 200 samples per neuron in the “neuron-wiggle” stage exactly for this purpose.

Truly smooth activations (sigmoid, tanh, GELU, Swish)

Here the derivative is continuous, so the clean “one-bit jump” disappears:

  1. No unique boundary:
    A sigmoid neuron transitions gradually over an O(w1)O(\|w\|^{-1}) wide slab instead of a zero-measure hyper-plane. Two neighbouring neurons’ transition regions easily overlap, so following a line μ(λ)\mu(\lambda) you see a sum of many smooth bumps; separating them is a de-convolution problem, not a simple change-point detection.

  2. Ambiguous scaling:
    For ReLU you can absorb any positive scalar into the next layer, but negative scalars are forbidden by the sign of the kink. With sigmoids the function σ(αu)\sigma(\alpha u) for α>0\alpha>0 is still monotone and can be re-absorbed by re-scaling the outgoing weights, so recovering the exact scale of ww (and its sign) from black-box queries alone becomes ill-posed.

  3. Higher-order derivatives help only with a gradient oracle:
    If the attacker could query f(x)\nabla f(x) exactly, one could in principle solve non-linear systems that involve σ()\sigma'(\cdot) in place of the ReLU mask. In the pure prediction-only model you would have to estimate those gradients with finite differences; the error grows as O(ε1)O(\varepsilon^{-1}) for first derivatives and O(ε2)O(\varepsilon^{-2}) for second, so the number of queries blows up quickly.

Known theoretical results mirror this: with sigmoids, even deciding consistency of a 2-layer 3-neuron network given labeled samples is NP-complete (Blum–Rivest 1993). Allowing chosen queries helps, but no polynomial-time black-box extraction algorithm is currently known.

Practical take-away

• Piece-wise-linear (kinky) activations ⇒ the Crypto ’20 / TII ’23 extraction works.

• Mild modification (leaky-ReLU, ReLU6, hard-sigmoid, etc.) ⇒ same attack with minor slope accounting.

• Smooth activations ⇒ sharp “critical-point” trick disappears; you would have to shift to optimisation-based model stealing (train a surrogate on many query/response pairs) or to side-channel / gradient-oracle attacks.

So the “sharpness” is not a small implementation detail—it is the linchpin that turns a messy non-linear identification problem into a clean linear-algebra one.

replyquoteparent

Generation Error: streaming error: received error while streaming: {"type":"tokens","code":"rate_limit_exceeded","message":"Request too large for o3 in organization org-cJxzFBGxo2GzbKflv8asSlTY on tokens per min (TPM): Limit 30000, Requested 30678. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.","param":null}

replyquoteparent