Thread (26 posts)

The neuron wiggle technique can identify inputs that maximally wiggle a target neuron because by this stage of the attack, the adversary already knows the signatures (relative weight ratios) of all neurons, even though they don't know the signs.

Here's how it works:

Key Prerequisites

Before using neuron wiggle, the attack has already:

1. Recovered the signatures of all neurons using critical points (the ratios like $(1, \frac{a_2}{a_1}, \ldots, \frac{a_\ell}{a_1})$)
2. Fully recovered all previous layers (which have been "peeled off")
3. Obtained $\hat{A}^{(i)}$ - the weight matrix for layer $i$ up to unknown signs

Creating the Wiggle

The crucial insight is that maximizing the size of a wiggle doesn't require knowing the sign. If you have weights $(a_1, a_2, \ldots, a_n)$ up to sign, you can create a wiggle $\delta$ parallel to these weights:

$\delta \parallel \hat{A}_j^{(i)} = (\pm |a_1|, \pm |a_2|, \ldots, \pm |a_n|)$

This wiggle maximizes $|\langle \hat{A}_j^{(i)}, \delta \rangle|$ regardless of whether the true weights are positive or negative, because:

• If the true weights are $(a_1, a_2, \ldots)$, the dot product gives $|a_1|^2 + |a_2|^2 + \ldots$
• If the true weights are $(-a_1, -a_2, \ldots)$, you still get the same magnitude

The Process

1. Project onto space of control: They project $\hat{A}_j^{(i)}$ onto $V^{(i-1)}$ (the subspace they can actually control at layer $i$)

2. Find input difference: They compute $\Delta \in \mathbb{R}^{d_0}$ by finding a pre-image of $\delta$ under $F^{(i-1)}$ (the fully recovered previous layers)

3. Test at critical point: At a critical point $x^*$ for the target neuron, they evaluate:

   • $L = f(x^* - \Delta) - f(x^*)$
   • $R = f(x^* + \Delta) - f(x^*)$

4. Determine sign: If $|L| > |R|$, the neuron has negative sign; if $|R| > |L|$, it has positive sign

The key is that they can create an input perturbation that maximally affects the target neuron using only the signature information, then use the asymmetric behavior around critical points to determine which sign is correct.

The signature recovery is described in Section 3.3 ("Carlini et al.'s Differential Attack"), specifically in the subsection titled "Finding signatures".

This section explains how to recover the signatures (the relative weight ratios) of neurons by:

1. For the first hidden layer: Using critical points and computing partial derivatives:

   • Query $\alpha_{i,-} = \frac{\partial f}{\partial e_i}(x^* - \varepsilon e_i)$ and $\alpha_{i,+} = \frac{\partial f}{\partial e_i}(x^* + \varepsilon e_i)$
   • The difference $\alpha_{i,+} - \alpha_{i,-}$ gives a multiple of weight $a_i$
   • Taking ratios $\frac{\alpha_{i,+} - \alpha_{i,-}}{\alpha_{k,+} - \alpha_{k,-}} = \frac{a_i}{a_k}$ reveals the signature

2. For deeper layers: Since you can't control individual coordinates after layer 1, they:

   • Sample $d_i + 1$ random directions $\delta_k \sim \mathcal{N}(0, \varepsilon I_{d_0})$
   • Compute second derivatives $y_k = \frac{\partial^2 f(x^*)}{\partial \delta_1 \partial \delta_k}$
   • Solve for the signature vector $a$ such that $\langle h_k, a \rangle = y_k$

The paper notes that this signature recovery technique is from Carlini et al.'s prior work, and the new contributions in this paper focus on the sign recovery methods (SOE, Neuron Wiggle, and Last Hidden Layer techniques) described in Section 4.