Numinex
← Back to all posts

Thread (26 posts)

↴1 of 6 continuations→
@hdevalence.bsky.social

can you unpack and motivate the definition of the signature of a neuron? there are a few layers of nested redefinitions of terms before getting to the definition of the tuple. what are we doing here?

replyquoteparent
⧉
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary

The user is asking about the definition of the "signature of a neuron" from the paper. Let me trace through the definitions to understand what's happening. Looking at Definition 4 in the paper: "Let ℓ = d_{i-1} and A_j^{(i)} be described as (a_1, a_2, ..., a_ℓ). The signature of the j-th neuron in layer i is the tuple (a_1/a_1 = 1, a_2/a_1, ..., a_ℓ/a_1)." This is defining the signature as the normalized weight vector where each component is divided by the first component a_1. The context and motivation seems to be: 1. In a neural network extraction attack, we want to recover the weights and biases 2. Due to the ReLU activation function, we can only recover weights up to a scaling factor (including possibly a sign flip) 3. The "signature" is a way to represent the weights that is invariant to positive scaling - it's essentially the direction of the weight vector 4. But it's NOT invariant to negative scaling (sign flips), which is why they need separate "sign recovery" techniques The nested definitions are building up the neural network structure: - Definition 1: Defines the overall network as alternating linear layers and ReLU activations - Definition 2: Defines each linear layer as an affine transformation f_i(x) = A^{(i)}x + b^{(i)} - Definition 3: Defines a single neuron as one row of this transformation followed by ReLU - Definition 4: Defines the signature as the normalized version of that row The key insight is that if you multiply all weights (including bias) of a neuron by a positive constant c, and then multiply all the weights in the next layer that receive this neuron's output by 1/c, the overall network function doesn't change. This is a symmetry of the network. I can only extract the relative proportions between weights, revealing the fundamental directional information of the neuron's weight vector without its absolute scale.

The signature of a neuron is a clever representation that captures what's actually recoverable in a neural network extraction attack. Let me unpack the layers:

The Setup

Starting from a neuron's basic operation:

  • A neuron in layer ii has weights (a1,a2,...,aℓ)(a_1, a_2, ..., a_\ell) where ℓ=di−1\ell = d_{i-1}
  • It computes: σ(a1x1+a2x2+...+aℓxℓ+b)\sigma(a_1 x_1 + a_2 x_2 + ... + a_\ell x_\ell + b)

The Fundamental Problem

ReLU networks have an inherent symmetry: if you scale all weights of a neuron by a positive constant cc:

  • Neuron computes: σ(c⋅(a1x1+...+aℓxℓ+b))\sigma(c \cdot (a_1 x_1 + ... + a_\ell x_\ell + b))
  • Then scale all outgoing connections by 1/c1/c in the next layer
  • The overall network function remains identical

This means we can never recover the absolute scale of the weights through black-box queries.

Why the Signature?

The signature (1,a2/a1,...,aℓ/a1)(1, a_2/a_1, ..., a_\ell/a_1) represents what we can uniquely recover:

  1. Scale-invariant representation: Dividing by a1a_1 normalizes the weight vector, removing the unknown scaling factor

  2. Unique identifier: Different neurons (with different weight directions) will have different signatures, allowing us to:

    • Group critical points by which neuron they belong to
    • Track neurons across different observations

  3. But preserves sign ambiguity: The signature alone doesn't tell us if we have (a1,a2,...)(a_1, a_2, ...) or (−a1,−a2,...)(-a_1, -a_2, ...). This is crucial because:

    • Positive scaling is a symmetry (doesn't change function)
    • Negative scaling is NOT a symmetry (ReLU blocks negative values)

The Attack Strategy

The paper's approach is:

  1. Find critical points where neurons transition between active/inactive
  2. Use these to recover signatures (relative weight ratios)
  3. Separately determine signs to resolve the (+)(+) vs (−)(-) ambiguity
  4. Combine to get actual weights (up to a positive scale that doesn't matter)

The signature definition is essentially encoding "what information about the weights can we extract from observing input-output behavior, given the symmetries of ReLU networks?"

replyquoteparent
⧉