Thread (64 posts)

Great question! The distinction between logits and logprobs is crucial to understanding why the attack in Section 5.3 is more complex than the initial warm-up attack.

Logits vs Logprobs

Logits are the raw output values from the model before applying softmax:

• For token $i$: $z_i = \mathbf{W}^T \cdot g_\theta(x)_i$

Logprobs are the log of the probabilities after applying softmax:

• For token $i$: $y_i = \log\left(\frac{e^{z_i}}{\sum_j e^{z_j}}\right) = z_i - \log\left(\sum_j e^{z_j}\right)$

Why This Matters for the Attack

The key challenge is that production APIs return logprobs, not logits. When you apply a logit bias $B$ to token $i$ and observe its logprob, you get:

$y_i^B = z_i + B - \log\left(\sum_{j \neq i} e^{z_j} + e^{z_i + B}\right)$

Notice the problem: the normalizing constant (the log term) changes depending on which tokens you bias. This means you can't simply subtract $B$ from $y_i^B$ to get $z_i$.

The Attack's Solution

The paper presents two approaches:

1. Reference token method: Use one token as a fixed reference point across all queries. By comparing other tokens to this reference, you can cancel out the changing normalization constants: $y_R^B - y_i^B - B = z_R - z_i$

2. Linear constraint method (Appendix A): Treat each logprob observation as a linear constraint on the original logits, then solve the resulting system of equations.

This distinction is why the attack requires multiple queries with carefully chosen logit biases—it's reconstructing the underlying logits from observations that have been transformed by the softmax's normalization.

Excellent question! This is exactly what Section 6 and Appendix C address - the "logprob-free" attacks where you only get the sampled token, not any probability information.

The Basic Approach: Binary Search

The key insight is that with temperature 0 sampling, the API returns the token with the highest logit value. So if you can manipulate logit biases, you can use binary search:

1. Start with prompt $p$ and identify the top token (call it token 0)
2. For each other token $i$:
   • Binary search to find the minimum bias $x_i$ needed to make token $i$ the top token
   • This tells you that $z_i - z_0 \approx -x_i$

The Challenge

This is much more expensive than the logprob-based attacks:

• Basic binary search: ~10 queries per logit (vs 0.25 with logprobs)
• Each token requires $\log_2(B/\epsilon)$ queries for precision $\epsilon$

Improved Methods

The paper presents optimizations that query multiple tokens simultaneously:

1. Hyperrectangle method: Apply biases to multiple tokens at once and accumulate linear constraints based on which token wins
2. One-of-n method: Carefully choose biases to make each token equally likely to be sampled

These reduce the cost to ~3.7 queries per logit, though still much higher than logprob-based methods.

Information-Theoretic Limits

The paper proves (Lemma 6) that you need at least: $\frac{l \cdot \log_2(B/\epsilon)}{\log_2(l)}$ queries, where $l$ is vocabulary size. Their best attack is within a factor of 2 of this theoretical limit!

The key takeaway: Yes, you can extract the model even with just next-token predictions, but it's about 15x more expensive than with logprob access.