Thread (60 posts)

The user is asking about why precision is extremely expensive to evaluate when simulations are using GPT-4 as in Bills et al. (2023). Looking at the paper, they mention this in section 4.3 "Finding simple explanations for features". The context is that they're trying to evaluate explanations for what features (latents) in their sparse autoencoders represent. The Bills et al. (2023) approach (from the linked paper "Language models can explain neurons in language models") uses GPT-4 to generate natural language explanations of what neurons/features are doing, and then uses GPT-4 again to simulate whether those explanations would predict when the neuron/feature activates. To evaluate precision, you need to check: of all the times the explanation predicts the feature should activate, how often does it actually activate? This requires: 1. Running GPT-4 on many text samples to see if the explanation predicts activation 2. Checking against ground truth activations The expense comes from needing to make many GPT-4 API calls for the simulation step - you need to test the explanation on a large corpus of text to get reliable precision estimates. Each test requires a GPT-4 call to simulate whether the explanation would predict activation on that text. In contrast, the N2G (Neuron to Graph) approach they use instead is much cheaper because it just uses n-gram patterns with wildcards rather than natural language explanations that need to be simulated with a language model.