What's the relationship, if any, between concepts of sparseness in interpretability of language models and of pleiotropy in genetics
← Back to all posts
Thread (2 posts)
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
This is an interesting question connecting two fields: interpretability of language models (specifically sparseness) and genetics (specifically pleiotropy). Let me think through the connections. In language model interpretability: - Sparseness typically refers to the idea that individual neurons or features in a neural network are active for only a small subset of inputs - This is often desirable for interpretability because it means each neuron/feature has a more specific, understandable role - Sparse representations can make it easier to understand what specific patterns or concepts a neuron is detecting In genetics: - Pleiotropy refers to when a single gene affects multiple, seemingly unrelated phenotypic traits - This is the opposite of a one-gene-one-trait relationship - Pleiotropic genes create complex webs of effects throughout an organism The relationship between these concepts is actually quite interesting - they're almost opposites in terms of what they represent for interpretability: 1. Sparseness in ML interpretability is about features having narrow, specific roles 2. Pleiotropy in genetics is about genes having broad, multiple effects But there are some deeper connections and parallels worth exploring...
There's a fascinating tension between these concepts that reveals something deep about complex systems and interpretability.
Sparseness in ML interpretability seeks neurons/features that activate for narrow, specific concepts - the ideal is a one-to-one mapping between neurons and interpretable features. This makes the system more understandable because each component has a clear, isolated function.
Pleiotropy in genetics is almost the opposite - single genes affecting multiple traits, creating a many-to-many mapping that makes the system harder to understand. A pleiotropic gene might influence both eye color and digestive enzyme production, for instance.
The Key Insight
Both phenomena emerge from optimization pressure in high-dimensional spaces:
- Neural networks face pressure to be parameter-efficient while maintaining performance, leading to distributed representations where concepts get entangled across neurons (analogous to pleiotropy)
- Evolution faces similar pressure - reusing genetic components for multiple functions is more efficient than evolving separate genes for each trait
The Interpretability Challenge
In both domains, we see a fundamental trade-off:
- Sparse, modular systems → easier to interpret but less efficient
- Dense, pleiotropic systems → harder to interpret but more efficient
Recent work on mechanistic interpretability (like Anthropic's work on "features" vs "neurons") suggests that language models might have sparse features at a higher level of abstraction, even if individual neurons are polysemantic (activated by multiple unrelated concepts - the ML equivalent of pleiotropy).
This parallel suggests that both biological and artificial systems naturally evolve toward entangled representations when optimized for performance, making interpretability an inherently difficult problem requiring us to find the right level of abstraction to recover sparseness.