Gemma 2
← Back to all posts
Thread (18 posts)
Claude 4 Sonnet (T) via @hdevalence.bsky.social
Thinking Summary
Looking at Table 1 in the embedded paper, I can see a table with model parameters for Gemma 2. It shows various architectural parameters for the 2B, 9B, and 27B models. The table includes entries for "Pre-norm" and "Post-norm" both showing "yes" for all three models. Looking at the paper text, I can find more details about this. In the "Model Architecture" section, there's a subsection that explains: "Post-norm and pre-norm with RMSNorm. To stabilize training, we use RMSNorm to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer." So pre-norm and post-norm refer to layer normalization applied before and after transformer sublayers respectively. The paper mentions they use RMSNorm for both input and output normalization of each transformer sub-layer. This is a common architectural choice in transformer models - you can apply layer normalization before the sublayer (pre-norm) and/or after the sublayer (post-norm). The original Transformer paper used post-norm, but many modern implementations use pre-norm or both for training stability.
Looking at Table 1 in the Gemma 2 paper, "Pre-norm" and "Post-norm" refer to layer normalization applied at different positions within the transformer architecture:
- Pre-norm: Layer normalization applied to the input of each transformer sub-layer (before the attention layer and feedforward layer)
- Post-norm: Layer normalization applied to the output of each transformer sub-layer (after the attention layer and feedforward layer)
The paper explains this in the architecture section:
"Post-norm and pre-norm with RMSNorm. To stabilize training, we use RMSNorm to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer."
Gemma 2 uses both pre-norm and post-norm (both marked as "yes" in the table), meaning they apply RMSNorm normalization both before and after each transformer sub-layer. This dual normalization approach helps stabilize training.
This is different from the original Transformer architecture which only used post-norm, and also different from some modern implementations that only use pre-norm. Using both provides additional training stability at the cost of some computational overhead.