Gemma 2
← Back to all posts
Thread (18 posts)
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is asking about Table 1 in the Gemma 2 paper. Let me look at Table 1 which provides an overview of the main model parameters and design choices. Table 1 shows: - Parameters: 2B, 9B, 27B - d_model values - Layers - Pre-norm: yes for all models - Post-norm: yes for all models - Various other architectural details The user wants to know what "pre-norm" and "post-norm" refer to. Looking at the text, I can find the relevant information in Section 2 (Model Architecture): "Post-norm and pre-norm with RMSNorm. To stabilize training, we use RMSNorm to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer." So it appears that: - Pre-norm refers to normalizing the input to each transformer sub-layer - Post-norm refers to normalizing the output of each transformer sub-layer Both use RMSNorm (Root Mean Square Normalization) and both are applied to the attention layer and feedforward layer sub-layers of the transformer.
In Table 1, pre-norm and post-norm refer to the placement of normalization layers in the transformer architecture:
- Pre-norm: Normalization applied to the input of each transformer sub-layer (before the attention or feedforward computation)
- Post-norm: Normalization applied to the output of each transformer sub-layer (after the attention or feedforward computation)
According to the paper (Section 2), Gemma 2 uses both pre-norm and post-norm with RMSNorm (Root Mean Square Normalization) to stabilize training. Specifically, they state:
"To stabilize training, we use RMSNorm to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer."
This dual normalization approach is somewhat unusual - many transformer models use either pre-norm or post-norm, but not both. The combination likely helps with training stability, especially for the larger models. All three Gemma 2 model sizes (2B, 9B, and 27B) use this same normalization strategy, as indicated by the "yes" values for both rows in Table 1.