Thread (18 posts)

Looking at Table 1 in the Gemma 2 paper, "Pre-norm" and "Post-norm" refer to layer normalization applied at different positions within the transformer architecture:

• Pre-norm: Layer normalization applied to the input of each transformer sub-layer (before the attention layer and feedforward layer)
• Post-norm: Layer normalization applied to the output of each transformer sub-layer (after the attention layer and feedforward layer)

The paper explains this in the architecture section:

"Post-norm and pre-norm with RMSNorm. To stabilize training, we use RMSNorm to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer."

Gemma 2 uses both pre-norm and post-norm (both marked as "yes" in the table), meaning they apply RMSNorm normalization both before and after each transformer sub-layer. This dual normalization approach helps stabilize training.

This is different from the original Transformer architecture which only used post-norm, and also different from some modern implementations that only use pre-norm. Using both provides additional training stability at the cost of some computational overhead.

In Table 1, pre-norm and post-norm refer to the placement of normalization layers in the transformer architecture:

• Pre-norm: Normalization applied to the input of each transformer sub-layer (before the attention or feedforward computation)
• Post-norm: Normalization applied to the output of each transformer sub-layer (after the attention or feedforward computation)

According to the paper (Section 2), Gemma 2 uses both pre-norm and post-norm with RMSNorm (Root Mean Square Normalization) to stabilize training. Specifically, they state:

"To stabilize training, we use RMSNorm to normalize the input and output of each transformer sub-layer, the attention layer, and the feedforward layer."

This dual normalization approach is somewhat unusual - many transformer models use either pre-norm or post-norm, but not both. The combination likely helps with training stability, especially for the larger models. All three Gemma 2 model sizes (2B, 9B, and 27B) use this same normalization strategy, as indicated by the "yes" values for both rows in Table 1.

In a Transformer block you have three main operations, executed in this order:

1. The residual input $x$ coming from the previous layer.
2. The sublayer (either self-attention or the feed-forward network, FFN).
3. The residual addition that adds the sublayer output back to $x$.

Where you place the normalisation step(s) relative to those three operations is what the paper’s Table 1 is calling “pre-norm” and “post-norm”.

• Pre-norm (a “yes” in the table) means:
 $y = \text{Sublayer}(\text{Norm}(x)) + x$
 i.e. the input is normalised before it is fed into the sublayer.

• Post-norm (also “yes”) means:
 $z = \text{Norm}(y)$
 i.e. after the residual addition the result is normalised again.

Gemma 2 therefore uses both: every attention or FFN sub-block is wrapped in a dual / sandwich normalisation scheme

x
│
├─► RMSNorm ─► Sublayer ─► + x ─► RMSNorm
│                                │
└────────────────────────────────┘

and it uses RMSNorm instead of LayerNorm for all of these normalisation steps.

Why both?

• Pre-norm is known to stabilise very deep Transformers and makes optimisation easier.
• Post-norm (the extra “sandwich” layer) helps keep the magnitude of activations bounded, improving training stability, calibration and in some cases final accuracy (ideas similar to DeepNorm, NormFormer, and Sandwich-LayerNorm).

So the two columns simply indicate that Gemma 2 applies RMSNorm both before and after each Transformer sub-layer.