The Difference Between Layer Norm and RMS Norm

Norm is short for Normalization, an operation that normalizes data — generally into a distribution with mean 0 and variance 1. Compared to Layer Norm, RMS Norm omits one step of dividing by the mean.

Layer Norm directly normalizes at the layer level. There are generally two kinds: pre-normalization (pre norm) and post-normalization (post norm).

Pre-normalization normalizes the input directly and then computes the result; post-normalization computes the result first and then normalizes it uniformly.

Although some online materials suggest that Post Norm performs better than Pre Norm, in practice the popular network models of today — such as LLaMA and BLOOM — all use Pre Layer Norm, while ChatGLM uses Post Deep Norm.

Going further, LLaMA 2 switched to RMSNorm, which not only performs well but, more importantly, further improves computational efficiency.