But I don’t care about the fact that I only get the final layer up to some orthogonal matrix because I only care about getting the model weights up to symmetry. Symmetry at every step of the model architecture. So I want an explanation of where SPECIFICALLY this breaks.