Super Weights in LLMs and the Failure of Selective Training

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude.

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important,…