Technology & AIJul 3, 2026
Teacher Supervision over Representation Equivalence Classes
Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match.
Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is…
Sign in to learn & save →
The frontier is open to all. Sign in to learn this from first principles and save it to your knowledge base.