Self-distillation

Self-distillation improves model performance in early training steps, then gradually declines as the model overfits to its own imperfect signal.