Self-distillation
Self-distillation improves model performance in early training steps, then gradually declines as the model overfits to its own imperfect signal.
Self-distillation improves model performance in early training steps, then gradually declines as the model overfits to its own imperfect signal.