View a PDF of the paper titled Distilling Invariant Representations with Dual Augmentation, by Nikolaos Giakoumoglou and 1 other authors
No PDF available, click to view other formats
Abstract:Knowledge distillation (KD) has been widely used to transfer knowledge from large, accurate models (teachers) to smaller, efficient ones (students). Recent methods have explored enforcing consistency by incorporating causal interpretations to distill invariant representations. In this work, we extend this line of research by introducing a dual augmentation strategy to promote invariant feature learning in both teacher and student models. Our approach leverages different augmentations applied to both models during distillation, pushing the student to capture robust, transferable features. This dual augmentation strategy complements invariant causal distillation by ensuring that the learned representations remain stable across a wider range of data variations and transformations. Extensive experiments on CIFAR-100 demonstrate the effectiveness of this approach, achieving competitive results in same-architecture KD.
Submission history
From: Nikolaos Giakoumoglou [view email]
[v1]
Sat, 12 Oct 2024 10:27:23 UTC (191 KB)
[v2]
Sun, 3 Nov 2024 22:31:31 UTC (1 KB) (withdrawn)
[v3]
Fri, 20 Dec 2024 22:10:44 UTC (1 KB) (withdrawn)
Source link
lol