A Measure of the System Dependence of Automated Metrics

View a PDF of the paper titled A Measure of the System Dependence of Automated Metrics, by Pius von D”aniken and 2 other authors

View PDF
HTML (experimental)

Abstract:Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect.