Measuring the Quality of Answers in Political Q&As with Large Language Models

View a PDF of the paper titled Measuring the Quality of Answers in Political Q&As with Large Language Models, by R. Michael Alvarez and Jacob Morrier

View PDF
HTML (experimental)

Abstract:This paper introduces a new approach for measuring the quality of answers in political question-and-answer sessions. We propose to measure answer quality based on the degree to which it allows to infer the initial question accurately. This measure of answer quality reflects how well the answer engages with and addresses the initial question. Drawing an analogy with semantic search, we demonstrate that this measurement approach can be implemented by fine-tuning a large language model on the corpus of observed questions and answers without additional labeled data. We showcase our approach within the context of the Question Period in the Canadian House of Commons, providing valuable insights into the correlates of answer quality. Our findings reveal significant variations in answer quality based on the party affiliation of the members of Parliament asking the question. Additionally, we find a meaningful correlation between answer quality and the topic raised in the question.