[Submitted on 27 Sep 2024]
View a PDF of the paper titled Exploring Language Model Generalization in Low-Resource Extractive QA, by Saptarshi Sengupta and 4 other authors
Abstract:In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize well to closed-domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to empirically explain the performance gap. Our findings suggest that: a) LLMs struggle with dataset demands of closed-domains such as retrieving long answer-spans; b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; c) Scaling model parameters is not always effective for cross-domain generalization; and d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.
Submission history
From: Saptarshi Sengupta [view email]
[v1]
Fri, 27 Sep 2024 05:06:43 UTC (1,361 KB)
Source link
lol
