LLM Evaluation For Text Summarization

Evaluating text summarization is difficult because there is no one correct solution, and summarization quality often depends on the summary’s context and purpose.

Metrics like ROUGE, METEOR, and BLEU focus on N-gram overlap but fail to capture the semantic meaning and context.

LLM-based evaluation approaches like BERTScore and G-eval aim to address these shortcomings by evaluating semantic similarity and coherence, providing a more accurate assessment.

Despite these advancements and the widespread use of LLM-generated summaries, ensuring robust and comprehensive evaluation remains an open problem and active area of research.

Text summarization is a prime use case of LLMs (Large Language Models). It aims to condense large amounts of complex information into a shorter, more understandable version, enabling users to review more materials in less time and make more informed decisions.

Despite being widely applied in sectors such as journalism, research, and business intelligence, evaluating the reliability of LLMs for summarization is still a challenge. Over the years, various metrics and LLM-based approaches have been introduced, but there is no gold standard yet.

In this article, we’ll discuss why evaluating text summarization is not as straightforward as it might seem at first glance, take a deep dive into the strengths and weaknesses of different metrics, and examine open questions and current developments.

How does LLM text summarization work? 

Summarization is a classic machine-learning (ML) task in the range of natural language processing (NLP). There are two main approaches:

• Extractive summarization creates a summary by selecting and extracting key sentences, phrases, and ideas directly from the original text. Accordingly, the summary is a subset of the original text, and no text is generated by the ML model. Extractive summarization relies on statistical and linguistic features—either explicitly or implicitly—such as word frequency, sentence position, and significance scores to identify important sentences or phrases.
• Abstractive summarization produces new text that conveys the most critical information from the original. It aims to identify the key information and generate a concise version. Abstractive summarization is typically performed with sequence-to-sequence models, a category to which LLMs with encoder-decoder architecture belong.

Dimensions of text summarization quality

There is no single objective measure for the quality of a summary, whether it’s created by a human or generated by an LLM. On the one hand, there are many different ways to convey the same information. On the other hand, what are the key pieces of information in a text is context-dependent and often debatable.

However, there are widely agreed-upon quality dimensions along which we can assess the performance of text summarization models:

• Consistency characterizes the summary’s factual and logical correctness. It should stay true to the original text, not introduce additional information, and use the same terminology.

• Relevance captures whether the summary is limited to the most pertinent information in the original text. A relevant summary focuses on the essential facts and key messages, omitting unnecessary details or trivial information.
• Fluency describes the readability of the summary. A fluent summary is well-written and uses proper syntax, vocabulary, and grammar.
• Coherence is the logical flow and connectivity of ideas. A coherent summary presents the information in a structured, logical, and easily understandable manner.

Metrics for text summarization

Metrics focus on the summary’s quality rather than its impact on any external task. Their computation requires multiple reference summaries crafted by human experts as ground truth. The quality and diversity of these reference summaries significantly influence the metric’s effectiveness. Poorly constructed references can lead to misleading scores.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE is one of the most common metrics used to evaluate the quality of summaries compared to human-written reference summaries. It determines the overlap of groups of words or tokens (N-grams) between the reference text and the generated summary.

ROUGE has multiple variants, such as ROUGE-N (for N-grams), ROUGE-L (for the longest common subsequence), and ROUGE-S (for skip-bigram co-occurrence statistics).

If the summarization is limited to key term extraction, ROUGE-1 is the preferred choice. For simple summarization tasks, it is better to use ROUGE-2 metrics. For a more structured summarization, ROUGE-L and ROUGE-S might be the best fit.

While ROUGE is popular for extractive summarization, it can also be used for abstractive summarization. A high value of the ROUGE score indicates that the generated summary preserves the most essential information from the original text.

How does the ROUGE metric work?

To understand how the ROUGE metrics work, let’s consider the following example:

• Human-crafted reference summary: The cat sat on the mat and looked out the window at the birds.
• LLM-generated summary: The cat looked at the birds from the mat.

ROUGE-1

1. Tokenize the summaries

First, we tokenize the reference and the generated summary into unigrams:

2. Calculate the overlap

Next, we count the overlapping unigrams between the reference and generated summaries:

Overlapping unigrams:

[‘The’, ‘cat’, ‘looked’, ‘at’, ‘the’, ‘birds’, ‘the’, ‘mat’]

There are eight overlapping unigrams.

3. Calculate precision, recall, and F1 score

a) Precision = Number of overlapping unigrams​ / Total number of unigrams in generated summary
Precision = 8/9 ​= 0.89

b) Recall = Number of overlapping unigrams​ / Total number of unigrams in reference summary
Recall = 8/14 = 0.57

c) F1 score = 2 × (Precision×Recall​) / (Precision+Recall)
F1 = 2 × (0.89×0.57) / (0.89+0.57) ​= 0.69

ROUGE-2

1. Tokenize the summaries

First, we tokenize the reference and the generated summary into bigrams:

2. Calculate the overlap

Next, we count the overlapping bigrams between the reference and generated summaries:

Overlapping bigrams:

[‘the cat’, ‘looked at’, ‘at the’, ‘the birds’, ‘the mat’]

There are five overlapping bigrams.

3. Calculate precision, recall, and F1 score

a) Precision = Number of overlapping bigrams / Total number of bigrams in generated summary
Precision = 5/8 ​= 0.625

b) Recall = Number of overlapping bigrams / Total number of bigrams in reference summary
Recall = 5/13 = 0.385

c) F1 score = 2 × (Precision×Recall​) / (Precision+Recall)
F1 = 2 × (0.625×0.385​) / (0.625+0.385​) ​= 0.476

ROUGE-L

1. Tokenize the summaries

First, we tokenize the reference and the generated summary into unigrams:

2. Find the largest overlap

The longest common sequence is [“the”, “cat”] with a length of two.

3. Calculate precision, recall, and F1 score

a) Precision = Length of longest common sequence / Total number of unigrams in generated summary
Precision = 2/9 = 0.22

b) Recall = Length of longest common sequence / Total number of unigrams
Recall = 2/14 = 0.143

c) F1 score = 2 × (Precision×Recall​) / (Precision+Recall)
F1 = 2 × (0.22 × 0.143)/(0.22 + 0.143) = 0.174

ROUGE-S

To calculate the ROUGE-S (ROUGE-Skip) score, we need to count skip-bigram co-occurrences between the reference and generated summaries. A skip-bigram is any pair of words in their respective order, allowing for gaps.

1. Tokenize the summaries

First, we tokenize the reference and the generated summary into unigrams:

2. Generate the skip-bigrams for reference and generate summaries.

Skip-bigrams for reference summary:

(“The”, “cat”), (“The”, “sat”), (“The”, “on”), (“The”, “the”), …

(“cat”, “sat”), (“cat”, “on”), (“cat”, “the”), …

(“sat”, “on”), (“sat”, “the”), (“sat”, “mat”), …

Continue for all combinations, allowing skips.

Skip-bigrams for generated summary:

(“The”, “cat”), (“The”, “looked”), (“The”, “at”), (“The”, “the”), …

(“cat”, “looked”), (“cat”, “at”), (“cat”, “the”), …

(“looked”, “at”), (“looked”, “the”), (“looked”, “birds”), …

Continue for all combinations, allowing skips.

3. Count the total number of skip-bigrams in the reference and the generated summary

There is no need to count the number of skip-bigrams manually. For a text with n words:

Number of skip-bigrams = (n x (n – 1)) / 2

Total skip-bigrams in reference summary: (14 × (14 − 1)) / 2 ​= 91

Total skip-bigrams in generated summary: (9 × (9 − 1)​) / 2 = 36

4. Calculate ROUGE-S score 

Finally, count the number of skip-bigrams in the reference summary that also appear in the generated summary. The ROUGE-S score is calculated as follows:

ROUGE-S = (2 × count of matching skip-bigrams​) /  (total skip-bigrams in reference summary + total skip-bigrams in generated summary)

The matching bi-grams in the reference and generated summary will be as follows:

(“The”, “cat”), (“The”, “looked”), (“The”, “at”), (“The”, “the”), (“cat”, “looked”), (“cat”, “at”), (“cat”, “the”), (“looked”, “at”), (“looked”, “the”), (“looked”, “birds”), (“at”, “the”), (“at”, “birds”), (“the”, “birds”)

Matching skip-bigrams: 13

ROUGE-S = (2 × 13) / (91 + 36) ​= 26 / 127​ ≈ 0.2047

Interpretation of ROUGE metrics

ROUGE is a recall-oriented metric that ensures that the generated summary includes as many relevant tokens of the reference summary as possible. Similar to information retrieval problems, we compute the precision, recall, and F1 score.

Focusing solely on achieving high ROGUE precision can result in missing important details, as we might generate fewer words to boost precision. Focusing too much on recall favors long summaries that include additional but irrelevant information. Typically, looking at the F1 score that balances both measures is best.

In our example, the high value of the ROUGE-1 F1 score indicates fairly good coverage of the key concepts, while the lower value of the ROUGE-2 F1 score indicates a change in verbs and missing connections between key terms.

Problems with ROUGE metrics

• Surface-level matching: ROUGE matches the exact N-grams from the reference and generated summaries. It fails to capture the semantic meaning and context of the text. ROUGE does not handle synonyms, meaning two semantically identical summaries with different wording have low ROUGE scores. Paraphrased content, which conveys the same meaning with different wording, receives a low ROUGE score despite being a good summary.
• Recall-oriented nature: ROUGE’s primary goal is to measure the completeness of the generated summary in terms of how much of the important content from the reference summary it captures. This can lead to high scores for longer summaries that include many reference terms, even if they contain irrelevant information.
• Lack of evaluation for coherence and fluency: ROUGE does not evaluate the text’s coherence, fluency, or overall readability. A summary that contains the right N-grams achieves a high ROUGE score, even if it is disjointed or grammatically incorrect.

METEOR (Metric for Evaluation of Translation with Explicit Ordering)

Extracting all important keywords from a text does not necessarily mean that the summary produced is of high quality. A logical flow of information should be maintained, even if the information is not presented in the same order as the original document.

When using an LLM, the generated summary likely contains different words or synonyms. In this case, metrics like ROGUE based on exact keyword matches will yield low scores even if the summary is of high quality.

METEOR is a summarization metric similar to ROGUE that matches words by reducing them to their root or base form through stemming and lemmatization. For example, “playing,” “plays,” “played,” and “playful” all become “play.”

Additionally, METEOR assigns higher scores to summaries that focus on the most important information from the source. Information that is repeated multiple times or irrelevant receives lower scores. It does so by calculating a fragmentation penalty by checking if a chunk is a sequence of matched words in the same order as they appear in the reference summary.

How does the METEOR metric work?

Let’s consider an example of a generated summary from an LLM and a human-crafted summary. 

• Human-crafted reference summary: The cat sat on the mat and looked out the window at the birds.
• LLM-generated summary: The cat looked at the birds from the mat.

1. Tokenize the summaries

First, we tokenize both summaries:

2. Identify exact matches

Next, we identify exact matches between the reference and generated summaries:

Exact matches:

[“the”, “cat”, “looked”, “at”, “the”, “birds”, “the”, “mat”]

3. Calculate precision, recall, and F1 score

a) Precision = Number of matched tokens / Total number of tokens in the generated summary
Precision = 8/9 = 0.89

b) Recall = Number of matched tokens ​ / Total number of words in reference summaryRecall = 8/14 = 0.57

c) Harmonic mean of precision and recall = (10×Precision×Recall​) / (Recall+9×Precision)
F-mean = (10×0.8889×0.5714) / (0.5714+9×0.8889) = 5.0793 / 8.4925​ ≈ 0.5980

4. Calculate the fragmentation penalty

Determine the number of “chunks.” A “chunk” is a sequence of matched tokens in the same order as they appear in the reference summary.

Chunks in the generated summary:

[“the”, “cat”], [“looked”, “at”, “the”, “birds”], [“the”, “mat”]

There are three chunks in the generated summary. The fragmentation penalty is calculated as:
P = 0.5 × (Number of chunks​) / (Number of matched words

P = 0.5 × 3/8 = 0.1875

5. Final METEOR score

The final METEOR score is calculated as follows:

METEOR = F-mean × (1−P) = 0.5980 × (1−0.1875) ≈ 0.5980×0.8125 ≈ 0.4866

Interpreting the METEOR score

The METEOR score ranges from 0 to 1, where a score close to 1 indicates a better match between the generated and reference text. METEOR is recall-oriented and ensures that the generated text captures as much information from the reference text.

The harmonic mean between precision and recall F-mean is biased towards recall and is the key indicator for the summary’s completeness. A low fragmentation penalty indicates that the summary is coherent and concise.

For our example, the METEOR score is approximately 0.4866, indicating a moderate level of alignment with the reference summary.

Problems with the METEOR metric

• Limited contextual understanding: METEOR does not capture the contextual relationship between words and sentences. As it focuses on word-level matching rather than sentence or paragraph coherence, it might misjudge the relevance and importance of information in the summary.

Despite improvements over ROUGE, METEOR still relies on surface forms of words and their alignments. This can lead to an overemphasis on specific words and phrases rather than understanding the deeper meaning and intent behind the text.

• Sensitivity to paraphrasing and synonym use: Although METEOR uses stemming for synonyms and paraphrasing, its effectiveness in capturing all possible variations is limited. It does not recognize semantically equivalent phrases that use different syntactic structures or less common synonyms.

BLEU (Bilingual Evaluation Understudy)

BLEU is yet another popular metric for evaluating LLM-generated text. Initially designed to evaluate machine translation, it is also used to evaluate summaries.

BLEU measures the correspondence between a machine-generated text and one or more reference texts. It compares the N-grams from the LLM-generated and reference texts and computes a precision score. These scores are then combined into an overall score through a geometric mean.

One advantage of BLEU compared to ROGUE and METEOR is that it can compare the generated text to multiple reference texts for a more robust evaluation. Also, BLEU includes a brevity penalty to prevent the generation of overly short texts that achieve high precision but omit important information.

How does the BLEU metric work?

Let’s use the same example we used above. 

1. Tokenize the summaries

First, we tokenize both summaries:

2. Calculate matching N-grams

Next, we find matching unigrams, bigrams, and trigrams and calculate the precision (matching N-grams / total N-grams in generated summary).

a) Unigrams (1-grams):

Matches:

[“the”, “cat”, “looked”, “at”, “the”, “birds”, “the”, “mat”]

Total unigrams in generated summary: 9

Precision: 8/9 = 0.8889

b) Bigrams (2-grams):

Matches:

[“the cat”, “at the”, “the birds”, “the mat”]

Total bigrams in generated summary: 8

Precision: 4/8 = 0.5

c) Trigrams (3-grams):

Matches:

[“the cat looked”, “cat looked at”, “looked at the”, “at the birds”, “the birds the”, “birds the mat”]

Total trigrams in generated summary: 7

Precision: 2/7 = 0.2857

d) Determine the brevity penalty

The brevity penalty is based on the length of the reference and the generated summary:

Length of the reference summary: 14 tokens
Length of the generated summary: 9 tokens
Brevity penalty: e^{(1−14 / 9)} = e^−0.5556 ≈ 0.5738

e) Calculate the BLEU score

Combined precision:
We combine the N-gram precisions with weights (usually uniform weights, e.g., 1/4 for 1-gram, 2-gram, 3-gram, 4-gram) and apply the brevity penalty.

P = (0.8889^0.25) × (0.5^0.25) × (0.2857^0.25)

P ≈ 0.927 × 0.84 × 0.76 ≈ 0.595

Calculate the final BLEU score by multiplying the brevity penalty and combined precision:

BLEU = BP × P ≈ 0.5738 × 0.595 ≈ 0.342

Interpreting the BLEU score

BLEU is a precision-oriented metric that evaluates the content present in the generated summary. The BLUE score ranges between 0 and 1, where a score close to 1 indicates a highly accurate summary, a score between 0.3 and 0.7 indicates a moderately accurate summary, and a score close to 0 indicates a lower quality of the generated summary.

BLEU is best used together with recall-oriented metrics like ROUGE and METEOR to evaluate the summary’s quality more comprehensively.

The calculated BLEU score for our example is 0.342, which means the LLM-produced text has moderate quality.

Problems with the BLEU score

• Surface-level matching: Similar to ROUGE and METEOR, BLEU relies on the exact N-gram matching between the generated text and reference text and fails to capture the semantic meaning and context of the text. BLEU does not handle synonyms or paraphrases well. Two summaries with the same meaning but different wording will have a low BLEU score due to the lack of exact N-gram matches.
• Effective short summaries are penalized: BLEU’s brevity penalty was designed to discourage overly short translations. It can penalize concise and accurate summaries that are shorter than the reference summary, even if they capture the essential information effectively.
• Higher order N-grams limitation: BLEU evaluates N-grams up to a certain length (typically 3 or 4). Longer dependencies and structures are not well captured, missing out on evaluating the coherence and logical flow of longer text segments.

LLM evaluation frameworks for summarization tasks

ROUGE and METEOR metrics focus on surface-level matching of N-grams and exact/stemmed/synonym matches, but they do not capture semantic meaning or context.

LLM evaluation frameworks, such as BERT and GPT, have been developed to address this limitation by focusing on understanding the actual meaning and coherence of the content.

BERTScore

BERTScore is an LLM-based framework that evaluates the quality of a generated summary by comparing it to a human-written reference summary. It leverages the contextual embeddings (vector representations of each word’s meaning and context) provided by pre-trained language models like BERT (Bidirectional Encoder Representations from Transformers).

BERTScore examines each word or token in the candidate summary and uses the BERT embeddings to determine which word in the reference summary is the most similar. It uses similarity metrics, majorly cosine similarity, to assess the closeness of the vectors.

Using the BERT model’s understanding of language, BERTScore finds the most related word from the generated summary in the reference summary. To get the overall score of semantic similarity between the reference summary and the candidate summary, all of these word similarities are compared. The higher the BERTScore, the better the summary generated by LLM models.

How does BERTScore work?

1. Tokenization and embedding extraction

First, we tokenize the candidate summary and the reference summary. Each token is converted into its corresponding contextual embedding using a pre-trained language model like BERT. Contextual embeddings consider the surrounding words to generate a meaningful vector representation for each word.

2. Cosine-similarity calculation

Next, we compute the pairwise cosine similarity between each embedded token in the candidate summary and each embedded token in the reference summary. The maximum similarity scores for each token are retained and then used to compute the precision, recall, and F1 scores.

a) Precision calculation: Precision is calculated by averaging the maximum cosine similarity for each token in the generated summary. For each token in the generated summary, we find the token in the reference summary that has the highest cosine similarity and average these maximum values.

b) Recall calculation: Recall is calculated in a similar manner. For each token in the reference summary, we find the token in the generated summary that has the highest cosine similarity and average these maximum values.

c) F1 score: The F1 score is the harmonic mean of the precision and recall.

Interpreting BERTScore

By calculating the similarity score for or all tokens, BERTScore takes into account both the syntactic and semantic relevance context of the generated summary compared to the human-crafted reference.

For the BERTScore, precision, recall, and F1 scores are all given equal importance. A high score for all these metrics indicates a high quality of the generated summary.

Problems with BERTScore

• High computational cost: Compared to the metrics discussed earlier, BERTScore requires significant computational resources to compute embeddings and measure similarity. This makes it less practical for large datasets or real-time applications.
• Dependency on pre-trained models: BERTScore relies on pre-trained transformer models, which may have biases and limitations inherited from their training data. This can affect the evaluation results, particularly for texts that differ significantly from the training domain of the pre-trained models.
• Difficulty in interpreting scores: BERTScore, being based on dense vector representations and cosine similarity, can be less intuitive to interpret compared to simpler metrics like ROUGE or BLEU. People may find it challenging to understand what specific scores mean in terms of text quality, which complicates debugging and improvement processes.
• Lack of standardization: There is no single standardized version of BERTScore, leading to variations in implementations and configurations. This lack of standardization can result in inconsistent evaluations across different implementations and studies.
• Overemphasis on semantic similarity: BERTScore focuses on capturing semantic similarity between texts. This emphasis can sometimes overlook other important aspects of summarization quality, such as coherence, fluency, and factual accuracy.

G-Eval 

G-Eval is another evaluation metric that harnesses the power of large language models (LLMs) to provide sophisticated, nuanced evaluations of text summarization tasks. It is an example of an approach known as LLM-as-a-judge. As of 2024, G-Eval is considered state-of-the-art for evaluating text summarization tasks.

G-Eval assesses the quality of the generated summary across four dimensions: coherence, consistency, fluency, and relevance. It passes prompts that include the generated and a reference summary to a GPT model. G-Eval uses four separate prompts, one for each dimension, and seeks a score between 1 to 5 from the LLM model.

How does G-Eval work?

• Input texts: Both the reference summary and the candidate (generated) summary are provided as inputs to the LLM.
• Criteria-specific prompts: Four prompts are used to guide the LLM to evaluate coherence, consistency, fluency, and relevance.

Here is the prompt template for evaluating the generated summary for a new article:

“””

You will be given one summary written for a news article.

Your task is to rate the summary on one metric.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

Relevance (1-5) – selection of important content from the source. The summary should include only important information from the source document. Annotators were instructed to penalize summaries which contained redundancies and excess information.

Evaluation Steps:

1. Read the summary and the source document carefully.

2. Compare the summary to the source document and identify the main points of the article.

3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.

4. Assign a relevance score from 1 to 5.

Example:

Source Text:

{{Document}}

Summary:

{{Summary}}

Evaluation Form (scores ONLY):

– Relevance:

“””

Different prompts for different evaluation criteria are available. Users can also create a custom prompt to capture additional dimensions.

• Scoring mechanism: The LLM outputs scores or qualitative feedback based on its understanding and evaluation of the summaries.
• Aggregate evaluation: Scores for different evaluation dimensions are aggregated to assess the summary comprehensively.

Problems with G-Eval

• Bias and fairness: Like any automated system, G-Eval can reflect biases in the training data or the choice of evaluation metrics. This can lead to unfair assessments of summaries, especially across different demographic or content categories.
• High computational cost: G-Eval uses GPT models, which require significant computational resources to compute embeddings and generate scores for different evaluation dimensions.
• Lack of calibration: Since an LLM provides the score based on a user-provided prompt, it is not calibrated. Thus, G-Eval is similar to asking different users to rate a summary on a five-star scale, but it is inconsistent across different summaries.

Open problems with current evaluation methods and metrics for LLM text summarization

One of the major issues with LLM text summarization evaluation is that metrics like ROUGE, METEOR, and BLEU rely on N-gram overlap and cannot capture the true meaning and context of the summaries. Particularly for abstractive summaries, they fall short of human evaluators.

Relying on human experts to write and assess reference summaries makes the evaluation process costly and time-consuming. Also, these evaluators can sometime suffer from subjectivity and variability making the standardization difficult across different evaluators. 

Another significant open challenge is evaluating the factual consistency. All metrics we discussed in this article do not effectively detect factual inaccuracies or misleading interpretation of the summarized source.

Current metrics also struggle sometimes to assess if the context and logic flow are preserved from the original piece of text. They fail to capture whether a summary includes all the critical information without unnecessary fluff or repetition.

It is likely that we will witness more advanced LLM-based evaluation methods in the coming years. The extensive use of LLMs for text summarization, including the integration of summarization features in search engines, makes research in this field highly popular and relevant.

Conclusion

After reading this article, you got a brief idea about the LLMs for text summarization. You have taken a look at different automated and LLM-based evaluation metrics like ROUGE, BLEU, METEOR, BERTScore, and G-Eval. You have been introduced to their working principle and the limitations that each of these metrics have. The best part is, that you need not implement these metrics from scratch, libraries like Hugging Face evaluate, Haystack, and LangChain provide ready-to-use implementations. 

While ROUGE, METEOR, and BLEU metrics are simple and fast to compute, they do not focus on the semantic matching of the generated summary with the reference one. While BERTScore and G-Eval try to resolve this issue, they have their own infrastructure requirements that can incur some costs. You can also use a combination of these metrics to make sure that your generated summary makes total sense. Apart from these LLM-based models, you can also fine-tune an open-source LLM to work as an LLM-as-a-Judge for your evaluation purpose.