You built a large language model that sounds smart. It writes fluent paragraphs and answers questions with confidence. But is it actually useful? That’s the trap many developers fall into. They assume fluency equals accuracy. In reality, a model can hallucinate facts while maintaining perfect grammar. To catch these errors, you need more than a gut feeling. You need task-specific scorecards.

These aren't generic report cards. They are targeted evaluation frameworks designed for specific jobs like summarization, question-answering (Q&A), and information extraction. Each task fails in different ways. A summary might miss key points; an answer might be correct but verbose; an extractor might split entities incorrectly. One-size-fits-all metrics fail here. Let’s break down how to build scorecards that actually tell you if your LLM is ready for production.

The Four Pillars of LLM Evaluation

Before diving into specific tasks, understand the framework. Industry standards, including those from Weights & Biases, suggest four core dimensions for any LLM evaluation:

• Quality: Is the output correct, helpful, and coherent?
• Outcome: Did the user achieve their goal? Did they click "thumbs up" or copy the text?
• Performance: How fast was it? How much did it cost per query?
• Safety & Compliance: Did it leak PII? Was it toxic or biased?

Your scorecard must track all four. Ignoring performance means you might have a great model that bankrupts you. Ignoring safety means you might have a fast model that gets sued. For this article, we’ll focus heavily on Quality, as that’s where most technical complexity lies.

Evaluating Summarization: Beyond ROUGE

Summarization is tricky because there’s no single "right" answer. Two humans might summarize the same news article differently, yet both be correct. This makes traditional metrics like ROUGE a set of n-gram overlap metrics used to evaluate the quality of machine summaries by comparing them to reference summaries insufficient on their own.

ROUGE measures surface-level similarity. If your model paraphrases "The company reported a profit" as "Earnings were positive," ROUGE sees zero overlap. It fails. This is why modern scorecards use a multi-layered approach.

The Ragas Approach: Question-Based Verification

A smarter way to judge summaries is to see if they retain the essence of the source. The Ragas an open-source framework for evaluating Retrieval-Augmented Generation (RAG) pipelines using LLM-as-a-judge techniques framework offers a clever solution called the Summarization Score. Here’s how it works:

1. Extract Keyphrases: Pull important concepts from the original document.
2. Generate Questions: Create yes/no questions based on those keyphrases (e.g., "Did the company report a profit?").
3. Test the Summary: Ask these questions to the generated summary.
4. Score: Calculate the ratio of correctly answered questions to total questions.

This method checks if the summary contains the critical information, not just the exact words. It also adds a conciseness penalty. If your summary is nearly as long as the source, it gets docked points. After all, a summary that doesn’t shorten the text isn’t a summary-it’s a copy.

Using BERTScore and G-Eval

For semantic understanding, look at BERTScore a metric that computes similarity between two texts using contextual embeddings from pre-trained transformer models like BERT. It compares token embeddings rather than raw strings. If "profit" and "earnings" have similar vectors, BERTScore recognizes the match. This solves ROUGE’s paraphrase blindness.

Then there’s G-Eval a framework that uses Large Language Models like GPT-4 to evaluate other LLM outputs based on defined criteria and chain-of-thought reasoning. Instead of math, you ask an LLM to grade the summary. You prompt it: "Rate the coherence and faithfulness of this summary on a scale of 1-5." G-Eval forces the judge to explain its reasoning first, which reduces random scoring. It’s expensive, but for high-stakes domains like legal or medical summaries, it’s worth the cost.

Judging Question-Answering (Q&A)

Q&A evaluation seems simpler: is the answer right or wrong? Not quite. Consider this question: "What is the capital of France?" Answer A: "Paris." Answer B: "The capital city of France is Paris." Both are correct. Answer C: "Paris is a city in Europe." Also correct, but less direct. Your scorecard needs to measure correctness, completeness, and relevance.

Reference-Based vs. Reference-Free

If you have gold-standard answers (like in SQuAD datasets), you can use exact match or F1 scores. These compare string overlaps. But in real-world RAG systems, you rarely have a single gold answer. You have a source document.

Here, you evaluate Faithfulness. Does the answer come from the provided context? Or did the model hallucinate? Use entailment models to check if the answer logically follows from the source text. If the model says "Apple released the iPhone 16 in 2024," but the source only mentions 2023 releases, the entailment score drops.

Also check Answer Relevancy. Sometimes models give a correct fact that doesn’t answer the specific question asked. If someone asks "When was Apple founded?" and the model says "Steve Jobs co-founded Apple," it’s true but irrelevant. Embedding similarity between the question and the answer helps detect this mismatch.

Information Extraction: Precision and Recall

Extraction tasks-pulling names, dates, or relationships-are unforgiving. There’s usually one right answer. However, boundaries matter. Extracting "New York City" vs. "New York" can change downstream results. Your scorecard must handle partial matches.

Handling Entity Boundaries

Standard precision/recall metrics often treat entity spans as binary. Either you got it, or you didn’t. But consider coreference. If the text says "The tech giant announced..." and later "Apple reported...", a good extractor links them. A bad one treats them as separate entities. Your evaluation must account for coreference resolution errors.

Use fuzzy matching for string comparison. Allow for minor variations in spelling or formatting. More importantly, evaluate Schema Adherence. If you asked for JSON output with fields "date," "amount," and "currency," did the model provide exactly that? Malformed JSON breaks pipelines. Add a strict schema validation step to your scorecard.

Building Your Custom Scorecard

Don’t rely on one metric. Triangulate. A robust evaluation pipeline looks like this:

1. Automated Baseline: Run ROUGE/BERTScore for quick coverage checks.
2. LLM Judge: Use G-Eval or Ragas for deeper semantic analysis.
3. Human Audit: Sample 5-10% of outputs for manual review. Humans catch subtle biases or tone issues that algorithms miss.

Tailor your criteria to the domain. A financial summary needs numerical accuracy above all else. A customer support chatbot needs empathy and brevity. Define these weights in your scorecard configuration.

Remember, evaluation isn’t a one-time test. It’s a continuous loop. As you update prompts or swap models, rerun the scorecard. Track trends over time. If BERTScore stays flat but human ratings drop, your automated metrics are lying to you. Adjust accordingly.