Eduardo Ordax’s Post

𝗧𝗵𝗲 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝗰𝗲 𝗼𝗳 𝗟𝗟𝗠 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻! 🧠 ✔   Although evaluating the outputs of LLMs is key, it remains a challenging task for many. Deciding the right set of evaluation metrics along with the strategy is imperative before moving to production. In this post I’ll help you with both! 1️⃣ 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 𝗦𝗰𝗼𝗿𝗲 (not ideal for LLMs as they don’t take any semantic into account) 🔸 BLEU: evaluates the output against annotated ground truths 🔸 ROUGE: evaluates text summaries from NLP and calculates recall 🔸 METOR: calculates scores by assessing precision and recall 🔸 Levenshtein distance: calculates the minimum number of single-character edits required to change one word into another 2️⃣ 𝗠𝗼𝗱𝗲𝗹-𝗕𝗮𝘀𝗲𝗱 𝗦𝗰𝗼𝗿𝗲 & 𝗡𝗼𝗻-𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 (rely on NLP models, more accurate but also more unreliable due to their probabilistic nature) ➡ 𝗡𝗼𝗻 𝗟𝗟𝗠 𝗦𝗰𝗼𝗿𝗲 🔸 NLI: uses Natural Language Inference models to classify whether an LLM output is logically consistent, contradictory, or unrelated. 🔸 BLEURT: uses pre-trained models like BERT to score LLM outputs on some expected outputs. ➡ 𝗟𝗟𝗠𝗘𝘃𝗮𝗹 🔸 G-Eval: uses LLMs to evaluate LLM outputs 🔸 Prometheus: fully open-source LLM that is comparable to GPT-4’s evaluation capabilities 3️⃣ 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 & 𝗠𝗼𝗱𝗲𝗹-𝗕𝗮𝘀𝗲𝗱 𝗦𝗰𝗼𝗿𝗲 🔸 BERTScore & MoverScore: relies on pre-trained language models like BERT and computes the cosine similarity. 🔸 GPTScore: uses the conditional probability of generating the target text as an evaluation metric. 🔸 SelfCheckGPT: It is a simple sampling-based approach that is used to fact-check LLM outputs. 🔸 QAG Score: leverages LLMs’ high reasoning capabilities to reliably evaluate LLM outputs. And finally, the 𝗟𝗟𝗠 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 will be defined based on: ❇ Do you have the groundtruth ❇ Does the model provide discrete outputs ❇ Do you want to want to automate the process #LLM #AI #GenAI #Evaluation