Improving LLM Interpretability for Business Teams

One of the major hurdles in adopting LLMs for many companies has been the risk of model-generated hallucinations and a lack of transparency. In response to this, there has been a concerted effort to enhance monitoring mechanisms and establish robust checks and balances around these models. This includes the development of more transparent evaluation models that align closely with human judgment (JudgeLM - https://lnkd.in/e8Xspek9), the implementation of customizable evaluation criteria tailored to specific business needs (FoFo - https://lnkd.in/eAMutjXJ), and open-source frameworks like DeepEval (https://lnkd.in/eYMB-Xiw ) which track aspects such as toxicity and hallucination using a variety of NLP models such as QA bi-encoders, vector similarity tools, and NLI models. This week, a few additional methods and frameworks were intoruduced: 1. Cleanlab's 𝗧𝗿𝘂𝘀𝘁𝘄𝗼𝗿𝘁𝗵𝘆 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹 (𝗧𝗟𝗠) incorporates a trust score for each output, enhancing reliability and transparency by indicating the likelihood of an output being accurate. This framework is particularly helpful for customer facing applications and where the cost of errors is high (https://lnkd.in/e9FUztgj) 2. 𝗣𝗿𝗼𝗺𝗲𝘁𝗵𝗲𝘂𝘀 𝟮 (https://lnkd.in/er_3nqGt) released by LG researchers is developed using weight merging of separately trained evaluators: one that directly scores the outputs (direct assessment) and another that ranks the outputs (pairwise ranking). In extensive benchmark tests across both direct assessment and pairwise ranking, Prometheus 2 achieved the highest correlations and agreement scores with human evaluators, demonstrating a substantial advancement over existing methods. 3. “When to Retrieve” (https://lnkd.in/euANnwWg) presents an innovative model, the 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗟𝗟𝗠 (𝗔𝗗𝗔𝗣𝗧-𝗟𝗟𝗠), which intelligently decides when to utilize external information retrieval to enhance its question-answering capabilities. ADAPT-LLM is trained to generate a special token ⟨RET⟩ when it needs more information to answer a question, signifying a retrieval is necessary. Conversely, it relies on its intrinsic knowledge when confident in its response. This model outperformed fixed strategies like always retrieving information or solely relying on its own memory. There is no doubt that significant investments are being made to enhance the robustness and reliability of LLMs, with stringent checks and balances being established. Meanwhile, tech giants like Google are not shying away from deploying LLMs in sensitive areas, as seen with their MedGemini project - a highly capable multimodal model specialized in medicine, demonstrating superior performance in medical benchmarks and tasks (https://lnkd.in/epbv63iN). These developments indicate a rapid progression towards broader and more impactful deployments of LLMs in critical sectors soon.

LLM Explainability: most explainability techniques just use source attribution today. For instance, source attribution might be adequate for Q&A where proof of provenance is straightforward when content is found in a single page. But it is not enough when you asked to summarize a 100-page document, and it is almost impossible to determine what information (or depth) the LLM is using to create the summary. To demystify the “black box” of LLMs, it is a good ideas to use a combination of techniques Eval & monitoring metrics + LLM visualization tools + ability to backtrack when response quality is not making sense. 👉 Define metrics for explainability that works well with LLM. For example start with triad metrics for RAG: Context relevance, Groundedness and Answer relevance (see picture) 👉 Use LLM to evaluate other LLM. Auto Evaluate response on perplexity, BLEU, ROUGE, DIVERSITY metrics works well. 👉 Leverage Visualization tools like BertViz and Phoenix that lets you visualize how the LLM black box is working 👉  The journey into LLM interpretability is not a solitary one. Engaging with the LLM Interpretability community (https://lnkd.in/enUG2zZj) is super helpful. The quest for explainability in LLMs is more than a technical challenge; it’s a step towards creating AI systems that are accountable, trustworthy, and aligned with human values. Here is a great paper on LLM explainability Survey : https://lnkd.in/eXthTvUy #llm #explainability

Can we look inside the “brain” of LLMs to find where financial concepts like risk, trading, or financial instruments are represented? While analyzing financial data, we identified a feature - Feature 471, in Gemma-2B model that consistently activated on statements about defaults, ratings downgrades, and credit risk. We present this along with many other research in our latest paper, “Beyond the Black Box: Interpretability of LLMs in Finance : https://lnkd.in/epqwXwaV. This is the first work in finance to examine the inner circuits of LLMs using mechanistic interpretability— bringing a neuroscience- perspective to understanding how LLMs reason about financial risk. This wasn’t general sentiment. It reflected credit-specific reasoning, emerging from the model’s internal structure - with no task-specific tuning. When we adjusted this feature’s activation, the model’s interpretation of credit quality shifted — producing more targeted, risk-aware outputs. This is mechanistic interpretability in action: identifying and modulating internal circuits to steer behavior — without retraining or prompt engineering. Of course, LLMs are complex systems, and attributing meaning to a single feature has its limits. Still, this approach opens new possibilities for adoption and explainability of LLMs in a highly regulated sector such as finance. Thanks to the researchers from Anthropic and Google DeepMind, whose work was leveraged for the paper. Shoutout to my co-author Ariye Shater. #FinanceAI #CreditRisk #MechanisticInterpretability #TradingAI #LLMs #ModelTransparency #FinNLP #AIAlignment