Andy Hall - Build verifiable explainability into financial services workflows with Automated Reasoning checks

Editor's Notes

• #4: * Sometimes the hallucination is big * Sometimes it’s subtle. Can you find it here? Take your time
• #5: * Surprise
• #6: * Back to our Gartner friends
• #7: I don’t want you to read me as bashing LLMs We want LLMs to hallucinate – it’s their super-power When it matters, we need to find a way to feel comfortable with using them
• #8: I don’t want you to read me as bashing LLMs We want LLMs to hallucinate – it’s their super-power When it matters, we need to find a way to feel comfortable with using them
• #9: The NAIC operates similarly to a trade association. It is a federation of insurance commissioners with a founding goal of education and identify leading practices. It does no regulatory authority. The NAIC can issue non-binding guidance, but it does not issue laws, rules, or regulation. There is no obligation to operate according to NAIC guidance. Dec 4, 2023 bulletin to align state insurance commissioners on the usage and controls needed to agentic systems.
• #10: Guaranteed Safe AI defined in May 10th 2024 Paper calls out the usage of a Safety Specification Read points on the slide here Verifier – provides the ability to explain actions and provides a deterministic mathematical proof
• #11: Automated Theorem Proving is a specialized branch of computer science that uses mathematical proof techniques and formal logical deduction to verify compliance with rules and requirements with absolute certainty under given assumptions. SMT Solvers (Satisfiability Modulo Theories) is a tool that aims to determine if a mathematical formula is satisfiable using a practical subset of inputs, and it generalizes SAT (Boolean Satisfiability problem) to more complex formulas involving real numbers and various data structures.
• #13: We want to be accurate – clearly identify factual claims that are incorrect Soundness to us means that we won’t draw false conclusion from true premises. We’d rather tell you we are not sure and you should clarify than tell you something is correct when it’s not We can be transparent. We can logically and verifiable explain why we believe something is correct
• #14: Unlike probabilistic approaches prevalent in machine learning, Automated Reasoning relies on formal mathematical logic to provide definitive guarantees about what can and can’t be proven. This approach mirrors the rigors of auditors verifying financial statements or compliance officers validating regulatory requirements, but with mathematical precision. By using rigorous logical frameworks and theorem-proving methodologies, Automated Reasoning can conclusively determine whether statements are true or false under given assumptions. This makes it exceptionally valuable for applications that demand high assurance and need to deliver unambiguous conclusions to their users.
• #15: Works well to validated factual claims and procedures Things that boil down to a yes/no, always/never answer
• #16: Step 1: Source Document, along with Intent Instructions are passed to the Automated Reasoning Service to build the Rules and Variables and create an AR policy Step 2: AR Policy is Created and Versioned Step 3: An AR Policy and version is associated with a Bedrock Guardrail Step 4: An ApplyGuardrail API call is made with the question and FM Response to the associated Bedrock Guardrail Step 5: The Automated Reasoning model is triggered with the inputs from the ApplyGuardrail API building logical representations of the input and FM response Step 6: An Automated Reasoning check is completed based on the created rules and variables from the source document, and the logical representation of the inputs Step 7: The results of the AR are shared with the user along with what rules, variables, and variable values where used in it’s determination, along with suggestions on what would make the assertion valid
• #25: Step 1: Source Document, along with Intent Instructions are passed to the Automated Reasoning Service to build the Rules and Variables and create an AR policy Step 2: AR Policy is Created and Versioned Step 3: An AR Policy and version is associated with a Bedrock Guardrail Step 4: An ApplyGuardrail API call is made with the question and FM Response to the associated Bedrock Guardrail Step 5: The Automated Reasoning model is triggered with the inputs from the ApplyGuardrail API building logical representations of the input and FM response Step 6: An Automated Reasoning check is completed based on the created rules and variables from the source document, and the logical representation of the inputs Step 7: The results of the AR are shared with the user along with what rules, variables, and variable values where used in it’s determination, along with suggestions on what would make the assertion valid
• #26: Together, avengers assemble, right?