CMU 2011 Watson Event
1 like456 views
This document provides a summary of IBM's Watson project to build a question answering system to compete on the game show Jeopardy!. It discusses the goals of capturing public imagination, engaging the scientific community, and being relevant to IBM customers. It outlines the technical challenges involved in natural language processing at scale to analyze evidence and answer open-domain questions with a high degree of confidence compared to human performance. It provides an overview of the DeepQA architecture and incremental progress over time in answering precision.
CMU 2011 Watson Event
- 1. © 2011 IBM Corporation© 2011 IBM Corporation Building Watson A Brief Overview of the Jeopardy! Challenge Dr. Mark Sherman IBM Software Group Strategy
- 2. © 2011 IBM Corporation Capture the imagination – The Next Deep Blue Engage the scientific community – Envision new ways for computers to impact society & science – Drive important and measurable scientific advances Be Relevant to IBM Customers – Enable better, faster decision making – Business Intelligence, Knowledge Discovery and Management, Government, Compliance, Publishing, Legal, Healthcare, Business Integrity, Customer Relationship Management, Web Self-Service, Product Support, etc. A Grand Challenge Opportunity 2
- 3. © 2011 IBM Corporation Informed Decision Making: Search vs. Expert Q&A Decision Maker Search Engine Finds Documents containing Keywords Delivers Documents based on Popularity Has Question Distills to 2-3 Keywords Reads Documents, Finds Answers Finds & Analyzes Evidence
- 4. © 2011 IBM Corporation Informed Decision Making: Search vs. Expert Q&A Expert Understands Question Produces Possible Answers & Evidence Delivers Response, Evidence & Confidence Analyzes Evidence, Computes Confidence Asks NL Question Considers Answer & Evidence Decision Maker
- 5. © 2011 IBM Corporation Informed Decision Making: Search vs. Expert Q&A Decision Maker Search Engine Finds Documents containing Keywords Delivers Documents based on Popularity Has Question Distills to 2-3 Keywords Reads Documents, Finds Answers Finds & Analyzes Evidence Expert Understands Question Produces Possible Answers & Evidence Delivers Response, Evidence & Confidence Analyzes Evidence, Computes Confidence Asks NL Question Considers Answer & Evidence Decision Maker
- 6. © 2011 IBM Corporation6 Broad Domain Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text. We do NOT attempt to anticipate all questions and build databases. In a random sample of 20,000 questions we found 2,500 distinct types*. The most frequent occurring <3% of the time. The distribution has a very long tail. And for each these types 1000’s of different things may be asked. *13% are non-distinct (e.g, it, this, these or NA) Even going for the head of the tail will barely make a dent We do NOT try to build a formal model of the world
- 7. © 2011 IBM Corporation7 What It Takes to compete against Top Human Jeopardy! Players Our Analysis Reveals the Winner’s Cloud Winning Human Performance Winning Human Performance Grand Champion Human Performance Grand Champion Human Performance Each dot – actual historical human Jeopardy! games More ConfidentMore Confident Less ConfidentLess Confident
- 8. © 2011 IBM Corporation8 What It Takes to compete against Top Human Jeopardy! Players Our Analysis Reveals the Winner’s Cloud Winning Human Performance Winning Human Performance 2007 QA Computer System2007 QA Computer System Grand Champion Human Performance Grand Champion Human Performance Each dot – actual historical human Jeopardy! games More ConfidentMore Confident Less ConfidentLess Confident Computers? Not So Good.
- 9. © 2011 IBM Corporation Baseline v0.1 12/07 v0.3 08/08 v0.5 05/09 v0.6 10/09 v0.7 04/10 v0.4 12/08 DeepQA: Incremental Progress in Answering Precision: 6/2007-4/2010 v0.2 05/08
- 10. © 2011 IBM Corporation One Jeopardy! question can take 2 hours on a single 2.6Ghz Core Optimized & Scaled out on 2880-Core IBM HPC using UIMA-AS, Watson is answering in 2-6 seconds. Question 100s Possible Answers 1000’s of Pieces of Evidence Multiple Interpretations 100,000’s scores from many simultaneous Text Analysis Algorithms100s sources . . . Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Synthesis Question & Topic Analysis Question Decomposition Hypothesis Generation Hypothesis and Evidence Scoring Answer & Confidence
- 11. © 2011 IBM Corporation Potential Business Applications Tech Support: Help-desk, Contact Centers Healthcare / Life Sciences: Diagnostic Assistance, Evidenced- Based, Collaborative Medicine Enterprise Knowledge Management and Business Intelligence Government: Improved Information Sharing and Security
- 12. © 2011 IBM Corporation The Core Technical Team Researchers and Engineers in NLP, ML, IR, KR&R and CL at IBM Labs and a growing number of universities
- 13. © 2011 IBM Corporation THANK YOU