Watson and Open Source Tools
- 1. Watson & Open Source Software Ivan Portilla IT Architect 5/8/12 portilla@gmail.com Sunday, May 20, 12 1
- 2. If I have seen further it is by standing on the shoulders of giants. Isaac Newton, Letter to Robert Hooke, February 5, 1675 Sunday, May 20, 12 2
- 3. Objectives By the end of this session, you should be able to: ü Describe the main characteristics of Watson QA system. ü Identify the key open source SW used in Watson. ü Recognize examples of Agile development best practices. 3 Sunday, May 20, 12 3
- 4. Disclaimers Sunday, May 20, 12 4
- 5. Disclaimer 1 ü This presentation represents the view of the author and does not represent the view of IBM. ü All opinions expressed in this presentation are strictly of the speaker, and do NOT represent those of IBM, IBM management, or anyone else. ü IBM and IBM (logo) are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. Sunday, May 20, 12 5
- 6. Disclaimer 2 I (We) do not work for the Watson team. Sunday, May 20, 12 6
- 7. Let’s Play Jeopardy Sunday, May 20, 12 7
- 8. Let’s Play Jeopardy BEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed. Sunday, May 20, 12 7
- 9. Let’s Play Jeopardy BEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed. Sunday, May 20, 12 7
- 10. Let’s Play Jeopardy BEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed. COMMON BONDS: trout, loose change in your pocket, and compliments. Sunday, May 20, 12 7
- 11. Let’s Play Jeopardy BEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed. COMMON BONDS: trout, loose change in your pocket, and compliments. Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north. Sunday, May 20, 12 7
- 12. Let’s Play Jeopardy BEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed. COMMON BONDS: trout, loose change in your pocket, and compliments. Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north. Sunday, May 20, 12 7
- 13. Let’s Play Jeopardy BEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed. COMMON BONDS: trout, loose change in your pocket, and compliments. Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north. Geography: Chile shares its longest land border with this country Sunday, May 20, 12 7
- 14. Natural Language Processing Understanding natural language is hard! Sunday, May 20, 12 8
- 15. Watson educa@on hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 Sunday, May 20, 12 9
- 16. A Brief History of Watson Sunday, May 20, 12 10
- 17. A Brief History of Watson § Deep Blue Ended in 1997 § Looking for a new research challenge § 2004, IBM Research manager Charles Lickel, § Ken Jennings § Started in 2005 • David Ferrucci • DeepQA in 2007 • Won Jeopardy Match, Feb 2011 hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 Sunday, May 20, 12 11
- 18. A Brief History of Watson § Deep Blue Ended in 1997 § Looking for a new research challenge § 2004, IBM Research manager Charles Lickel, § Ken Jennings § Started in 2005 • David Ferrucci • DeepQA in 2007 • Won Jeopardy Match, Feb 2011 hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 Sunday, May 20, 12 11
- 19. 11/2010 4/2010 10/2009 5/2009 12/2008 Precision 8/2008 5/2008 12/2007 Baseline hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 Sunday, May 20, 12 12
- 20. What is Watson? ü Understands natural language. ü Generates & evaluates hypothesis for beAer outcomes. ü Adapts & learns from user selec@ons and responses. hAp://www.ibm.com/innova@on/us/watson/ Sunday, May 20, 12 13
- 21. Watson metrics Development Team: 25 people Project Duration: 4 years Hardware: 90 IBM Power-750 servers 2880 Power7 cores @ 80+ TFLOPS 20 TB Disk, 16 TB RAM (memory) 10 Gbps network hAp://na11.apachecon.com/talks/19932 hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 Sunday, May 20, 12 14
- 22. Open Source Software Sunday, May 20, 12 15
- 23. OSS -‐ Linux hAp://video.linux.com/videos/linuxcon-‐vancouver-‐day-‐2-‐1/ Sunday, May 20, 12 16
- 24. Open Source too hAp://ocw.mit.edu/index.htm hAp://@.arc.nasa.gov/opensource/ Sunday, May 20, 12 17
- 25. How does it work? Sunday, May 20, 12 18
- 26. Learning To Rank (Basic Architecture) Hypothesis Keywords Evidence Scoring Watson by R.Yates Sunday, May 20, 12 19
- 27. Watson Architecture hAp://en.wikipedia.org/wiki/Watson_(computer) Sunday, May 20, 12 20
- 28. Who is the 44th President of the United States? Sunday, May 20, 12 21
- 29. Who is the 44th President of the United States? Who is the 44th President of the United States? Watson by R.Yates Sunday, May 20, 12 22
- 30. Who is the 44th President of the United States? 'Who' is the '44th' 'President' of the 'United States'? Who is the 44th President of the United States? Lexical Focus Keywords Answer Type Can be replaced by the correct answer to make a → Person true statement Sunday, May 20, 12 23
- 31. Who is the 44th President of the United States? Keywords: 44th President United States Question Question Hypothesis Analysis Generation Watson by R.Yates Sunday, May 20, 12 24
- 32. Who is the 44th President of the United States? Keywords: 44th President United States Question Question Hypothesis Analysis Generation Sunday, May 20, 12 25
- 33. Who is the 44th President of the United States? Primary Search Question Question Hypothesis Analysis Generation Sunday, May 20, 12 26
- 34. Who is the 44th President of the United States? Barack Obama George W. Bush Harvard Law School Illinois Primary Search Question Question Hypothesis Analysis Generation Watson by R.Yates Sunday, May 20, 12 27
- 35. Who is the 44th President of the United States? Who is the 44th President of the United States? Barack Obama Who is the 44th President of the United States? George W. Bush Answer Who is the 44th President of the United States? Scoring Harvard Law School Who is the 44th President of the United States? Illinois Question Scoring Question Hypothesis Scoring Analysis Generation Scoring Sunday, May 20, 12 28
- 36. Who is the 44th President of the United States? Who is the 44th President of the United States? Barack Obama Who is the 44th President of the United States? Who is the 44th President of the United States? George W. Bush Answer Answer Scoring Contextual → Person Scoring Answer Answer Who is the 44th President of the United States? Answer Scoring Scoring Is Barack Obama a Person? .90 Harvard Law School Scoring Is George W. Bush a Person? .90 Is Harvard Law School a Person? .10 Who is the 44th President of the United States? Is Illinois a Person? .15 Illinois Question Scoring Question Hypothesis Scoring Analysis Generation Scoring Watson by R.Yates Sunday, May 20, 12 29
- 37. Who is the 44th President of the United States? Barack Obama is the 44th President of the United States George W. Bush is the 44th President of the United States Harvard Law School is the 44th President of the United States Illinois is the 44th President of the United States Contextual Answer Scoring Question Scoring Question Hypothesis Scoring Analysis Generation Scoring Unstructured Informa@on Management Applica@ons -‐ UIMA Sunday, May 20, 12 30
- 38. Who is the 44th President of the United States? Barack Hussein Obama II (i/bəәˈrɑːk huːˈseɪn oʊˈbɑːməә/; born August 4, 1961) is the 44th and current President of the United States. George Walker Bush (born July 6, 1946) is an American politician who served as the 43rd President of the United States from 2001 to 2009 and the 46th Governor of Texas from 1995 to 2000. Barack Obama is the 44th President of the United States George W. Bush is the 44th President of the United States Harvard Law School is the 44th President of the United States Illinois is the 44th President of the United States Question Barack Obama .95 George W. Bush .80 Scoring Harvard Law School .05 Question Hypothesis Scoring Illinois.10 Analysis Generation Scoring Unstructured Informa@on Management Applica@ons -‐ UIMA Watson by R.Yates Sunday, May 20, 12 31
- 39. Who is the 44th President of the United States? Candidate Answer Answer Evidence retrieval & Confidence Scoring scoring Barack Obama 0.90 0.90 .95 George W. Bush 0.90 0.80 .65 Evidence Harvard Law School 0.10 0.05 Retrieval .05 Illinois 0.15 0.10 .10 Trained Question Models Scoring Question Hypothesis Scoring Analysis Generation Scoring Unstructured Informa@on Management Applica@ons -‐ UIMA Watson by R.Yates Sunday, May 20, 12 32
- 40. DeepQA Massively Parallel Probabilistic Evidence-Based Architecture Learned Models help combine and weigh the Evidence Evidence Balance Answer Sources & Combine Models Models Sources Question Answer Evidence Models Models Scoring Retrieval Primary Candidate & Scoring Models Models Search Answer Generation Ques@on & Final Confidence Ques@on Hypothesis Hypothesis and Evidence Topic Synthesis Merging & Decomposi@on Genera@on Scoring Analysis Ranking Hypothesis Hypothesis and Merging & Answer & Evidence Scoring Ranking Genera@on Confidence ... ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker Sunday, May 20, 12 33
- 41. OSS in Watson 1.0 Sunday, May 20, 12 34
- 42. OSS in Watson üUIMA, UIMA-‐AS üHadoop, Map Reduce üLucene, Indri ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker Sunday, May 20, 12 35
- 43. UIMA hAp://uima.apache.org/ Sunday, May 20, 12 36
- 44. UIMA-‐Asynchronous Scaleout UIMA AS provides more flexible and powerful scale out capability. Innovate 2011, How Does It Work? The Architecture of Watson. Grady Booch Sunday, May 20, 12 37
- 45. Think Hadoop A framework for storing & processing big data. üUp to 4,000 machines üUp to 20 PB High reliability done in soiware: üAutomated failover for data & computa@on üImplemented in Java hAp://hadoop.apache.org/mapreduce/ Sunday, May 20, 12 38
- 46. Map reduce hAp://hadoop.apache.org/mapreduce/ Sunday, May 20, 12 39
- 47. UIMA pipelines in Hadoop Mul@ple threads, each runing a UIMA pipeline Thread Thread Input Mapper . . . Output Thread Thread Thread Reducer Mapper Thread . . . ~5000 “splits” Shuffle/Sort . . Thread Thread . . ~50 -‐ 100 . . “mappers” ~400-‐800 threads . . Thread . Thread Mapper . . . Thread Reducer Hadoop Thread Hadoop Distributed Thread Distributed File Mapper Thread File . . . System Thread System Thread hAp://blogs.apache.org/founda@on/entry/apache_innova@on_bolsters_ibm_s 41 Sunday, May 20, 12 40
- 48. Architecture hAp://lucene.apache.org Sunday, May 20, 12 41
- 49. Indri &Lemur Indri is a text search engine developed at Umass & CMU. Indri is part of the Lemur project. hAp://lemurproject.org/indri/ Sunday, May 20, 12 42
- 50. Watson answers in 2-‐6 seconds Ques@on 1000’s of 100s Possible 100,000’s scores from many simultaneous Text Pieces of Evidence Analysis Algorithms 100s sources Answers Mul@ple Interpreta@ons Ques@on & Ques@on Hypothesis Hypothesis and Evidence Final Confidence Synthesis Topic Analysis Decomposi@on Genera@on Scoring Merging & Ranking Hypothesis Hypothesis and Evidence Genera@on Scoring Answer & Confidence . . . ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker © 2011 IBM Corporation Sunday, May 20, 12 43
- 51. Other OSS in Watson hAps://www.ibm.com/developerworks/mydeveloperworks/blogs/InsideSystemStorage/entry/ ibm_watson_how_to_build_your_own_watson_jr_in_your_basement7?lang=en Sunday, May 20, 12 44
- 52. J-‐Archive data hAp://www.j-‐archive.com/showgame.php?game_id=3577 Sunday, May 20, 12 45
- 53. Development Process üWar room setting with continuous collaboration. üWeekly integration. üResults driven with E2E regression testing. ü About 8,000 experiments ü 10 GBs of test data/wk. ü Agile development Innovate 2011, How Does It Work? The Architecture of Watson. Grady Booch Sunday, May 20, 12 46
- 54. Other OSS hAp://manning.com/ hAp://www.apache.org Sunday, May 20, 12 47
- 55. Take Away OSS is powerful and scalable enough for the Watson team, what about your project? Sunday, May 20, 12 48
- 56. Resources IBM Journal of Research and Development hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 IBM Watson hAp://www.ibm.com/innova@on/us/watson/ hAp://www.research.ibm.com/deepqa/index.shtml Nova hAp://www.pbs.org/wgbh/nova/tech/smartest-‐machine-‐on-‐earth.html Sunday, May 20, 12 49
- 57. Review of Objectives Now that you have completed this session, you are able to: ü Describe the main characteristics of Watson QA system. ü Identify the key open source tools used in Watson. ü Recognize examples of Agile development best practices. 51 Sunday, May 20, 12 50
- 58. …any final questions ? 52 Sunday, May 20, 12 51