ApacheCon NA 2011 report
Download as PPT, PDF1 like1,132 views
The document summarizes the speaker's experience at ApacheCon NA 2011. It discusses several keynotes and sessions attended, including talks on building secure software, the success of Hadoop, Watson's use of Apache technologies, and new features in Lucene 4.0 like improved performance through UTF8 encoding and faster querying through new query types and indexing approaches. The document also mentions several projects that build user interfaces and experiences on top of Solr, like Prism, Blacklight, TwigKit, and Ajax Solr.
ApacheCon NA 2011 report
- 1. ApacheCon NA 2011 Report 2011/12/19 @ijokarumawak
- 2. About myself Nutch Cloudera Certified Hadoop Developer Hadoop Administrator CouchDB JP
- 3. ApacheCon http://na11.apachecon.com/ 2 days training 3 days sessions Keynotes, 5 tracks Over 80 sessions Slide and audio files http://lanyrd.com/2011/apachecon-north-america/
- 4. Why did I go there? Because I wanted to! Nov 5,6: CouchHack Nov 7: CouchConf Berlin Nov 3: Left Japan Nov 14: Came back Nov 9-11: ApachCon Nov 12: Apach BarCamp Image from: http:// en.wikipedia.org/wiki/File:World_map_blank_gmt.svg
- 5. Keynote| Building in Security and Innovation David A. Wheeler A specialist at developing Secure Open Source Software The importance of developing secure software Do not make the same mistake Learn how to make it secure before start to develop it
- 6. Keynote | The Apache Way Done Right: The Success of Hadoop Eric Baldeschwieler co-founder and the CEO of History of Hadoop Difficulty of leading a huge community “ Being optimistic and good things will happen.”
- 7. Keynote | Watson, a Reasoning System: based on Apache Inside! David Boloker CTO of IBM's Emerging Internet Technology group IBM’s Watson won Jeopardy Commercialization of Watson Its target is medical field
- 8. Lucene/Solr Meet up Discussion with core committers of Lucene/Solr Erik Hatcher Chris Hostetter Simon Willnauer We are supposed to drink beer, aren't we?
- 9. Sessions I attended to Lucene 4.0 - next generation open source search Simon Willnauer Solr Flair Erik Hatcher And more… 20 sessions! http://www.atware.co.jp/category/column/apachecon-na-2011/
- 10. Lucene 4.0 - next generation open source search - by Simon Willnauer
- 11. about the author Lucene core committer Project Management Committee chair (PMC) Berlin Buzzwords co-founder http://berlinbuzzwords.de/ Community portal targeting OpenSource Search http:// www.searchworkings.org /
- 12. Lucene 4.0 The latest is currently Lucene 3.5.0 When does the Lucene 4.0 come out? Any time. He doesn’t know.
- 13. IndexWriter & IndexReader Talk to a Directory (file system) Just a factory for input and output streams From Lucene4 Flex API on the Codec layer Codec Defines the file format Data structures Fields, term dictionaries You can use MySQL as a backup (it’s not a good idea though) 90% won’t get in touch 10% might be researchers Backward compatibility File System Directory Codec Flex API IndexWriter & Reader
- 14. Storing Strings in UTF8 Lucene 3 uses UTF16 From Lucene 4, UTF8 Performance will improve when you switched to Lucene 4
- 15. PostingsFormat PostingsFormat can be defined per field field:uid = Pulsing – PostingsFormat Usually 1 doc per uid Inlines postings into term dictionary Safes additional disc lookup field:spell = Memory – PostingsFormat Spelling correction doesn’t need posting list traversal Large amount of key lookups Load terms into RAM field:body = Default – PostingsFormat Primary Key lookup 170K qps -> 550K qps with Memory PostingsFormat Term Dictionary Posting List Term Posting List RAM Terms
- 16. IndexDocValues Lucene uses inverted index ( Term to Doc ) It’s not good at to get a value of certain field from a document Fast access to a certain field’s value for every document To sort documents or to display doc’s values not only its ID Stored Fields It works but it’s not an efficient way It’s designed for bulk read FieldCache ( on RAM ) Undo the entire work in the indexing time to make an array (un-inverting) It works well until certain size of the index It can be a problem under real-time or near-real-time usecases IndexDocValue 1 value per field, type safe It can reside on disk Reading 10M docs from a disc FieldCache: 3161 ms DocValues: 90 ms Term Doc Doc Doc How to sort docs?
- 17. DWPT (Document Writer Per Thread) In Lucene 3 IndexWriter merges segments and flushes it to the disk While flushing data, multi-threaded IndexWriter takes a break From Lucene 4 IndexWriter doesn’t merge data anymore It flushes its own segment to the disc simultaneously less RAM more Concurrency
- 18. Automaton Query Automaton Query RegExp: (ftp|http).* Fuzzy: dogs~1 Fuzzy-Prefix: (dogs~1).* Fuzzy query was too slow to use in production Prior to 4.0, Fuzzy query took the simple yet horribly costly brute force approach In Lucene 3 this is about 0.1 - 0.2 QPS Now it’s 50 QPS, 20k% improvement! http://java.dzone.com/news/lucenes-fuzzyquery-100-times
- 19. Solr Flair by Erik Hatcher
- 20. Solr Flair User Interfaces User Interactions Ajax suggestion Did you mean? – Spell Checking Facet Cluster .. So on
- 21. wt = velocity http://wiki.apache.org/solr/VelocityResponseWriter Solritas /browse
- 22. Prism https://github.com/lucidimagination/Prism Requires Lucid Works Enterprise JRuby with Sinatra gem installed Production use of LucidWorks Enterprise requires an annual subscription It’s free to play :’)
- 23. blacklight http://projectblacklight.org/ Ruby on Rails DEMO http:// demo.projectblacklight.org / Being used by Universities University of Versinia http:// search.lib.virginia.edu/catalog?portal = all&q = lucene Stanford University http:// searchworks.stanford.edu/?q = lucene+in+action&search_field =search
- 24. VUFind http:// vufind.org / blacklight competitor library resource portal PHP DEMO http://vufind.org/demo/
- 25. TwigKit http:// twigkit.com / JSP tag library Search UI components Samples http:// twigkit.com/components.html
- 26. Ajax Solr https://github.com/evolvingweb/ajax-solr Javascript library goes with JQuery DEMO http://evolvingweb.github.com/ajax-solr/examples/reuters/index.html
- 27. ApacheCon 2012 ApacheCon EUROPE November 2012 Germany!!?
- 28. Thank you!