Discussion on the Distributed Search Engine
0 likes766 views
The document discusses FARE, a peer-to-peer distributed search system that allows for real-time indexing without a master-slave configuration. FARE uses consistent hashing to distribute data and queries across nodes connected in a Chord chain. The system supports real-time indexing through direct RPC communication between nodes for sharing data and querying using an inverted index structure.
Discussion on the Distributed Search Engine
- 1. Discussion on the P2P Distributed Search System Yusuke FUJISAKA, CyberAgent inc. 2011 6 18
- 2. • • • • • • 2011 6 18
- 3. Agenda • • • • • 2011 6 18
- 4. • •( :) 2011 6 18
- 5. • 2 • 1 100 →10 /sec. • TB • 2011 6 18
- 6. • • • • • ... 2011 6 18
- 7. • ( TB ) • • • •… 2011 6 18
- 8. • Namazu • Senna • Lucene • Solr • Hyper Estraier • etc. ... 2011 6 18
- 9. • Solr ( ) Broker Search Application Slave Index Data Master Index 2011 6 18
- 10. • Solr ( ) Broker Search Application Slave Index Data Master Index 2011 6 18
- 11. • Solr ( ) Broker Search Application Slave Index Data Master Index 2011 6 18
- 12. • • • • 2011 6 18
- 13. • → • 2011 6 18
- 14. • • Zoie (with Lucene) • Caffeine (Google) • ... 2011 6 18
- 15. FARE system • • Fast Autonomous Retrieval Engine • 2011 6 18
- 16. 1.No Master-Slave • 2.Peer to Peer • 3.Realtime indexing • 2011 6 18
- 17. No Master-Slave • … • 1 →Primary mode • →Secondary mode 2011 6 18
- 18. Primary mode • <Setting> ... Content = Title Content = Body Appendix = PostID Appendix = URL Appendix = PostTime 2011 6 18
- 19. Primary mode • <Setting> ./fare --primary SETTING 2011 6 18
- 20. Secondary mode 10.0.1.1 10.0.1.2 10.0.1.3 ./fare --secondary 10.0.1.1 2011 6 18
- 21. Secondary mode 10.0.1.1 10.0.1.2 Setting, Data 10.0.1.3 ./fare --secondary 10.0.1.1 2011 6 18
- 22. Secondary mode 10.0.1.1 10.0.1.2 10.0.1.3 ./fare --secondary 10.0.1.2 2011 6 18
- 23. Secondary mode 10.0.1.1 10.0.1.2 Setting, Data 10.0.1.3 ./fare --secondary 10.0.1.2 2011 6 18
- 24. Secondary mode 10.0.1.1 10.0.1.2 Setting, Data 10.0.1.3 ./fare --secondary 10.0.1.2 2011 6 18
- 25. • • • Consistent hashing 2011 6 18
- 26. • Chord chain F A B E C D 2011 6 18
- 27. • Chord chain F Hash: 0xEF459AB... A B E C D 2011 6 18
- 28. • Chord chain F Hash: 0xEF459AB... A B E Hash: 0x6D137EA... C D 2011 6 18
- 29. •F F A B E C D 2011 6 18
- 30. F A B E (replication=3) C D 2011 6 18
- 31. • • 2011 6 18
- 32. •1 1 2011 6 18
- 33. 1 Node 1: 56% Node 2: 20% 2 3 Node 3: 24% 2011 6 18
- 34. 1 Node 1: 56% Node 2: 20% 2 3 Node 3: 24% 2011 6 18
- 35. 2 3 Node 1 56% 42% Node 2 20% 32% 1 1 Node 3 24% 26% 2 3 1 3 2 1 1 2 2 1 2011 6 18
- 36. 2 3 Node 1 56% 42% Node 2 20% 32% 1 1 Node 3 24% 26% 2 3 1 3 2 1 1 2 2 1 2011 6 18
- 37. Indexing • RPC ( ) • → • → 2011 6 18
- 38. Queries Invert Index Contents 2011 6 18
- 39. Skip pointer … Dictionary … Invert index … ID Skip pointer … Content … Appendix … 2011 6 18
- 40. Indexing F ID: 12345 Title: ... A Content Body: ... Term URL: ... MessagePack-RPC Time:... Term B E C D 2011 6 18
- 41. Indexing F A Content Term B Hash Hash E C D 2011 6 18
- 42. Searching MessagePack-RPC F Term Analyze Term A Query Term B E C D 2011 6 18
- 43. Searching Intersection F A ID: 12, 24, 35, 49, ... ID: 12, 30, 49, 55, ... B E ID: 7, 12, 30, 49, ... C D 2011 6 18
- 44. Searching F Output A ID: 49 ID: 12 Title: B Body:... E C D 2011 6 18
- 45. • 1000 • • • … 2011 6 18
- 46. Beacon F beacon A B E C D 2011 6 18
- 47. Beacon F beacon A “live” B E C D 2011 6 18
- 48. F … × A B E C D 2011 6 18
- 49. Running F × Suspend A B E C D 2011 6 18
- 50. Suspend dead “F” F × Node dead A B E C D 2011 6 18
- 51. A B E C D 2011 6 18
- 52. Suspend F Running A B E C D 2011 6 18
- 53. Beacon F A B E C D 2011 6 18
- 54. Future work • • • • 2011 6 18
- 55. Source codes • : • http://code.google.com/p/fujene/ 2011 6 18
- 56. Bibliography (1) I. Stoica, et al.; Chord: A Scalable Peer- to-peer Lookup Service for Internet Applications; SIGCOMM 2001; October 2001 (2) D. Karger, et al.; Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web; STOC ’97; 1997 2011 6 18
- 57. Bibliography (3) C. D. Manning, et al.; An Introduction to Information Retrieval; Cambridge UP; 2009 (4) T. Luu, et al.; ALVIS Peers: A Scalable Full-text Peer-to-Peer Retrieval Engine; P2PIR ’06; Nov. 2006 2011 6 18