![Micro Sharding
map reduce hdfs
Retrieval Indexer #1
for (t in types) Retrieval Indexer #2
yield [key:(t, h(key)%shardCnt), Retrieval Indexer #n
value:doc]
Content Indexer #1
Content Indexer #2
Content Indexer #m
n,m >> number of search nodes](https://image.slidesharecdn.com/hpts-cloud-search-120510151406-phpapp01/85/HPTS-talk-on-micro-sharding-with-Katta-7-320.jpg)
HPTS talk on micro-sharding with Katta
- 2. Text Retrieval Task • Text viewed as a sequences of terms in fields • Document and position for each term are indexed • Query is a sequence of terms (typically many more than user actually types)
- 3. Text Retrieval • Scores computed by merging occurrences of terms in query • Only top scoring documents are kept • Deletion and document edits done by adding new documents and keeping deletion list • (this is all standard ... Lucene is the best known example)
- 4. Traditional Scaling Sharding 1..n/4 n/4+1...n/2 n/2+1...3n/4 3n/4+1...n ? shard shard shard shard shard 1 2 3 4 5 Replication shard shard shard shard shard 1 2 3 4 5 shard shard shard shard shard 1 2 3 4 5
- 5. Consistent Hashing 1 0 0 1 0 1
- 6. Problems • Presumes objects can be moved individually • Has very high insertion/deletion rate • Has disordered access patterns • Often exhibits content/placement correlations
- 7. Micro Sharding map reduce hdfs Retrieval Indexer #1 for (t in types) Retrieval Indexer #2 yield [key:(t, h(key)%shardCnt), Retrieval Indexer #n value:doc] Content Indexer #1 Content Indexer #2 Content Indexer #m n,m >> number of search nodes
- 8. Search Architecture Retrieval Engine #1 presentation federator Retrieval Engine #2 layer federator Retrieval Engine #n Content Engine #1 Content Engine #m
- 9. Control Architecture federator Retrieval Engine #2 zookeeper katta HDFS master indexer
- 10. Scenario: Node Start ● Node starts, tells ZK it exists and has no shards ● Master notified by ZK, looks at shard placement ● Imbalance exists so Master assigns shards to new node ● Node notified by ZK, downloads shard, tells ZK ● Master notified by ZK, looks at shard placement, unassigns shard somewhere
- 11. Scenario: Node Crash ● ZK detects node connection loss and session expiration ● Master is notified by ZK that node ephemeral file has vanished, looks at shard placements ● If under-replication exists, Master assigns shards to other nodes ● Nodes are notified by ZK, download shards, tell ZK ● Master is notified by ZK, no action needed
- 12. Summary of Master ● Master is notified of node set or shard set change ● Master examines current state of cluster ● If shards are under-replicated, add assignments ● If shards are over-replicated, delete assignments ● If cluster is imbalanced, add assignments ● Rinse, repeat
- 13. Quick Results • No deletion/insertion in indexes at runtime • Reloading micro-shards allows large sequential transfers • Multiple shards allows very simple threading of search • Random placement guided by balancing policy gives near optimal motion • Node addition and failure are simple, reliable • = Random sharding also near optimal local global statistics, 2x query time improvement load balancing uniform management
- 14. Building Blocks • EC2 - elastic compute • Zookeeper - reliable coordination • Katta - shard and query management • Hadoop - map-reduce, RPC for Katta • Lucene - candidate set retrieval, index file storage • Deepdyve search algorithms - segment scoring
- 15. Building Blocks • EC2 - elastic compute • Zookeeper - reliable coordination • Katta - shard and query management • Hadoop - map-reduce, RPC for Katta • Lucene - candidate set retrieval, index file storage • Deepdyve search algorithms - segment scoring
- 16. Zookeeper • Replicated key-value in-memory store • Minimal semanticsversion create, read, replace specified sequential and ephemeral files notifications • orderingstrict correctness guarantees strict Very quorum writes no blocking operations no race conditions • High speed 200,000 reads per second 50,000 updates per second,
- 17. Building Blocks • EC2 - elastic compute • Zookeeper - reliable coordination • Katta - shard and query management • Hadoop - map-reduce, RPC for Katta • Lucene - candidate set retrieval, index file storage • Deepdyve search algorithms - segment scoring
- 18. Katta Interface • -Simple Interface for query, vertical broadcast for update Client horizontal broadcast InodeManaged - add/removeShard • Pluggable Application Interface • current returnReturn Policy Given Pluggable state return < 0 => done return 0 => return result, allow updates return n => wait at most n milliseconds • Comprehensivetimes Results, exceptions, arrival Results
- 19. Horizontal/Vertical Broadcast 1..n/4 n/4+1...n/2 n/2+1...3n/4 3n/4+1...n shard shard shard shard 1 2 3 4 Replication shard shard shard shard 1 2 3 4 shard shard shard shard 1 2 3 4
- 20. Operations federator Retrieval Engine #2 zookeeper katta HDFS master indexer
- 21. Impact of Cloud Approach • Scale-free programming • Deployed in EC2 (test) or in private farm (production) • No single point of failure • Real-time scale up/down • Extensible to real-time index updates
- 22. Lessons • Random document to shard assignments no correlations in is good ⇒ strong bounds on node variations in search time ⇒ local statistics are as good as global statistics no structure in shard to node assignments ⇒ node failure is not correlated to documents ⇒ load balancing and rebalancing is trivial ⇒ threaded search is trivial
- 23. More Lessons • Randomized clustering requires good coordination Zookeeper makes that easy • Good coordination means not having to say you’re sorry Masters coordinate but don’t participate
- 24. Resources ● My blog – http://tdunning.blogspot.com/ ● The web-site – www.deepdyve.com ● Source code – Katta (sourceforge) – Hadoop (Apache) – Lucene (Apache)