Scaling search with Solr Cloud
12 likes5,227 views
This document discusses scaling search with Apache SolrCloud. It provides an introduction to Solr and how scaling search was difficult in previous versions due to manually managing shards and replicas. SolrCloud makes scaling easier by utilizing ZooKeeper for centralized configuration and management across a cluster. Nodes can be added to a SolrCloud cluster and will automatically be configured and assigned as shards or replicas. This allows for effortless scaling, fault tolerance, and load balancing. The document promotes upcoming features in Solr 4 and demonstrates indexing and querying in a SolrCloud cluster.
Scaling search with Solr Cloud
- 1. 1 Jan Høydahl Scaling search with Cominvent AS SolrCloud
- 2. 2 Jan Høydahl 1995: Developer telecom 1998: Java developer 2000: Search - FAST 2006: Lucene 2007: new Cominvent() 2009: Lucene/Solr 2011: Lucene committer 2012: Lucene PMC > 100 projects
- 3. 3
- 4. 4 About Cominvent Business critical search Domain knowledge & best practices: Consulting Training Support
- 5. 5 SolrTraining.com Next course in Oslo: SEPTEMBER 2012 MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Calendar from www.calendar-of-2012.com
- 6. 6 http://www.meetup.com/Oslo-Solr-Community CommunityZone talk: «Solr 101» Thursday 14:20
- 7. 7 http://www.meetup.com/Oslo-Solr-Community next MeetUp:
- 8. 8 ApacheCon Europe 2012 • Sinsheim, Germany • Lucene/Solr track • November 5-8 • www.apachecon.eu
- 9. 9 Agenda • Intro to Solr • Scaling search - before • Introduction to SolrCloud • Demo with Wikipedia data • Plans for Solr going forward • Q&A
- 10. 10 Intro to Solr
- 11. 11 Apache Solr Search Server
- 13. 13
- 14. 14 Areas of use
- 15. 15 Example: e-commerce www.libris.no
- 16. 16 Boosting by function Boosting on review popularity and sales numbers: log(sum(popularity,numsold))
- 17. 17 Auto suggest & phonetic normalization
- 18. 18 Example: classifieds/auctions www.finn.no
- 19. 18 Example: classifieds/auctions www.finn.no
- 20. 19 Who use Apache Lucene/Solr™ ? ..and many more: http://wiki.apache.org/solr/PublicServers
- 21. 20 Versions • Current stable = 3.6.1 • Latest release = 4.0-beta • Next release = 4.0-FINAL --- «soon» :-) v3.6.1 v1.1 v1.3 v1.4 v3.1 v3.3 v3.6 v4.0a v4.0ß v4.0 01/2007 09/2008 11/2009 03/2011 06/2011 04/2012 06/2012 08/2012 ??/2012 07/2012
- 23. 22 Why scale? • One single Solr server handles... –millions of documents (per shard) –hundreds of queries per second (per replica) • We need to scale if... –data volume increases –query volume increases –we need high availability / fault tolerance
- 24. 23 Scaling search - before Solr shard 1 - config, schema - synonyms
- 25. 23 Scaling search - before - Add shard node - Manually copy config - Manually index to right shard - Manually shards query parameter Solr shard 1 Solr shard 2 - config, schema - config, schema - synonyms - synonyms
- 26. 23 Scaling search - before - Add shard node - Manually copy config - Manually index to right shard - Manually shards query parameter Solr shard 1 Solr shard 2 - config, schema - config, schema - synonyms - synonyms Solr 1 replica Solr 2 replica - config, schema - config, schema - synonyms - synonyms - Add replica node - Copy config - Setup poll based replication - No indexing failover - Monitor every node
- 27. 24 Solr Cloud
- 28. 25 What is SolrCloud? • New in Solr 4.0 • Easier scaling ZooKeeper: «Because • Centralized config coordinating distributed systems is a Zoo» • Fault tolerant indexing and querying • Using Apache ZooKeeper as «registry»
- 32. 26 What is SolrCloud Logical collection
- 33. 26 What is SolrCloud Logical collection Soft commit Transaction log
- 34. 27 Scaling search - with SolrCloud Solr master 1 ZK aware Apache ZooKeeper
- 35. 27 Scaling search - with SolrCloud - Add shard node, point it to ZK - It assumes the role of shard 2 - Automatic document distribution - Automatic querying across cluster - Centralized config & monitoring Solr master 1 Solr master 2 ZK aware ZK aware Apache ZooKeeper
- 36. 27 Scaling search - with SolrCloud - Add shard node, point it to ZK - It assumes the role of shard 2 - Automatic document distribution - Automatic querying across cluster - Centralized config & monitoring Solr master 1 Solr master 2 ZK aware ZK aware Apache ZooKeeper Solr replica 1 Solr replica 2 ZK aware ZK aware - Add replica node(s) - Auto role assignment - Push based replication - Indexing failover - Leader election through ZK
- 37. 27 Scaling search - with SolrCloud Solr master 1 Solr master 2 ZK aware ZK aware Apache ZooKeeper Solr replica 1 Solr replica 2 ZK aware ZK aware
- 38. 27 Scaling search - with SolrCloud Solr master 1 Solr master 2 ZK aware ZK aware Apache ZooKeeper Solr replica 1 Solr replica 2 master ZK aware ZK aware
- 39. 27 Scaling search - with SolrCloud Solr master 1 Solr master 2 replica ZK aware ZK aware Apache ZooKeeper Solr replica 1 Solr replica 2 master ZK aware ZK aware
- 40. 28 Configuration Solr master 1 Solr master 2 ZK aware ZK aware ZK Solr replica 1 Solr replica 2 ZK aware ZK aware
- 41. 28 Configuration -DzkRun Solr master 1 -Dcollection.configName=jz Solr master 2 -DnumShards=2 ZK aware -Dbootstrap_confdir=./solr/coll/conf ZK aware ZK Solr replica 1 Solr replica 2 ZK aware ZK aware
- 42. 28 Configuration -DzkRun Solr master 1 -Dcollection.configName=jz Solr master 2 -DnumShards=2 -DzkHost=localhost:xxxx ZK aware -Dbootstrap_confdir=./solr/coll/conf ZK aware ZK Solr replica 1 Solr replica 2 -DzkHost=localhost:xxxx -DzkHost=localhost:xxxx ZK aware ZK aware
- 43. 29 Demo indexing & querying
- 44. 30 Solr 4.0 and beyond • Other news in v4.0 FINAL (expected later this autumn) –NRT –Real-time GET –Smaller index & memory footprint –New «modern» Admin GUI –Incremental updates –Pseudo-join • Future plans –More shard distribution mechanisms –Re-balancing cluster (split shards) –...
- 45. 31 Recap • Apache Solr open source enterprise search • Scaling Solr was hard • Solr 4.0 with SolrCloud makes it easy :) –Centralized config –Effortless scaling of cluster –Fault tolerant indexing & querying • Download the 4.0-beta today, 4.0-FINAL soon
- 46. 32 Remember Next Solr course in Oslo: SEPTEMBER 2012 MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 CommunityZone talk: «Solr 101» 17 18 19 20 21 22 23 Thursday 14:20 24 25 26 27 28 29 30 Calendar from www.calendar-of-2012.com www.solrkurs.no
- 47. 33 ? Jan Høydahl Cominvent AS @cominvent