Solr @ Etsy - Apache Lucene Eurocon
6 likes1,599 views
Solr provides concise summaries of key points from the document: 1. Solr discusses its search architecture including the use of Thrift for service encapsulation and reduced network traffic. Only IDs are returned from searches to reduce index size and enable easy scaling of primary key lookups. 2. Load balancing is discussed, including an algorithm that hashes the query and number of servers to provide server affinity while distributing load evenly. 3. Replication of the index is covered, including challenges with multicast and an implementation using BitTorrent to efficiently replicate files.
![Load Balancing
Server Affinity Algorithm
$serversNew = array();
[“host2”, “host3”, “host1”, “host4”]
$numServers = count($servers);
while($numServers > 0) {
// Take the first 4 chars of the md5sum of the server count
// and the query, mod the available servers
$key = hexdec(substr(md5($numServers . '+' . $query),0,4))%($numServers);
$keySet = array_keys($servers);
$serverId = $keySet[$key];
// Push the chosen server onto the new list and remove it
// from the initial list
array_push($serversNew, $servers[$serverId]);
unset($servers[$serverId]);
--$numServers;
}](https://image.slidesharecdn.com/eurocon-trimmed-111027161108-phpapp01/85/Solr-Etsy-Apache-Lucene-Eurocon-21-320.jpg)
![Load Balancing
Server Affinity Algorithm
$key = hexdec(substr(md5($query),0,4))
“jewelry” [“host2”, “host3”, “host1”, “host4”]
“scarf” [“host2”, “host3”, “host1”, “host4”]](https://image.slidesharecdn.com/eurocon-trimmed-111027161108-phpapp01/85/Solr-Etsy-Apache-Lucene-Eurocon-22-320.jpg)
![Load Balancing
Server Affinity Algorithm
$key = hexdec(substr(md5($numServers . '+' . $query),0,4))%(count($servers));
“jewelry” [“host2”, “host3”, “host1”, “host4”]
“scarf” [“host2”, “host1”, “host4”, “host3”]](https://image.slidesharecdn.com/eurocon-trimmed-111027161108-phpapp01/85/Solr-Etsy-Apache-Lucene-Eurocon-23-320.jpg)
![Replication
Multicast Rsync?
[15:25] <engineer> patrick: i'm gonna test multi-rsyncing some indexes
from host1 to host2 and host3 in prod. I'll be watching the graphs and
what not, but let me know if you see anything funky with the network
[15:26] <patrick> ok
....
[15:31] <keyur> is the site down?](https://image.slidesharecdn.com/eurocon-trimmed-111027161108-phpapp01/85/Solr-Etsy-Apache-Lucene-Eurocon-30-320.jpg)
![Solr InterOp
QParsers
@Override
public Query parse() throws ParseException {
TermQuery exactFullNameQuery = new TermQuery(new Term("full_name", qstr));
exactFullNameQuery.setBoost(4.0f);
String[] userQueryTerms = qstr.split("s+");
Query firstLastQuery = null;
if (2 == userQueryTerms.length)
firstLastQuery = parseAsFirstAndLast(userQueryTerms[0], userQueryTerms[1]);
else
firstLastQuery = parseAsFirstOrLast(userQueryTerms);
DisjunctionMaxQuery realNameQuery = new DisjunctionMaxQuery(0);
realNameQuery.add(exactFullNameQuery);
realNameQuery.add(firstLastQuery);
return realNameQuery;
}](https://image.slidesharecdn.com/eurocon-trimmed-111027161108-phpapp01/85/Solr-Etsy-Apache-Lucene-Eurocon-46-320.jpg)
Solr @ Etsy - Apache Lucene Eurocon
- 1. Solr@
- 2. Things I’m not going to talk about: A/B Testing i18n Continuos Deployment
- 3. About Us
- 6. 10+ Million Listings 500 qps
- 9. Architecture Overview Thrift struct Listing { 1: i64 listing_id } struct ListingResults { 1: i64 count, 2: list<Listing> listings } service Search { ListingResults search(1:string query) }
- 10. Architecture Overview Thrift Generated Java server code: public class Search { public interface Iface { public ListingResults search(String query) throws TException; } Generated PHP client code: class SearchClient implements SearchIf { /**...**/ public function search($query) { $this->send_search($query); return $this->recv_search(); }
- 11. Architecture Overview Thrift Why use Thrift? • Service Encapsulation • Reduced Network Traffic
- 12. Architecture Overview Thrift Why only return IDs? • Index Size • Easy to scale PK lookups
- 14. Architecture Overview Search Server • Identical Code + Hardware • Roles/Behavior controlled by Env variables • Single Java Process • Solr running as a Jetty Servlet • Thrift Servers • Smoker
- 15. Architecture Overview Search Server Master-specific processes: • Incremental Indexer • External File Field Updaters
- 16. Load Balancing
- 21. Load Balancing Server Affinity Algorithm $serversNew = array(); [“host2”, “host3”, “host1”, “host4”] $numServers = count($servers); while($numServers > 0) { // Take the first 4 chars of the md5sum of the server count // and the query, mod the available servers $key = hexdec(substr(md5($numServers . '+' . $query),0,4))%($numServers); $keySet = array_keys($servers); $serverId = $keySet[$key]; // Push the chosen server onto the new list and remove it // from the initial list array_push($serversNew, $servers[$serverId]); unset($servers[$serverId]); --$numServers; }
- 22. Load Balancing Server Affinity Algorithm $key = hexdec(substr(md5($query),0,4)) “jewelry” [“host2”, “host3”, “host1”, “host4”] “scarf” [“host2”, “host3”, “host1”, “host4”]
- 23. Load Balancing Server Affinity Algorithm $key = hexdec(substr(md5($numServers . '+' . $query),0,4))%(count($servers)); “jewelry” [“host2”, “host3”, “host1”, “host4”] “scarf” [“host2”, “host1”, “host4”, “host3”]
- 24. Load Balancing Server Affinity Results 2% 20%
- 25. Load Balancing Server Affinity Caveats • Stemming / Analysis • Be wary of query distribution
- 26. Replication
- 30. Replication Multicast Rsync? [15:25] <engineer> patrick: i'm gonna test multi-rsyncing some indexes from host1 to host2 and host3 in prod. I'll be watching the graphs and what not, but let me know if you see anything funky with the network [15:26] <patrick> ok .... [15:31] <keyur> is the site down?
- 33. Replication Bit Torrent POC Using BitTornado:
- 34. Replication Bit Torrent + Solr Fork of TTorent: https://github.com/etsy/ttorrent Multi-File Support Performance Enhancements
- 35. Replication Bit Torrent + Solr
- 36. Replication Bit Torrent + Solr
- 37. Replication Bit Torrent + Solr
- 38. Replication Bit Torrent + Solr
- 39. Solr InterOp
- 40. QParsers
- 41. “writing query strings is for suckers”
- 43. Solr InterOp QParsers http://host:8393/solr/person/select/?q=_query_:%22{!dismax %20qf=$fqf%20v=$fnq}%22%20OR%20(_query_:%22{!dismax%20qf=$fiqf %20v=$fiq}%22%20AND%20(_query_:%22{!dismax%20qf=$lwqf%20v=$lwq} %22%20OR%20_query_:%22{!dismax%20qf=$lqf%20v=$lq}%20%22))&fnq= %22giovanni%20fernandez-kincade %22&fqf=full_name^4&fiq=giovanni&fiqf=first_name^2.0%20first_name_s yn&qt=standard&lwq=fernandez-kincade*&lwqf=last_name&lq=fernandez- kincade&lqf=last_name^3
- 44. Solr InterOp QParsers http://host:8393/solr/person/select/?q={!personrealqp}giovanni %20fernandez-kincade
- 45. Solr InterOp QParsers class PersonNameRealQParser extends QParser { public PersonNameRealQParser(String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) { super(qstr, localParams, params, req); }
- 46. Solr InterOp QParsers @Override public Query parse() throws ParseException { TermQuery exactFullNameQuery = new TermQuery(new Term("full_name", qstr)); exactFullNameQuery.setBoost(4.0f); String[] userQueryTerms = qstr.split("s+"); Query firstLastQuery = null; if (2 == userQueryTerms.length) firstLastQuery = parseAsFirstAndLast(userQueryTerms[0], userQueryTerms[1]); else firstLastQuery = parseAsFirstOrLast(userQueryTerms); DisjunctionMaxQuery realNameQuery = new DisjunctionMaxQuery(0); realNameQuery.add(exactFullNameQuery); realNameQuery.add(firstLastQuery); return realNameQuery; }
- 47. Solr InterOp QParsers The QParserPlugin that returns our new QParser: public class PersonNameRealQParserPlugin extends QParserPlugin { public static final String NAME = "personrealqp"; @Override public void init(NamedList args) {} @Override public QParser createParser(String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) { return new PersonNameRealQParser(qstr, localParams, params, req); } }
- 48. Solr InterOp QParsers Registering the plugin in solrconfig.xml: <queryParser name="personrealqp" class="com.etsy.person.solr.PersonNameRealQParserPlugin" />
- 49. Custom Stemmer
- 51. Solr InterOp Custom Stemmer banded, banding, birding, bouldering, bounded, buffing, bundler, canning, carded, circled, coupler, dangler, doubler, firring, foiling, hooper, japanned, lipped, napped, papered, pebbled, pitted, pocketed, reductive, ricer, rooter, roper, seeded, shouldered, silvered, skinning, spindling, staining, stitcher, strapped, threaded, yellowing
- 52. Solr InterOp Custom Stemmer First we extend KStemmer and intercept stem calls: public class LStemmer extends KStemmer { /**.....**/ @Override String stem(String term) { String override = overrideStemTransformations.get(term); if(override != null) return override; return super.stem(term); } }
- 53. Solr InterOp Custom Stemmer Then create a TokenFilter that uses the new Stemmer: final class LStemFilter extends TokenFilter { /**.....**/ protected LStemFilter(TokenStream input, int cacheSize) { super(input); stemmer = new LStemmer(cacheSize); } @Override public boolean incrementToken() throws IOException { /**....**/ }
- 54. Solr InterOp Custom Stemmer Create a FilterFactory that exposes it: public class LStemFilterFactory extends BaseTokenFilterFactory { private int cacheSize = 20000; @Override public void init(Map<String, String> args) { super.init(args); String cacheSizeStr = args.get("cacheSize"); if (cacheSizeStr != null) { cacheSize = Integer.parseInt(cacheSizeStr); } } @Override public TokenStream create(TokenStream in) { return new LStemFilter(in, cacheSize); } }
- 55. Solr InterOp Custom Stemmer And finally plug it into your analysis chain: <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="solr/common/conf/stopwords.txt"/> <filter class="com.etsy.solr.analysis.LStemFilterFactory" /> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer>
- 56. Thanks!