Solr on HDFS - Past, Present, and Future: Presented by Mark Miller, Cloudera
10 likes4,548 views
Solr is a search engine that can run indexes stored on HDFS for improved fault tolerance and operations. Past attempts had Solr directly run on HDFS, but were slow. The current approach uses a block cache to improve performance by keeping frequently accessed index blocks in memory. Future work includes improving the block cache and enabling HDFS-only replication for SolrCloud replicas to reduce unnecessary data copying.
![Inside the Block Cache.
ConcurrentLinkedHashMap<BlockCacheKey,BlockCacheLocation>
ByteBuffer[] banks
int numberOfBlocksPerBank
Each ByteBuffer of size ‘blockSize’.
Used locations tracked by ‘lock’ bitset.](https://image.slidesharecdn.com/solronhdfsfinal-markmiller-141124123956-conversion-gate02/85/Solr-on-HDFS-Past-Present-and-Future-Presented-by-Mark-Miller-Cloudera-12-320.jpg)
Solr on HDFS - Past, Present, and Future: Presented by Mark Miller, Cloudera
- 2. Solr on HDFS Past, Present, and Future Mark Miller, Cloudera
- 3. About Me Lucene Committer, Solr Committer. Works for Cloudera. A lot of work on Lucene, Solr, and SolrCloud.
- 4. Some Basics Solr A distributed, fault tolerant search engine using Lucene as it’s core search library. HDFS A distributed, fault tolerant filesystem that is part of the Hadoop project.
- 5. Solr on HDFS Wouldn’t it be nice if Solr could run on HDFS. If you are running other things on HDFS, it simplifies operations. If you are building indexes with MapReduce, merging them into your cluster becomes easy. You can do some other kind of cool things when you have are using a shared file system. Most attempts in the past have not really caught on.
- 6. Solr on HDFS in the Past. • Apache Blur is one of the more successful marriages of Lucene and HDFS. • We borrowed some code from them to seed Solr on HDFS. • Others have copied indexes between local filesystem and HDFS. • Most people felt that running Lucene or Solr straight on HDFS would be too slow.
- 7. How HDFS Writes Data Remote Remote Remote Remote Local Solr Write An attempt is made to make a local copy and as many remote copies as necessary to satisfy the replication factor configuration.
- 8. Co-Located Solr and HFDS Data Nodes HDFS HDFS HDFS HDFS Solr Solr Solr Solr We recommend that HDFS data nodes and Solr nodes are co-located so that the default case involves fast, local data.
- 9. Non Local Data • BlockCache is first line of defense, but it’s good to get local data again. • Optimize is more painful option. • An HDFS affinity feature could be useful. • A tool that simply wrote out a copy of the index with no merging might be interesting.
- 10. HdfsDirectory • Fairly simple and straightforward implementation. • Full support required making the Directory interface a first class citizen in Solr. • Largest part was making Replication work with non local filesystem directories. • With large enough ‘buffer’ sizes, works reasonably well as long as the data is local. • Really needs some kind of cache to be reasonable though.
- 11. “The Block Cache” A replacement for the OS filesystem cache, especially for the case when there is no local data. Even with local data, making it larger will beneficially reduce HDFS traffic in many cases. Block Cache HDFS Solr
- 12. Inside the Block Cache. ConcurrentLinkedHashMap<BlockCacheKey,BlockCacheLocation> ByteBuffer[] banks int numberOfBlocksPerBank Each ByteBuffer of size ‘blockSize’. Used locations tracked by ‘lock’ bitset.
- 13. The Global Block Cache The initial Block Cache implementation used a separate Block Cache for every unique index directory used by Solr in HDFS. There are many limitations around this strategy. It hinders capacity planning, it’s not very efficient, and it bites you at the worst times. The Global Block Cache is meant to be a single Block Cache to be used by all SolrCore’s for every directory. This makes sizing very simple - determine how much RAM you can spare for the Block Cache and size it that way once and forget it.
- 14. Performance In many average cases, performance looks really good - very comparable to local filesystem performance, though usually somewhat slower. In other cases, adjusting various settings for the Block Cache can help with performance. We have recently found some changes to improve performance.
- 15. Tuning the Block Cache Sizing By default, each ‘slab’ is 128 MB. Raise the slab count to increase by 128 MB slabs. Block Size (8 KB default) Not originally configurable, but certain use cases appear to work better with 4 KB.
- 16. HDFS Transaction Log We also moved the Transaction Log to HDFS. Implementation has held up okay, some improvements needed, a large replay performance issue improved. The HDFSDirectory and Block Cache have had a much larger impact. No truncate support in HDFS, so we work around it by replaying the whole log in some failed recovery cases where local filesystem impl just drops the log.
- 17. The autoAddReplicas Feature A new feature that is currently only available when using a shared filesystem like HDFS. The Overseer monitors the cluster state and fires off SolrCore create command pointing to existing data in HDFS when a node goes down.
- 18. The autoAddReplicas Feature 2 HDFS HXDFS HDFS HDFS Solr SXolr Solr Solr
- 19. The Future At Cloudera, we are building an Enterprise Data Hub. In our vision, the more that runs on HDFS, the better. We will continue to improve and push forward HDFS support in SolrCloud.
- 20. Block Cache Improvements Apache Blur has a Block Cache V2. Uses variable sized blocks. Optionally uses Unsafe for direct memory management. The V1 Block Cache has some performance limitations. * Copying bytes from off heap to IndexInput buffer. * Concurrent access of the cache. * Sequential reads have to pull a lot of blocks from the cache. * Each DirectByteBuffer has some overhead, including a Cleaner object that can affect GC and add to RAM reqs.
- 21. HDFS Only Replication When Using Replicas Currently, if you want to use SolrCloud replicas, data is replicated both by HDFS and by Solr. HDFS replication factor = 1 is not a very good solution. autoAddReplicas is one possible solution. We will be working on another solution where only the leader writes to an index in HDFS while replicas read from it.
- 22. The End Mark Miller @heismark