Friends of Solr - Nutch & HDFS
7 likes4,402 views
Nutch is an open source web crawler built on Hadoop that can be used to crawl websites at scale. It integrates directly with Solr to index crawled content. HDFS provides a scalable storage layer that Nutch and Solr can write to and read from directly. This allows building indexes for Solr using Hadoop's MapReduce framework. Morphlines allow defining ETL pipelines to extract, transform, and load content from various sources into Solr running on HDFS.
![MorphLines
SOLR_LOCATOR : { collection : collection1 zkHost : "127.0.0.1:9983" batchSize : 100 } morphlines : [ { id : morphline1 importCommands : ["org.kitesdk.**", "org.apache.solr.**"] commands : [ { readAvroContainer { ... } } { extractAvroPaths {...} } { convertTimestamp {...} } { sanitizeUnknownSolrFields {...} } { loadSolr {...} } .... ] } ]](https://image.slidesharecdn.com/solr-sep-meetup-nutch-hdfs-141004143616-conversion-gate01/85/Friends-of-Solr-Nutch-HDFS-35-320.jpg)
Friends of Solr - Nutch & HDFS
- 1. Friends of Solr - “Nutch and HDFS” Saumitra Srivastav saumitra.srivastav@glassbeam.com Bangalore Apache Solr Group September-2014 Meetup
- 2. Friends
- 4. What is Nutch? -Distributed framework for large scale web crawling -but does not have to be large scale at all -Based on Apache Hadoop -Direct integration with Solr
- 5. Overview Seed (URLs) Solr Nutch Crawl Fetch Parse
- 6. Overview
- 7. Components -CrawlDB -Info about URLs -LinkDB -Info about links to each URL -Segments -set of URLs that are fetched as a unit
- 8. Segments 1.crawl_generate -set of URLs to be fetched 2.crawl_fetch -status of fetching each URL 3.content -raw content retrieved from each URL 4.parse_text -parsed text of each URL 5.parse_data -outlinks and metadata parsed from each URL 6.crawl_parse -outlink URLs, used to update the crawldb
- 9. Scale -Scalable storage -HDFS -Scalable crawling -Map-Reduce -Scalable search -SolrCloud -Scalable backend -Gora
- 10. Features -Fetcher -Multi-threaded fetcher -Queues URLs per hostname / domain / IP -Limit the number of URLs for round of fetching -Default values are polite but can be made more aggressive
- 11. Features -Crawl Strategy -Breadth-first but can be depth-first -Configurable via custom ScoringFilters
- 12. Features -Scoring -OPIC (On-line Page Importance Calculation) by default -LinkRank -Protocols -Http, file, ftp, https -Respects robots.txt directives
- 13. Features -Scheduling -Fixed or adaptive -URL filters -Regex, FSA, TLD, prefix, suffix -URL normalisers -Default, regex
- 14. Features -Parsing with Apache Tika -Hundreds of formats supported -But some legacy parsers as well -Plugins -Feeds, Language Identification etc. -Pluggable indexing -Solr, ES etc.
- 15. Common crawled fields -url -content -title -anchor -site -boost -digest -segment -host -type -arbitrary metadata
- 16. Setup -Download binary and unzip -http://nutch.apache.org/downloads.html -Conf Directory
- 17. Solr Schema
- 19. Indexing crawled data to Solr -Add agent.name in nutch-default.xml -Copy fields from schema.xml to a core/collection in Solr -create seed directory -bin/crawl <seedDir> <crawlDir> <solrURL> <numberOfRounds>
- 20. Friend #2 HDFS
- 21. Why integrate with Hadoop? -Hadoop is NOT AT ALL needed to scale your Solr installation -Hadoop is NOT AT ALL needed for Solr distributed capabilities
- 22. Why integrate with Hadoop? -Integrate Solr with HDFS when your whole pipeline is hadoop based -Avoid moving data and indexes in and out -Avoid multiple sinks -Avoid redundant provisioning for Solr -Individual nodes disk, etc
- 23. Solr + Hadoop -Read and write directly to HDFS -build indexes for Solr with Hadoop's map-reduce
- 24. Lucene Directory Abstraction Class Directory { listAll(); createOutput(file, context); openInput(file, context); deleteFile(file); makeLock(file); clearLock(file); ... ... }
- 25. HdfsDirectory
- 26. Index in HDFS -writing and reading index and transaction log files to the HDFS -does not use Hadoop Map-Reduce to process Solr data -Filesystem cache needed for Solr performance -HDFS not fit for random access
- 27. Block Cache -enables Solr to cache HDFS index files on read and write -LRU semantics -Hot blocks are cached
- 28. Transaction Log -HdfsUpdateLog -Extends updateLog -Triggered by setting the UpdateLog dataDir to something that starts with hdfs:/ -no additional configuration
- 29. Running Solr on HDFS Cloud mode java -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs -Dsolr.hdfs.home=hdfs://localhost:5432/solr/ -DzkHost=localhost:2181 -jar start.jar
- 30. Map-Reduce index building -Scalable index creation via map-reduce -https://github.com/markrmiller/solr-map-reduce- example
- 31. Map-Reduce index building -initial implementations sent documents from reducer to SolrCloud over http -Not scable -Reducers create indexes in HDFS
- 32. Map-Reduce index building -Reducers create indexes in HDFS -merge the indexes down to the correct number of ‘shards’ -zookeeper aware -Go-Live
- 34. MorphLines -A morphline is a configuration file that allows you to define ETL transformation pipelines -replaces Java programming with simple configuration steps -Extract content from input files, transform content, load content -Uses Tika to extract content from a large variety of input documents
- 35. MorphLines SOLR_LOCATOR : { collection : collection1 zkHost : "127.0.0.1:9983" batchSize : 100 } morphlines : [ { id : morphline1 importCommands : ["org.kitesdk.**", "org.apache.solr.**"] commands : [ { readAvroContainer { ... } } { extractAvroPaths {...} } { convertTimestamp {...} } { sanitizeUnknownSolrFields {...} } { loadSolr {...} } .... ] } ]
- 36. Map-Reduce index building bin/hadoop --config /tmp/hadoop/sample1 jar ~/softwares/solr/solr-4.10.0/dist/solr-map-reduce-*.jar -D 'mapred.child.java.opts=-Xmx500m' -libjars "$HADOOP_LIBJAR" --morphline-file /tmp/readAvroContainer.conf --zk-host localhost:2181 --output-dir hdfs://localhost/outdir --collection twitter --log4j log4j.properties --go-live --verbose "hdfs://localhost/indir"
- 37. Thanks -Attributions •Julien Nioche’s slides on “Large scale crawling with Apache Nutch” •Mark Miller’s slides on “First Class Integration of Solr with Hadoop” -Connect •saumitra.srivastav@glassbeam.com •saumitra.srivastav7@gmail.com •https://www.linkedin.com/in/saumitras •@_saumitra_ -Join: •http://www.meetup.com/Bangalore-Apache-Solr-Lucene-Group/