Indexing Big Data on Amazon AWS
1 like640 views
The document discusses strategies for indexing big data in the cloud, focusing on tools like Solr and various programming languages such as Bash, Python, and Java. It emphasizes an agile approach to project management, including making quick iterations and testing batch processing across multiple instances. Key metrics and techniques for scaling and error recovery are highlighted, alongside alternatives like CloudFormation and Hadoop.
Indexing Big Data on Amazon AWS
- 1. Indexing Big Data in the Cloud
- 2. Me Scott Stults Co-Founder of OpenSource Connections Solr / Lucene Bash / Python / Java Indexing Big Data in the Cloud 2
- 3. Eric Indexing Big Data in the Cloud 3
- 4. Big Data Indexing Big Data in the Cloud 4
- 5. Big Data Wrangler Indexing Big Data in the Cloud 5
- 6. How? Address a Real Project Be Agile Make Small Mistaeks Fast Succeed BIG Indexing Big Data in the Cloud 6
- 7. USPTO Goals Prototype Search UX Prove Solr: Scales Integrates Excels Indexing Big Data in the Cloud 7
- 8. Scale? Indexing Big Data in the Cloud 8
- 9. Our Approach KISS YAGNI (This space intentionally left blank) Indexing Big Data in the Cloud 9
- 10. Minimal Flair Indexing Big Data in the Cloud 10
- 11. Record Everything! Indexing Big Data in the Cloud 11
- 12. Some Numbers Doc Count 1.1 Million Zip Files 313 Docs per Zip File 4,000 Zip File Size 75M File Size 300M Indexing Big Data in the Cloud 12
- 13. Testing Start some servers Process a batch Check the clock Indexing Big Data in the Cloud 13
- 14. start_nodes start_nodes() { ec2-run-instances ami-1b814f72 --block-device-mapping '/dev/sdb=snap-48adde35::true' --block-device-mapping '/dev/sdi1=:10:false' --block-device-mapping '/dev/sdi2=:10:false' --block-device-mapping '/dev/sdi3=:20:false' --instance-type m1.large --key uspto-proto --instance-count $MAX_NODES --group default > ~/run-output } Indexing Big Data in the Cloud 14
- 15. Gut Check How fast can we do this? What can we do in parallel? Indexing Big Data in the Cloud 15
- 16. Scaling Raise our instance limit xargs -P GNU parallel Indexing Big Data in the Cloud 16
- 17. Shortcomings SSH? Error recovery One Solr Indexing Big Data in the Cloud 17
- 18. Alternatives CloudFormation Puppet / Chef Multiple Cores / Shards Hadoop Indexing Big Data in the Cloud 18
- 19. Success Indexing Big Data in the Cloud 19
- 20. Victory Lap Indexing Big Data in the Cloud 20
- 21. Instances / Time Indexing Big Data in the Cloud 21
- 22. Thank You https://github.com/sstults/patent-indexing @scottstults #o19s Indexing Big Data in the Cloud 22