Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher Bradford, Open Source Connections)
4 likes2,662 views
Christopher Bradford presented lessons learned from using Apache Spark at the US Patent and Trademark Office to improve their process of loading data from Cassandra into Solr. The initial Spark implementation performed poorly due to opening and closing a Solr connection for each document. Optimizations like opening a single connection per partition and pushing documents in batches significantly improved performance, resulting in a solution that was 5 times faster than the original process. Future work involves further optimizing this Spark job and exploring additional uses of Spark.
Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher Bradford, Open Source Connections)
- 1. Lessons Learned with Spark at the US Patent & Trademark Office Christopher Bradford Big Data Architect at OpenSource Connections
- 2. Christopher Bradford Twitter: @bradfordcp GitHub: bradfordcp
- 4. Exploring Search Technologies - EST
- 5. EST – Technology Stack
- 6. EST – Data Loading CSS Ingestion (CSS2C) Solr Ingestion (C2S)
- 7. EST – C2S Process Note: some connections are omitted for clarity
- 8. EST – C2S Process (Scaled Out) Note: some connections are omitted for clarity
- 9. EST – C2S Review Did it work? Why change it? How could we make it better?
- 11. EST – Old C2S Process Note: some connections are omitted for clarity
- 12. EST – Spark C2S Process Note: some connections are omitted for clarity
- 13. How did this work out? Poorly
- 14. Poor Performance joinedRDD = … joinedRDD.foreach() document = … // build document sc = new SolrConnection() sc.push(document) sc.disconnect() // Job is done
- 15. Poor Performance sc = new SolrConnection() sc.push(document) sc.disconnect()
- 16. Optimum Performance joinedRDD = … sc = new SolrConnection() joinedRDD.foreach() document = … // build document sc.push(document) sc.disconnect() // Job is done joinedRDD = … joinedRDD.foreachPartition() sc = new SolrConnection() partition.foreach() document = … // build document sc.push(document) sc.disconnect() // Job is done Almost
- 17. The Solution! joinedRDD = … joinedRDD.mapPartitions() sc = new SolrConnection() partition.foreach() document = … // build document sc.push(document) sc.close() return partition.rows .collect() joinedRDD = … joinedRDD.mapPartitions() sc = new SolrConnection() partition.foreach() document = … // build document sc.push(document) sc.close() return partitions.rows.count .collect()
- 18. Results?
- 19. Solr Indexing
- 20. Better Solr Indexing Note: some connections are omitted for clarity
- 21. EST – Spark C2S Process v2 Note: some connections are omitted for clarity
- 22. Success? YUP 5x faster than the original C2S process (with optimizations)
- 23. What’s Next? • Optimization of the C2S Spark job • More Spark jobs • Newer version of Spark & DSE • Scala Spark jobs instead of Java