MapR-DB Elasticsearch Integration
2 likes3,238 views
This document discusses MapR's integration with Elasticsearch. It introduces MapR-DB, a scalable NoSQL database, and describes how MapR replicates data from MapR-DB tables to Elasticsearch in near real-time. The replication architecture uses gateway nodes to stream data changes from MapR-DB to Elasticsearch. It also covers data type conversions and future extensions, such as supporting additional external sinks like Spark streaming.
MapR-DB Elasticsearch Integration
- 1. © 2015 MapR Technologies© 2015 MapR Technologies MapR-Elasticsearch Integration Brian Parkison bparkison@maprtech.com
- 2. © 2015 MapR Technologies Agenda Introduction to MapR-DB Integration with Elasticsearch Extending to other external sinks
- 3. © 2015 MapR Technologies© 2015 MapR Technologies Introduction to MapR-DB
- 4. © 2015 MapR Technologies 4 MapR-DB and MapR Enterprise Database Edition Hadoop-Enabled Architecture Proven Production Success Consistent High Performance at Any Scale •True operational, real-time analytics •Reduced complexity and cost versus separate clusters deployments •Leverage existing HBase skillsets/expertise •Zero downtime •Zero database administration •Integrated security •High throughput •Continuous low latency •Extreme database scalability
- 5. © 2015 MapR Technologies 5 MapR-DB: The Best In-Hadoop Database ▪ ▪ ▪ HBase JVM HDFS JVM ext3/ext4 Disks Typical Stack Tables/Files MapR-DB The most scalable, enterprise-grade, NoSQL database that supports online applications and analytics
- 6. © 2015 MapR Technologies 6 Consistent, Low Read Latency --- MapR-DB Read Latency --- Other’s Read Latency
- 7. © 2015 MapR Technologies© 2015 MapR Technologies Integration with Elasticsearch
- 8. © 2015 MapR Technologies Motivations Key-value stores are very fast, but don’t provide efficient searches for anything but the row key - Secondary indices don’t allow for the types of free-form text searches ES supports Customer demand to be able to do complex queries of their data, without: - Full table scans - Application-level in-sync updates to Elasticsearch Provide near real-time access to MapR-DB data in ES queries - at internet-scale volumes Robust fault-tolerant architecture with no changes to operational application
- 9. © 2015 MapR Technologies Inside Lonely Planet
- 10. Lets take THIS And Lets Come up with THIS ● What activities can I do on a ‘$’ budget? ● Where can I see some “Howler monkeys”? ● Can I zipline with arenal volcano views? ● Tell me some “$$”, mexican restaurants that are within 5 miles of Monteverde. ● Give me all the hotels with 8.5 or higher user rating that are less than 5 years old.
- 11. © 2015 MapR Technologies MapR Server Volume 1 Volume 1 DB Client Operations ES Client operations MapR-DB Cluster Volume 1 Tabl e 1 Tabl e 2 Tabl e n MapR Server Gateway Nodes Replication Gateway & ES Client Repl Stream Write ES Cluster Replica Replication Architecture MapR-DB Write
- 12. © 2015 MapR Technologies Elasticsearch client management Supports both node and transport Elasticsearch clients Flushed to Elasticsearch every 128KB, as part of a BulkRequest Multiple ES clients started on each Gateway - Flushes can happen in parallel - Ordering maintained for any individual row/document
- 13. © 2015 MapR Technologies Data Type Conversions Gateway converts change-log stream into JSON documents and builds an UpdateRequest ES assumes everything is a string, by default Add a mapping to Elasticsearch before setting up replication https://www.elastic.co/guide/en/elasticsearch/reference/1.4/mapping-core-types.html Use a custom conversion class - Implement com.mapr.fs.external.es.MapRESConverter - Provide path to JAR file and name when setting up replication - May need to do this in conjunction with manually creating a mapping in ES
- 14. © 2015 MapR Technologies© 2015 MapR Technologies Future Extensions
- 15. © 2015 MapR Technologies Supporting other External Sinks ES integration is a specific implementation of a generic external replication interface Could easily add support for other sinks in the future - e.g. Spark streaming Could also open up the API, allowing others access to the replication stream
- 16. © 2015 MapR Technologies© 2015 MapR Technologies Q & A
- 17. © 2015 MapR Technologies Find my presentation and other related resources here: http://events.mapr.com/Elastic (you can also find this link in the event’s page at meetup.com) Slidecks Whiteboard & demo videos Free Hadoop Sandbox Free On-Demand Training Free eBooks And more...