Real-Time Analytics with MemSQL and Spark

Editor's Notes

• #12: - Distributed In-Memory Database - Built for real-time analytics and transactions Familiar SQL Interface Spark integration out-of-the-box - Native Kafka Ingestion What did they want to do? - highly scalable infrastructure that collects, stores and processes user engagement data in real-time higher performance event logging Reliable log transport and storage ability to query real-time data
• #13: user clicks Pin or repin event is pushed to Apache Kafka Storm, Spark and other custom built log readers process these events in real-time log persistence service called Secor that reliably writes these events to Amazon S3 (zero data loss, overcoming its weak eventual consistency model).  self-serve big data platform loads the data from S3 into many different Hadoop clusters for batch processing In house tools Singer (logger) & Secor (replicator) asynchronously replicating local logs from app servers to centralized S3 location using Kafka for transport Kafka was great for throughput, but needed a way to derive value, e.g. run SQL against these datasets in real time A few days later this data would hit Redshift and be queryable
• #14: - took several days to access analytics and make available to data science team (too late, A/B testing, advertising) - no SQL Interface - 5.5 M rows / second for one topic, 1.7 M rows / second for another, with the lowest throughput being 132k rows / second - data needs to be filtered as well as enriched - At LEAST once semantics
• #17: user clicks Pin or repin event is pushed to Apache Kafka Storm, Spark and other custom built log readers process these events in real-time log persistence service called Secor that reliably writes these events to Amazon S3 (zero data loss, overcoming its weak eventual consistency model).  self-serve big data platform loads the data from S3 into many different Hadoop clusters for batch processing In house tools Singer (logger) & Secor (replicator) asynchronously replicating local logs from app servers to centralized S3 location using Kafka for transport Kafka was great for throughput, but needed a way to derive value, e.g. run SQL against these datasets in real time A few days later this data would hit Redshift and be queryable
• #19: Goes both ways
• #20: easily repeatable success days to seconds now has a source of record for sharing relevant user engagement data and metrics their data analyst and with key brands Pinterest and their partners can get a better understanding of user behavior and provide more value to the Pinner community Cheaper the ability to identify (and react to) developing trends as they happen provides insight into how users are engaging with Pins across the globe in real-time helps Pinterest become a better recommendation engine- SQL interface for engineering and data science teams fast ad-hoc query execution on real-time data to allow the execution of SQL queries on the real-time events as they arrive
• #21: easily repeatable success days to seconds now has a source of record for sharing relevant user engagement data and metrics their data analyst and with key brands Pinterest and their partners can get a better understanding of user behavior and provide more value to the Pinner community Cheaper the ability to identify (and react to) developing trends as they happen provides insight into how users are engaging with Pins across the globe in real-time helps Pinterest become a better recommendation engine- SQL interface for engineering and data science teams fast ad-hoc query execution on real-time data to allow the execution of SQL queries on the real-time events as they arrive
• #26: Pull up Ops Pull up a terminal and create the database Deploy Spark Create a Streamliner pipeline Create a Pipeline pipeline Expose the UI Ad-Hoc queries, Tableau, and custom reporting