Tachyon meetup San Francisco Oct 2014
1 like967 views
This document outlines the agenda for a Tachyon Meetup in San Francisco. The agenda includes discussing the xPatterns architecture, BDAS++, demos of Tachyon internals and APIs, and lessons learned. BDAS++ refers to enhancements made to Tachyon to support Spark SQL and the Spark job server. Lessons learned focus on issues discovered like partial in-memory file storage bugs and best practices for Tachyon usage.
Tachyon meetup San Francisco Oct 2014
- 1. Tachyon in xPatterns Tachyon Meetup SF 1 Oct 2014
- 2. Agenda • xPatterns architecture • BDAS++ • Demos • Tachyon internals (apis) • Lessons learned 2
- 3. 3
- 4. BDAS++ • Tachyon patch (https://github.com/amplab/tachyon/pull/482) • Jaws, xPatterns http spark sql server http://github.com/Atigeo/http-spark-sql- 4 server Backward compatible with Shark and Spark 0.x stack • Spark Job Server multiple Spark contexts in same JVM, job submission in Java + Scala https://github.com/Atigeo/spark-job-rest • Mesos framework starvation bug submitted patch… detailed Tech Blog link soon at http://xpatterns.com • *SchedulerBackend update, 0.9.0 patches (shuffle spill, Mesos fine-grained)
- 5. Demos … 5
- 6. Lessons learned • partial in-memory file storage bug • journal file on hdfs -> backup of local master disk • hdfs api • RawTable in Shark • persist(OFF_HEAP) temporary storage • RDD.persist() OFF_HEAP > MEMORY_SER_AND_DISK • native API: getInStream(CACHE|NO_CACHE) -> local workers • do not evict blocks when streaming to Tachyon/hdfs • Tachyon > Spark JVM Cache for long running jobs • kryo/defaultCodec/sequenceFile format to minimize memory footprint • 25million emails/month 2TB, 3-45 nodes, 120-170GB of RAM for Tachyon 6
- 7. Q & A 7
- 8. © 2013 Atigeo, LLC. All rights reserved. Atigeo and the xPatterns logo are trademarks of Atigeo. The information herein is for informational purposes only and represents the current view of Atigeo as of the date of this presentation. Because Atigeo must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Atigeo, and Atigeo cannot guarantee the accuracy of any information provided after the date of this presentation. ATIGEO MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Editor's Notes
- #4: The physical architecture diagram for our largest customer deployment, demonstrating the enterprise-grade attributes of the platform: scalability, high availability, performance, resilience, manageability while providing means for geo-failover (warehouse), geo-replication (real-time DB), data and system monitoring, instrumentation, backup & restore. Cassandra rings are DC-replicated across EC2 east and west coast regions, data between geo-replicas synchronized in real time through an ipsec tunnel (VPC-to-VPC). Geo-replicated apis behind an AWS Route 53 DNS service (latency based resource records sets) and ELBs ensures users requests are served from the closest geographical location. Failure to an entire region (happened to us during a big conference!) does not affect our availability and SLAs. User facing dashboards are served from Cassandra (real-time store), with data being exported from a data warehouse (Shark/Hive) build on top a Mesos-managed Spark/Hadoop cluster. Export jobs are instrumented and provide a throttling mechanism to control throughput. Export jobs run on the east-coast only, data is synchronized in real time with the west coast ring. Generated apis are automatically instrumented (Graphite) and monitored (Nagios).
- #6: Referral Provider Network: one of the 6 applications that we built for our healthcare customer using the xPatterns APIs and tools on the new beyond Hadoop infrastructure: ELT Pipeline, Export to NoSQL API. The dashboard for the RPN application was built using D3.js and angular against the generic api published by the export tool. The application allows for building a graph of downstream and upstream referred and referring providers, grouped by specialty and with computed aggregates like patient counts, claim counts and total charged amounts. RPN is used for both fraud detection and for aiding a clinic buying decision, by following the busiest graph paths. The dataset behind the app consists of 8 billion medical records, from which we extracted 1.7 million providers (Shark warehouse) and built 53 million relationships in the graph (persisted in Cassandra)