Apache Apex & Bigtop
Download as PPTX, PDF2 likes459 views
Apache Apex is a platform and runtime engine designed for developing scalable, fault-tolerant distributed applications, leveraging Hadoop for resource management and storage. It integrates with a variety of data sources and libraries, enabling both streaming and batch data processing. The document also outlines the installation process using the Bigtop framework and provides resources for further engagement with the Apache Apex community.
Apache Apex & Bigtop
- 1. © 2016 DataTorrent Chinmay Kolhatkar Committer, Apache Apex Engineer, DataTorrent June 21, 2016 Apache Apex-Bigtop
- 2. © 2016 DataTorrent Agenda 2 • About Apache Apex • Apex Platform Overview • Apex - Native Hadoop Integration • Apex Malhar Library • Apex as a Bigtop component • Installing Bigtop Apex • Apex Docker sandbox • Apex Docker sandbox Demo
- 3. © 2016 DataTorrent About Apache Apex 3 • Platform and runtime engine that enables development of scalable and fault-tolerant distributed applications • Hadoop native (Hadoop >= 2.2) No separate service to manage stream processing Streaming Engine built into Application Master and Containers • Process streaming or batch big data • High throughput and low latency • Library of commonly needed business logic • Write any custom business logic in your application
- 4. © 2016 DataTorrent Apex Platform Overview 4
- 5. © 2016 DataTorrent Apex - Native Hadoop Integration 5 • YARN is the resource manager • HDFS used for storing any persistent state
- 6. © 2016 DataTorrent Apex Malhar Library 6 RDBMS • Vertica • MySQL • Oracle • JDBC NoSQL • Cassandra, Hbase • Aerospike, Accumulo • Couchbase/ CouchDB • Redis, MongoDB • Geode Messaging • Kafka • Solace • Flume, ActiveMQ • Kinesis, NiFi File Systems • HDFS/ Hive • NFS • S3 Parsers • XML • JSON • CSV • Avro • Parquet Transformations • Filters • Rules • Expression • Dedup • Enrich Analytics • Dimensional Aggregations (with state management for historical data + query) Protocols • HTTP • FTP • WebSocket • MQTT • SMTP Other • Elastic Search • Script (JavaScript, Python, R) • Solr • Twitter
- 7. © 2016 DataTorrent Apex as Bigtop component 7 • Uses Bigtop framework for ease of deployment Deployment using puppet recipes and Vagrant Can spawn multiple node clusters for docker, VM & OpenStack • Generates a deployable binaries for Apex engine RPM - CentOS 5 & 6, Fedora 20, OpenSuse 42.1 DEB - Ubuntu 14.04 & 16.04, Debian 8 • Allows validating installations Package Test Smoke Test
- 8. © 2016 DataTorrent • Add Bigtop Repository http://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/ • Install bigtop-hadoop For Debian: apt-get install hadoop* For RPM: yum install hadoop* • Download bigtop-apex from bigtop CI https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/ • Install Apex: For Debian: dpkg -i apex_3.4.0-1_all.deb For RPM: rpm -i apex-3.4.0-1.el6.noarch.rpm Installing Bigtop Apex Bigtop 1.1.0 (Current) 8
- 9. © 2016 DataTorrent • Add Bigtop Repository (Future URL) http://www.apache.org/dist/bigtop/bigtop-1.2.0/repos/ • Install apex For Debian: apt-get install apex For RPM: yum install apex Installing Bigtop Apex Bigtop 1.2.0 (Next Release) 9
- 10. © 2016 DataTorrent • A quick starter Apex docker image: https://hub.docker.com/r/chinmayk/apex/ • Preconfigured and running components HDFS (namenode, secondarynamenode, datanode) YARN (resourcemanager, nodemanager, timelineserver) • Preconfigured and installed component Apex • Get started: Step1: docker pull chinmayk/apex Step2: docker run -it chinmayk/apex:ubuntu-14.04 Apex Docker sandbox 10
- 11. © 2016 DataTorrent Apex Docker sandbox (contd.) 11
- 12. © 2016 DataTorrent Resources 12 • Apache Apex website - http://apex.apache.org/ • Subscribe - http://apex.apache.org/community.html • Download - http://apex.apache.org/downloads.html • Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex • Facebook - https://www.facebook.com/ApacheApex/ • Meetup - http://www.meetup.com/topics/apache-apex • SlideShare - http://www.slideshare.net/ApacheApex/presentations • More Examples - https://github.com/DataTorrent/examples • Startup Program – Free Enterprise License for Startups, Educational Institutions, Non-Profits - https://www.datatorrent.com/startups/ • Cloud Trial - https://www.datatorrent.com/download/cloud-trial/
- 13. © 2016 DataTorrent We Are Hiring 13 • jobs@datatorrent.com • Back-End Engineers • Front-End Engineers • QA Automation Engineers • Solutions Engineers