Apache Cassandra Lunch #94: StreamSets and Cassandra
Download as PPTX, PDF0 likes158 views
This document details a discussion led by Arpan Patel about connecting StreamSets and Cassandra, focusing on the features of StreamSets, including its data integration capabilities and various deployment options. It highlights the supported Cassandra versions, authentication methods, and the use of a demo for data transformations. Additionally, it emphasizes a scalable architecture involving Cassandra, Spark, and Kafka for engineering and operations.
Apache Cassandra Lunch #94: StreamSets and Cassandra
- 1. Version 1.0 StreamSets and Cassandra In Cassandra Lunch #94, Arpan Patel will discuss how to connect StreamSets and Cassandra. Arpan Patel Engineer @ Anant
- 2. Streamsets ● Data Integration Platform Built for DataOps ● Build streaming, batch, CDC, ETL, and ML pipelines from a single UI and deploy data and workloads to any cloud ● DataOps Platform ○ Free tier (no cc required) with Data Collector Engine, Transform Engine, Control Hub ○ Self Managed Deployments via Docker ○ 2 active jobs, 2 active users, 10 published pipelines
- 3. Streamsets ● Control Hub ● Data Collector Engine ○ Open-source ● Transformer Engine ○ Can natively execute on Apache Spark, Snowflake, AWS EMR, Google Cloud Dataproc, and Databricks platforms ● Pre-built connectors and native integrations ○ Applications, Big Data, SQL/NoSQL DBs, Storage/Warehouses, Streaming ○ Tons of Sources + Destinations ● StreamSets Academy + Tutorials
- 4. Cassandra Connector ● https://docs.streamsets.com/platform- datacollector/latest/datacollector/UserGuide/Destinations/ Cassandra.html ● Supported Apache Cassandra Versions ○ 1.2, 2.x, 3.x ● Additional Auth ○ DSE ○ Kerberos ○ SSL ○ TLS ● Batch Writes ○ Logged -> distributed batch log and atomic ○ Unlogged -> can write partial batches of records to Cassandra
- 5. Demo ● Spin up Data Collector Deployment from Control Hub + Docker ● Get CSV data from GitHub ● Make data transformations + arithmetic ● Spin Up Cassandra on Docker + Connect in Control Hub ● Write to Cassandra
- 6. Strategy: Scalable Fast Data Architecture: Cassandra, Spark, Kafka Engineering: Node, Python, JVM,CLR Operations: Cloud, Container Rescue: Downtime!! I need help. www.anant.us | solutions@anant.us | (855) 262-6826 3 Washington Circle, NW | Suite 301 | Washington, DC 20037