Data Engineer's Lunch #57: StreamSets for Data Engineering
Download as PPTX, PDF0 likes163 views
The document discusses StreamSets, a data integration platform designed for data engineering, enabling the creation of streaming, batch, CDC, ETL, and ML pipelines through a single user interface. It highlights features such as free tier access, self-managed deployments via Docker, and native integrations with various platforms. The document also mentions a demo showcasing the creation and management of data pipelines.
Data Engineer's Lunch #57: StreamSets for Data Engineering
- 1. Version 1.0 StreamSets for Data Engineering In Data Engineer's Lunch #57, we will discuss StreamSets and how it can be used for data engineering. Arpan Patel Engineer @ Anant
- 2. Streamsets ● Data Integration Platform Built for DataOps ● Build streaming, batch, CDC, ETL, and ML pipelines from a single UI and deploy data and workloads to any cloud ● DataOps Platform ○ Free tier (no cc required) with Data Collector Engine, Transform Engine, Control Hub ○ Self Managed Deployments via Docker ○ 2 active jobs, 2 active users, 10 published pipelines
- 3. Streamsets ● Control Hub ● Data Collector Engine ○ Open-source ● Transformer Engine ○ Can natively execute on Apache Spark, Snowflake, AWS EMR, Google Cloud Dataproc, and Databricks platforms ● Pre-built connectors and native integrations ○ Applications, Big Data, SQL/NoSQL DBs, Storage/Warehouses, Streaming ○ Tons of Sources + Destinations ● StreamSets Academy + Tutorials
- 4. Demo ● Spin up Data Collector Deployment from Control Hub + Docker ● Create Sample Pipeline and Preview Data ● Schedule / Run Pipeline Job and View Metrics ● Spin up Transformer Engine Deployment from Control Hub + Docker ● Create Sample ETL Pipeline and Preview Data ● Submit ETL Pipeline to Local Spark
- 5. Strategy: Scalable Fast Data Architecture: Cassandra, Spark, Kafka Engineering: Node, Python, JVM,CLR Operations: Cloud, Container Rescue: Downtime!! I need help. www.anant.us | solutions@anant.us | (855) 262-6826 3 Washington Circle, NW | Suite 301 | Washington, DC 20037