Delivering the power of data using Spring Cloud DataFlow and DataStax Enterprise Cassandra NoSQL database
1 like658 views
This document discusses using Spring Cloud Data Flow and DataStax Enterprise to provide app developers insight into app performance. It summarizes DataStax Enterprise as an integrated platform for multi-model databases including key-value, tabular, JSON and graph models to support transactional, search and analytics workloads. It also discusses using Spring Cloud Data Flow to define and orchestrate data pipelines to source, process, analyze and flow data to downstream processes leveraging DataStax Enterprise.
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterprise Cassandra NoSQL database
- 1. PCF Metrics – App Dev Providing App Developers insight into app performance Delivering the Power of Data Using Spring Cloud Data Flow and DataStax Enterprise Gilbert Lau, Partner Architect, DataStax gilbert.lau@datastax.com Wayne Lund, Advisory Platform Architect, Pivotal wlund@pivotal.io
- 2. • Multi-model including key value, tabular, JSON, and graph • Mixed workloads o Transactional o Search o Analytics • Developer tooling • Administration and monitoring 2 An integrated platform So you can focus on your business. DataStax Enterprise © 2017 DataStax, All Rights Reserved. DataStax is a registered trademark of DataStax, Inc. and its subsidiaries in the United States and/or other countries. Apache Cassandra, Apache, Spark, and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.
- 3. Instant insight for real-time experiences Search Analytics Graph
- 4. Indexing & Search Streaming Analytics Graph Batch Analytics Integrated multi-model/mixed workload platform
- 5. DSE Search Search + Cassandra Your Application CQL Live indexing engine with powerful search • Automatic indexing on insert • Higher ingestion throughput • Distributed query optimization Compared to self-managed: • No separate search cluster to manage • Probably less total hardware required • No “Split Brain” data inconsistencies • No ETL or synch to build and maintain • No app level data management code
- 6. DSE Analytics Your Application Real Time Operations Cassandra Analytics Analytics Queries Your Analytics Real Time Replication Single DSE Custer Streaming, ad-hoc, and batch • High-performance • Workload management • SQL reporting Compared to self-managed: • No ETL • True HA without Zookeeper
- 7. DSE Graph A scalable, distributed graph database that is optimized for storing, traversing and querying complex graph data in real time • Value data between relationships • DSE Analytics and Search integrated • Perfect for use cases: Customer360, Recommendations, Fraud Detection
- 8. Enterprise-grade Security External Authentication • External validation of authorized users • Leverages Kerberos & LDAP/AD • Single sign-on to all data domains Transparent Data Encryption • Protects sensitive data at rest via encryption • No changes needed at app level Data Auditing • Audit trail of all accesses and changes • Control to audit only what’s needed • Uses log4j interface to ensure performance & efficient audit
- 9. The Most Innovative Companies Use DataStax
- 10. Spring Cloud Dataflow Architecture ❖ The Spring Cloud Dataflow server uses Spring Cloud Deployer to target various runtimes like: ➢ Pivotal Cloud Foundry ➢ pcfDEV (developer cf) ➢ YARN ➢ K8S ➢ Mesos ➢ Others ❖ Multiple ways to access dataflow-server ➢ Commandline shell ➢ Admin UI with specific views for apps, streams, tasks, analytics and more ➢ Restful URL for clients
- 11. Spring Cloud Dataflow Tile for PCF ❖ We are using the SCDF tile available from network.pivotal.io. ❖ This tile is currently in private beta but will be out in Jan as a public beta once SCDF-1.3 OSS has been Released. ❖ This allows you to use a managed service that will automatically provision MySQL for storing the meta-data required for streams and tasks ❖ It will also automatically provision Rabbit as the default binder for the message bus. ❖ It will also automatically bind your apps to the real-time analytics service (Redis) for runtime statistics. ❖ When you deploy a stream it will automatically create the apps and bind them to the services mentioned.
- 12. Spring Cloud Dataflow Tile for PCF ❖ We are using the SCDF tile available from network.pivotal.io. ❖ This allows you to use a managed service that will automatically provision MySQL for storing the meta-data required for streams and tasks ❖ It will also automatically provision Rabbit as the default binder for the message bus. ❖ It will also automatically bind your apps to the real-time analytics service (Redis) for runtime statistics. ❖ When you deploy a stream it will automatically create the apps and bind them to the services mentioned.
- 13. Spring Cloud Dataflow Tile for PCF ❖ We are using the SCDF tile available from network.pivotal.io. ❖ This allows you to use a managed service that will automatically provision MySQL for storing the meta-data required for streams and tasks ❖ It will also automatically provision Rabbit as the default binder for the message bus. ❖ It will also automatically bind your apps to the real-time analytics service (Redis) for runtime statistics.
- 14. Spring Cloud Dataflow Tile for PCF ❖ We are using the SCDF tile available from network.pivotal.io. ❖ This allows you to use a managed service that will automatically provision MySQL for storing the meta-data required for streams and tasks ❖ It will also automatically provision Rabbit as the default binder for the message bus. ❖ It will also automatically bind your apps to the real-time analytics service (Redis) for runtime statistics. ❖ When you deploy a stream it will automatically create the apps and bind them to the services mentioned (e.g. rabbit & redis). .
- 15. Spring Cloud Dataflow Tile for PCF ❖ We are using the SCDF tile available from network.pivotal.io. ❖ This allows you to use a managed service that will automatically provision MySQL for storing the meta-data required for streams and tasks ❖ It will also automatically provision Rabbit as the default binder for the message bus. ❖ It will also automatically bind your apps to the real-time analytics service (Redis) for runtime statistics. ❖ When you deploy a stream it will automatically create the apps and bind them to the services mentioned (e.g. rabbit & redis). ❖ Selecting an app will provide the details
- 16. Spring Cloud Dataflow Apps
- 17. Spring Cloud Dataflow stream pipeline DSL Stream DSL Meetup-rsvp | groovy-filter | groovy-transform | cassandra Target Runtime Data Flow Spring Boot Applications Cassandra meetup-rsvp groovy-filter groovy-transform
- 18. Spring Cloud Dataflow Dashboard ❖ Create streams visually by dragging and dropping apps from the palette onto the canvas ❖ The DSL is automatically created for you in the edit window where you can modify either visually or from the UI. ❖ You have the same syntax for the DSL available for you with the commandline shell.
- 19. Spring Cloud Dataflow Dashboard ❖ Create streams visually by dragging and dropping apps from the palette onto the canvas ❖ The DSL is automatically created for you in the edit window where you can modify either visually or from the UI. ❖ You have the same syntax for the DSL available for you with the commandline shell. ❖ When you select an app and settings a list of the properties that you can set are available for you.
- 20. Spring Cloud Dataflow Dashboard ❖ Create streams visually by dragging and dropping apps from the palette onto the canvas ❖ The DSL is automatically created for you in the edit window where you can modify either visually or from the UI. ❖ You have the same syntax for the DSL available for you with the commandline shell. ❖ When you select an app and settings a list of the properties that you can set are available for you. ❖ You also have the same view available at runtime.
- 21. Spring Cloud Dataflow and Taps ❖ SCDF also supports Taps. ❖ They can be created from the dataflow-shell or from teh UI as shown here ❖ They use the existing stream to tap properties. In this example we are showing cities with a bubble chart and venues with a pie chart ❖
- 22. Spring Cloud Dataflow Results ❖ We have a database populated with filtered and processed data ❖ And a Web App taking advantage of everything that Datastax gives us for spatial searching 08a99aff-cb5f-4f53-95ce-d947183f7857@cqlsh:meetups > select count(*) from meetup; count ------- 10586 (1 rows)
- 23. What is for DataStax for PCF? • A fully integrated capability on Pivotal Cloud Foundry • The always-on data platform for micro-services on PCF • The leading distributed database on PCF
- 24. Demo (Streaming Data Pipeline) Cassandra + Search / Analytics Message Broker Meetup consumer application (source) DataStax writer Application (sink) DataStax Studio
- 25. Cassandra + Search / Analytics (Pivotal Tile) Demo (Spring Boot App) (Stateless Spring Boot App) DataStax Studio
- 26. Key Takeaways DataStax Enterprise serving as the always-on data platform to store real-time data sets and generate meaningful insights Using Spring Cloud Data Flow to define and orchestrate dataflows to source, process, analyze and flow data where downstream processes can leverage DataStax Enterprise Service Broker tile for micro-services or cloud apps to access a DSE cluster
- 27. Share the demos with your colleagues https://github.com/wxlund/s1p-meetup-rsvp-demo Next Steps