Real-time Centralized Data Platform
Download as PPTX, PDF0 likes110 views
The document outlines a centralized real-time data platform utilizing Spark and Apache Cassandra to design, build, and manage business applications. It details various use cases including data streaming, processing, and integration with multiple cloud services, while emphasizing scalability and stability through microservices architecture. The platform supports both real-time analytics and batch processing across diverse data sources.
Real-time Centralized Data Platform
- 1. Real-time Centralized Data Platform Using Spark and Apache Cassandra
- 2. Business Platform Success We design, build, and manage business platforms by leveraging DataStax, Sitecore, Salesforce, Quickbooks and other cloud software.
- 3. Use Cases ● Stream data inputs from different types of streams such as Rabbit, Kinesis, Kafka, SQS...etc into one standardized platform. ● Conduct streaming analytics / data processing on this data in realtime. ● Batch process data in or out from various sources such as S3, SQL, Dynamo, etc.
- 4. Spark Datasource / App / Environments ● DSE Analytics Runs Spark + Cassandra On the Same Nodes
- 5. ● Real-time Processing for Streams ● Realtime Structured Streaming or Batch Processing for SQL/CSV/etc. ● Real-time availability to other systems.
- 6. Microservice ● Segment Services by ○ Datacenter ○ Keyspace ○ Table ● Depending on Scalability Needs
- 7. Message Assurance ● One Cluster ● 2 Virtual Machine Data Centers ● 1 Kubernetes Container Datacenter ● VM DCs have RF=3 for Stability ● K8S DC has RF=2 for Speed ● Allows for Speed + Stability at Scale
- 8. Other Patterns ● ETL on Spark ○ Streaming ETL w/ Structured Streaming ○ Batch ETL w/ all Data Sources ● API Platforms on Cassandra ○ Dreamfactory ■ Generated API Layer ○ MediaWiki RESTBase ■ API CACHE ■ Mimics Dynamo/Google Storage ○ Aerobase UnifiedPush Server ■ Back end as a Service ■ Implement Pub/Sub
- 9. Data & Analytics Cassandra, DataStax, Kafka, Spark Customer Experience Sitecore Information Systems Salesforce, Quickbooks, and more www.anant.us | solutions@anant.us | (855) 262-6826 3 Washington Circle, NW | Suite 301 | Washington, DC 20037