What's new in MariaDB AX webinar
2 likes211 views
The document outlines the features and updates of MariaDB AX, focusing on its capabilities for modern analytics and data warehousing. It highlights advancements in data ingestion, custom analytics, and enhanced high availability and disaster recovery options. Key updates include bulk and streaming data adapters, user-defined functions, and integration with Apache Kafka for real-time data processing.
What's new in MariaDB AX webinar
- 1. What’s new in MariaDB AX, the database for modern analytics & data warehousing Dipti Joshi Director Product Management MariaDB
- 2. Conrad Hotel, New York City February 26–27, 2018 m18.mariadb.com M|18, the second annual MariaDB user conference, is where global MariaDB experts and practitioners meet to exchange ideas, best practices and success stories. Join us in NYC to share journeys on open source strategies and infrastructure modernization with MariaDB. -2017 SPONSORS- YOTTA ZETTA EXA
- 3. MariaDB AX Analytics made easy – simple, fast, scalable… and open source
- 4. Customer Use Case Industry: healthcare (Medicaid) Data: surveys Use case: decision support system Details: 1. Identify trends and patterns 2. Determine population cohorts 3. Predict health outcomes 4. Anticipate funding / capacity 5. Recommend intervention Can’t do complex queries on current hardware with Oracle and snowflake schemas Limited to optimizing for simple, known queries (2-3 columns) Replaced with ColumnStore > a single table > 2.5 million rows, 248 columns > complex, ad-hoc queries > query 20+ columns in seconds
- 5. Customer Use Cases By industry Finance Identify trade patterns Detect fraud and anomolies Predict trading outcomes Manufacturing Simulations to improve design/yield Detect production anomalies Predict machine failures (sensor data) Telecom Behavioral analysis of customer calls Network analysis (perf and reliability) Healthcare Find genetic profiles/matches Analyze health vs spending Predict viral outbreaks
- 6. MariaDB AX MariaDB Server MariaDB MaxScale MariaDB ColumnStore Parallel queries Distributed storage No indexes Automatic partitioning Read optimized High compression Low disk IO ColumnStore Storage ColumnStore Storage ColumnStore Storage MariaDB Server ColumnStore MariaDB Server ColumnStore MariaDB MaxScale MariaDB Server ColumnStore ColumnStore Storage MariaDB MaxScale
- 7. MariaDB AX Goals 3. Streamline and simplify the process of ingesting data 2. Make it easier to perform custom, complex analytics 1. Expand high availability/disaster recovery options
- 8. MariaDB AX What was there Manual import Manual backup/restore Window functions Aggregate functions User-defined functions Cross-engine joins ColumnStore StorageMariaDB Server ColumnStoreInnoDB Applications / Spark MariaDB MaxScale
- 9. MariaDB AX What’s new MariaDB ColumnStore 1.1 Streaming data adapters Bulk data adapters User defined Window functions Distributed aggregates Spark support Phase I: SQL (JDBC) Phase II: data adapters High availability Local storage (GlusterFS) Parallel backup/restore ColumnStore Storage Backup/Restore GlusterFS MariaDB Server ColumnStore Applications / Spark Bulk Data Adapters User Defined Window Functions Streaming Data Adapters User Defined Aggregate Functions MariaDB MaxScale
- 10. CERTIFICATION What’s new in MariaDB AX INGESTION ANALYTICS Applications, Apache Kafka, MariaDB MaxScale User-defined aggregate and window functions HA / DR GlusterFS support, Parallel backup/restore DATA TYPES Text, BLOB columns SECURITY Auditing Tableau
- 11. Extend high availability/disaster recovery options
- 12. GlusterFS Volume Replication High availability GlusterFS can replicate files within a volume - HA without the need for an expensive SAN ColumnStore storage nodes can read other files within a volume - simple, automatic failover ColumnStore Storage (dbroot1) ColumnStore Storage (dbroot2) MariaDB Server ColumnStore MariaDB Server ColumnStore ColumnStore Storage (dbroot3) /dbroot1 /dbroot2 /dbroot2 /dbroot3 /dbroot3 /dbroot1
- 13. High availability GlusterFS can replicate files within a volume - HA without the need for an expensive SAN ColumnStore storage nodes can read other files within a volume - simple, automatic failover MariaDB Server ColumnStore MariaDB Server ColumnStore GlusterFS Volume /dbroot1 /dbroot2 /dbroot2 /dbroot3 /dbroot3 /dbroot1 Replication ColumnStore Storage (dbroot1) ColumnStore Storage (dbroot2) ColumnStore Storage (dbroot3)
- 14. GlusterFS Volume High availability GlusterFS can replicate files within a volume - HA without the need for an expensive SAN ColumnStore storage nodes can read other files within a volume - simple, automatic failover MariaDB Server ColumnStore MariaDB Server ColumnStore /dbroot2 /dbroot3 /dbroot3 /dbroot1 Replication ColumnStore Storage (dbroot2) ColumnStore Storage (dbroot3)
- 15. /home/user/columnstoreBackupData/pm1dbroot1 /home/user/columnstoreBackupData/pm2dbroot2 /home/user/columnstoreBackupData/pm3dbroot3 Parallel backup/restore Parallel backup/restore using rsync - faster backup and restore Support incremental backup and restore - faster backup and restore Consolidate data from multiple storage nodes in a single backup location - simplified, automatic backups and restores ColumnStore Storage ColumnStore Storage MariaDB Server ColumnStore MariaDB Server ColumnStore ColumnStore Storage Backup and restore tool rsync /data1/* rsync /data2/* rsync /data3/*
- 16. Make it easier to perform custom, complex analytics
- 17. User-defined aggregate and window functions User-defined distributed aggregate functions - custom analytical functions and better performance User-defined window functions Example: calculate a weighted sum (revenue) $1-10 (0.5) $11-100 (1.0) $100+ (1.5) MariaDB Server ColumnStore MariaDB Server ColumnStore ColumnStore Storage ColumnStore Storage ColumnStore Storage $10 $5 $100 $100 $200 $300 Column WSUM $4 $2 $8 $4 $20 $20 Column WSUM $12 $6 $60 $60 $300 $450 Column WSUM WSUM = $405 WSUM = $26 WSUM = $516 WSUM = $947
- 18. Streamline and simplify the process of data ingestion
- 19. Organizations need to make data available for analysis as soon as it arrives Machine learning results need to be stored where other business/data analysts work with them Time to insight and time to action are now competitive differentiators for businesses Motivation
- 20. Bulk data adapters Applications can use bulk data adapters to collect and write data - on-demand data loading Bypass SQL interface, parser and optimizer - faster writes C++ Python Java More on the way MariaDB Server ColumnStore Application ColumnStore Storage ColumnStore StorageColumnStore Storage Write API Write API Write API MariaDB Server ColumnStore Bulk Data Adapter 1. For each row a. For each column i. bulkInsert->setColumn b. bulkInsert->writeRow 2. bulkInsert->commit * Buffer 100,000 rows by default
- 21. Customer Use Case Industry: biotechnology (genetics) Data: genotypes Use case: genetic profiling Details: 1. Find genetic mates for cattle 2. Predict meat production 3. Gene/DNA analysis Had to convert to CSV files and schedule import jobs (cron) Always receiving new genetic data Migrated to data adapter (Python) > streamline import process > remove steps / possible error > remove delays > import data on demand > immediate customer access
- 22. Streaming data adapters – MaxScale CDC Stream all writes from MariaDB TX to MariaDB AX automatically and continuously - ensure analytical data is up to date and not stale, no need for batch jobs, manual processes or human intervention MariaDB Server InnoDB MariaDB Server ColumnStore MariaDB MaxScale ColumnStore Storage ColumnStore StorageColumnStore Storage Write API Write API Write API MariaDB Server ColumnStore Streaming Data Adapter (CDC Client) CDC Server
- 23. Streaming data adapters – Apache Kafka Stream all messages published to Apache Kafka topics to MariaDB AX automatically and continuously - enable data from many sources to be streamed and collected for analysis without complex code MariaDB Server ColumnStore Apache Kafka ColumnStore Storage ColumnStore StorageColumnStore Storage Write API Write API Write API MariaDB Server ColumnStore Streaming Data Adapter (Kafka Client) Topic Topic Topic
- 24. The big picture – putting it all together
- 25. AnalyticsOperations Ingestion Apache Kafka Streaming Data Adapters Data Services Bulk Data Adapters Spark / Python / ML Bulk Data Adapters Transaction (OLTP) MariaDB Server InnoDB MariaDB MaxScale Web/Mobile Services MariaDB MaxScale Analytics (OLAP) MariaDB Server ColumnStore
- 26. Demo TX to AX data streaming
- 27. Reach me Resources Download Documentation https://mariadb.com/kb/en/library/mariadb-columnstore/ Blogs https://mariadb.com/blog-tags/columnstore https://mariadb.com/blog-tags/big-data dipti.joshi@mariadb.com MariaDB ColumnStore 1.1 https://mariadb.com/downloads/mariadb-ax Bulk Data Adapters and Streaming Data Adapters https://mariadb.com/downloads/mariadb-ax/data-adapters MariaDB ColumnStore Backup/Restore Tool https://mariadb.com/downloads/mariadb-ax/tools-ax
- 28. What’s new in MariaDB AX Summary Improved HA/DR GlusterFS support Parallel backup/restore Streamlined data ingestion Streaming data adapters Bulk data adapters Complex, custom analytics User-defined aggregate functions User-defined window functions Text and binary columns Spark integration (in-progress) JDBC (SQL) Direct (data adapter)
- 29. Thank you