Relational databases for BigData
Download as PPTX, PDF0 likes160 views
The document discusses big data and open-source relational database management systems (RDBMSs). It begins by introducing the speaker and their background in cloud architecture and enterprise databases. The rest of the document summarizes various RDBMS options for big data, including PostgreSQL and extensions like Citus and Timescale, as well as columnar databases like Greenplum and Clickhouse. It compares the systems based on their SQL support, integration capabilities, data ingestion styles and performance, support for wide tables versus star schemas, horizontal scaling abilities, and availability as managed cloud services.
Relational databases for BigData
- 1. BigData and open-source RDBMSs Alexander Tokarev
- 2. Who am I 1. Cloud architect: - Solution design - Best practices - Performance tuning 2. Enterprise databases Oracle - 2001, Oracle 8.1.7 â 19c 3. Cloud databases: Redshift, Athena, BigQuery, SnowFlake
- 3. Agenda 1.Big Data 2.Postgres ecosystem 3.Others vendors 4.Conclusion 5.Q&A
- 5. Big Data in RDBMS 1.Velocity 2.Volume
- 6. Big Data ingestion nature 1.batch/micro-batch 2.realtime
- 7. Big Data nature 1.Time series data: date, numeric metrics âą Append-only mostly âą Huge volume of inserts 2.Analytical data: date, dimensions, measures âą A lot of dimensions âą Dimensions are subjects for update âą Moderate ingestion speed
- 8. Big Data data models 1.Star-style normalized 2.Wide tables 3.Nested structures
- 10. Postgres 1.Classical RDBMS 2.Very good for OLTP 3.Our experience â max DB size 5 TB 4.Partitioning is fine from PG 12 5.Row-based database 6.Search conditions predefined
- 11. Postgres improvements types 1.Fork + more control - major version faster 2.Extension + in sync with main version - not all changes are possible
- 12. Timescale 1.Postgres extension 2.Time-series optimized database (like Prometheus) 3.Ingestion 10x faster than Postgres 4.Own partitioning implementation 5.Time-series-oriented SQL 6.Attempts to create stream processing with SQL 7.SQL is perfect to extract by dates
- 13. MPP databases + Very fast Horizontally scalable - Different modeling approach Like good disks Hard to fined managed databases
- 14. Citus 1.Postgres extension 2.Owned by Microsoft 3.MPP tailored for OLTP â sharding for fast ingestion 4.Vendor states for OLAP as well â not true 5.SQL is perfect to extract data by shard key
- 15. Greenplum 1.Postgres fork 2.OLAP workload optimized MPP database 3.Perfect partitioning with automatic data lifecycle management 4.A lot of compression algorithms 5.Up to 100 TB clusters very fast 6.A lot of integrations: files, HDFS 7.Perfect SQL support â perfect for analytical queries
- 16. Big data platforms 1.Set of opensource components 2.Components for storage, processing, management 3.Storage components different by tiers: - in-memory for hot - MPP for warm - Hadoop for cold
- 17. Big data platform Arenadata DB
- 18. Clickhouse 1. DBMS written from scratch 2. Written by developers for developers â Yandex.Metrica 3. Wide table optimized MPP columnar database 4. Extremely fast batch ingestion + InMemory buffer tables 5. Very fast queries for wide tables 6.Aggregations during ingestion + real-time materialized views 7. Hundreds TB clusters 8. Huge number of integrations: files, HDFS, Kafka, JDBC 9. Self-sufficient for ELT/ETL 10. Own SQL dialect
- 19. Summary Postgres Citus Timescale Greenplum Clickhouse Columnar storage - - - + + SQL richness ++ + + + - Integrations - - - + +++ In-flight aggregations - - -+ - +++ Ingestion style Per row Per row Per row Bulk Huge batches Ingestion performance + + ++++ ++ +++ Wide table oriented - - + + +++ Star-schema oriented - - - + - Horizontal scaling - + ++ ++ +++ Managed in cloud +++ + + - -+
- 20. More questions?