Demystifying Real time Analytics with TiDB
0 likes44 views
The document discusses the capabilities of TiDB, an open-source, distributed HTAP database that supports real-time analytics and integrates with Tiflash for efficient processing. It highlights the architecture, advantages of TiDB, and the functionality of Tiflash in handling real-time data insights, including scaling, query optimization, and replication management. Use cases for real-time analytics such as fraud detection and market analysis are mentioned, along with challenges related to data volume and integration.
![❏ You can specify read queries to use replicas of specific engines with TiDB as shown below:
Config file:
[isolation-read]
engines = ["tikv", "tidb", "tiflash"]
SESSION:
set SESSION tidb_isolation_read_engines = "engine list separated
by commas";
Engine Isolation](https://image.slidesharecdn.com/demystifyingrealtimeanalyticswithtidb-240621104222-5fe308f4/85/Demystifying-Real-time-Analytics-with-TiDB-27-320.jpg)
![❏ You can force the TiDB to use TiFlash replica as below with manual hint in query.
select /*+ read_from_storage(tiflash[table_name]) */ ... from
table_name;
Manual Hint](https://image.slidesharecdn.com/demystifyingrealtimeanalyticswithtidb-240621104222-5fe308f4/85/Demystifying-Real-time-Analytics-with-TiDB-28-320.jpg)
Demystifying Real time Analytics with TiDB
- 1. Demystifying Real Time Analytics With TiDB Unlocking the Power of TiFlash for Real-Time Data Insights Kabilesh PR Co-Founder, Mydbops LLP 33rd MyWebinar - Mydbops
- 2. About Me Kabilesh PR ❏ Interested in Open Source DB technologies ❏ Keen Interest in MySQL, TiDB & Distributed SQL’s ❏ Active Tech Speaker/Blogger ❏ Pingcap Certified TiDB Professional ❏ AWS Database Speciality ❏ Founding Partner, Mydbops
- 3. Focus on MySQL, MongoDB, PostgreSQL, TiDB, Cassandra Consulting Services Consulting Services Managed Services 24*7 DBA Team Targeted Engagement Mydbops Services
- 4. ❏ Introduction ❏ TiDB Architecture ❏ Understanding Real-time Analytics ❏ Analytical Engine - TiFlash ❏ Enabling TiFlash ❏ Queries with TiFlash Agenda
- 5. Introduction
- 6. TiDB is an Open Source , Distributed HTAP database compatible with MySQL Protocol. Introduction 2
- 9. MySQL compatible, the TiDB SQL Layer separates compute from storage to make scaling simpler,
- 10. The Placement Driver functions as a orchestrator. Responsible for TSO, scheduling, shard maintenance, metadata and much more
- 11. Tikv is ROW based, Transactional storage, Offers high-availability, strong consistency that can auto-scale to hundreds of node with petabyte data scale
- 12. Advantages of TiDB Open Source No Vendor lock-in with a database that’s 100% open source. Horizontal Scaling Grants total transparency into data workloads without manual sharding. Horizontal Scaling Grants total transparency into data workloads with automatic sharding. High Availability Guarantees auto-failover and self-healing for continuous data access. MySQL Compatibility Enjoy the most MySQL compatible distributed SQL database on the planet. Multi-Cloud Deploy database clusters anywhere in the world. Mixed Workloads Streamlined tech stack makes it easier to produce real-time analytics. Robust Security Protect data with enterprise-grade encryption both in-flight and at-rest.
- 15. ❏ Real-time analytics: Process of analyzing data as it is created, collected, and processed to provide immediate insights to enable prompt decision-making. ❏ Use- Case: Real-time Fraud detection, Market Analysis, Personalized Recommendations, Demand forecasting ❏ Challenges: Data Volume and Velocity Integration Data Quality Cost
- 16. ● With TiDB Realtime insights as in when the business happens. ● Easy Integration and maintenance.
- 17. Real-Time Analytical Engine = TiFlash
- 18. ❏ An Integrated columnar storage engine built exclusively for analytical workload. ❏ It's tightly integrated with TiKV and uses Clickhouse co-processor for providing MPP (Massively Parallel Processing) analytical queries. What is TiFlash?
- 19. Data Sync with TiFlash
- 20. Data Sync to TiFlash is done using the extended Raft-Learner Algorithm
- 21. Enabling TiFlash
- 22. ❏ Adding a TiFlash node online won't impact the OLTP workload. Tiflash_servers: - host: 10.0.1.10 #tiup cluster scale-out <cluster-name> scale-out-topology.yaml ❏ After adding a TiFlash node, replication won’t starts by default. ❏ Replication to TiFlash can be at the table level or schema level. ALTER TABLE table_name SET TIFLASH REPLICA count; ALTER DATABASE db_name SET TIFLASH REPLICA count; ❏ Monitoring of TiFlash replication: SELECT * FROM information_schema.tiflash_replica; Enabling TiFlash
- 23. Scaling TiFlash
- 24. ❏ Scaling out and scaling in TiFlash nodes is done online and won't impact the OLTP workload. Nodes Addition: Tiflash_servers: - host: 10.0.1.10 - host: 10.0.1.12 #tiup cluster scale-out <cluster-name> scale-out-topology.yaml Adjust the table / Schema replica count ALTER TABLE table_name SET TIFLASH REPLICA count; ALTER DATABASE db_name SET TIFLASH REPLICA count; Node Removal: Set the replica count to 0 for table ALTER TABLE table_name SET TIFLASH REPLICA 0; #tiup cluster scale-in <cluster-name> --node <tiflash_node_id> Scaling TiFlash
- 26. ❏ TiDB Optimizer automatically determines to use TiFlash replicas based on the COST. ❏ This works even in mix of workloads. Smart Selection
- 27. ❏ You can specify read queries to use replicas of specific engines with TiDB as shown below: Config file: [isolation-read] engines = ["tikv", "tidb", "tiflash"] SESSION: set SESSION tidb_isolation_read_engines = "engine list separated by commas"; Engine Isolation
- 28. ❏ You can force the TiDB to use TiFlash replica as below with manual hint in query. select /*+ read_from_storage(tiflash[table_name]) */ ... from table_name; Manual Hint
- 29. TiFlash Modes
- 30. ❏ This mode enables the execution of queries in parallel across multiple nodes. ❏ TiDB automatically determines when to select MPP based on the optimizer’s cost estimation. tidb_allow_mpp ,tidb_enforce_mpp- Control variables MPP Mode
- 31. ❏ With FastScan, TiFlash provides more efficient query performance but sacrifices the data consistency. ❏ This mode is disabled by default. ❏ Query results might include old data of a table. ❏ Enable / Disable using tiflash_fastscan FastScan Mode
- 32. Any Questions?
- 33. Thank You