Building near real-time HTAP solutions using Synapse Link for Azure Cosmos DB

Editor's Notes

• #3: What are the requirements of modern applications: Apps & services generate growing volumes of operational data Users expect these apps & services to be milli-second response times and always-on Businesses have to serve users who are globally distributed Hence businesses have chosen Azure Cosmos DB as their operational database of choice courtesy ..
• #4: Generate insights over growing volumes of operational data at scale
• #5: Data volume =>OLTP performance impact
• #6: To ensure that there is no performance impact on your applications or OLTP workloads, today data engineers Create pipelines for data movement from Cosmos DB to ADLS. Operationalize the pipelines Monitor and manage the pipelines. Then the same needs to be done where the data lands – Ensure the data is in the right format , for example., columnar formats for analytics workloads Create and manage the storage account where the data is moved Only after these can you start to use the flexibility of your analytic workloads for ex. with Spark to transform or enrich and serve with SQL. But your analytical workloads are not running against the latest state of operational data due periodic ingest of data.
• #7: We’re excited to announce Azure Synapse Link for Azure Cosmos DB as a cloud-native hybrid transactional and analytical processing (HTAP) capability that enables you to analytics over operational data in Azure Cosmos DB with no ETL and no performance impact on your transactional workloads. Azure Synapse Link finally breaks down the barrier that has long existed between the OLTP and OLAP systems. You can now generate near real-time analytics over your operational data in Cosmos DB at scale…. With a ‘Single click’
• #8: So what’s the magic powering Synapse Link? First animation: So far, your operational data in Cosmos DB is internally stored in a row-oriented ‘transactional store’. This store is optimized for transactional reads/writes & operational queries. Second animation: Now, we’re excited to bring you a fully-managed native ‘analytical store’ within Azure Cosmos DB container. Analytical store is a fully-isolated column-oriented store, optimized for typical analytical queries over large volumes of data. Your inserts, updates, deletes to operational data are automatically synced from transactional store to analytical store in near real-time within minutes. This ‘auto-sync’ capability does not consume RUs allocated for your operational workloads. Third animation: Now with Synapse Link, you can connect Cosmos DB Analytical store with Synapse Analytics with a ‘single click’. This allows you to interchangeably query operational data in Cosmos DB using Spark & SQL runtimes. This enables you to build machine learning pipelines, BI dashboards and run big-data analytics in near real-time on your operational data. Talking points (4 mins): Introduce Synapse Link as cloud-native HTAP capability to be able to run near real time analytics over operational data with no ETL & no perf impact on transactional workloads Synapse Link for Cosmos DB consists of two main components – first is built-in isolated column store called ‘Analytical Store’ and second native integration with the Spark & SQL runtimes of Azure Synapse Analytics Talk about Analytical Store: Auto-sync as native capability to handle inserts, updates, deletes to operational data are automatically synced from transactional store to analytical store in near real-time. In public preview, expected latency is 2 minutes with further optimizations going on to reduce this to 10s of seconds Auto-sync transforms your operational data from row-format in the transactional store into a column-format in the analytical store, optimized for complex analytical queries and large scans No RU impact Support for global distribution for analytics from local copy Talk about Synapse Runtime support: Spark SQL Serverless
• #9: DEMO