Discovering Insights - Azure Data Explorer Unleashed
0 likes9 views
"Discovering Insights: Azure Data Explorer Unleashed," is about Azure Data Explorer (ADX), a big data analytics cloud platform optimized for interactive, ad-hoc queries. It covers various aspects of ADX, including its infrastructure, Kusto Query Language (KQL), data ingestion methods, data management, cross-queries, data sharing, and advanced analytics. The presentation also includes demos and tips for query optimization, highlighting the platform's capabilities in handling large-scale data analytics efficiently.
Discovering Insights - Azure Data Explorer Unleashed
- 1. Discovering Insights: Azure Data Explorer Unleashed Callon Campbell Microsoft MVP | Azure @flying_maverick
- 2. Callon Campbell Azure Architect | Developer Adastra Microsoft MVP | Azure (2018-2024) 25 years enterprise development with Microsoft technologies – .NET (C#), Azure, ASP.NET, Desktop, SQL, and Mobile Passionate about serverless and cloud-native application development, with focus on app migration and modernization, app integration and data analytics Blogging at https://TheFlyingMaverick.com, and on @flying_maverick Speaker at community events and meetups Organizer of “Canada’s Technology Triangle .NET User Group” in Kitchener, Ontario About me
- 3. Agenda What is Azure Data Explorer Infrastructure Kusto Query Language Demos Q&A
- 4. Azure Data Explorer (ADX) Any append- only stream of records Relational query model: Filter, aggregate, join, calculated columns, … Fully- managed Rapid iterations to explore the data High volume High velocity High variance (structured, semi- structured, free-text) PaaS, Vanilla, Database A big data analytics cloud platform optimized for interactive, ad-hoc queries Purposely built
- 5. Query Tools Orchestration Notebook Connectivity Connectors TDS, JDBC, ODBC API and Client Libraries REST API Export Visualization ADX Dashboard Data Ingestion ecosystem Managed pipelines SDKs Tools One Click Ingestion LightIngest Connection & plugins Data Formats Data Consumption ecosystem Batching Streaming Azure Data Explorer Architecture
- 6. Batching vs streaming ingestion Batching • Optimized for high ingestion throughput • Preferred method and most performant • Data is batched according to properties • Set ingestion batching policy on databases or tables • Default max batching value is 5 minutes, 1000 items or total of 1 GB • 4 GB data size limit for a batch ingestion command Streaming • Ongoing data ingestion from a streaming source • Near real-time latency for small sets of data per table • Initially ingested to row store • Then moved to column store extents • Steaming can be done using ADX client library or supported pipelines NOTE: The recommendation is to ingest files between 100 MB and 1 GB.
- 7. Tools Connection & plugins Managed pipelines SDKs Data Management and Ingestion Data Explorer Batching Ingestion Management Data Management Connections Data Explorer Engine 1 One Click Ingestion LightIngest
- 8. Cross Queries Between Data Explorer databases and clusters 1 Query SQL pool from Data Explorer Query Azure Monitor from Data Explorer Query Cosmos DB from Data Explorer Query Data Lake from Data Explorer
- 9. Data Sharing Share data within the company • Load balancing • Data-as-a-service • Hub and Spoke model • Chargeback No Maintenance • In-Place Data Sharing • No data pipeline to maintain • Near real-time updates 1 Leader Ingest Query DB1 DB2 DB3 Symbolic link Azure Blob Storage Compute R/W R/W R/W Follower DB3 DB4 Azure Blob Storage Compute R/W Query R
- 10. When to use ADX
- 12. Using ADX Commands to Manage Tables .create table .create-merge table .drop table .alter column .rename column
- 13. What is a Kusto query? • A Kusto query is a read-only request to process data and return results. • Has one or more query statements and returns data in a tabular or graph format. • Statements are sequenced by a pipe (|). Data flows, or is piped, from one operator to the next. • The data is filtered/manipulated at each step and then fed into the following step. It's like a funnel, where you start out with an entire data table. • Each time the data passes through another operator, it's filtered,
- 14. KQL Concepts Relational operators (filters, union, joins, aggregations, …) Each operator consumes tabular input and produces tabular output Can be combined with ‘|’ (pipe). Ease to write, read, change
- 15. KQL Basic operators for data exploration … | count • Counts records in input table (e.g. T) … | take 10 • Get few records - convenient to start get familiar with the data • No actual order ensured … | where Timestamp > ago(1) and UserId == ‘abdcdef’ • Filtering on a specific fields … | project Col1, Col2, … • Choose some columns (great if input table has dozens of coluns) …| extend NewCol1=Col1+Col2 • Introduces new calculated columns … | render timechart • Plot the data (in KE and KWE) while exploring
- 16. SQL to KQL Try the ‘EXPLAIN’ operator as follows Use SQL to KQL Cheat Sheet
- 17. Query Optimization Tips Use Materialized Views Use Time Filters First! Avoid filtering on calculated columns Use case-insensitive operators when possible tinyurl.com/ ADXQueryBestPractices
- 18. Schema is Relational, Lightweight, Dynamic Databases Authorization boundary Transaction boundary But not query boundary! Supporting cross-database and cross-cluster queries Tables Rectangular Columns Supported types: boolean, integer, real, decimal, dates, timespan, string, dynamic (JSON) Stored functions (views) Materialized views Schema
- 19. Stored Functions Essence: Reusable function, defined and used in a Database scope View is parameter-less function Schema: scalar or tabular Special powers: Can override table with the same name. Safe and Secure: Control commands are forbidden. Applications: Sharing queries between users/applications Abstracting complex logic from other applications SQL-compat tools connecting to Kusto via Views to run high perf queries
- 20. Update Policy (inner ETL/Trigger) Essence: Triggered ingestion into another table. Semantics: Attached to ‘target’ table and points on ‘source’ table. Transformation is an arbitrary Kusto query. Special powers: Source table is scoped to the newly ingested data only. Applications: Transform data schema (lightweight ETL) De-multiplexing data stream into several tables Use target tables with longer retention Reduce duplications
- 23. Demos
- 24. Did you know The Azure Data Explorer (ADX), formerly known by its internal code name “Kusto”, has an interesting origin story. Back in 2014, the team needed a name that captured the essence of their mission: exploring vast oceans of data. Inspired by the legendary oceanographer Jacques Cousteau, they chose the internal code name "Kusto". Just like Cousteau explored the depths of the ocean, the Kusto project aimed to tackle the challenges of fast and scalable log and telemetry analytics.
Editor's Notes
- #3: Tonight I’ll be showing you have you can get new insights from your data with Azure Data Explorer. The Azure Data Explorer (ADX), formerly known by its internal code name “Kusto”, has an interesting origin story. Back in 2014, the team needed a name that captured the essence of their mission: exploring vast oceans of data. Inspired by the legendary oceanographer Jacques Cousteau, they chose the internal code name "Kusto"1. Just like Cousteau explored the depths of the ocean, the Kusto project aimed to tackle the challenges of fast and scalable log and telemetry analytics.
- #4: Big Data – Data that has High Volume, Velocity, variety – Structured, Semi-Structured, free text data. Append only DB. Analytics – Can have relational query model with tables, columns, joins, filters and aggregates. Cloud Platform – Platform as as Service where we don’t have to take care of infrastructure. With little configuration of Scale we can provision cluster, DB and start working with it. We can take care of scaling using autoscale or you can use manual scale. Optimized – any data that is timestamped can fit right into ADX. It is optimized for timeseries data. All columns are compressed and indexed. Interactive , ad hoc queries – ADX has native Kusto Query Language for gaining data insights. Has some similarity to SQL in terms of operators/joins etc Use KQL to – finding patterns, trends, forecasting, anomalies etc. Create near realtime dashboards
- #5: Azure Data Explorer is a fully managed, high-performance, big data analytics platform that makes it easy to analyze high volumes of data in near real time. The Azure Data Explorer toolbox gives you an end-to-end solution for data ingestion, query, visualization, and management. Big Data – Data that has High Volume, Velocity, variety – Structured, Semi-Structured, free text data. Append only DB. Analytics – Can have relational query model with tables, columns, joins, filters and aggregates. Cloud Platform – Platform as as Service where we don’t have to take care of infrastructure. With little configuration of Scale we can provision cluster, DB and start working with it. We can take care of scaling using autoscale or you can use manual scale. Optimized – any data that is timestamped can fit right into ADX. It is optimized for timeseries data. All columns are compressed and indexed. Interactive , ad hoc queries – ADX has native Kusto Query Language for gaining data insights. Has some similarity to SQL in terms of operators/joins etc Use KQL to – finding patterns, trends, forecasting, anomalies etc. Create near realtime dashboards
- #6: Azure Data Explorer Architecture Overview Time series database Azure PaaS cluster Data storage benefits from compression Ingest only, mostly querying, not data manipulation
- #8: https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-overview#batching-vs-streaming-ingestion Small batches ingested get merged and optimized behind the scenes.
- #9: You can ingest data in Data Explorer in many ways. Using our SDKs. Use managed pipelines where Data Explorer will automatically read the data from IoT Hub, EventHub or Storage\Data Lake, Cosmos DB, etc. Use any of the 1st party connectors and agents to push data from Kafka, Spark, Logstach, PowerApp, FluentBit, and Telegraph agents. Use Azure Data Factory\Synapse Pipelines to bring data from wide variety of sources. Data Explorer support automated batching of the data for efficient ingestion and can ingest up 200 MB/sec csv files per 16 cores compute. It support wide variety of formats such as JSON, Parquet, Orc, Avro and more.
- #10: You may have scenarios where you may want to create multiple Data Explorer clusters for storing different types of data or host different pool is different regions or different divisions in your company many have their own Data Explore clusters. In all these scenarios cross database\cluster enable you to easily cross query between database, between ADX clusters in same region or other regions. Data Explorer is integrated with rest of the Azure ecosystem, in cases where you don’t want to ingest the data in Data Explorer – you can query it easily from SQL, Azure Monitor, Cosmos DB, and Data Lake.
- #11: Thanks to the separation of storage and compute in Data Explorer pool. You can have one Data Explorer pool for ingesting and querying the data and share the databases with multiple Data Explorer pools. This enables you to load balance, offer data-as-a-service, build Hub and Spoke model, or chargeback to you internal teams. Its all automated and real-time caching, you don’t need to maintain any pipeline.
- #12: Azure Data Explorer is ideal for enabling interactive analytics capabilities over high velocity, diverse raw data. Use the following decision tree to help you decide if Azure Data Explorer is right for you:
- #15: https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/data-ingestion/ingest-from-storage https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/data-ingestion/ingest-from-query https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/data-ingestion/streaming-ingestion-schema-changes https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/data-ingestion/ingest-inline
- #16: Very fast Ideal for time-series data Built-in Functions Read-Only
- #23: Update policy is like an internal ETL. It can help you manipulate or enrich the data as it gets ingested into the source table (e.g. extracting JSON into separate columns, creating a new calculated column, joining the new records with a static dimension table that is already in your database, etc). For these cases, using an update policy is a very common and powerful practice. Each time records get ingested into the source table, the update policy's query (which we'll define in the update policy) will run on them (and only on newly ingested records - other existing records in the source table aren’t visible to the update policy when it runs), and the results of the query will be appended to the target table. When you trigger an update policy with a command that adds data to a source table, data also appends to a target table. The target table can have a different schema, retention policy, and other policies from the source table. For example, a high-rate trace source table can contain data formatted as a free-text column. The target table can include specific trace lines, with a well-structured schema generated from a transformation of the source table's free-text data using the parse operator. Performance impact Update policies can affect cluster performance, and ingestion for data extents is multiplied by the number of target tables. It's important to optimize the policy-related query. You can test an update policy's performance impact by invoking the policy on already-existing extents, before creating or altering the policy, or on the function used with the query.