Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturdays Guatemala

Editor's Notes

• #6: CONTEXT SOURCES INFRASTRUCTURE DESTINATIONS
• #14: DEMO: LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer (ADX). The utility can pull source data from a local folder or from an Azure blob storage container. LightIngest is most useful when you want to ingest a large amount of data, because there is no time constraint on ingestion duration. When historical data is loaded from existing system to ADX, all records receive the same the ingestion date. Now, the -creationTimePattern argument allows users to partition the data by creation time, not ingestion time. The -creationTimePattern argument extracts the CreationTime property from the file or blob path; the pattern doesn't need to reflect the entire item path, just the section enclosing the timestamp you want.
• #15: LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer (ADX). The utility can pull source data from a local folder or from an Azure blob storage container. LightIngest is most useful when you want to ingest a large amount of data, because there is no time constraint on ingestion duration. When historical data is loaded from existing system to ADX, all records receive the same the ingestion date. Now, the -creationTimePattern argument allows users to partition the data by creation time, not ingestion time. The -creationTimePattern argument extracts the CreationTime property from the file or blob path; the pattern doesn't need to reflect the entire item path, just the section enclosing the timestamp you want.
• #16: LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer (ADX). The utility can pull source data from a local folder or from an Azure blob storage container. LightIngest is most useful when you want to ingest a large amount of data, because there is no time constraint on ingestion duration. When historical data is loaded from existing system to ADX, all records receive the same the ingestion date. Now, the -creationTimePattern argument allows users to partition the data by creation time, not ingestion time. The -creationTimePattern argument extracts the CreationTime property from the file or blob path; the pattern doesn't need to reflect the entire item path, just the section enclosing the timestamp you want.
• #21: EXAMPLE of QUEUED INGESTION https://docs.microsoft.com/en-us/azure/kusto/api/netfx/kusto-ingest-queued-ingest-sample Example of INLINE INGESTION https://docs.microsoft.com/it-it/azure/kusto/management/data-ingestion/ingest-inline
• #24: https://notebooks.azure.com/riccardo-zamana/projects/azuresaturday2019
• #26: Se usi HAS usi indici, contains non usa indici Spiega come fai i between
• #32: FARE UN PO DI PROVE FARE la distinct per introdurre la SUMMERIZE
• #33: .create-or-alter function with (folder = "AzureSaturday2019", docstring = "Func1", skipvalidation = "true") MyFunction1(i:long) {TBL_LAB0X | limit 100 | where minimum_nights > i} MyFunction1(80); explain SELECT name, minimum_nights from TBL_LAB0X .create-or-alter function with (folder = "AzureSaturday2019", docstring = "Func1", skipvalidation = "true") MyFunction1(i:long) {TBL_LAB0X | project name, minimum_nights | limit 100 | where minimum_nights > i | render columnchart} MyFunction1(80);
• #34: T | summarize Hits=count() by bin(Duration, 1s) 
• #52: Kusto is built to support tables with a huge number of records (rows) and large amounts of data. To handle such large tables, each table's data is divided into smaller "tablets" called data shards or extents (the two terms are synonymous). The union of all the table's extents holds the table's data. Individual extents are kept smaller than a single node's capacity, and the extents are spread over the cluster's nodes, achieving scale-out. An extent is a like a type of mini-table. It contains data and metadata and information such as its creation time and optional tags that are associated with its data. Additionally, the extent usually holds information that lets Kusto query the data efficiently. For example, an index for each column of data in the extent, and an encoding dictionary, if column data is encoded. As a result, the table's data is the union of all the data in the table's extents. Extents are immutable and can never be modified. It may only be queried, reassigned to a different node, or dropped out of the table. Data modification happens by creating one or more new extents and transactionally swapping old extents with new ones. Extents hold a collection of records that are physically arranged in columns. This technique is called columnar store. It enables efficient encoding and compression of the data, because different values from the same column often "resemble" each other. It also makes querying large spans of data more efficient, because only the columns used by the query need to be loaded. Internally, each column of data in the extent is subdivided into segments, and the segments into blocks. This division isn't observable to queries, and lets Kusto optimize column compression and indexing. To maintain query efficiency, smaller extents are merged into larger extents. The merge is done automatically, as a background process, according to the configured merge policy and sharding policy. Merging extents reduces the management overhead of having a large number of extents to track. More importantly, it allows Kusto to optimize its indexes and improve compression. Extent merging stops once an extent reaches certain limits, such as size, since beyond a certain point, merging reduces rather than increases efficiency. When a Data partitioning policy is defined on a table, extents go through another background process after they're created (post-ingestion). This process reingests the data from the source extents and creates homogeneous extents, in which the values of the column that is the table's partition key all belong to the same partition. If the policy includes a hash partition key, all homogeneous extents that belong to the same partition will be assigned to the same data node in the cluster.
• #58: Azure Data Share creates a symbolic link between two ADX cluster. Sharing occurs in near-real-time (no data pipeline) ADX Decouples the storage and compute Allows customers to run multiple compute (read-only) instances on the same underlying storage You can attach a database as a follower database, which is a read-only database on a remote cluster. You can share the data at the database level or at the cluster level. The cluster sharing the database is the leader cluster and the cluster receiving the share is the follower cluster. A follower cluster can follow one or more leader cluster databases. The follower cluster periodically synchronizes to check for changes. The queries running on the follower cluster use local cache and don't use the resources of the leader cluster.
• #60: FAI ESEMPIO DI EXPORT
• #70: Fai vedere NOTEBOOKS
• #73: PROVA SU VSC CTRL+P => kuskus Poi: cluster(adxclu001).database('db001').table('TBL_LAB01') | count