From the course: Microsoft Azure Data Engineer Associate (DP-203) Cert Prep by Microsoft Press
Join today to access over 24,700 courses taught by industry experts.
Troubleshoot failed Spark jobs - Azure Tutorial
From the course: Microsoft Azure Data Engineer Associate (DP-203) Cert Prep by Microsoft Press
Troubleshoot failed Spark jobs
“
- [Speaker] And here, this is just a callback to what we did a couple lessons ago, that if you are using HD Insight, you've got the Apache Ambari dashboard to rely on and Azure Databricks, you can load the Spark UI and take a look in the web console and in, it's not on the slide but in Synapse you've got Log Analytics is your main go-to.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.
Watch this course anytime, anywhere.
Get started with a free trial today.
Contents
-
-
(Locked)
Learning objectives36s
-
(Locked)
Design a partition strategy for files3m 58s
-
(Locked)
Design a partition strategy for analytical workloads1m 57s
-
(Locked)
Design a partition strategy for efficiency and performance2m 56s
-
(Locked)
Design a partition strategy for Azure Synapse Analytics2m 2s
-
(Locked)
Identify when partitioning is needed in Azure Data Lake Storage Gen223m 46s
-
(Locked)
-
-
(Locked)
Learning objectives40s
-
(Locked)
Design star schemas3m 13s
-
(Locked)
Design slowly changing dimensions2m 23s
-
(Locked)
Design a dimensional hierarchy28s
-
(Locked)
Design a solution for temporal data1m 16s
-
(Locked)
Design for incremental loading51s
-
(Locked)
Design analytical stores2m 18s
-
(Locked)
Design metastores in Azure Synapse Analytics and Azure Databricks24m 10s
-
(Locked)
-
-
(Locked)
Learning objectives29s
-
(Locked)
Implement compression1m 42s
-
(Locked)
Implement partitioning42s
-
(Locked)
Implement sharding18s
-
(Locked)
Implement different table geometries with Azure Synapse Analytics pools2m 3s
-
(Locked)
Implement data redundancy3m 55s
-
(Locked)
Implement distributions27s
-
(Locked)
Implement data archiving13m 19s
-
(Locked)
-
-
(Locked)
Learning objectives31s
-
(Locked)
Transform data by using Apache Spark2m 29s
-
(Locked)
Transform data by using Transact-SQL1m 5s
-
(Locked)
Transform data by using Data Factory1m 35s
-
(Locked)
Transform data by using Azure Synapse pipelines1m 24s
-
(Locked)
Transform data by using Stream Analytics19m 50s
-
(Locked)
-
-
(Locked)
Learning objectives46s
-
(Locked)
Develop batch processing solutions by using Data Factory, Data Lake, Spark, Azure Synapse pipelines, PolyBase, and Azure Databricks1m 14s
-
(Locked)
Create data pipelines2m
-
(Locked)
Design and implement incremental data loads1m 20s
-
(Locked)
Design and develop slowly changing dimensions36s
-
(Locked)
Handle security and compliance requirements2m 35s
-
(Locked)
Scale resources21m 11s
-
(Locked)
-
-
(Locked)
Learning objectives37s
-
(Locked)
Configure the batch size2m 26s
-
(Locked)
Design and create tests for data pipelines3m 31s
-
(Locked)
Integrate Jupyter and Python Notebooks into a data pipeline1m 15s
-
(Locked)
Handle duplicate data23s
-
(Locked)
Handle missing data36s
-
(Locked)
Handle late-arriving data7m 39s
-
(Locked)
-
-
(Locked)
Learning objectives39s
-
(Locked)
Upsert data1m 52s
-
(Locked)
Regress to a previous state2m 14s
-
(Locked)
Design and configure exception handling1m 44s
-
(Locked)
Configure batch retention1m 2s
-
(Locked)
Revisit batch processing solution design1m 16s
-
(Locked)
Debug Spark jobs by using the Spark UI24m 55s
-
(Locked)
-
-
(Locked)
Learning objective46s
-
(Locked)
Develop a stream processing solution by using Stream Analytics, Azure Databricks, and Azure Event Hubs1m 53s
-
(Locked)
Process data by using Spark structured streaming1m 52s
-
(Locked)
Monitor for performance and functional regressions1m 34s
-
(Locked)
Design and create windowed aggregates1m 50s
-
(Locked)
Handle schema drift21m 50s
-
(Locked)
-
-
(Locked)
Learning objectives47s
-
(Locked)
Process time series data1m 53s
-
(Locked)
Process across partitions2m 9s
-
(Locked)
Process within one partition1m
-
(Locked)
Configure checkpoints and watermarking during processing1m 2s
-
(Locked)
Scale resources1m 49s
-
(Locked)
Design and create tests for data pipelines1m 20s
-
(Locked)
Optimize pipelines for analytical or transactional purposes15m 26s
-
(Locked)
-
-
(Locked)
Learning objectives34s
-
(Locked)
Trigger batches1m 53s
-
(Locked)
Handle failed batch loads1m 50s
-
(Locked)
Validate batch loads45s
-
(Locked)
Manage data pipelines in Data Factory and Synapse pipelines1m 16s
-
(Locked)
Schedule data pipelines in Data Factory and Synapse pipelines22s
-
(Locked)
Implement version control for pipeline artifacts56s
-
(Locked)
Manage Spark jobs in a pipeline12m 1s
-
(Locked)
-
-
(Locked)
Learning objectives40s
-
(Locked)
Implement data masking1m 49s
-
(Locked)
Encrypt data at rest and in motion1m 40s
-
(Locked)
Implement row-level and column-level security18s
-
(Locked)
Implement Azure RBAC1m 31s
-
(Locked)
Implement POSIX-like ACLs for Data Lake Storage Gen254s
-
(Locked)
Implement a data retention policy21s
-
(Locked)
Implement a data auditing strategy15m 28s
-
(Locked)
-
-
(Locked)
Learning objectives40s
-
(Locked)
Manage identities, keys, and secrets across different data platforms2m 20s
-
(Locked)
Implement secure endpoints: Private and public1m 38s
-
(Locked)
Implement resource tokens in Azure Databricks1m 34s
-
(Locked)
Load a DataFrame with sensitive information54s
-
(Locked)
Write encrypted data to tables or Parquet files34s
-
(Locked)
Manage sensitive information16m 51s
-
(Locked)
-
-
(Locked)
Learning objectives30s
-
(Locked)
Implement logging used by Azure Monitor1m 9s
-
(Locked)
Configure monitoring services1m 15s
-
(Locked)
Measure performance of data movement57s
-
(Locked)
Monitor and update statistics about data across a system1m 12s
-
(Locked)
Monitor data pipeline performance13s
-
(Locked)
Measure query performance10m 15s
-
(Locked)
-
-
(Locked)
Learning objectives32s
-
(Locked)
Compact small files1m 9s
-
(Locked)
Rewrite user-defined functions (UDFs)1m 26s
-
(Locked)
Handle skew in data1m 50s
-
(Locked)
Handle data spill1m 29s
-
(Locked)
Tune shuffle partitions1m 7s
-
(Locked)
Find shuffling in a pipeline21s
-
(Locked)
Optimize resource management12m
-
(Locked)
-
-
(Locked)
Learning objectives31s
-
(Locked)
Tune queries by using indexers1m 53s
-
(Locked)
Tune queries by using cache55s
-
(Locked)
Optimize pipelines for analytical or transactional purposes1m 38s
-
(Locked)
Optimize pipeline for descriptive versus analytical workloads1m 28s
-
(Locked)
Troubleshoot failed Spark jobs30s
-
(Locked)
Troubleshoot failed pipeline runs1m 14s
-
(Locked)