SlideShare a Scribd company logo

Table Partitioning in SQL Server: A Magic Solution for Better Performance? (Pragmatic Works)

Cathrine Wilhelmsen, profile picture
Cathrine Wilhelmsen

The document provides an introduction to table partitioning in SQL Server. It explains what a partitioned table is, the key components like partition key, partition function, and partition scheme. It discusses why table partitioning is used, including benefits like partition elimination to improve query performance, ability to backup and restore partitions individually, and perform maintenance tasks like indexing and statistics updates on partitions. It also covers techniques like partition switching which allows fast loading and archiving of data without physically moving it. The document uses examples and diagrams to illustrate these concepts and components of table partitioning.

Data & Analytics
Related topics:
Women in Technology
1 of 58
Downloaded 236 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Most read
15
16
17
18
19
20
21
22
23
24
25
Most read
26
27
28
29
30
31
Most read
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
Table Partitioning in SQL Server A Magic Solution for Better Performance?
Cathrine Wilhelmsen @cathrinew cathrinewilhelmsen.net Data Warehouse Architect Business Intelligence Developer
You? New to table partitioning "Everything is slow" Today? Basic Introduction What, Why & How
A Magic Solution for Better Performance? Spoiler Alert!
A Magic Solution for Better Performance? Implementing table partitioning is not a trivial task and can actually cause worse performance...
A Magic Solution for Better Performance? ...but don't worry, I made plenty of mistakes so you can avoid them 
Enterprise Edition only
What? Partition Key Partition Function Partition Scheme Why? Backup & Restore Maintenance Load & Archive How? Partition Elimination Switch, Split & Merge Sliding Windows
Table Partitioning Basics
What is a partitioned table? 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Data is physically stored in groups of rows called partitions Each partition can be accessed and maintained separately Partitioning is not visible to users, it behaves like one logical table
Partition Key 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Data is partitioned based on a single column, the Partition Key The Partition Key should always be used as a filter in queries This ensures Partition Elimination: only relevant partitions are accessed
Partition Function 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... The Partition Function defines how to partition the data It specifies boundary values, the points between two partitions It specifies if the boundary value belongs to its left (upper) partition or its right (lower) partition
Partition Function: Range Left and Range Right 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ...
Partition Function: Range Left and Range Right Range Left means the boundary value is the last value in the left partition CREATE PARTITION FUNCTION pfLeft (INT) AS RANGE LEFT FOR VALUES (20,30,40); Range Right means the boundary value is the first value in the right partition CREATE PARTITION FUNCTION pfRight (INT) AS RANGE RIGHT FOR VALUES (20,30,40); ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40
Partition Scheme 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... The Partition Scheme maps logical partitions to physical filegroups Filegroups? Data files on one or more disks Backed up and restored individually Can be Read-Only
Map all partitions to one filegroup FILEGROUP 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ...
Map partitions to separate filegroups FILEGROUP1 (Read-Only) FILEGROUP2 (Read-Only) 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... FILEGROUP3 FILEGROUP4
How are partitions mapped to filegroups? ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40 The partition function specified the boundary values and partitions:
How are partitions mapped to filegroups? CREATE PARTITION SCHEME psLeft AS PARTITION pfLeft TO (FG1, FG2, FG3, FG4); CREATE PARTITION SCHEME psRight AS PARTITION pfRight TO (FG1, FG2, FG3, FG4); ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40 The partition scheme uses the partition function...
How are partitions mapped to filegroups? CREATE PARTITION SCHEME psLeft AS PARTITION pfLeft TO (FG1, FG2, FG3, FG4); CREATE PARTITION SCHEME psRight AS PARTITION pfRight TO (FG1, FG2, FG3, FG4); ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40 FG1 FG2 FG4FG3 FG1 FG2 FG4FG3 ...to map each partition to filegroups:
Filegroups Partition Scheme Partitioned Table 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... A partitioned table is created on a partition scheme instead of directly on a filegroup The partition scheme uses the partition key to store rows in the correct partition and filegroup based on the definition specified in the partition function
Table Partitioning Basics
Why Table Partitioning?
Partition Elimination SELECT COUNT(*) FROM Table WHERE Year = 2012; SELECT COUNT(*) FROM Table;
Partition Elimination SELECT COUNT(*) FROM Table WHERE Year = 2012; SELECT COUNT(*) FROM Table;
Backup & Restore Partitions Filegroups can be backed up and restored individually If each partition is mapped to a separate filegroup, partitions with the most critical data can be restored first
Backup & Restore Partitions Filegroups can be read-only If each partition is mapped to a separate filegroup, partitions with historical, unchanging data can be excluded from regular backups
Index Maintenance per Partition Rebuild and reorganize indexes per partition Rebuild indexes online per partition was introduced in SQL Server 2014 Set data compression per partition ALTER INDEX IndexName ON TableName REBUILD PARTITION = 2 WITH (ONLINE = ON);
Statistics Maintenance per Partition Update statistics on specific partitions instead of scanning and updating the whole table UPDATE STATISTICS TableName (StatisticsName) WITH RESAMPLE ON PARTITIONS (3,5); CREATE STATISTICS StatisticsName ON TableName (ColumnName) WITH INCREMENTAL = ON; Incremental Statistics was introduced in SQL Server 2014
Load & Archive: Partition Switching Partitions can be switched between tables, called switching in or switching out Partition switching is a metadata operation that updates the location of the data, no data is physically moved Extremely fast compared to inserting into or deleting from a large table 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... ALTER TABLE Table1 SWITCH PARTITION 5 TO Table2 PARTITION 5; SELECT $PARTITION.pf(2014);
Load & Archive: Switch out 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Called switch out when you move data out of a table (archive)
Load & Archive: Switch out 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... Called switch out when you move data out of a table (archive)
Load & Archive: Switch in 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2016-01-01 ... ... 2016-12-31 ... ... Called switch in when you move data into a table (load)
Load & Archive: Switch in 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2016-01-01 ... ... 2016-12-31 ... ... Called switch in when you move data into a table (load)
Sliding Windows The Sliding Windows technique automates data loading, data archiving and partition management It keeps a certain number of partitions by continuously switching out the oldest data and switching in new data 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ...
Sliding Windows: Split & Merge 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Partitions are not actually added or removed, they are split or merged Be careful! Splitting and merging partitions can cause data movement!!
Sliding Windows: Split & Merge 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Split one partition in two by adding a new boundary value ALTER PARTITION FUNCTION pf () SPLIT RANGE ('2013-06-01'); Data movement will occur if there is data on both sides of the new boundary value!
Sliding Windows: Split & Merge 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Merge two partitions to one by removing a boundary value ALTER PARTITION FUNCTION pf () MERGE RANGE ('2014-01-01'); Data movement will occur if there is data on both sides of the old boundary value!
Sliding Windows Steps 1. Add new filegroup and file 2. Create switch out (archive) table 3. Create switch in (load) table and load it with new data 4. Split to add new partition 5. Switch in new partition 6. Switch out old partition 7. Merge to remove old partition 8. Delete switch out and switch in tables 9. Delete old file and filegroup
Sliding Windows Demo
Alternative Sliding Windows Steps 1. Add new filegroup and file 2. Create switch out (archive) table 3. Switch out old partition 4. Merge to remove old partition 5. Create switch in (load) table and load it with new data 6. Split to add new partition 7. Switch in new partition 8. Delete switch out and switch in tables 9. Delete old file and filegroup
Why Table Partitioning?
Case Study: What Went Wrong?
Background: Financial Data Warehouse Periodic Snapshot Fact Tables Daily, Weekly and Monthly Balances SQL Server, DB2, Oracle, .csv, .txt Loaded at different times during the day 1 2 3 4 5 6 7 Pension Insurance Bank Fund
First version Daily fact table: Keep 1 day per source 1 2 Pension 6 7 Bank Fund Monthly fact table: Keep 36 months per source 3 4 5 Insurance
Second version Daily fact table: Keep 1 day per source 1 2 Pension Monthly / Weekly fact table: Keep 36 months for some sources Keep 106 weeks for some sources 3 4 5 Insurance ! 6 7 Bank Fund
Third version Daily fact table: Keep 1 day per source 1 2 Pension Monthly / Weekly / Daily fact table: Keep 36 months for some sources Keep 106 weeks for some sources Keep 7 days for some sources 3 4 5 6 7 Insurance Bank Fund !!
"Everything is slow"
"Let's try partitioning"
Daily fact table partitioning 1 ... ... 2 ... ... 3 ... ... 4 ... ... 5 ... ... 6 ... ... 7 ... ... Partition Key: source number One partition per source Helped with deadlock issues Easy to switch data in and out
Monthly / Weekly / Daily fact table partitioning 12011 ... ... 12012 ... ... 12013 ... ... 12014 ... ... 19990 ... ... 19991 ... ... 19992 ... ... Partition Key: Source number + Period type Period type: YYYY for monthly data 9991 for weekly data 9992 for daily data Helped with deadlock issues, but…
Monthly / Weekly / Daily fact table partitioning 12011 ... ... 12012 ... ... 12013 ... ... 12014 ... ... 19990 ... ... 19991 ... ... 19992 ... ... Partition key was difficult to remember and was not used in queries Data had to be inserted into and deleted from partitions with existing data, instead of switching partitions in and out
What went wrong? "The usual suspects": Conflicting priorities, changing requirements and high pressure to deliver on time Fact tables were expanded, not changed, when new sources and requirements were implemented Partitioning was implemented on fact tables with multiple grains, instead of correcting the fact tables first
What did we learn?
test, test, test, test
References and Resources: Table Partitioning Brent Ozar Unlimited: SQL Server Table Partitioning Resources brentozar.com/sql/table-partitioning-resources/ Bradley Ball: Partitioning in SQL Server 2012 pragmaticworks.com/Training/FreeTraining/ViewWebinar/WebinarID/541
References and Resources: Further Reading Partial Backups in SQL Server 2014 msdn.microsoft.com/en-us/library/ms191539.aspx Piecemeal Restores in SQL Server 2014 msdn.microsoft.com/en-us/library/ms177425.aspx Benjamin Nevarez: SQL Server 2014 Incremental Statistics benjaminnevarez.com/2015/02/2014-incremental-statistics/
Thank you! @cathrinew cathrinewilhelmsen.net no.linkedin.com/in/cathrinewilhelmsen contact@cathrinewilhelmsen.net

More Related Content

ODP
Partitioning
Reema Gajjar
 
ODP
Ms sql-server
Md.Mojibul Hoque
 
PDF
Always on in sql server 2017
Gianluca Hotz
 
PPSX
Oracle Table Partitioning - Introduction
MyOnlineITCourses
 
PPTX
T shaped people
Syed Wajahat Iftikhar
 
PPTX
Azure fundamentals
Raju Kumar
 
KEY
Trees In The Database - Advanced data structures
Lorenzo Alberton
 
PPTX
Oracle Database Sequence
Eryk Budi Pratama
 
Partitioning
Reema Gajjar
 
Ms sql-server
Md.Mojibul Hoque
 
Always on in sql server 2017
Gianluca Hotz
 
Oracle Table Partitioning - Introduction
MyOnlineITCourses
 
T shaped people
Syed Wajahat Iftikhar
 
Azure fundamentals
Raju Kumar
 
Trees In The Database - Advanced data structures
Lorenzo Alberton
 
Oracle Database Sequence
Eryk Budi Pratama
 

What's hot (20)

PDF
Introduction to column oriented databases
ArangoDB Database
 
PDF
Multi-language database design appoaches
SudabaSolaimankhil
 
PPTX
Introduction of ssis
deepakk073
 
PDF
Oracle Performance Tuning Fundamentals
Enkitec
 
PDF
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
PPTX
Oracle sql high performance tuning
Guy Harrison
 
PDF
ORACLE ARCHITECTURE
Manohar Tatwawadi
 
PDF
Partitioning tables and indexing them
Hemant K Chitale
 
PDF
Oracle Database SQL Tuning Concept
Chien Chung Shen
 
PPTX
1. SQL Basics - Introduction
Varun A M
 
PDF
PostgreSQL Tutorial For Beginners | Edureka
Edureka!
 
PPTX
DAX (Data Analysis eXpressions) from Zero to Hero
Microsoft TechNet - Belgium and Luxembourg
 
PPT
1 - Introduction to PL/SQL
rehaniltifat
 
PDF
Indexes and Indexing in Oracle 12c
Oren Nakdimon
 
PPTX
Free Training: How to Build a Lakehouse
Databricks
 
PDF
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Aaron Shilo
 
PDF
Azure Data Factory V2; The Data Flows
Thomas Sykes
 
PPTX
Lect 08 materialized view
Bilal khan
 
PDF
Data Warehouse Concepts | Data Warehouse Tutorial | Data Warehousing | Edureka
Edureka!
 
PDF
Moving to Databricks & Delta
Databricks
 
Introduction to column oriented databases
ArangoDB Database
 
Multi-language database design appoaches
SudabaSolaimankhil
 
Introduction of ssis
deepakk073
 
Oracle Performance Tuning Fundamentals
Enkitec
 
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
Oracle sql high performance tuning
Guy Harrison
 
ORACLE ARCHITECTURE
Manohar Tatwawadi
 
Partitioning tables and indexing them
Hemant K Chitale
 
Oracle Database SQL Tuning Concept
Chien Chung Shen
 
1. SQL Basics - Introduction
Varun A M
 
PostgreSQL Tutorial For Beginners | Edureka
Edureka!
 
DAX (Data Analysis eXpressions) from Zero to Hero
Microsoft TechNet - Belgium and Luxembourg
 
1 - Introduction to PL/SQL
rehaniltifat
 
Indexes and Indexing in Oracle 12c
Oren Nakdimon
 
Free Training: How to Build a Lakehouse
Databricks
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Aaron Shilo
 
Azure Data Factory V2; The Data Flows
Thomas Sykes
 
Lect 08 materialized view
Bilal khan
 
Data Warehouse Concepts | Data Warehouse Tutorial | Data Warehousing | Edureka
Edureka!
 
Moving to Databricks & Delta
Databricks
 
Ad

Viewers also liked (20)

PDF
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Sacrame...
Cathrine Wilhelmsen
 
PDF
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Minnesota)
Cathrine Wilhelmsen
 
PDF
Biml Academy 2 - Lesson 5: Importing source metadata into Biml
Cathrine Wilhelmsen
 
PDF
Biml for Beginners: Speed up your SSIS development (SQLSaturday Iceland)
Cathrine Wilhelmsen
 
PDF
Microsoft SQL Server - Files and Filegroups
Naji El Kotob
 
DOCX
Ssis partitioning and best practices
Vinod Kumar
 
PDF
Partitioning Tables and Indexing Them --- Article
Hemant K Chitale
 
PPTX
Responding to extended events in near real time
Gianluca Sartori
 
PPTX
Table partitioning in Oracle Database
Aakash Mehndiratta
 
PPTX
Partitioning on Oracle 12c - What changed on the most important Oracle feature
Luis Marques
 
PPTX
TehDays Basel - Auditing in sql server 2012 - charley hanania - tech days bas...
Charley Hanania
 
PDF
Implementing Auditing in SQL Server
David Dye
 
PPTX
SQL Server Index and Partition Strategy
Hamid J. Fard
 
KEY
SQL Server: Security
LearnNowOnline
 
PDF
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Go...
Cathrine Wilhelmsen
 
PDF
whitepaper_advanced_analytics_with_tableau_eng
S. Hanau
 
PDF
Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)
Cathrine Wilhelmsen
 
PDF
Don't Repeat Yourself - Agile SSIS Development with Biml and BimlScript (SQL ...
Cathrine Wilhelmsen
 
PPT
PMO OBjectives - a quick guide
PM Majik
 
PPTX
SQL Server Reporting Services 2008
VishalJharwade
 
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Sacrame...
Cathrine Wilhelmsen
 
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Minnesota)
Cathrine Wilhelmsen
 
Biml Academy 2 - Lesson 5: Importing source metadata into Biml
Cathrine Wilhelmsen
 
Biml for Beginners: Speed up your SSIS development (SQLSaturday Iceland)
Cathrine Wilhelmsen
 
Microsoft SQL Server - Files and Filegroups
Naji El Kotob
 
Ssis partitioning and best practices
Vinod Kumar
 
Partitioning Tables and Indexing Them --- Article
Hemant K Chitale
 
Responding to extended events in near real time
Gianluca Sartori
 
Table partitioning in Oracle Database
Aakash Mehndiratta
 
Partitioning on Oracle 12c - What changed on the most important Oracle feature
Luis Marques
 
TehDays Basel - Auditing in sql server 2012 - charley hanania - tech days bas...
Charley Hanania
 
Implementing Auditing in SQL Server
David Dye
 
SQL Server Index and Partition Strategy
Hamid J. Fard
 
SQL Server: Security
LearnNowOnline
 
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Go...
Cathrine Wilhelmsen
 
whitepaper_advanced_analytics_with_tableau_eng
S. Hanau
 
Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)
Cathrine Wilhelmsen
 
Don't Repeat Yourself - Agile SSIS Development with Biml and BimlScript (SQL ...
Cathrine Wilhelmsen
 
PMO OBjectives - a quick guide
PM Majik
 
SQL Server Reporting Services 2008
VishalJharwade
 
Ad

Similar to Table Partitioning in SQL Server: A Magic Solution for Better Performance? (Pragmatic Works) (20)

PPT
MBA
claraclan
 
PDF
pdfcoffee.com_i-openwells-basics-training-3-pdf-free.pdf
Jalal Neshat
 
PDF
From SQL to Pandas
Walter Gentile
 
PDF
Datacolor match textile 1.0 user guide
pinelope
 
PDF
Rtips123
Mahendra Babu
 
PDF
_Super_Study_Guide__Data_Science_Tools_1620233377.pdf
nielitjanarthanam
 
PDF
Trading volume mapping R in recent environment
Nagi Teramo
 
PDF
Getting_started_teamcenterhdfdjshfkdsjfhjksdfhdkjsfhkdsjf
imranoci123
 
PDF
Backtrack tutorial
khadikhadi
 
DOCX
Getting started with power map preview september refresh
Ming Gu
 
PDF
A practical guide to implementing tivoli storage manager on as 400 sg245472
Banking at Ho Chi Minh city
 
PDF
Why PostgreSQL for Analytics Infrastructure (DW)?
Huy Nguyen
 
PDF
Jaspersoft and Clarity PPM - Advanced Reporting with Data Warehouse
Thiago Bottoni
 
PDF
lum_117_whatsnew_en
Jim Miller, MBA
 
PDF
lum_117_whatsnew_en
James Miller, MBA
 
PDF
SQL tips and techniques April 2014
Nick Ivanov
 
PDF
Practical Recipes for Daily DBA Activities using DB2 9 and 10 for z/OS
Cuneyt Goksu
 
PPTX
Really using Oracle analytic SQL functions
Kim Berg Hansen
 
PDF
IBM manual Mainframe Utilities DFSMS.pdf
Flavio787771
 
PDF
3 d autocad
Cleverson Leite
 
MBA
claraclan
 
pdfcoffee.com_i-openwells-basics-training-3-pdf-free.pdf
Jalal Neshat
 
From SQL to Pandas
Walter Gentile
 
Datacolor match textile 1.0 user guide
pinelope
 
Rtips123
Mahendra Babu
 
_Super_Study_Guide__Data_Science_Tools_1620233377.pdf
nielitjanarthanam
 
Trading volume mapping R in recent environment
Nagi Teramo
 
Getting_started_teamcenterhdfdjshfkdsjfhjksdfhdkjsfhkdsjf
imranoci123
 
Backtrack tutorial
khadikhadi
 
Getting started with power map preview september refresh
Ming Gu
 
A practical guide to implementing tivoli storage manager on as 400 sg245472
Banking at Ho Chi Minh city
 
Why PostgreSQL for Analytics Infrastructure (DW)?
Huy Nguyen
 
Jaspersoft and Clarity PPM - Advanced Reporting with Data Warehouse
Thiago Bottoni
 
lum_117_whatsnew_en
Jim Miller, MBA
 
lum_117_whatsnew_en
James Miller, MBA
 
SQL tips and techniques April 2014
Nick Ivanov
 
Practical Recipes for Daily DBA Activities using DB2 9 and 10 for z/OS
Cuneyt Goksu
 
Really using Oracle analytic SQL functions
Kim Berg Hansen
 
IBM manual Mainframe Utilities DFSMS.pdf
Flavio787771
 
3 d autocad
Cleverson Leite
 

More from Cathrine Wilhelmsen (20)

PDF
Fra utvikler til arkitekt: Skap din egen karrierevei ved å utvikle din person...
Cathrine Wilhelmsen
 
PDF
One Year in Fabric: Lessons Learned from Implementing Real-World Projects (PA...
Cathrine Wilhelmsen
 
PDF
Data Factory in Microsoft Fabric (MsBIP #82)
Cathrine Wilhelmsen
 
PDF
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Cathrine Wilhelmsen
 
PDF
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Cathrine Wilhelmsen
 
PDF
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Cathrine Wilhelmsen
 
PDF
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Cathrine Wilhelmsen
 
PDF
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Cathrine Wilhelmsen
 
PDF
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Cathrine Wilhelmsen
 
PDF
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
Cathrine Wilhelmsen
 
PDF
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Cathrine Wilhelmsen
 
PDF
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Cathrine Wilhelmsen
 
PDF
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Cathrine Wilhelmsen
 
PDF
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Cathrine Wilhelmsen
 
PDF
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Cathrine Wilhelmsen
 
PDF
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Cathrine Wilhelmsen
 
PDF
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
Cathrine Wilhelmsen
 
PDF
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Cathrine Wilhelmsen
 
PDF
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
Cathrine Wilhelmsen
 
PDF
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Cathrine Wilhelmsen
 
Fra utvikler til arkitekt: Skap din egen karrierevei ved å utvikle din person...
Cathrine Wilhelmsen
 
One Year in Fabric: Lessons Learned from Implementing Real-World Projects (PA...
Cathrine Wilhelmsen
 
Data Factory in Microsoft Fabric (MsBIP #82)
Cathrine Wilhelmsen
 
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Cathrine Wilhelmsen
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Cathrine Wilhelmsen
 
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Cathrine Wilhelmsen
 
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Cathrine Wilhelmsen
 
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Cathrine Wilhelmsen
 
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Cathrine Wilhelmsen
 
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
Cathrine Wilhelmsen
 
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Cathrine Wilhelmsen
 
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Cathrine Wilhelmsen
 
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Cathrine Wilhelmsen
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Cathrine Wilhelmsen
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Cathrine Wilhelmsen
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Cathrine Wilhelmsen
 
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
Cathrine Wilhelmsen
 
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Cathrine Wilhelmsen
 
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
Cathrine Wilhelmsen
 
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Cathrine Wilhelmsen
 

Recently uploaded (20)

PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
thirishadevan81
 
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
CHINTU5000
 
PPTX
Supervised vs unsupervised machine learning algorithms
agarwal18harsh08
 
PPTX
Business Ppt On Nestle.pptx huunnnhhgfvu
ashaurya327
 
PPTX
climate analysis of Dhaka ,Banglades.pptx
rrawjatun
 
PDF
.pdf is not working space design for the following data for the following dat...
jerinjoy242
 
PDF
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
ngaviet5
 
PPTX
IB Computer Science - Internal Assessment.pptx
agarwal18harsh08
 
PPT
Quality review (1)_presentation of this 21
abobaker13
 
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
chepkoitcheruiyot
 
PPTX
05. PRACTICAL GUIDE TO MICROSOFT EXCEL.pptx
macomputermacomputer
 
PDF
Lecture1 pattern recognition............
ZafriFarid
 
PDF
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
sriram270905
 
PPTX
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
rohullahansari5
 
PPTX
IBA_Chapter_11_Slides_Final_Accessible.pptx
SrikantKapoor1
 
PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
PPTX
1_Introduction to advance data techniques.pptx
AshutoshDeshmukh33
 
PPT
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
JanakiRaman206018
 
PPTX
Business Acumen Training GuidePresentation.pptx
stephvshelton19
 
PPT
Chapter 2 METAL FORMINGhhhhhhhjjjjmmmmmmmmm
JanakiRaman206018
 
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
thirishadevan81
 
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
CHINTU5000
 
Supervised vs unsupervised machine learning algorithms
agarwal18harsh08
 
Business Ppt On Nestle.pptx huunnnhhgfvu
ashaurya327
 
climate analysis of Dhaka ,Banglades.pptx
rrawjatun
 
.pdf is not working space design for the following data for the following dat...
jerinjoy242
 
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
ngaviet5
 
IB Computer Science - Internal Assessment.pptx
agarwal18harsh08
 
Quality review (1)_presentation of this 21
abobaker13
 
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
chepkoitcheruiyot
 
05. PRACTICAL GUIDE TO MICROSOFT EXCEL.pptx
macomputermacomputer
 
Lecture1 pattern recognition............
ZafriFarid
 
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
sriram270905
 
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
rohullahansari5
 
IBA_Chapter_11_Slides_Final_Accessible.pptx
SrikantKapoor1
 
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
1_Introduction to advance data techniques.pptx
AshutoshDeshmukh33
 
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
JanakiRaman206018
 
Business Acumen Training GuidePresentation.pptx
stephvshelton19
 
Chapter 2 METAL FORMINGhhhhhhhjjjjmmmmmmmmm
JanakiRaman206018
 

Table Partitioning in SQL Server: A Magic Solution for Better Performance? (Pragmatic Works)

  • 1. Table Partitioning in SQL Server A Magic Solution for Better Performance?
  • 3. You? New to table partitioning "Everything is slow" Today? Basic Introduction What, Why & How
  • 4. A Magic Solution for Better Performance? Spoiler Alert!
  • 5. A Magic Solution for Better Performance? Implementing table partitioning is not a trivial task and can actually cause worse performance...
  • 6. A Magic Solution for Better Performance? ...but don't worry, I made plenty of mistakes so you can avoid them 
  • 8. What? Partition Key Partition Function Partition Scheme Why? Backup & Restore Maintenance Load & Archive How? Partition Elimination Switch, Split & Merge Sliding Windows
  • 10. What is a partitioned table? 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Data is physically stored in groups of rows called partitions Each partition can be accessed and maintained separately Partitioning is not visible to users, it behaves like one logical table
  • 11. Partition Key 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Data is partitioned based on a single column, the Partition Key The Partition Key should always be used as a filter in queries This ensures Partition Elimination: only relevant partitions are accessed
  • 12. Partition Function 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... The Partition Function defines how to partition the data It specifies boundary values, the points between two partitions It specifies if the boundary value belongs to its left (upper) partition or its right (lower) partition
  • 13. Partition Function: Range Left and Range Right 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ...
  • 14. Partition Function: Range Left and Range Right Range Left means the boundary value is the last value in the left partition CREATE PARTITION FUNCTION pfLeft (INT) AS RANGE LEFT FOR VALUES (20,30,40); Range Right means the boundary value is the first value in the right partition CREATE PARTITION FUNCTION pfRight (INT) AS RANGE RIGHT FOR VALUES (20,30,40); ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40
  • 15. Partition Scheme 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... The Partition Scheme maps logical partitions to physical filegroups Filegroups? Data files on one or more disks Backed up and restored individually Can be Read-Only
  • 16. Map all partitions to one filegroup FILEGROUP 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ...
  • 17. Map partitions to separate filegroups FILEGROUP1 (Read-Only) FILEGROUP2 (Read-Only) 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... FILEGROUP3 FILEGROUP4
  • 18. How are partitions mapped to filegroups? ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40 The partition function specified the boundary values and partitions:
  • 19. How are partitions mapped to filegroups? CREATE PARTITION SCHEME psLeft AS PARTITION pfLeft TO (FG1, FG2, FG3, FG4); CREATE PARTITION SCHEME psRight AS PARTITION pfRight TO (FG1, FG2, FG3, FG4); ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40 The partition scheme uses the partition function...
  • 20. How are partitions mapped to filegroups? CREATE PARTITION SCHEME psLeft AS PARTITION pfLeft TO (FG1, FG2, FG3, FG4); CREATE PARTITION SCHEME psRight AS PARTITION pfRight TO (FG1, FG2, FG3, FG4); ...20 21-30 41...31-40 ...19 20-39 40...30-39 20 30 40 20 30 40 FG1 FG2 FG4FG3 FG1 FG2 FG4FG3 ...to map each partition to filegroups:
  • 21. Filegroups Partition Scheme Partitioned Table 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... A partitioned table is created on a partition scheme instead of directly on a filegroup The partition scheme uses the partition key to store rows in the correct partition and filegroup based on the definition specified in the partition function
  • 24. Partition Elimination SELECT COUNT(*) FROM Table WHERE Year = 2012; SELECT COUNT(*) FROM Table;
  • 25. Partition Elimination SELECT COUNT(*) FROM Table WHERE Year = 2012; SELECT COUNT(*) FROM Table;
  • 26. Backup & Restore Partitions Filegroups can be backed up and restored individually If each partition is mapped to a separate filegroup, partitions with the most critical data can be restored first
  • 27. Backup & Restore Partitions Filegroups can be read-only If each partition is mapped to a separate filegroup, partitions with historical, unchanging data can be excluded from regular backups
  • 28. Index Maintenance per Partition Rebuild and reorganize indexes per partition Rebuild indexes online per partition was introduced in SQL Server 2014 Set data compression per partition ALTER INDEX IndexName ON TableName REBUILD PARTITION = 2 WITH (ONLINE = ON);
  • 29. Statistics Maintenance per Partition Update statistics on specific partitions instead of scanning and updating the whole table UPDATE STATISTICS TableName (StatisticsName) WITH RESAMPLE ON PARTITIONS (3,5); CREATE STATISTICS StatisticsName ON TableName (ColumnName) WITH INCREMENTAL = ON; Incremental Statistics was introduced in SQL Server 2014
  • 30. Load & Archive: Partition Switching Partitions can be switched between tables, called switching in or switching out Partition switching is a metadata operation that updates the location of the data, no data is physically moved Extremely fast compared to inserting into or deleting from a large table 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... ALTER TABLE Table1 SWITCH PARTITION 5 TO Table2 PARTITION 5; SELECT $PARTITION.pf(2014);
  • 31. Load & Archive: Switch out 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Called switch out when you move data out of a table (archive)
  • 32. Load & Archive: Switch out 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... Called switch out when you move data out of a table (archive)
  • 33. Load & Archive: Switch in 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2016-01-01 ... ... 2016-12-31 ... ... Called switch in when you move data into a table (load)
  • 34. Load & Archive: Switch in 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... 2016-01-01 ... ... 2016-12-31 ... ... Called switch in when you move data into a table (load)
  • 35. Sliding Windows The Sliding Windows technique automates data loading, data archiving and partition management It keeps a certain number of partitions by continuously switching out the oldest data and switching in new data 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2012-01-01 ... ... 2012-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ...
  • 36. Sliding Windows: Split & Merge 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Partitions are not actually added or removed, they are split or merged Be careful! Splitting and merging partitions can cause data movement!!
  • 37. Sliding Windows: Split & Merge 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Split one partition in two by adding a new boundary value ALTER PARTITION FUNCTION pf () SPLIT RANGE ('2013-06-01'); Data movement will occur if there is data on both sides of the new boundary value!
  • 38. Sliding Windows: Split & Merge 2012-01-01 ... ... 2012-12-31 ... ... 2013-01-01 ... ... 2013-12-31 ... ... 2014-01-01 ... ... 2014-12-31 ... ... 2015-01-01 ... ... 2015-12-31 ... ... Merge two partitions to one by removing a boundary value ALTER PARTITION FUNCTION pf () MERGE RANGE ('2014-01-01'); Data movement will occur if there is data on both sides of the old boundary value!
  • 39. Sliding Windows Steps 1. Add new filegroup and file 2. Create switch out (archive) table 3. Create switch in (load) table and load it with new data 4. Split to add new partition 5. Switch in new partition 6. Switch out old partition 7. Merge to remove old partition 8. Delete switch out and switch in tables 9. Delete old file and filegroup
  • 41. Alternative Sliding Windows Steps 1. Add new filegroup and file 2. Create switch out (archive) table 3. Switch out old partition 4. Merge to remove old partition 5. Create switch in (load) table and load it with new data 6. Split to add new partition 7. Switch in new partition 8. Delete switch out and switch in tables 9. Delete old file and filegroup
  • 43. Case Study: What Went Wrong?
  • 44. Background: Financial Data Warehouse Periodic Snapshot Fact Tables Daily, Weekly and Monthly Balances SQL Server, DB2, Oracle, .csv, .txt Loaded at different times during the day 1 2 3 4 5 6 7 Pension Insurance Bank Fund
  • 45. First version Daily fact table: Keep 1 day per source 1 2 Pension 6 7 Bank Fund Monthly fact table: Keep 36 months per source 3 4 5 Insurance
  • 46. Second version Daily fact table: Keep 1 day per source 1 2 Pension Monthly / Weekly fact table: Keep 36 months for some sources Keep 106 weeks for some sources 3 4 5 Insurance ! 6 7 Bank Fund
  • 47. Third version Daily fact table: Keep 1 day per source 1 2 Pension Monthly / Weekly / Daily fact table: Keep 36 months for some sources Keep 106 weeks for some sources Keep 7 days for some sources 3 4 5 6 7 Insurance Bank Fund !!
  • 50. Daily fact table partitioning 1 ... ... 2 ... ... 3 ... ... 4 ... ... 5 ... ... 6 ... ... 7 ... ... Partition Key: source number One partition per source Helped with deadlock issues Easy to switch data in and out
  • 51. Monthly / Weekly / Daily fact table partitioning 12011 ... ... 12012 ... ... 12013 ... ... 12014 ... ... 19990 ... ... 19991 ... ... 19992 ... ... Partition Key: Source number + Period type Period type: YYYY for monthly data 9991 for weekly data 9992 for daily data Helped with deadlock issues, but…
  • 52. Monthly / Weekly / Daily fact table partitioning 12011 ... ... 12012 ... ... 12013 ... ... 12014 ... ... 19990 ... ... 19991 ... ... 19992 ... ... Partition key was difficult to remember and was not used in queries Data had to be inserted into and deleted from partitions with existing data, instead of switching partitions in and out
  • 53. What went wrong? "The usual suspects": Conflicting priorities, changing requirements and high pressure to deliver on time Fact tables were expanded, not changed, when new sources and requirements were implemented Partitioning was implemented on fact tables with multiple grains, instead of correcting the fact tables first
  • 54. What did we learn?
  • 56. References and Resources: Table Partitioning Brent Ozar Unlimited: SQL Server Table Partitioning Resources brentozar.com/sql/table-partitioning-resources/ Bradley Ball: Partitioning in SQL Server 2012 pragmaticworks.com/Training/FreeTraining/ViewWebinar/WebinarID/541
  • 57. References and Resources: Further Reading Partial Backups in SQL Server 2014 msdn.microsoft.com/en-us/library/ms191539.aspx Piecemeal Restores in SQL Server 2014 msdn.microsoft.com/en-us/library/ms177425.aspx Benjamin Nevarez: SQL Server 2014 Incremental Statistics benjaminnevarez.com/2015/02/2014-incremental-statistics/