Jss 2015 in memory and operational analytics
Download as PPTX, PDF1 like347 views
This document contains summaries of presentations and information about the #JSS2015 conference on SQL Server 2015 organized by GUSS. It provides information on speakers David Barbarin and Frédéric Pichaut and topics to be covered including columnstore architecture, columnstore improvements in SQL 2016, in-memory OLTP architecture and improvements, and remaining unsupported in-memory features.
![#JSS2015
SQL 2016 - Improved Scaling - Storage
• SQL Server 2014
• SQL Server 2016
– Multiple offline checkpoint threads
– Goal: 1GB/s of log generation [work in progress]
–
Single Offline Checkpoint Thread
Del Tran2
(TS 450)
Del Tran3
(TS 250)
Del
Tran1(TS150)
Insert into
Hekaton T1
Log in SQL
Table
Del Tran1
(TS150)
Del Tran2
(TS 450)
Del Tran3
(TS 250)
Insert into
Hekaton T1SQL Transaction log
disk
SQL Transaction log Del Tran2
(TS 450)
Del Tran3
(TS 250)
Del
Tran1(TS150)
Insert into
Hekaton T1
Log in SQL
Table
Del Tran1
(TS150)
Del Tran2
(TS 450)
Del Tran3
(TS 250)
Insert into
Hekaton T1
disk
Offline Checkpoint Thread Offline Checkpoint Thread Offline Checkpoint Thread](https://image.slidesharecdn.com/jss2015-inmemoryandoperationalanalytics-151204092114-lva1-app6891/85/Jss-2015-in-memory-and-operational-analytics-12-320.jpg)
![#JSS2015
SQL 2016 In-Memory Improvements
• Native Procedure:
• ALTER PROC and sp_recompile (online and recompilation Durant execution)
• Nested native procedures
• Natively Compiled Scalar UDFs (Access from both native and interop)
• Native Inline Table-Valued Functions
• EXECUTE AS CALLER
• Security and math built-ins
• Full collations support in native modules
• Query surface area
• Subqueries
• LIKE operator
• {LEFT|RIGHT} OUTER JOIN
• Disjunction (OR, NOT)
• UNION [ALL]
• SELECT DISTINCT
• Subqueries (EXISTS, IN, scalar)
• Parallel scan for memory-optimized indexes
• MARS (Multiple Active Result Sets) support
• TDE (Transparent Data Encryption) (on-disk data files encrypted once TDE is enabled)](https://image.slidesharecdn.com/jss2015-inmemoryandoperationalanalytics-151204092114-lva1-app6891/85/Jss-2015-in-memory-and-operational-analytics-14-320.jpg)
Jss 2015 in memory and operational analytics
- 1. #JSS2015 Les journées SQL Server 2015 Un événement organisé par GUSS @GUSS_FRANCE
- 2. #JSS2015 Les journées SQL Server 2015 Un événement organisé par GUSS In-Memory 2016 Operational Analytics David Barbarin Frédéric Pichaut
- 3. #JSS2015 Merci à nos sponsors
- 4. #JSS2015 Speakers David Barbarin http://blog.developpez.com/mikedavem/ david.barbarin@dbi-services.com @mikedavem Frédéric Pichaut Senior Escalation Engineer EMEA, Microsoft France @FPichaut
- 5. #JSS2015 Columnstore Architecture C1 C2 C3 C4 C5 Row group Set of rows (typically 1 million rows) Segment Contains values from one column for the row group Unit of transfer between disk and memory 1- Encoding Dictionary > Value: Base / Scale > Bit packing > RLE … Vertipaq 2- Binary compression (3)- Archive compression Xpress 8 + + C1 C2 C3 C2 C4 C1 C5 C3 C5 LOB C1 C2 C3 C4 C5 Tuple mover
- 6. #JSS2015 Columnstore & Batch mode Scan Predicate I/O significantly reduced by > Encoding & compression > Segment elimination Next challenge: CPU usage – how to improve performance in this area? Get row Filtered row X X X Batch object Column vector C1 C2 Scan Predicate Get batch Filtered batch Process multiple rows in batch for efficiency (~1K rows) > Using of SIMD instructions > Optimized for 64 bits values of the register > Significant reduction in function calls = less CPU time(7 – 40x) Reduction of CPU latency and cache misses > Optimized for CPU L2 cache and avoiding cache misses > Aggressive memory pre-fetch (sequential vs random)
- 7. #JSS2015 SQL 2016 Columnstore Improvements • Tables: • Primary key, foreign keys, CDC, Triggers, temporal tables • Change tracking (NCCI only) • Transaction isolation level SI and RCSI • Availability groups and readonly secondary replicas support • CI introduced in operational analytics • Disk-based table + NCCI (updatable and filter capabilities) • CCI + nonclustered indexes • In-memory table + NCCI • ALTER TABLE .. REORGANIZE for dealing with fragmentation • New or enhanced DMVS • sys.dm_db_column_store_row_group_* • sys.dm_db_index_* and sys.dm_xtp_*
- 8. #JSS2015 SQL 2016 Columnstore Improvements • Batch mode • Sort operator • Aggregate operators and distinct • AVG, SUM, COUNT, CHECKSUM_AGG, STDEV, STDEVP • Windows functions • COUNT, COUNT_BIG, SUM, AVG, MIN, MAX • Analysis functions • LAG, LEAD, FIRST_VALUE, LAST_VALUE, PERCENTILE_CONT, PERCENTILE_DISC, CUME_DIST, PERCENT_RANK • Batch mode with maxdop 1 • Parallelized insert into the delta stores • String predicate pushdown
- 9. #JSS2015 Relational Table (Clustered Index/Heap) Btree Index Hot Deletebitmap Delta rowgroups Operational analytics: columnstore indexes & In-Memory tables Unified view for OLTP & DW developers High performance past period DW Data Lifecycle Real-time operational OLTP Create Columnstore only on cold data – using filtered predicate to minimize maintenance create nonclustered columnstore index ….. where order_status = ‘SHIPPED’ CCI w / High compression Deletebitmap Delta rowgroups
- 10. #JSS2015
- 11. #JSS2015 Memory-optimized Table Filegroup SQL Server.exe Hekaton Engine: Memory_optimized Tables & Indexes TDS Handler and Session Management OLTP In-Memory Architecture Natively Compiled SPs and Schema Buffer Pool for Tables & Indexes Proc/Plan cache for ad-hoc T- SQL and SPs Client App Transaction Log Query Interop Non-durable TableT1 T4T3 T2 T1 T4T3 T2 T1 Tables Indexes Interpreter for TSQL, query plans, expressions T1 T4T3 T4T3 Checkpoint & Recovery Access Methods Parser, Catalog, Algebrizer, Optimizer Hekaton Compiler Hekaton Component Key Existing SQL Component Generated .dll Data File Group T4T3 T4T3
- 12. #JSS2015 SQL 2016 - Improved Scaling - Storage • SQL Server 2014 • SQL Server 2016 – Multiple offline checkpoint threads – Goal: 1GB/s of log generation [work in progress] – Single Offline Checkpoint Thread Del Tran2 (TS 450) Del Tran3 (TS 250) Del Tran1(TS150) Insert into Hekaton T1 Log in SQL Table Del Tran1 (TS150) Del Tran2 (TS 450) Del Tran3 (TS 250) Insert into Hekaton T1SQL Transaction log disk SQL Transaction log Del Tran2 (TS 450) Del Tran3 (TS 250) Del Tran1(TS150) Insert into Hekaton T1 Log in SQL Table Del Tran1 (TS150) Del Tran2 (TS 450) Del Tran3 (TS 250) Insert into Hekaton T1 disk Offline Checkpoint Thread Offline Checkpoint Thread Offline Checkpoint Thread
- 13. #JSS2015 SQL 2016 In-Memory Improvements • Tables: • ALTER TABLE (offline, requires 2X memory) • Identity columns • Indexes on NULLable columns • COLUMNSTORE indexes • FOREIGN KEY, CHECK, UNIQUE constraints. • Change HASH index bucket_count through index REBUILD • Add/drop index supported • Stats improvements: auto-update and sampled stats • More than 8 indexes. • DML Triggers (Only AFTER triggers; no INSTEADOF; Natively compiled) • 2TB of user data in durable tables (in SQL 2014 Max 256GB) • Full collations support
- 14. #JSS2015 SQL 2016 In-Memory Improvements • Native Procedure: • ALTER PROC and sp_recompile (online and recompilation Durant execution) • Nested native procedures • Natively Compiled Scalar UDFs (Access from both native and interop) • Native Inline Table-Valued Functions • EXECUTE AS CALLER • Security and math built-ins • Full collations support in native modules • Query surface area • Subqueries • LIKE operator • {LEFT|RIGHT} OUTER JOIN • Disjunction (OR, NOT) • UNION [ALL] • SELECT DISTINCT • Subqueries (EXISTS, IN, scalar) • Parallel scan for memory-optimized indexes • MARS (Multiple Active Result Sets) support • TDE (Transparent Data Encryption) (on-disk data files encrypted once TDE is enabled)
- 15. #JSS2015 Remaining In-Memory Unsupported Features Tables • DDL triggers and Transactional DDL • Data types: XML, Spatial, CLR types, datetimeoffset, rowversion, sql_variant • ALTER TABLE ONLINE • Cross-container transaction limitations: snapshot/snapshot, serializable/serializable; SAVEPOINT • Online Migration of disk-based tables to memory-optimized Interop: cross-database queries, MERGE INTO, locking hints Native Compilation • CASE, MERGE, JOIN in UPDATE/DELETE; DML with OUTPUT; VIEWs • Natively Compiled Table-Valued Functions • Automatic and statement-level recompile • Access to disk-based tables • Must be schema-bound; no dynamic T-SQL • Parallelism; limitations in query operators (hash join/agg, merge join) Management: replication; DB snapshot; CDC; compression
- 16. #JSS2015 In-Memory OLTP Table Updateable CCI Tail DRT Range Index Hash Index Hot Relational Table (Clustered Index/Heap) Btree Index Hot Deletebitmap Delta rowgroups Operational analytics: columnstore indexes & In-Memory tables Unified view for OLTP & DW developers High performance past period DW Data Lifecycle Real-time operational OLTP Create Columnstore only on cold data – using filtered predicate to minimize maintenance create nonclustered columnstore index ….. where order_status = ‘SHIPPED’ CCI w / High compression Deletebitmap Delta rowgroups
- 17. #JSS2015
- 18. #JSS2015#JSS2015 Les évaluations des sessions, c’est important !! http://GUSS.Pro/jss
- 19. #JSS2015 Merci à nos volontaires…
- 20. #JSS2015#JSS2015
Editor's Notes
- #20: On parlait des speakers, il y a une chose qui leur tient à cœur !