Understanding the Transaction Log, Your Key to Unlocking Greater Throughput
Download as PPTX, PDF1 like1,052 views
The document discusses the importance of the SQL Server transaction log in managing database transactions, covering its structure, types of log records, and the implications of log management on performance and recovery. It explores concepts like virtual log files, recovery models, and checkpoint processes while also highlighting monitoring techniques for better throughput. Additionally, it provides practical advice on architecture, coding practices, and server configurations that impact log efficiency.
Understanding the Transaction Log, Your Key to Unlocking Greater Throughput
- 1. UNDERSTANDING THE TRANSACTION LOG; YOUR KEY TO UNLOCKING GREATER THROUGHPUT Richard Douglas • Sales Engineer, SQL Sentry • Social Media: @SQLRich, LinkedIn, • Blog Info: http://blogs.sqlsentry.com/author/RichardDouglas/; http://SQL.RichardDouglas.co.uk
- 2. Tuning blog: http://www.sqlperformance.com/ E-mail ebooks@sqlsentry.com for free copies of our $10 e-books:
- 3. YOUR HOST • Richard Douglas • Sales Engineer at SQL Sentry • Blog: http://SQL.RichardDouglas.co.uk • Twitter: @SQLRich • Email: RDouglas@SQLSentry.com • Slides: http://www.Slideshare.net/SQLRich
- 4. AGENDA • When the log is used • Internals • Hardware • Database Architecture • Coding • Monitoring • Wrap Up
- 5. Logs, that’s all about backups right?
- 6. SQL Server will use the transaction log for almost every transaction
- 7. Data File SNI Phase 0 Phase 1 Phase 2 Optimizer Command Parser Query Executor Access Methods Buffer Manager Pre-Opt Execution Plan TDS TDS Query Tree Language Event Plan Cache Buffer Cache Buffer Pool Storage Engine Relational Engine Protocol Layer Results SELECT Statement Buffer Pool Extension
- 8. Log File Data File SNI Phase 0 Phase 1 Phase 2 Optimizer Command Parser Query Executor Access Methods Buffer Manager Pre-Opt Execution Plan TDS TDS Query Tree Language Event Plan Cache Buffer Cache Buffer PoolStorage Engine Relational Engine Protocol Layer Results Transaction Manager INSERT Statement
- 9. RECOVERY MODELS • Simple • Full • Bulk Logged Data File Log File SQL Database Automated administration Manual Administration Note: Recovery model will be defaulted to the one in the Model database
- 10. MINIMALLY LOGGED OPERATIONS • Bulk import operations* – Bcp – Bulk Insert – INSERT / SELECT (External Tools) • SELECT * INTO • .Write in UPDATE statements (new values only) • WRITETEXT and UPDATETEXT (new values only, marked for deprecation) • Some DDL Statements – CREATE INDEX – ALTER INDEX REBUILD / DBCC DBREINDEX (marked for deprecation) – DROP INDEX (Index pages fully logged) • When using Transactional Replication this will be fully logged.
- 11. TRANSACTION LOG INTERNALS • The Transaction Log is segmented into pieces called Virtual Log Files (VLF’s) • There will always be at least one “active” VLF • Having too many VLF’s is thought to be a bad thing. • How many is too many and how are they calculated? Active VLF Inactive VLF Inactive VLF Inactive VLF Inactive VLF Yes, another 9 sets like this would make 50 shades of grey.
- 12. CALCULATING VLFS • Pre2014 the numberof VLF’s is determined by the size at creation / growth time. – Under 64MB 4VLF’s – >=64and> 1GB 8 VLF’s – >= 1GB 16 VLF’s • In 2014, if the growth size is less than 1/8 the size of the log Then create1 single VLF at the size of the growth size Else use the pre2014 formula • But wait, there’s more. VLF’s aren’t the smallest segment of the Transaction Log …
- 13. LOG BLOCKS • Each VLF will have a numberof “Log Blocks” inside it • These blocks varyin size: – Anywherebetween 512bytesand60KB • A new block is createdwhen: – The data insidethe blockhits the 60KB limit – A commitor abortoccurswithin a transaction • Each log block contains a parity bit of 64 or128 which is flipped everytime it is used. • Inside each log block we have…
- 14. LOG RECORDS • Log records are identified by Log Sequence Numbers or LSN’s – VLF Sequence nbr: Logblock nbr : Logrecordnbr • Each data page contains the LSN of the last recordthat made changes to that page. • Log records are written in the orderthey occur. i.e.SQL Serverwill notwait for a transaction to finish before writing away another concurrent transaction. • Log records often storebefore and after images of changes. • Theycontain a list of pages that this change has affected. • Log records are stored in the buffer pool (log cache)until flushed to disk Look for WAITLOG wait type.
- 15. CHECKPOINT • Data pages are markedas “dirty” when they are changed. • Thecheckpoint process writes all dirty pages to disk and marksthem as cleared. • Checkpoint is an optimisation process, it reducesthe amount of writes to disk to keepdata up to date.
- 16. INTERESTING CHECKPOINT TRACE FLAGS • 828 –Checkpoint resumes the behaviour that it exhibited before you installed SQLServer2000 SP3 when you enabletrace flag 828 on a computer that is running SQL Server2000 SP4 - KB906121 • 3031 –In 2005 turns NO_LOG and TRUNCATE_ONLY backups to checkpoints in all recovery modes http://bit.ly/1b7RFNV • 3502 –Writes information about checkpoints tothe errorlog • 3503 –Indicates whether the checkpointat the end of automatic recoverywas skipped for a database. • 3504 –Writes further info about checkpoints to the errorlog • 3505 –Disables automatic checkpoints
- 17. MAKING STUFF GO FASTER Server / IO Architecture Database Architecture Coding
- 18. HARDWARE CHOICES • IO Architecture can be a nightmare • Local vs Shared • Specifying drives • RAID Levels o Hardware RAID with large cash cache
- 19. DATABASE ARCHITECTURE • Get the log size right early on • Consider High Availability Overhead • Only log what you need • Beware of “evil” page splits • Delayed Durability
- 20. HIGH AVAILABILITY OVERHEAD • Log Shipping • Database Mirroring • AlwaysOn Availability Groups • AlwaysOn Failover Clusters • Replication
- 21. WATCH YOUR INDICES The following add overhead, beware: • Duplicate indices • Unused indices • Subsets of indices Use the Dynamic Management Object Sys.dm_db_Index_Usage_Stats to find out how useful an index really is.
- 22. CODING • “Temp Tables vs Table Variable” debate o Blog post - http://bit.ly/1ZmvRtt • “Rollback Gotchas” o Blog post - http://bit.ly/1Z4ZfEm • Batching transactions
- 23. MONITORING THE LOG • Dynamic Management Objects • Perfmon counters • Wait statistics • Reading the log! Sys.dm_io_virtual_file_stats Lots of Log based counters in SQLServer:Databases LogBuffer and WriteLog are common
- 25. VLF RESULTS High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 Update 02 Log Backup 02 Update 03 Log Backup 03 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 Log Backup 02 Update 03 Log Backup 03 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 219 209 25 27 204 210 24 25 27 Log Backup 02 Update 03 Log Backup 03 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 219 209 25 27 204 210 24 25 27 Log Backup 02 237 217 21 16 231 219 18 14 15 Update 03 Log Backup 03 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 219 209 25 27 204 210 24 25 27 Log Backup 02 237 217 21 16 231 219 18 14 15 Update 03 216 210 25 27 205 211 27 29 27 Log Backup 03 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 219 209 25 27 204 210 24 25 27 Log Backup 02 237 217 21 16 231 219 18 14 15 Update 03 216 210 25 27 205 211 27 29 27 Log Backup 03 239 217 21 15 231 219 17 15 15 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 219 209 25 27 204 210 24 25 27 Log Backup 02 237 217 21 16 231 219 18 14 15 Update 03 216 210 25 27 205 211 27 29 27 Log Backup 03 239 217 21 15 231 219 17 15 15 Backup Full 0 1 1 0 0 0 0 0 0 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Update 01 1673 516 222 54 1479 517 168 50 36 Log Backup 01 247 217 22 15 234 219 18 15 15 Update 02 219 209 25 27 204 210 24 25 27 Log Backup 02 237 217 21 16 231 219 18 14 15 Update 03 216 210 25 27 205 211 27 29 27 Log Backup 03 239 217 21 15 231 219 17 15 15 Backup Full 0 1 1 0 0 0 0 0 0 Totals 2831 1587 337 154 2584 1595 272 148 135 0 200 400 600 800 1000 1200 1400 1600 1800 Update 01 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD 0 50 100 150 200 250 300 Update 01 Log Backup 01 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD Some outliers have been removed to improve readability 0 50 100 150 200 250 300 Update 01 Log Backup 01 Update 02 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD 0 50 100 150 200 250 300 Update 01 Log Backup 01 Update 02 Log Backup 02 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD 0 50 100 150 200 250 300 Update 01 Log Backup 01 Update 02 Log Backup 02 Update 03 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD 0 50 100 150 200 250 300 Update 01 Log Backup 01 Update 02 Log Backup 02 Update 03 Log Backup 03 High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD 0 50 100 150 200 250 300 Update 01 Log Backup 01 Update 02 Log Backup 02 Update 03 Log Backup 03 Backup Full High - 2012 External Low - 2012 External High - 2012 SSD Low - 2012 SSD High - 2014 External Low - 2014 External High - 2014 SSD Low - 2014 SSD Optimal - 2014 SSD
- 26. 2014 HIGH VLF WAITS 39 58 Update 02 LOGBUFFER QDS_SHUTDOWN_QUEUE 46 52 Update 03 LOGBUFFER QDS_SHUTDOWN_QUEUE 7 10 4 20 7 49 Update 01 LATCH_EX LCK_M_U PREEMPTIVE_OS_FILEOPS PREEMPTIVE_OS_FLUSHFILEBUFFERS PREEMPTIVE_OS_WRITEFILEGATHER QDS_SHUTDOWN_QUEUE
- 27. THANK YOU! • Slides will be available at http://Slideshare.net/SQLRich • More information at: o SQLSkills, et al • E-mail ebooks@sqlsentry.com for free copies of our e-books: o Just tell them where you met me • My contact info for other questions: o RDouglas@SQLSentry.com o Twitter: @SQLRich o Blogs: http://SQL.RichardDouglas.co.uk / http://http://blogs.sqlsentry.com/author/RichardDouglas