Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSaturday Portugal)
1 like1,409 views
The document provides tips to help data warehouse developers become more efficient. It summarizes a presentation on tools and techniques for improving productivity when developing data warehouses and ETL processes. The presentation covers topics like visual features in SQL Server Management Studio, shortcuts, templates, snippets, query analysis, activity monitoring, and using Biml to generate SSIS packages from metadata. Attendees are encouraged to visit the speaker's website for more details and resources from the presentation.
Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSaturday Portugal)
- 1. Tools and Tips: From Accidental to Efficient Data Warehouse Developer Cathrine Wilhelmsen - SQLSaturday Lisbon 2015
- 2. Session description You have probably heard about the Accidental DBA, but what about the Accidental Data Warehouse developer? We stumbled into the world of data warehousing, learned dimensional modeling and work with T-SQL and SSIS daily. We're masters of googling solutions to our problems and make sure our complex ETL processes run without errors. We deliver data to business users... but we don't deliver data as fast as we want. You might not be able to rewrite your entire data warehouse or change your team's processes over night, but there are many things you can do to increase your own productivity and become a more efficient data warehouse developer. In this session I will show you some of what I've learned and discovered that has made me burst out "Oh wow! Why did I not know this yesterday!?" including query improvements, free tools and scripts, SSMS features and even a couple of things I used to think were only useful for those scary DBAs.
- 3. Thank you to the main sponsors!
- 4. Say thank you to organizers and volunteers! They spend their FREE time to give you this event Because they are crazy Because they want YOU to learn from the BEST IN THE WORLD
- 5. Paulo Matos:
- 10. Murilo Miranda:
- 11. Quilson Antunes:
- 12. Important Activities Sponsor Sessions with Raffles • 15:10 - Rumos, BI4ALL, Bold Int, CozyRoc, Pythian WIT (Women in Technology) • 15:10 at BizSpark Room (Ground Floor) SQLClinic Challenges • 17:00 BI (Steph Locke)
- 13. Cathrine Wilhelmsen @cathrinew cathrinewilhelmsen.net Data Warehouse Architect Business Intelligence Developer
- 15. once upon a time...
- 17. how I felt...
- 19. how I want to be...
- 22. Tip #1: Visual Information
- 24. Status Bar and Tab Text
- 25. Results in Separate Tab
- 26. Tab Groups - Vertical
- 27. Tab Groups - Horizontal
- 28. Split –one query in two windows
- 30. Query Shortcuts
- 31. Oh, by the way... SELECT SUM(rows) FROM sys.partitions WHERE index_id IN (0, 1) AND object_id = OBJECT_ID('Sales'); ...querying sys.partitions can be better and faster than COUNT(*)
- 32. Keyboard Shortcuts Assign shortcuts you frequently use Remove shortcuts you accidentally click (no more "ooops") msdn.microsoft.com/en-us/library/ms174205.aspx
- 33. Magic keys! HOME END PG UP PG DNCTRL ALT SHIFT TAB
- 34. Show / Hide Query Results CTRL R
- 35. Toggle Full Screen ALTSHIFT ENTER
- 37. Change database while writing query CTRL U
- 38. Column / Multi-Line Editing SHIFTALT
- 39. Comment / Uncomment CTRL K CTRL C Comment Line CTRL K CTRL U Uncomment Line
- 40. Tip #3: Search in SSMS
- 41. Free Tool: Redgate SQL Search red-gate.com/products/sql-development/sql-search/
- 42. Free Tool: Redgate SQL Search
- 43. Tip #4: Templates and Snippets
- 44. Templates Template Browser Drag & Drop Templates Create Templates CTRL ALT T
- 45. Template Parameters Replace Template Parameters with actual values CTRL SHIFT M
- 46. Snippets CTRL K CTRL X Insert Snippet CTRL K CTRL S Surround With Snippet
- 47. Advanced Snippets and Formatting Redgate SQL Prompt (Licensed) ApexSQL Complete / Refactor SSMS Tools Pack (Licensed) SSMS Boost Poor Man's T-SQL Formatter dbForge SQL Complete (Licensed) red-gate.com apexsql.com ssmstoolspack.com ssmsboost.com poorsql.com devart.com/dbforge
- 48. Redgate SQL Prompt Demo
- 49. Tip #5: Registered Servers and Multiserver Queries
- 50. Registered Servers Save and group servers Is the server running? Multiserver Queries View Registered Servers CTRL ALT G
- 51. Manage services from SSMS
- 54. Tip #6: SARGable Queries
- 55. SARGable queries "The query can efficiently seek using an index to find the correct rows searched for in WHERE or JOIN clauses" Compare it to finding a person in a phone book (We'll pretend we still use phone books)
- 56. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon SARGable queries Find all rows where Name starts with "T"
- 57. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon SARGable queries Find all rows where Name starts with "T"
- 58. Non-SARGable queries "The query has to scan each row in the table to find the correct rows searched for in WHERE or JOIN clauses" Compare it to finding a person in a phone book (We'll keep pretending we still use phone books)
- 59. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon Non-SARGable queries Find all rows where Name contains "al"
- 60. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon Non-SARGable queries Find all rows where Name contains "al"
- 61. WHERE Name LIKE '%al%' WHERE Name LIKE 'T%' WHERE LEFT(Name,1,1) = 'T' SARGable or Non-SARGable?
- 62. WHERE Name LIKE '%al%' WHERE Name LIKE 'T%' WHERE LEFT(Name,1,1) = 'T' SARGable or Non-SARGable?
- 63. WHERE CAST(EpisodeDate AS DATE) = '20050114' WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501' WHERE YEAR(EpisodeDate) = 2005 WHERE EpisodeDate >= '20050101' AND EpisodeDate < '20060101' SARGable or Non-SARGable?
- 64. WHERE CAST(EpisodeDate AS DATE) = '20050114' WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501' WHERE YEAR(EpisodeDate) = 2005 WHERE EpisodeDate >= '20050101' AND EpisodeDate < '20060101' SARGable or Non-SARGable?
- 65. WHERE Survivors < 40000 WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000 WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000 SARGable or Non-SARGable?
- 66. WHERE Survivors < 40000 WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000 WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000 SARGable or Non-SARGable?
- 68. Tip #7: Query Analysis
- 69. Execution Plans Display Estimated Execution Plan CTRL L Include Actual Execution Plan CTRL M
- 70. Execution Plans See how a query was or will be executed:
- 71. Details in Tooltips (ok)
- 72. Details in Properties (better)
- 73. Free Tool: SQL Sentry Plan Explorer sqlsentry.com/products/plan-explorer
- 74. Free Tool: SQL Sentry Plan Explorer answers.sqlperformance.com
- 75. Free Book: SQL Server Execution Plans by Grant Fritchey red-gate.com/community/books
- 76. Tip #8: Query Statistics
- 77. Statistics IO SET STATISTICS IO OFF; SET STATISTICS IO ON;
- 78. Statistics Time SET STATISTICS TIME OFF; SET STATISTICS TIME ON;
- 79. Free Tool: Statistics Parser by Richie Rump statisticsparser.com
- 80. Client Statistics Include Client Statistics SHIFT SALT
- 81. Client Statistics Compare multiple query executions:
- 82. Tip #9: Activity Monitoring
- 83. Free Script: sp_WhoIsActive by Adam Machanic sqlblog.com/blogs/adam_machanic
- 84. Free Script: sp_WhoIsActive by Adam Machanic
- 85. Tip #10: Generate SSIS Packages with Biml
- 86. Business Intelligence Markup Language Easy to read and write XML dialect Generate SSIS packages from metadata
- 87. What do I need? Free add-in for BIDS / SSDT-BI bidshelper.codeplex.com
- 88. How does it work? Create many SSIS packages from one Biml file
- 89. …what do you need me to do after lunch? Of course I can create 200 SSIS Packages!
- 90. Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Connections> <OleDbConnection Name="Source" ConnectionString="…" /> </Connections> <Packages> <Package Name="EmptyPackage"> … </Package> </Packages> </Biml>
- 91. From Biml to SSIS
- 92. From Biml to SSIS
- 93. The magic is in the Extend Biml with C# or VB.NET code blocks Import database structure and metadata Loop over tables and columns Add expressions to replace static values
- 94. BimlScript syntax <#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #> <# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "..."); #> <# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #> <Packages> <# foreach (var table in AW2014DB.TableNodes) { #> <Package Name="Load_<#=table.Name#>"> … </Package> <# } #> </Packages>
- 95. Biml for SSIS demo
- 96. …BimlBreak the rest of the week Biml on Monday…
- 97. Bonus Tip: Be social in your community! #SQLHelp
- 98. cathrinewilhelmsen.net/efficient Not enough details? Too fast? Don't worry! Slide deck, links and resources:
- 99. Go talk to the sponsors!
- 100. Thank you! Enjoy your lunch @cathrinew cathrinewilhelmsen.net no.linkedin.com/in/cathrinewilhelmsen contact@cathrinewilhelmsen.net cathrinewilhelmsen.net/efficient