When indexes are not enough
Download as PPTX, PDF0 likes389 views
The document discusses innovative alternative methods for improving code performance and maintainability in data solutions, specifically focusing on scenarios where indexes fall short. It presents the concept of lateral thinking for problem-solving and provides practical examples, such as managing free spaces and generating loan schedules. The session emphasizes the importance of exploring creative approaches and using declarative, set-based methods to enhance scalability and efficiency in database management.
When indexes are not enough
- 1. Blazing fast queries: when indexes are not enough Davide Mauri | dm@sensoriainc.com | @mauridb Director SW Engineering & Lead data architect Sensoria Inc.
- 2. About Me Microsoft Data Platform MVP since 2006 Working with the Microsoft Data Platform from 1997 • Specialized in Data Solution Architecture, Database Design, Performance Tuning, High-Performance Data Warehousing, BI, Big Data • Very strong developer background Loves this community! • President of UGISS (Italian SQL Server UG) for 11 Years • AppDev VG Leader since 2017 Regular Speaker @ Microsoft Data Platform events Director SW & Cloud @ Sensoria: http://www.sensoriafitness.com/ E-mail: dm@sensoriainc.com-Twitter: @mauridb Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx
- 3. About this session Discuss and demonstrate alternative ways of solving daily problem such in a way you will have Easier, more maintainable, more understandable code base Better performance Better scalability All without using indexes…because maybe you already used them and performances are still not good! It will “just” require a mind shift.
- 4. Free Space Problem Given a group of elements of defined size, find where there is enough free space to make sure that all elements will be stored near to each other This sample can be applied to warehouses, cinemas, stadiums, and so on…
- 6. Free Spaces – Most common solution 3 4 5 6 7 9 A. Set the “Free Space Counter” (FSP) to 1 B. Start from the first free seat (n), store this number as First Free Space (FFS) C. If next free space is n+1 then increase the FSC D. If next free space is not n+1, store it somewhere along with FFS E. restart from point A 1 2 5 1
- 7. Free Spaces Common solution demo
- 8. Considerations Is such solution the best? …or just the first one that came to our mind? Is really *that* easy? Not really: code is much more complex that one first thought Is scalable? Let’s say we have a warehouse with a million of small spaces. Will it perform well?
- 9. Lateral Thinking Let’s try to solve the problem with some creativity, using the so-called “Lateral Thinking”: Lateral thinking is solving problems through an indirect and creative approach, using reasoning that is not immediately obvious and involving ideas that may not be obtainable by using only traditional step-by-step logic https://en.wikipedia.org/wiki/Lateral_thinking
- 10. First steps in lateral thinking A. If SQL Server could see the picture above would be able to see that there are groups of free spaces Then we could simply use “GROUP BY” B. This means that there is some hidden information in the picture and we have to figure out how to turn it into something a RDBMS can handle
- 11. First steps in lateral thinking A. Let’s enumerate all the seats (free and occupied) B. And then enumerate all the free seats C. Do a simple difference…. D. And now we can use a GROUP BY 1 2 3 4 5 6 7 8 9 10… 1 2 3 4 5 6 7 … 2 2 2 2 2 3 3
- 12. Free Spaces Lateral Thinking Solution Demo
- 13. Lateral thinking results Code is so much simpler! Performances and scalability are vastly improved
- 14. Lateral thinking Seems hard? Well, at the beginning everything is hard, right? Remember where you were a kid? Easy: A little less easy: But it is quite clear who’s going to win in a competition, right? The only way to improve is exercising!
- 15. Let’s start! 1st problem: Loans Knowing the value of the requested loan, interests and number for rates, generate a row for each rate, from the beginning up until the end of the loan Loan Value # Rates Payment Frequency % Interest € 10.000,00 12 1 5,00% € 20.000,00 12 6 6,00% € 30.000,00 12 6 5,50% Rate Date € 875,00 January-20 € 875,00 February-20 € 875,00 March-20 € 875,00 April-20 € 875,00 May-20 € 875,00 June-20 € 875,00 July-20 € 875,00 August-20 € 875,00 September-20 € 875,00 October-20 € 875,00 November-20 € 875,00 December-20
- 16. Loans Demo
- 17. Groups and Correlated Values Given a list of machines statuses, tell the latest reported status of each machine Controller A B C Controller Status TimeStamp A StandBy 1001 B StandBy 1001 C StandBy 1003 D Working 1004 A Working 1004 B StandBy 1006 D StandBy 1009 D Working 1010 D Working 1011
- 18. Groups and Correlated Values Demo
- 19. Stocks Given a table with stock transactions get the High, Low, Open, Close & Volume values for each stock, each hour, each day SymbolID TransactionDateTime Price Volume 1 2008-01-01 09:15:21.000 75.800 2589 1 2008-01-01 09:25:44.000 68.200 4386 1 2008-01-01 09:29:31.000 74.300 2837 1 2008-01-01 09:34:42.000 68.900 2937 1 2008-01-01 09:39:13.000 72.300 4513 1 2008-01-01 09:43:35.000 67.300 838 1 2008-01-01 09:51:57.000 73.800 1380 1 2008-01-01 09:56:42.000 68.700 4190 tran_hour from_datetime to_datetime symbol_id high low volume open close 9 2008-01-01 09:15:21.000 2008-01-01 09:56:42.000 1 75.800 67.300 23670 75.800 68.700
- 20. Stocks Demo
- 21. Value Packing Now, from the same machines statuses table, we need to “pack” all the ranges Controller Status TimeStamp A StandBy 1001 B StandBy 1002 B StandBy 1003 B StandBy 1004 A Working 1005 A Working 1006 A Working 1007 B Working 1008 B Working 1009 A StandBy 1010 A StandBy 1011 B StandBy 1012 Controller Status From To A StandBy 1001 1001 B StandBy 1002 1004 A Working 1005 1007 B Working 1008 1009 A StandBy 1010 1011 B StandBy 1012 1012
- 23. Conclusions Don’t stop at the first solution that comes to your mind. Try to figure out if there is a different one 1 Try to use a declarative, set-based, approach: in RDBMS it is usually the best shot • Declarative: you tell the system which result you want not how to obtain it • Leverage Windows Functions that are really a powerful feature 2 Avoid the Row-By-Agonizing-Row (RBAR) whenever possible! It is complex, it doesn’t scale, it doesn’t perform well 3