Firebird: cost-based optimization and statistics, by Dmitry Yemanov (in English)
- 1. Cost-based OptimizationandStatistics in FirebirdDmitry YemanovThe Firebird Projecthttp://www.firebirdsql.org
- 2. IntroductionOptimizer decides how to find all the information required in the most efficient way it can
- 3. Different queries and/or fetch strategies may benefit from different data access paths
- 4. Some information should exist in order to help the optimizer in guessing about the best access path
- 7. Cost-based (statistics)Rule-based OptimizationHeuristical definitions
- 8. Indexed retrieval is better than a full table scan(and indexed loop join is better than a merge join)
- 9. B-tree has three levels of depth
- 10. Compound indices are better than simple ones
- 11. Drawbacks
- 12. Indices could be bad for some operations
- 13. User intentions are not taken into account
- 14. Not ready for “ad hoc” queriesCost-based OptimizationKey points
- 15. Every operation has an associated cost value
- 16. Cost value is calculated using statistical data
- 17. Cost is aggregated from bottom up in the access path
- 18. Drawbacks
- 21. Requires up-to-date statisticsBasic TermsSelectivity
- 22. Represents a fraction of rows from a row set
- 23. Lies in the value range 0.0 to 1.0
- 24. Cardinality
- 25. Represents number of rows in a row set
- 26. Base cardinality is the number of rows in a base tableUnderstanding of CostCost
- 27. Is a function of the estimated cardinalities
- 28. Represents computational complexity of the retrieval
- 29. Measurement
- 30. Cost value linearly depends on the number of logical reads required to perform an operation
- 31. Logical read is equal to a single page fetch
- 32. Cost value may also take into account auxiliary steps such as an external sortingCost Measurement (example)Full table scan
- 33. cost = base cardinality
- 35. cost = b-tree level + 1
- 36. Range index scan
- 37. cost = b-tree level + N + selectivity * base cardinality(N represents the number of the required leaf page fetches and thus depends on the average key length)Cost Aggregation (example)Final Row Setcost = 9000SELECT *FROM T1 JOIN T2 ON T1.PK = T2.FKWHERE T1.VAL + T2.VAL < 100ORDER BY T1.NUMSortcost = 9000Filtercost = 7000Loop Joincost = 6000Full Scancost = 1000Index Scancost = 5
- 38. StatisticsInformation describing data amounts and distribution of values on different levels(table, index, column)
- 39. Stored in a database or estimated at runtime
- 40. Collected by request or automaticallyCore StatisticsNumber of Rows in a Table (Base Cardinality)
- 41. Small tables:number of used record slots on the data pages
- 42. Large tables:number of used data pages / average record length
- 43. Estimated at runtimevia scanning pointer or data pagesCore Statistics (continued)Index Selectivity
- 44. 1 / number of distinct keys in the index
- 45. Maintained per segment: (A), (A, B), (A, B, C)
- 46. Assumes uniform distribution of values
- 47. Calculated during index creation or upon request(SET STATISTICS statement)
- 48. Stored on the index root page
- 49. Visible in RDB$INDICES and RDB$INDEX_SEGMENTSDecisions Based on Core StatisticsFull Table Scan over Indexed Retrieval
- 50. Selectivity close to 1.0 suggests a full scan
- 51. What Indices to Use
- 52. Compare index selectivities and index scan costs
- 53. Consider segment operations for compound indices
- 54. Calculate selectivities for AND and OR operations
- 55. Order of Streams in Loop Joins
- 56. Calculate costs for different join ordersand choose the best oneAdvanced StatisticsTable level
- 57. Average page fill factor
- 58. Average row length(both help with a better base cardinality estimation)
- 59. Number of rows(allows to avoid the runtime pages scan)Advanced Statistics (continued)Index level
- 60. B-tree depth
- 61. Average key length(both help with a better cost estimation for index scans)
- 62. Clustering factor(allows to prefer an index navigationover an external sort under some conditions;also could be used to avoid filling the sparse bitmap)Clustering FactorIndex Key 1Data Page 12Index Key 2Data Page 25Data Page 12Index Key 3Data Page 28Data Page 13Index Key 4Data Page 44Data Page 14Index Key 5Data Page 57Bad Clustering FactorGood Clustering Factor
- 63. Advanced Statistics (continued)Column level
- 64. Selectivity(core feature, required to estimate costs)
- 65. Number of NULLs(useful for selectivity estimations for IS [NOT] NULL)
- 66. Value distribution histogram(allows selectivity estimations for non-uniform value distributions)Sample Histograms1. Non-Selective Column'A''B''C''D'2. Selective Column1 5 5 5 10 20 50 50 80 100
- 67. Decisions Based on Advanced StatisticsSort Aggregation vs Hash Aggregation
- 68. Selectivity of columns being grouped by
- 69. Loop Join vs Merge Join vs Hash Join
- 70. Cardinality of tables and filtering predicates
- 71. Index Usage
- 72. Number of NULLs or histogram
- 73. Index Navigation vs External Sorting
- 74. Clustering factorThe Firebird Projectwww.firebirdsql.org