7 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019
- 1. 7 Database Mistakes You are making
- 2. Hello! I am Dave Stokes MySQL Community Manager Slides for this talk online at slideshare.net/davidmstokes 2
- 3. 1. Backing up data != Safety I do not care about your cron job, cute scripts, or any of that if what you backup can not be used!
- 4. Backups are vital Use mysqldump/mysqlpump, Xtrabackup, MySQL Enterprise Backup, etcetera Async Replication, back up down stream server LVM Snapshots Something else in house DO ALL THE ABOVE AND LOOK FOR MORE!!! 4
- 5. But restoration is more vital Be able to restore: β’ Entire server β’ Schema β’ Table β’ Row And make sure you are not the only one at your company with this knowledge β’ Document β’ Train β’ Practice!!!! 5
- 6. Paranoia is not your enemy! 6 Murphyβs law is optimistic
- 7. And make sure you remember to backup β’ Binary logs β’ Stored Procedures β’ Views β’ Admin databases β’ Encryption keys β’ Configuration files (my.cnf, firewall, etc.) And store them someplace safe (minimum two places) 7
- 8. What is your companyβs record retention policy????! β’ Suppose your corporate lawyers ask (or are asked by a court) to show what data you had on hand five years ago? (copy of data, database, database software, hardware/software to support) β’ Are you subject to the new European rules where may have to βforgetβ data of individuals? What if the USA gets similar rules? β’ Medical/Legal/Research rules? 8
- 9. 2. Keep on top of software updates Do not skips updates
- 10. Keep your software updated! β’ New features β’ Bug Fixes β’ Security vulnerabilities Good for your resume! (ask your local FoxPro or dBase II professional) 10
- 11. And Keep a copy of the server backup before the upgrade around (and a copy from right after the upgrade) 11
- 12. 3. Monitor Those who do not study the past will pay heavily for the repeats
- 13. βNobody ever bitches that the database is running too fast!β 13 Daveβs First Law of Databases
- 14. What is your database doing RIGHT NOW?!?!?!? 1. RDMS a. Queries b. Logins/connections c. Index management d. Analytics e. Overhead f. Logging g. Replication 2. Hardware A. RAM B. Buffers C. Logins D. Network traffic E. Logging F. Paging G. Disk I/O H. Overhead 14
- 15. Big concept Does your paycheck and/or the paycheck of your boss depend on a reliable database? Then make sure you know what the #%@$ is is doing!! 15
- 16. Options Enterprise Level MySQL Enterprise Manager, Percona Manager, Solar Winds, etc - Yes, they cost money Instance Level MySQL Workbench PHPMyAdmin (Say hi to the Script Kiddies) If you do decide to βroll your ownβ you are not spending your time wisely. Your time has value. 16
- 17. Options Enterprise Level MySQL Enterprise Manager, Percona Manager, Solar Winds, etc - Yes, they cost money Instance Level MySQL Workbench PHPMyAdmin (Say hi to the Script Kiddies) If you do decide to βroll your ownβ you are not spending your time wisely. Your time has value. If you do roll your own do you do your own dentistry? 17
- 18. 4. Understand User Authentication 18
- 19. MySQL Authentication is Promiscuous It is VERY easy to have the same username with different hosts (they are a pair) AND different permissions for each of those pairs! 19
- 20. Connection flow β’ $mysql βu joe βp -> sent to server β’ Server checks to see if host is okay to connect β’ Server may prompt for authentication string β’ User and auth string checked β’ Is user over resource limits? β’ Connection established -> prompt set to user 20
- 21. How you can get different privileges for βthe sameβ account Joe @ 172.12.10.x Read, write, update, delete Joe @ 10.10.x.x Read, write, update, delete, drop Joe @ % Everything!! 21 Joeβs developer account when he joined the company Joeβs account for use in the data center Joeβs account created at 3AM fighting problems with legacy application
- 22. How you can get different privileges for βthe sameβ account Joe @ 172.12.10.x Read, write, update, delete Joe @ 10.10.x.x Read, write, update, delete, drop Joe @ % EVERYTHING!! 22 Joeβs developer account when he joined the company Joeβs account for use in the data center Joeβs account created at 3AM fighting problems with legacy application The Mysql daemon look for the most generous match First!!!!
- 23. Big concept So someone guesses Joeβs password is βJoeISNumber1β from outside the 10.10 and 172.12 networks and ends up with privs for EVERYTHING!! 23
- 24. 5. Indexes -> too many, too few 24
- 25. Big concept The cost based optimizer determines the cheapest way to return the requested data, sort of like a GPS, based on past query statistics and available indexes. And like a GPS it can be fooled And may not have latest information. 25
- 26. Full Table Scan A Full Table Scan is when every row of a table (file) needs to be read to see if it match the search criteria. SELECT name FROM city WHERE District='Texas'; There is no INDEX on the District column 26
- 27. Index Scan An index lets you go directly to the matching record(s). SELECT name FROM city WHERE CountryCode='USA'; There is an INDEX on the CountryCode column so it does not need to read all the rows! 27
- 28. So letβs index every column!!!! 28
- 29. Indexes should be considered a parallel table to your data table. So each time you add, delete, or modify an indexed column, that parallel table has to be updated TOO! That overhead is expensive!!! And the more options you have for indexes, the more the optimizer has to consider -- That adds a factorial to the complexity for each possible index 29 Why you do not want to index every column!!
- 30. Indexing options 1. Compound indexes use more than one column a. A Year-Month-Date index works for Y-M-D, Y-M, and Y b. But not for D or M-D (leftmost only!) 2. For βmultiple shortβ columns, consider hashes a.SELECT * FROM tbl_name WHERE hash_col=MD5(CONCAT(val1,val2)) AND col1=val1 AND col2=val2; 1. Optimizer Hints a. Comments in your query to direct optimizer b.SELECT /*+ JOIN_ORDER(t1, t2) JOIN_PREFIX(t2, t1) */ ... FROM t1, t2; 30
- 31. Indexing options 1. Compound indexes use more than one column 1. A Year-Month-Date index works for Y-M-D, Y-M, and Y 2. But not for D or M-D (leftmost only!) 2. Index prefix of column for wide columns 3. For βmultiple shortβ columns, consider hashes 1. SELECT * FROM tbl_name WHERE hash_col=MD5(CONCAT(val1,val2)) AND col1=val1 AND col2=val2; 4. Optimizer Hints 1. Comments in your query to direct optimizer 2. SELECT /*+ JOIN_ORDER(t1, t2) JOIN_PREFIX(t2, t1) */ ... FROM t1, t2; 31
- 32. Sys Schema -- Performance statistics You can use MySQL Workbench to access the SYS Schema to see a list of UNUSED indexes β’ Do not look at after a reboot as there will not be enough usage data collected β’ Make sure it is not a rarely used index that is not vital at quarter or year end 32
- 33. Histograms - Like Indexes without the overhead (for some data) A histogram is an accurate representation of the distribution of numerical data. For databases, a histogram is an approximation of the data distribution within a specific column. 33
- 34. Creating a histogram mysql> ANALYZE TABLE customer UPDATE HISTOGRAM ON c_mktsegment WITH 1024 BUCKETS; +---------------+-----------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+-----------+----------+---------------------------------------------------------+ | dbt3.customer | histogram | status | Histogram statistics created for column 'c_mktsegment'. | +---------------+-----------+----------+---------------------------------------------------------+ 34
- 35. Creating a histogram mysql> ANALYZE TABLE customer UPDATE HISTOGRAM ON c_mktsegment WITH 1024 BUCKETS; +---------------+-----------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+-----------+----------+---------------------------------------------------------+ | dbt3.customer | histogram | status | Histogram statistics created for column 'c_mktsegment'. | +---------------+-----------+----------+---------------------------------------------------------+ 35 Do you know the cardinality of your indexes? Hit rate? Active data set size?
- 36. Invisible indexes Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations. ALTER TABLE t1 ALTER INDEX i_idx INVISIBLE; ALTER TABLE t1 ALTER INDEX i_idx VISIBLE; 36
- 37. 6. Learn to Optimize Queries 37
- 38. Query optimization is a skill, not a black art. Even if you use an ORM (lazy), you need to learn how to use EXPLAIN and read a query plan. 38
- 39. 1. Buy a copy of High Performance MySQL and/or the MySQL 5.0 Certification Guide and read sections on query optimizations a. Both are dated but valuable 2. Learn to use VISUAL EXPLAIN from MySQL Workbench 3. Read DAILY your slow query log a. Not all slow queries are bad (monthly report) b. A query that was running well that now shows up on the slow query log is BAD 4. Optimize the most frequently run queries FIRST 39 Best ways to learn how to optimize queries
- 40. 7. Use JSON columns 40
- 41. Oracle, SQL Server, MySQL, PostgreSQL* have added JSON data types β PG actually added two JSON data types Vendors have added Native JSON data type
- 42. Mutability JSON data type let you have a place store rapidly evolving or undetermined data. JSON Data Types are extremely useful! Replace many-to-many joins If you have to do repeated index- lookup/data-dives for βstubβ data consider refactoring that data into a JSON column 42
- 43. MySQl Document Store β’ New API β The X Devapi β’ New Protocol β’ Allows you to use MySQL as a NoSQL JSON Document Store β’ No need to set up relational tables, indexes, or normalize data β’ Emphasis on CRUD β can use database before having βperfectβ knowledge of the data β’ Easily mutable β’ Works with relational tables too (mix & match) 43
- 44. JSON_Table THE JSON_TABLE is a function that lets you temporarily turn unstructured JSON data into a relational table for processing with SQL commands select country_name, IndyYear from countryinfo, json_table(doc,"$" columns (country_name char(20) path "$.Name", IndyYear int path "$.IndepYear")) as stuff where IndyYear > 1992; +----------------+----------+ | country_name | IndyYear | +----------------+----------+ | Czech Republic | 1993 | | Eritrea | 1993 | | Palau | 1994 | | Slovakia | 1993 | +----------------+----------+ 44
- 45. Generated Columns from JSON data If you find some of that JSON data is needed to be used in SQL searches then extract that information into its own column using a Generated Column CREATE TABLE stuff (c JSON, g INT GENERATED ALWAYS AS (c->"$.id")), INDEX i (g)) ; 45
- 46. Please buy my book If you are interested in using the JSON data type with MySQL but find the documentation hard to understand or are just looking for a compact reference guide, then you need my book! 46