7 Ways To Crash Postgres
5 likes13,556 views
7 ways to crash Postgres 1. Do not apply updates and remain on outdated versions of PostgreSQL. 2. Run out of disk space by allowing the database to grow without monitoring disk usage. This can result in errors and panics. 3. Delete important database files and directories which causes the database to fail to start. 4. Set memory settings too high and overload the system memory, triggering out of memory kills of the PostgreSQL process. 5. Use faulty hardware without monitoring for failures which can lead to corrupted blocks and index errors. 6. Allow too many open connections without connection pooling which can prevent new connections. 7. Accumulate zombie locks by not closing transactions, slowing down
7 Ways To Crash Postgres
- 1. 7 ways to crash Postgres Josh Berkus PostgreSQL Project LinuxConf.AU 2013
- 4. Don't 1 Apply Updates
- 5. 1. ignore update announcements 2. keep running version X.X.0 or X.X.1 3. wait for disaster to strike
- 6. 9. 2. 2
- 7. 9. 2. 3
- 9. 9.2.2: Prevent database-wide crash and restart when a new child process is unable to create a pipe for its latch. 9.1.6: Fix misbehavior when default_transaction_isolation is set to serializable. Symptoms include crashes at process start on Windows, and crashes in hot standby operation.
- 10. Upgrade! EOL after 5 years
- 11. 8.3 is now EOL
- 12. Out of 2 Disk Space
- 13. 1. install DB on volume which is 80% full 2. don't monitor disk space usage 3. let database grow
- 14. ERROR: could not write block 31840050 of temporary file: No space left on device PANIC: could not write to file "pg_xlog/xlogtemp.49680" : No space left on device
- 15. Out of 2a Disk Space Archiving Edition
- 16. 1. set up transaction log archiving to a standby 2. stop file use/deletion on the standby 3. don't monitor
- 17. 4. disk on the standby fills up with archive logs 5. disk on the master fills up with archive logs 6. failover impossible
- 18. freeing up space takes more space ● VACUUM FULL ● REINDEX ● DELETE ● pg_reorg
- 20. 3 Deleting Stuff
- 23. 2012-09-25 08:27:03 UTC ERROR: could not access status of transaction 500185903 2012-09-25 08:27:03 UTC DETAIL: Could not open file "pg_clog/01DD": No such file or directory.
- 24. 4 Out of Memory
- 25. 1. set shared_buffers to 80% of RAM 2. set work_mem to 1GB 3. ramp up to 1000 connections
- 26. 1. run Postgres on cloud server 2. add a JVM 3. add Apache/mod_php 4. add MySQL 5. add a second Postgres version
- 27. 0. run server with default Linux memory management settings
- 28. kernel: postgres invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 LOG: background writer process (PID 22999) was terminated by signal 9: Killed LOG: terminating any other active server processes
- 29. 5 Bad Hardware
- 30. 1. buy brand-new hardware 2. install PostgreSQL 3. don't test 4. deploy in production
- 31. 1. deploy production hardware 2. don't monitor SMART 3. don't monitor syslog 4. run for a couple years
- 32. ERROR: could not read block 1 in file "base/18511/13872": read only 0 of 8192 bytes FATAL: failed to re-find parent key in index "257969064" for split pages 8366/12375
- 33. Too 6 Many Connections
- 34. 1. build application with ad-hoc DB connections 2. no connection pooling 3. program auto-reconnect on timeout 4. raise max_connections to 2000
- 35. CRITICAL: server postgres2 is not accepting connections
- 36. 7 Zombie Locks
- 37. 1. program application so it forgets to close transactions 2. run for a few hours or days 3. watch the locks pile up
- 38. DBI ERROR timeout after 300 seconds: "UPDATE sessions SET last_seen = now()WHERE session_key = 'a16g32b12cd0';"
- 39. 1. no updates 2. out of disk space 3. deleting stuff 4. out of RAM 5. bad hardware 6. too many connections 7. zombie locks
- 41. more elephants ● Postgres Pizza BOF: – Wed Night, 8:30PM, B&G Snooker Hall th ● pgDay Melbourne: Feb 5 – 2013.pgday.org.au ● pgCon May 21-24 Ottawa ● me: www.databasesoup.com – josh@postgresql.org This talk is copyright 2013 Josh Berkus, and is licensed under the Creative Commons Attribution License. Most images are the property of their respective copyright holders, and are used here for purposes of satire.