a look at the postgresql engine
0 likes562 views
The document discusses PostgreSQL's internal architecture and components. It describes the data area, which stores data files on disk, and key directories like pg_xlog for write-ahead logs. It explains the buffer cache and clock sweep algorithm for managing memory, and covers the multi-version concurrency control (MVCC) which allows simultaneous transactions. TOAST storage is also summarized, which stores large data values externally.
a look at the postgresql engine
- 1. PostgreSQL: a look to the engine The Ravenous Bugblatter Beast of Traal Federico Campoli Brighton PostgreSQL Users Group 13 June 2016 Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 1 / 43
- 2. Table of contents 1 Don’t panic! 2 Infinite Improbability Drive 3 Time is an illusion. Lunchtime doubly so 4 The Pan Galactic Gargle Blaster 5 Mostly harmless Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 2 / 43
- 3. Don’t panic! Table of contents 1 Don’t panic! 2 Infinite Improbability Drive 3 Time is an illusion. Lunchtime doubly so 4 The Pan Galactic Gargle Blaster 5 Mostly harmless Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 3 / 43
- 4. Don’t panic! Don’t panic! Copyright by Kreg Steppe - https://www.flickr.com/photos/spyndle/ Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 4 / 43
- 5. Don’t panic! Don’t panic! Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 5 / 43
- 6. Infinite Improbability Drive Table of contents 1 Don’t panic! 2 Infinite Improbability Drive 3 Time is an illusion. Lunchtime doubly so 4 The Pan Galactic Gargle Blaster 5 Mostly harmless Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 6 / 43
- 7. Infinite Improbability Drive Infinite Improbability Drive Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 7 / 43
- 8. Infinite Improbability Drive The data area The PostgreSQL’s data area is the directory where the cluster stores the data on durable storage. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 8 / 43
- 9. Infinite Improbability Drive The directory base The default location when a new database is created without the TABLESPACE clause. Inside, for each database there is a folder with numeric names An optional folder pgsql tmp is used for the external sorts The base location is mapped as pg default in the pg tablespace system table Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 9 / 43
- 10. Infinite Improbability Drive The directory base Each database directory contains files with numeric names where postgres stores the relation’s data. The maximum size a data file can grow is 1 GB, a new file is generated with a numerical suffix Each data file is organised in fixed size pages of 8192 bytes The data files are called file nodes. Their relationship with the logical relations is stored in the pg class system table Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 10 / 43
- 11. Infinite Improbability Drive The directory pg global The directory pg global contains the data files used by the relations shared across the cluster. There is also a small 8kb file named pg control. This is a very critical file where PostgreSQL stores the cluster’s vital data like the last checkpoint location. A corrupted pg control prevents the cluster’s start. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 11 / 43
- 12. Infinite Improbability Drive The directory pg tblspc Contains the symbolic links to the tablespaces. Very useful to spread tables and indices on different physical devices Combined with the logical volume management can improve dramatically the performance... or drive the project to a complete failure The objects tablespace location can be safely changed but this require an exclusive lock on the affected object the view pg tablespace maps the objects name and identifiers Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 12 / 43
- 13. Infinite Improbability Drive The directory pg xlog The write ahead logs are stored in this directory Is probably the most important and critical directory in the cluster Each WAL segment is 16 Mb Each segment contains the records describing the tuple changed in the volatile memory In case of crash or an unclean shutdown the WAL are replayed to restore the cluster’s consistent state The number of segments is automatically managed by the database Putting the location on a dedicated and high reliable device is vital for performance and reliability Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 13 / 43
- 14. Infinite Improbability Drive Data pages Each block is structured almost the same, for tables and indices. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 14 / 43
- 15. Infinite Improbability Drive Page header Each page starts with a 24 bytes header followed by the tuple pointers. Those are usually 4 bytes each and point the physical tuples are stored in the page’s end Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 15 / 43
- 16. Infinite Improbability Drive The tuples Now finally we can look to the physical tuples. For each tuple there is a 27 bytes header. The numbers are the bytes used by the single values. The user data can be either the data stream or a pointer to the out of line data stream. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 16 / 43
- 17. Infinite Improbability Drive TOAST with Marmite please Copyright by David Martyn Hunt TOAST, the best thing since sliced bread TOAST is the acronym for The Oversized Attribute Storage Technique The attribute is also known as field The TOAST can store up to 1 GB in the out of line storage (and free of charge) Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 17 / 43
- 18. Infinite Improbability Drive TOAST with Marmite please Fixed length data types like integer, date, timestamp do not are not TOASTable. The data is stored after the tuple header. Varlena data types as character varying without the upper bound, text or bytea are stored in line or out of line. The storage technique used depends from the data stream size, and the storage method assigned to the attribute. Depending from the storage strategy is possible to store the data in external relations and/or compressed using the fast zlib algorithm. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 18 / 43
- 19. Infinite Improbability Drive TOAST with Marmite please TOAST permits four storage strategies (shamelessly copied from the on line manual). PLAIN prevents either compression or out-of-line storage; This is the only possible strategy for columns of non-TOAST-able data types. EXTENDED allows both compression and out-of-line storage. This is the default for most TOAST-able data types. Compression will be attempted first, then out-of-line storage if the row is still too big. EXTERNAL allows out-of-line storage but not compression. Use of EXTERNAL will make substring operations on wide text and bytea columns faster at the penalty of increased storage space. MAIN allows compression but not out-of-line storage. Actually, out-of-line storage will still be performed for such columns, but only as a last resort. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 19 / 43
- 20. Infinite Improbability Drive TOAST with Marmite please When the out of line storage is used the data is encoded in bytea and eventually split in multiple chunks. An unique index over the chunk id and chunk seq avoid either duplicate data and speed up the lookups Figure : Toast table Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 20 / 43
- 21. Time is an illusion. Lunchtime doubly so Table of contents 1 Don’t panic! 2 Infinite Improbability Drive 3 Time is an illusion. Lunchtime doubly so 4 The Pan Galactic Gargle Blaster 5 Mostly harmless Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 21 / 43
- 22. Time is an illusion. Lunchtime doubly so Time is an illusion. Lunchtime doubly so Copyright by Federico Campoli Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 22 / 43
- 23. Time is an illusion. Lunchtime doubly so The magic of the MVCC t xmin contains the xid generated at tuple insert t xmax contains the xid generated at tuple delete t cid contains the internal command id to track the sequence inside the same transaction Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 23 / 43
- 24. Time is an illusion. Lunchtime doubly so The magic of the MVCC The PostgreSQL’s consistency is achieved using MVCC which stands for Multi Version Concurrency Control. The base logic seems simple. A 4 byte unsigned integer called xid is incremented by 1 and assigned to the current transaction. Every committed xid which value is lesser than the current xid is considered in the past and then visible to the current transaction. Every xid which value is greater than the current xid is in the future and then invisible to the current transaction. The commit status is managed in the $PGDATA using the directory pg clog where small 8k files tracks the transaction statuses. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 24 / 43
- 25. Time is an illusion. Lunchtime doubly so The magic of the MVCC In this model there is no UPDATE field. Every time a row is updated it simply generates a new version with the field t xmin set to the current XID value. The old row is marked dead just by writing the same XID in the t xmax. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 25 / 43
- 26. The Pan Galactic Gargle Blaster Table of contents 1 Don’t panic! 2 Infinite Improbability Drive 3 Time is an illusion. Lunchtime doubly so 4 The Pan Galactic Gargle Blaster 5 Mostly harmless Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 26 / 43
- 27. The Pan Galactic Gargle Blaster The Pan Galactic Gargle Blaster Copyright by Federico Campoli Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 27 / 43
- 28. The Pan Galactic Gargle Blaster A little history Back in the days, when the world was young, PostgreSQL memory was managed by a simple MRU algorithm. The new 8.x development cycle introduced a powerful algorithm called Adaptive Replacement Cache (ARC) where two self adapting memory pools managed the most recently used and most frequently used buffers. Because a software patent on the algorithm shortly after the release 8.0 the buffer manager was replaced by the two queue algorithm. The release 8.1 adopted the clock sweep memory manager still in use in the latest version because of its flexibility. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 28 / 43
- 29. The Pan Galactic Gargle Blaster The clock sweep The buffer manager’s main goal is to keep cached in memory the most recently used blocks and adapt dynamically for the most frequently used blocks. To do this a small memory portion is used as free list for the buffers available for memory eviction. Figure : Free list Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 29 / 43
- 30. The Pan Galactic Gargle Blaster The clock sweep The buffers have a reference counter which increase by one when the buffer is pinned, up to a small value. Figure : Block usage counter Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 30 / 43
- 31. The Pan Galactic Gargle Blaster The clock sweep Shamelessly copied from the file src/backend/storage/buffer/README There is a ”free list” of buffers that are prime candidates for replacement. In particular, buffers that are completely free (contain no valid page) are always in this list. To choose a victim buffer to recycle when there are no free buffers available, we use a simple clock-sweep algorithm, which avoids the need to take system-wide locks during common operations. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 31 / 43
- 32. The Pan Galactic Gargle Blaster The clock sweep It works like this: Each buffer header contains a usage counter, which is incremented (up to a small limit value) whenever the buffer is pinned. (This requires only the buffer header spinlock, which would have to be taken anyway to increment the buffer reference count, so it’s nearly free.) The ”clock hand” is a buffer index, NextVictimBuffer, that moves circularly through all the available buffers. NextVictimBuffer is protected by the BufFreelistLock. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 32 / 43
- 33. The Pan Galactic Gargle Blaster It’s bigger in the inside The algorithm for a process that needs to obtain a victim buffer is: 1 Obtain BufFreelistLock. 2 If buffer free list is nonempty, remove its head buffer. If the buffer is pinned or has a nonzero usage count, it cannot be used; ignore it and return to the start of step 2. Otherwise, pin the buffer, release BufFreelistLock, and return the buffer. 3 Otherwise, select the buffer pointed to by NextVictimBuffer, and circularly advance NextVictimBuffer for next time. 4 If the selected buffer is pinned or has a nonzero usage count, it cannot be used. Decrement its usage count (if nonzero) and return to step 3 to examine the next buffer. 5 Pin the selected buffer, release BufFreelistLock, and return the buffer. (Note that if the selected buffer is dirty, we will have to write it out before we can recycle it; if someone else pins the buffer meanwhile we will have to give up and try another buffer. This however is not a concern of the basic select-a-victim-buffer algorithm.) Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 33 / 43
- 34. The Pan Galactic Gargle Blaster It’s bigger in the inside Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 34 / 43
- 35. The Pan Galactic Gargle Blaster It’s bigger in the inside Since the version 8.3 the buffer manager have the ring buffer strategy. Operations which require a large amount of buffers in memory, like VACUUM or large tables sequential scans, have a dedicated 256kb ring buffer, small enough to fit in the processor’s L2 cache. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 35 / 43
- 36. Mostly harmless Table of contents 1 Don’t panic! 2 Infinite Improbability Drive 3 Time is an illusion. Lunchtime doubly so 4 The Pan Galactic Gargle Blaster 5 Mostly harmless Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 36 / 43
- 37. Mostly harmless Mostly harmless Copyright by Federico Campoli Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 37 / 43
- 38. Mostly harmless The XID wraparound failure XID is a 4 byte unsigned integer. Every 4 billions transactions the value wraps PostgreSQL uses the modulo − 231 comparison method For each value 2 billions XID are in the future and 2 billions in the past When a xid’s age becomes too close to 2 billions VACUUM freezes the xmin value to an hardcoded xid forever in the past Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 38 / 43
- 39. Mostly harmless The XID wraparound failure If for any reason an xid reaches 10 millions transactions from the wraparound failure the database starts emitting scary messages WARNING: database "mydb" must be vacuumed within 5770099 transactions HINT: To avoid a database shutdown, execute a database-wide VACUUM in "mydb". If a xid’s age reaches 1 million transactions from the wraparound failure the database simply shut down and can be started only in single user mode to perform the VACUUM. Anyway, the autovacuum deamon, even if turned off starts the required VACUUM long before this catastrophic scenario happens. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 39 / 43
- 40. Mostly harmless Questions? Questions? Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 40 / 43
- 41. Mostly harmless Boring legal stuff All the images copyright is owned by the respective authors. The sources the author’s attribution is provided with a link alongside with image. The section’s titles are quotes from Douglas Adams hitchhiker’s guide to the galaxy. No copyright infringement is intended. Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 41 / 43
- 42. Mostly harmless Contacts and license Twitter: 4thdoctor scarf Blog:http://www.pgdba.co.uk Brighton PostgreSQL Meetup: http://www.meetup.com/Brighton-PostgreSQL-Meetup/ This document is distributed under the terms of the Creative Commons Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 42 / 43
- 43. Mostly harmless PostgreSQL: a look to the engine The Ravenous Bugblatter Beast of Traal Federico Campoli Brighton PostgreSQL Users Group 13 June 2016 Federico Campoli (Brighton PostgreSQL Users Group) PostgreSQL: a look to the engine 13 June 2016 43 / 43