Designing Information Structures For Performance And Reliability

Editor's Notes

• #4: Keep relevant data closest to the CPU in memory once it has been read from disk. More memory reduces the need for costly “page-in” operations from disk by reducing the need to “page-out” data to make space for new data. Memory bus speed is still much slower than CPU bus speeds, often becoming a bottleneck as CPU speeds increase. It’s important to have the fastest memory speed and FSB that your chipset will support. More CPU cores allows you to parallelize workloads. A multithreaded database takes advantage of multi-processing bydistributing a query into several threads across multiple CPUs, drastically increasing the query’s efficiency while reducing its process time. Faster disks with high bandwidth and low seek times maximize read performance into memory for CPUs to process complex queries. OLAP databases benefit from this because they scan large datasets frequently. Using RAID allows you to aggregate disk I/O by striping data across several spindles, drastically decreasing the time it takes to read data into memory and write back onto the disks during commits, while also providing massive storage space, redundancy and fault-tolerance.
• #5: Set realistic goals, know the hardware’s expected limitations....Measure current performance....Analyze the results (research upgrades and possible performance problems)Modify the system..Benchmark again....Repeat as needed
• #6: Client issues a query across the network ....Database server searches cache and memory for database extents...if they’re not found in memory, they’re located on disk...Disks then seek out the blocks containing the database extents and begin loading the data into memory...Memory pages are then fed into CPU’s cache and ultimately into the CPU for processing...Results are found and sent back to the client
• #7: Each CPU has several coresInternal Clock Speed: processes per second in MHz or GHz (advertised)External Clock Speed: speed the FSB is accessed (typical bottleneck)Memory Clock Speed: speed at which RAM is given requests for data (another bottleneck)PostgreSQL is multi-process, one Unix process per DB connection.A single connection can only use on CPU, not multithreaded.
• #8: CPU speed has increased roughly 70% each year, memory speed hasn’t kept up.DDR Memory (double data rate) allows for sending data to the CPU at the top and bottom of the clock cycle (sine wave). Doubling throughput....still bottleneck.Memory tradeoff is typically speed for capacity, faster = less capacity/more expensiveFurther out from the CPU you go, the slower and greater capacity the storage. Also, disk is the only permanent storage....also holds swap.
• #9: 1st Generation Xeon multiprocessing bottlenecks....still bottlenecks today, but less so...Shared FSB between processors....halves bandwidth to memory....second processor competes for FSB bandwidth...Memory access delayed between controller and memory bankBandwidth between I/O controller (Southbridge) and memory controller (Northbridge) congested...as is bandwidth from Northbridge to the expansion slots.
• #10: Here you see how each Intel processor shares a common FSB bandwidth, dividing the bandwidth per CPU. Access to memory must be at the reduced bandwidth, through the northbridge memory controller, and into the memory banks.AMD’s approach places a Northbridge controller directly on each processor, so there’s no external chipsets to deal with. Each processor features three point-to-point HyperTransport links, delivering 3.2 GB/s of bandwidth in each direction (6.4GB/s full duplex). So AMD’s scalability was better in the earlier days of Xeon multiprocessing.
• #11: “Second Generation” (Harpertown) Xeon Processors E5200/E5400:==========================================Each CPU has a clock speed of 2GHz, 12MB of L2 cache, and a FSB of 1333MHz (1600MHz max on other models). The read bandwidth for each DDR2 667-MHz memory channel is 5.325 GB/s which gives a total read bandwidth of 21.3 GB/s for four memory channelsWrite memory bandwidth through the same four channels is 10.7 GB/s write memory bandwidth for the same four memory channels. Overall Effective bandwidth to memory is then 32 GB/s ... 21.3 GB/s read and 10.7 GB/s write.5500-series "Nehalem-EP" (Gainestown) adds: (December 2008)=================================Integrated memory controller supporting 2-3 DDR3 memory channelsPoint to Point processor interconnect called “QuickPath” (like AMD’s HyperTransport), bypassing FSBHyperthreading, doubling each core
• #12: There’s 3 delays associated with reading or writing to a hard drive:Seek Time, Rotational Delay, and Transfer TimeSeek Time is the time it takes for the drive’s read/write head to be physically moved into the correct place for the data being sought.Rotational Delay is the time required for the addressed area of the disk to rotate into a position where it is accessible by the read/write head….typically measured in milliseconds.Transfer Time is the time it takes to transfer data from the disk through the read/write head, across the storage bus, into memory for processing by the CPU.Seek Time/Rotational Delay is heavily influenced by the disk’s rotational speed (RPMs), data location on the actual platters, how many platters the disk has, and the diameter of the platters.Generally speaking, the faster a disk spins, the lower its seek times will be. Also, the further outside the circumference of the platter data is located, the faster it will be sought and lower it’s rotational delay will be.Bandwidth/Throughput (Transfer Time): Once data is located, this is the raw throughput rate at which data is transferred from disk into memory. This can be aggregated using RAID, which will be discussed later.SATA-I bandwidth is 1Gb/s which translates into ~ 150MB/s real speed.SATA-II and SAS bandwidth is 3Gb/s, which translates into ~ 300MB/s real speed.Generally speaking, the higher the data density of the platter, the more data will be sent through the read/write head per block…resulting in higher throughput and lower transfer times.
• #13: Buffer/Cache:Writeback-cacheData normally written to disk by the CPU is first written into disk’s the cache. This allows for higher write performance with the risk that data stored in cache isn’t flushed to disk before a power During idle machine cycles, the data are written from the cache into memory or onto disk. Write back caches improve performance, because a write to the high-speed cache is faster than to normal RAM or disk….this cache aids in Addressing the disk-to-memory subsystem bottleneck.I’ve enabled write-back caching on all of our RAID arrays. RAID will be discussed later.
• #14: 4. Track Data Density :Defines how much information can be stored on a given track. The higher the track data density, the more information the disk can store on one track. If a disk can store more data on one track it does not have to move the head to the next track as often. This means that the higher the recording density the lower the chances are that the head will have to be moved to the next track to get the required data.
• #15: RAID: (n = number of drives in array) “Redundant Array of Inexpensive Disks”. RAID systems improve performance by allowing the controller to exploit the capabilities of multiple hard disks to get around performance-limiting mechanical issues that plague individual hard disks. Different RAID implementations improve performance in different ways and to different degrees, but all improve it in some way. RAID0 “Striping” (n) : Fastest due to no parity…raw cumulative speed. Single drive failure causes the entire array to fail. “All-or-none” RAID1 “Mirroring” (n/2): Each drive is mirrored, speed and capacity is ½ of RAID0, requires even number of disks in order to be divided. Entire source or mirror array can go bad before data is jeopardized.RAID5 “Striping w/Parity” (n – 1): Fast, with a drive set aside for fault-tolerance. Only one drive can fail before the array is lost.RAID6 “Striping with dual Parity” (n -2): Fast, with 2 drives set aside for fault tolerance. Two drives can fail before the array is lost.
• #16: Normal PCI’s bandwidth is 132MB/sAGP8x is 2.1GB/sPCI Express outperforms PCI significantly:PCIe is bidirectional/full-duplex...allowing data to flow in both directions simultaneously (doubling throughput):PCIe 1x = 500MB/s (250MB/s each way)PCIe 2x = 1GB/s (500MB/s each way)PCIe 4x = 2GB/s (1GB/s each way)PCIe 8x = 4GB/s (2GB/s each way)PCIe 16x = 8GB/s (4/GB each way)Even PCIe 32x = 16GB/s (8GB/s each way)So, to open the Internally, you want to use PCIe, not just plain PCI which is old and slow in comparison.AGP is also obsolete due to PCIe’s introduction, now graphics cards use this interface as well.Regular PCI is a bottleneck in modern computers.All of our 2950 servers have PERC6/i RAID controllers built in, the “i” means “integrated’ on the motherboard.I found that our throughput was significantly slower than what we expected, despite having 6 SATA-II drives even in RAID0. The settings we selected for the RAID Virtual Drives were: Stripe Element Size 64KB Read Policy: Adaptive Read-Ahead (to optimize large read operations) Write Policy: Write BackWe were seeing read speeds in RAID5 of approximately 150-225MB/s across 4 drives..which we knew was way too slow given the hardware.After rebuilding the array several times and searching around on the Internet, I came across DELL’s PERC firmware update site, which showed that a newer release was available: v.6.2.0-0013“performance enhancements including significant improvements in random-write performance, multi-threaded write performance, and reduction in maximum and average I/O response times.”I couldn’t flash the PERC controllers without a floppy, so I had to create a Linux based FreeDOS bootable CD with the updated PERC firmware in a subdirectory, allowing me to successfully flash the controller’s BIOS. Later, I discovered that DELL’s OpenManage CD provides a tool to handle BIOS updates, however, I wasn’t able to get this working....so the FreeDOS solution worked out.I also dug around and found that I could set filesystem read-ahead parameters through “hdparm” in Linux that would allow me to tell the OS to read ahead 2048 blocks whenever a read operation was performed. I set this in /etc/rc.local to persist after a reboot.Once the PERC controller was flashed and linux filesystem readahead was set, performance increased dramatically:We’re now seeing just over 500MB/s reads in RAID5/6.This significantly reduces the time it takes to load tables into memory for complex queries, thereby reducing overall query execution time. Performance is now on par with GreenPlum without having to pay $40,000/year licensing.
• #18: Keep relevant data closest to the CPU in memory once it has been read from disk. More memory reduces the need for costly “page-in” operations from disk by reducing the need to “page-out” data to make space for new data. Memory bus speed is still much slower than CPU bus speeds, often becoming a bottleneck as CPU speeds increase. It’s important to have the fastest memory speed and FSB that your chipset will support. More CPU cores allows you to parallelize workloads. A multithreaded database takes advantage of multi-processing bydistributing a query into several threads across multiple CPUs, drastically increasing the query’s efficiency while reducing its process time. Faster disks with high bandwidth and low seek times maximize read performance into memory for CPUs to process complex queries. OLAP databases benefit from this because they scan large datasets frequently. Using RAID allows you to aggregate disk I/O by striping data across several spindles, drastically decreasing the time it takes to read data into memory and write back onto the disks during commits, while also providing massive storage space, redundancy and fault-tolerance.
• #19: Keep relevant data closest to the CPU in memory once it has been read from disk. More memory reduces the need for costly “page-in” operations from disk by reducing the need to “page-out” data to make space for new data. Memory bus speed is still much slower than CPU bus speeds, often becoming a bottleneck as CPU speeds increase. It’s important to have the fastest memory speed and FSB that your chipset will support. More CPU cores allows you to parallelize workloads. A multithreaded database takes advantage of multi-processing bydistributing a query into several threads across multiple CPUs, drastically increasing the query’s efficiency while reducing its process time. Faster disks with high bandwidth and low seek times maximize read performance into memory for CPUs to process complex queries. OLAP databases benefit from this because they scan large datasets frequently. Using RAID allows you to aggregate disk I/O by striping data across several spindles, drastically decreasing the time it takes to read data into memory and write back onto the disks during commits, while also providing massive storage space, redundancy and fault-tolerance.
• #20: Database Types:OLAP (Online Analytical Processing):Provides big picture, supports analysis, needs aggregate data, evaluates all datasets quickly, uses a multidimensional model.DB size is typically 100GB to several TB (even petabytes)Mostly read-only operations, lots of scans, complex queries.Benefits from multi-threading, parallel processing, and fast drives with highread throughput/low seek times.Key Performance Metrics: Query throughput/Response time.OLTP (Online Transactional Processing): Provides detailed audit, supports operations, needs detailed data, finds one dataset quickly, uses a relational model.DB size typically < 100GBShort, atomic transactions…read/write.Key Performance Metrics: Transaction Throughput, Availability
• #21: Database Types:The time and expense involved in retrieving answers from databases means that a lot of business intelligence information often goes unused. The reason: most operational databases (OLTP) are designed to store your data, not to help you analyze it. The solution: an online analytical processing (OLAP) database, a specialized database designed to help you extract business intelligence information from your data.
• #24: A connection from an application program to the PostgreSQL server has to be established.The parser stagechecks the query transmitted by the application program for correct syntax and creates a query tree. The rewrite systemtakes the query tree created by the parser stage and looks for any rules (stored in the system catalogs) to apply to the query tree. It performs the transformations given in the rule bodies. The planner/optimizer takes the (rewritten) query tree and creates a query planthat will be the input to the executor. It does so by first creating all possible paths leading to the same result. For example if there is an index on a relation to be scanned, there are two paths for the scan. One possibility is a simple sequential scan and the other possibility is to use the index. Next the cost for the execution of each path is estimated and the cheapest path is chosen. The executor recursively steps through the plan tree and retrieves rows in the way represented by the plan. The executor makes use ofthe storage systemwhile scanning relations, performs sorts and joins, evaluates qualifications and finally hands back the rows derived.
• #25: GreenPlum and PostgreSQL:We found that despite the claims above, GreenPlum was overpriced, slow, and problematic. Furthermore, our GreenPlum PSA database grew to exceed the hardware we had in place, requiring us to constantly have to manually delete old tables. To replace GreenPlum while maintaining the table structures already in place, we opted to go with PostgreSQL, aware that it’s not pre-optimized for OLAP/data-warehouse applications.The mindset in doing this was that we could tweak PostgreSQL to mimic the actual performance we saw from GreenPlum without having to pay an expensive license.Understanding how to tweak PostgreSQL to mimic the performance of GreenPlum requires an understanding of PostgreSQL query execution characteristics and its tweak file concepts.
• #26: Max_connections sets the maximum number of client connections per server. Several performance parameters use “max_connections” as part of their formula for tweaking Postgresql.Shared buffers: As the name implies, this is the maximum shared memory allowed to PostgreSQL. Too much and you risk paging.Working Memory:You need to consider what you set max_connections to in order to size this parameter correctly. This is a setting where data warehouse systems, where users are submitting very large queries, can readily make use of many gigabytes of memory. This size is applied to each and every sort done by each user, and complex queries can use multiple working memory sort buffers. Set it to 50MB, and have 30 users submitting queries, and you are soon using 1.5GB of real memory. Furthermore, if a query involves doing merge sorts of 8 tables, that requires 8 times work_mem.
• #27: Max_connections sets the maximum number of client connections per server. Several performance parameters use “max_connections” as part of their formula for tweaking Postgresql.Shared buffers: As the name implies, this is the maximum shared memory allowed to PostgreSQL. Too much and you risk paging.Working Memory:You need to consider what you set max_connections to in order to size this parameter correctly. This is a setting where data warehouse systems, where users are submitting very large queries, can readily make use of many gigabytes of memory. This size is applied to each and every sort done by each user, and complex queries can use multiple working memory sort buffers. Set it to 50MB, and have 30 users submitting queries, and you are soon using 1.5GB of real memory. Furthermore, if a query involves doing merge sorts of 8 tables, that requires 8 times work_mem.
• #29: Partitioning refers to splitting what is logically one large table into smaller physical pieces. Partitioning can provide several benefits: Query performance can be improved dramatically in certain situations, particularly when most of the heavily accessed rows of the table are in a single partition or a small number of partitions. The partitioning substitutes for leading columns of indexes, reducing index size and making it more likely that the heavily-used parts of the indexes fit in memory. When queries or updates access a large percentage of a single partition, performance can be improved by taking advantage of sequential scan of that partition instead of using an index and random access reads scattered across the whole table. Bulk loads and deletes can be accomplished by adding or removing partitions, if that requirement is planned into the partitioning design. ALTER TABLE is far faster than a bulk operation. It also entirely avoids the VACUUM overhead caused by a bulk DELETE. Seldom-used data can be migrated to cheaper and slower storage media.
• #31: Vacuuming ensures the databases remain ACID.... Atomic, Consistent, Isolated, and Durable.Atomicity: Guarantees that either all of the tasks of a transaction are performed or none.Consistency: Only valid data will ever be written to the database.Isolation: other operations cannot access or see the data in an intermediate state during a transaction.Durability: once the user has been notified of the transaction’s success, the transaction will persist and not be undone….thus surviving a system failure.
• #32: disk I/O was roughly 90MB/s Database volume was constantly around 90% capacity, causing Mike to have to manually delete tables…space was at a premium.Licensing with GreenPlum was expensive ($20,000/6 months….$40,000/yr.) made sense to leverage the underlying free open source code and scrap the proprietary distributed DB solution
• #33: Performance challenges with this server:Limited capacity, drives were small and slow (150MB/s).RAID controller’s SATA-1 interface didn’t recognize higher capacity SATA-II drives, despite SATA-II backwards compatibility.No floppy drive or USB boot capability…making it difficult to flash the controller and BIOS for SATA-II backwards compatibility. No PCIe expansion bays, only PCI-X, ruling out high-performance external enclosures.PostgreSQL requires significant configuration tweaks to realize decent performance.PostgreSQL isn’t multithreaded. A single query process (regardless of its complexity) uses only 1 CPU core.
• #34: Performance challenges with this server:This is our third (and current) generation PSA boxDELL PowerEdge 2950, with dual Xeon Quad Core processors @ 2.5GHz16GBDDR memory @ 667MHz bus speed (x2, DDR)1333MHz FSB6 SATA-II1TB7200RPM Drives, configured in a single 5TBRAID5 Array…1 drive can fail. Throughput is across 5 spindles…effective capacity is roughly 4.5TB.Drive Setup: Used PERC6/i BIOS Menu for RAID configuration (hardware RAID, OS transparent), battery backup is enabled for write-caching.Virtual Drive1 (Physical drives 1 and 2): RAID1 for system (1TB), 64KB Stripe Element Size, Write-Back enabledVirtual Drive2 (Physical drives 3,4,5,6): RAID5 for PostgreSQL data (3TB), 64KB Stripe Element Size, Write-Back enabledI/O Performance: Read:    507MB/sWrite:   401MB/s
• #35: Through process of elimination and online research (Google and Postgresql forums) we have setteled on the above settings in the PSA server’s configuration file:max_connections = 25Mike confirmed that only 10-15 PSA ever really connect at any given time, so this setting allows for spikes while remaining conservative enough to not inflate “work_mem” as work_mem uses max_connections in its memory allocation formula.shared_buffers = 4096MB This number comes from best practices, ¼ total physical memory (16GB)