CPU Caches

Editor's Notes

• #5: To improve performance, we need to understand the impact of our decisions in the physical world My goal is help everyone understand how each physical level of caching within a CPU works Capped the talk below virtual memory (memory management for multitasking kernels), as that is specific to the operating system running on the machine Not enough time to cover all of the levels of caching that exist on CPUs
• #7: 2011 Sandy Bridge and AMD Fusion integrated Northbridge functions into CPUs, along with processor cores, memory controller and graphics processing unit Gil Tene of Azul claims the Vega architecture is a true SMP compared to those of AMD/Intel – circular architecture with linkages between every core
• #8: NUMA architectures have more trouble migrating processes across sockets because L3 isn’t warmed Initially, it has to look for a core with the same speed accessing memory resources and enough resources for the process; if none are found, it then allows for degradation
• #9: All algorithms are dominated by that - you have to send data somewhere to have an operation performed on it, and then you have to send it back to where it can be used
• #10: Memory controllers were moved onto the processor die on AMD64 and Nehalem
• #11: Be careful about what's on them, because if a line holds multiple variables and the state of one changes, coherency must be maintained, killing performance for parallel threads on an SMP machine If the critical word in the line is in the last block, takes 30 cycles or more to arrive after the first word. However, the memory controller is free to request the blocks in a different order. The processor can specify the "critical word", and the block that word resides in can be retrieved first for the cache line by the memory controller. The program has the data it needs and can continue while the rest of the line is retrieved and the cache is not yet in a consistent state, called "Critical Word First & Early Restart” Note that with pre-fetching, the critical word is not known - if the core requests the cache line while the pre-fetching is in transit, it will not be able to influence the block ordering and will have to wait until the critical word arrives in order; position in the cache line matters, faster to be at the front than the rear
• #12: Replacement policy determines where in the cache an entry of main memory will go Tradeoff - if data can be mapped to multiple places, all of those places have to be checked to see if the data is there, costing power and possibly time; but more associativity means less cache misses Set associative has a fixed number of places and can be 2-16-way; skewed does as well, but uses a hash to place the second cache line in the group, only up to 2-way that I know of
• #13: LRU can be expensive, tracks each cache line with “age bits” - each time a cache line is used, all others have to be updated PLRU is used where LRU is prohibitively expensive – “almost always” discards one of the most LRU lines and no guarantee it was the absolutely LEAST recently used is sufficient 2WSA for caches where even PLRU is too slow – address of an item is used to put it in one of two positions, and the LRU of those two is discarded Direct mapped evicts whatever was in that position before it, regardless of when it was last used
• #14: Write through caching: slow, but predictable, resulting in lots of FSB traffic Write back caching: mark flag bit as dirty, when evicted the processor notes and sends cache line back to main memory, instead of just dropping it; have to be careful of false sharing and cache coherency Write combining caching: used by graphics cards, groups writes to main memory for speed Uncacheable: dynamic memory values that can change without warning, used for commodity hardware
• #15: In an exclusive cache (AMD), when the processor needs data it is loaded by cache line into L1d, which evicts a line to L2, which evicts a line to L3, which evicts a line to main memory; each eviction is progressively more expensive Exclusive is necessary if the L2 is less than 4-8x the size of L1, otherwise duplication of data affects L2’s local hit rate In an inclusive cache (Intel), eviction is much faster - if another processor wants to delete data, it only has to check L2 because it knows L1 will have it as well; exclusive will have to check both, which is more expensive; good for snooping
• #16: For intra-socket communication, Sandy Bridge has a 4-ring architecture linking cores, LLC and GPU; prior to Sandy Bridge, required direct linkages GT/s - how many times signals are sent per second Not a bus, but a point to point interface for connections between sockets in a NUMA platform, or from processors to I/O and peripheral controllers DDR = double data rate, two data transferred per clock cycle, Processors "snoop" on each other, and certain actions are announced on external pins to make changes visible to others; the address of the cache line is visible through the address bus Sandy Bridge can transfer on both the rise & fall of the clock cycle and is full duplex HTX could theoretically be up to 32 bits per transmission, but hasn't been done yet that I know of
• #17: Request for Ownership (RFO): a cache line is held exclusively for read by one processor, but another requests for write; cache line is sent, but not marked shared by the original holder - it is marked Invalid and the other cache becomes Exclusive. This marking happens in the memory controller. Performing this operation in the last level cache is expensive, as is the I->M transition. If a Shared cache line is modified, all other views of it must be marked invalid via an announcement RFO message. If Exclusive, no need to announce. Processors will try to keep cache lines in an exclusive state, as the E to M transition is much faster than S to M. RFOs occur when a thread is migrated to another processor and the cache lines have to be moved once, or when two threads absolutely must share a line; the costs are a bit smaller when shared across two cores on the same processor. Collisions on the bus can happen, latency can be high with NUMA systems, sheer traffic can slow things down - all reasons to minimize traffic. Since code almost never changes, instruction caches do not use MESI but rather just a SI protocol. If code does change, a lot of pessimistic assumptions have to be made by the controller.
• #18: The number of circuits per datum means that SRAM can never be dense.
• #19: One transistor, one capacitor Reading contiguous memory is faster than random access due to how you read - you get one write combining buffer at a time from each of the memory banks, 33% slower. 240 cycles to get data from here E7-8870 (Westmere) can hold 4TB of RAM, E5-2600 Sandy Bridge "only" 1TB DDR3-SDRAM: Double Data Rate, Synchronous Dynamic Has a high-bandwidth three-channel interface Reduces power consumption 30% over DDR2 Data is transferred on the rising and falling edges of a 400-1066 MHz I/O clock of the system DDR4 and 5 are coming, or are here with GPGPUs. DDR3L is for lower energy usage
• #21: Westmere was the 32nm die shrink of Nehalem from 45nm. Ivy Bridge uses 22nm, but the Sandy Bridge architecture I'm ignoring Oracle SPARC here, but note that Oracle is supposedly building a 16K core UltraSPARC 64 Current Intel top of the line is Xeon E7-8870 (8 cores per socket, 32MB L3, 4TB RAM), but it's a Westmere-based microarchitecture and doesn’t have the Sandy Bridge improvements The E5-2600 Romley is the top shelf new Sandy Bridge offering - it's specs don't sound as mighty (8 cores per socket, max 30MB L3, 1TB RAM). Don't be fooled, the Direct Data IO makes up for it as disks network cards can do DMA (Direct Memory Access) from L3, not RAM, decreasing latency by ~18%. Also has two memory read ports, whereas previous architectures only had one and were a bottleneck for math-intensive applications Sandy Bridge is 4-30MB (8MB is currently max for mobile platforms, even with Ivy Bridge), includes the processor graphics
• #23: The problem with these numbers is that while they're not far off, the lowest level cache interactions (registers, store buffers, L0, L1) are truly measured in cycles, not time Clock cycles are also known as Fetch and Execute or Fetch-Decode-Execute Cycle (FDX), retrieve instruction from memory, determine actions required and carry them out; ~1/3 of a nanosecond
• #24: Sandy Bridge has 2 load/store operations for each memory channel Low values are because the pre-fetcher is hiding real latency in these tests when “striding” memory
• #25: Types of registers include: instruction data (floating point, integer, chars, small bit arrays, etc) address conditional constant etc.
• #26: Store buffers disambiguate memory access and manage dependencies for instructions occurring out of program order If CPU 1 needs to write to cache line A but doesn't have the line in its L1, it can drop the write in the store buffer while the memory controller fetches it to avoid a stall; if line is already in L1 it can store directly to it (and bypass store buffer), possibly causing invalidate requests to be sent out
• #27: Macro ops have to be decoded into micro ops to be handled by the CPU Nehalem used L1i for macro ops decoding and had a copy of every operand required, but Sandy Bridge caches the uops (Micro-operations), boosting performance in tight code (small, highly profiled and optimized, can be secure, testable, easy to migrate) Not the same as the older "trace" cache, which stored uops in the order in which they were executed, which meant lots of potential duplication; no duplication in the L0, storing only unique decode instructions Hot loops (where the program spends the majority of its time) should be sized to fit here
• #28: Instructions are generated by the compiler, which knows rules for good code generation Code has predictable patterns and CPUs are good at recognizing them, which helps pre-fetching Spatial and temporal locality is good Nehalem could load 128 bits per cycle, but Sandy Bridge can load double that because load and store address units can be interchanged - thus using 2 128-bit units per cycle instead of one
• #29: When the working set size random access is greater than the L2 size, cache misses start to grow Caches from L2 up are "unified", having both instructions and data (not in terms of shared across cores)
• #30: Sandy Bridge allows only exclusive access per core Last Level Cache (LLC)
• #32: Doesn't have to be contiguous in an array, you can be jumping in 2K chunks without a performance hit As long as it's predictable, it will fetch the memory before you need it and have it staged Pre-fetching happens with L1d, can happen for L2 in systems with long pipelines Hardware prefetching cannot cross page boundaries, which slows it down as the working set size increases. If it did and the page wasn't there or is invalid, the OS would have to get involved and the program would experience a page fault it didn't initiate itself Temporally relevant data may be evicted fetching a line that will not be used
• #33: The pre-fetcher will bring data into L1d for you. The simpler your code, the better it can do this If your code is complex, it will do it wrong, costing a cache miss and forcing it to lose the value of pre-fetching and having to go out to an outer layer to get that instruction again Compulsory/Capacity/Conflict are miss types
• #34: Short lived data is not cheap, but variables scoped within a method running on the JVM are stack allocated and very fast Think about the affect of contending locking. You've got a warmed cache with all of the data you need, and because of contention (arbitrated at the kernel level), the thread will be put to sleep and your cached data is sitting until you can gain the lock from the arbitrator. Due to LRU, it could be evicted. The kernel is general purpose, may decide to do some housekeeping like defragging some memory, futher polluting your cache. When your thread finally does gain the lock, it may end up running on an entirely different core and will have to rewarm its cache. Everything you do will be a cache miss until its warm again. CAS is better, checking the value before you replace it with a new value. If it's different, re-read and try again. Happens in user space, not at the kernel, all on thread Algos get a lot harder, though - state machines with many more steps and complexity. And there is still a non-negligible cost. Remember the cycle time diffs for different cache sizes; if the workload can be tailored to the size of the last level cache, performance can be dramatically improved Note that for large data sets, you may have to be oblivious to caching
• #35: Eventually, caches have to evict, cache misses cost hundreds of cycles
• #36: Hyperthreading shares all CPU resources except the register set Intel CPUs limit to two threads per core, allows one hyper thread to access resources (like the arithmetic logic unit) while another hyper thread is delayed, usually by memory access. Only more efficient if the combined runtime is lower than a single thread, possible by overlapping wait times for different memory access that would ordinarily be sequential. The only variable is the number of cache hits Note that the effectively shared L1d & L2 reduce the available caches and bandwidth for each thread to 50% when executing two completely different threads of execution (questionable unless the caches are very large), inducing more cache misses, therefore hyper threading is only useful in a limited set of situations. Can be good for debugging with SMT (simultaneous multi-threading) Contrast this with Bulldozer's "modules" (akin to two cores), which has dedicated schedulers and integer units for each thread in a single processor
• #37: If n is not very large, an array will beat it for performance Linked lists and trees have pointer chasing which are bad for striding across 2K cache pre-fetching Java's hashmap uses chained buckets, where each hash's bucket is a linked list Clojure/Scala vectors are good, because they have groupings of contiguous memory in use, but do not require all data to be contiguous like Java's ArrayList Cliff Click has his lock-free library, Boundary has added with their own implementations Fastutil is additive, no removal, but that's not necessarily a bad thing with proper tombstoning and data cleanup
• #38: We can have as much RAM/heap space as we want now. And requirements for RAM grow at about 100x per decade. But are we now bound by GC? You can get 100GB of heap, but how long do you pause for marking/remarking phases and compaction? Even on a 2-4 GB heap, you're going to get multi-second pauses - when and how often? IBM Metronome collector is very predictable. Azul around one millisecond for lots of garbage with C4.
• #39: Graphics card processing (OpenCL) is great for specific kinds of operations, like floating point arithmetic. But they don't perform well for all operations, and you have to get the data to them.
• #41: Number of cores is large enough that traditional multi-processor techniques are no longer efficient[citation needed] — largely because of issues with congestion in supplying instructions and data to the many processors. Network-on-chip technology may be advantageous above this threshold
• #42: From HP, due to come out this year or next, may change memory fundamentally. If the data and the function are together, you get the highest throughput and lowest latency possible. Could replace DRAM and disk entirely. When current flows in one direction through the device, the electrical resistance increases; and when current flows in the opposite direction, the resistance decreases. When the current is stopped, the component retains the last resistance that it had, and when the flow of charge starts again, the resistance of the circuit will be what it was when it was last active. Is write endurance good enough for anything but storage?
• #43: Thermally-driven phase change, not an electronic process. Faster because it doesn't need to erase a block of cells before writing. Flash degrades at 5000 writes per sector, where PRAM lasts 100 million writes. Some argue that PRAM should be considered a memristor as well.