Design and evaluation of an io controller for data protection

Editor's Notes

• #3: With increasing capacity, the rate of failures in storage devices increases as well. RAID technology masks device failures, assuming that the data is available in untouched condition. With the possibility of data corruption, other forms of data protection become necessary as well: Versioning, to recover from user errors Checksums, to protect against silent data corruption events These additional data protection techniques need persistent state (”metadata”), to be maintained & accessed at high I/O rates along the common I/O path.
• #6: It is very important to guarantee the constant overheads for allocation or de-allocation of buffer. The allocation of buffers for the I/O commands and their completions are different from other traditional buffers used by the controller. Command buffers allocation is done on the controller side, but are filled in by the host. Likewise, completion buffers are allocated on the host, but they are filled in by the controller.
• #7: I/O commands are services in stages, with several events in each stage (e.g: DMA start/completion, disk i/o start/completion) Option #1 is to design an event-driven FSM. Option #2 is to design a thread-based system, where each threads incorporates handling several of the many events. In our prototype, we took Option #2, using full Linux OS. This gives us a programmable infrastructure to build advanced storage features more easily (in line with our goals in the prototype). However, with more software layers we have less control over timing of events and interactions.
• #8: Error detection and correction (EDC) in DARC is done by using redundancy schemes (e.g. RAID-1), where the corrupted data are rebuilt using the parity or redundant data block copies. Redundancy schemes, such as RAID-1, 5, or 6, are commonly used for availability and reliability purposes in all installed storage systems that are supposed to handle disk failures. DARC also uses the same redundant data blocks to correct silent data errors, based on the supposition that the probability of data corruption on both copies of one disk block are very small. A similar hypothesis is made for other redundancy schemes, such as RAID-5/6, but the possibility of a valid data restoration gets high with the amount of redundancy preserved. The likelihood of a second data error occurring within the same group typically, 4-5 data blocks and 1-2 parity blocks in RAID-5/6 is higher for the two data block copies in RAID-10. The checksum capabilities of DMA engines in this platform suffers insignificant overhead when used during the DMA data transfers to/from the host. Therefore, checksums are used to protect host accessible data blocks. An advantage of only protecting host-accessible data blocks is that, RAID restoration, which is unaware of the existence of persistent checksums. Therefore, RAID restoration don’t need to be modified. Furthermore, the RAID restoration procedure can have benefit of the data structure storing the checksums to determine which blocks can be safely avoided, as they have not been written so far. This would reduce the RAID restoration time.
• #9: If the stored checksum does not match the computed, we proceed to check the mirror copies. This requires the capability to map the “virtual” block# to physical block locations, on the devices that make up the RAID-10 volume. After this map is made available, the mirror copies of the referenced block are retrieved and their corresponding checksums are computed. These checksums are then compared to one another. If they match, we update the stored checksum and proceed to complete the read request. Otherwise, if one of them matches the stored checksum, we proceed to re-issue the original read request, after synchronizing the mirror copies.
• #10: PIO  simple (load/store instructions in specific memory regions), but incur CPU overhead. Not suitable for large data volumes. DMA  high throughput (at PCIe rates), but complex to initiate and to detect their completion. I/O command consists of SCSI CDB & DMA segments, with host-side addresses. I/O completion consist of status-code & reference to corresponding commands.
• #11: Figure 5 demonstrates the how of I/O issue and completion from the host to the controller, and back. Commands are transferred from the host to the controller, using Programmed IO, in a circular first in first out (FIFO) queue The FIFO queues are statically allotted, and carry of fixed size elements. Commands, completions and other control information may consume multiple sequential queue elementsof the FIFO queues.
• #13: Typically, I/O controllers have an on-board cache. We have done some work in our prototype to incorporate a cache, and we will discuss the design issues involved. Cache on the controller serves 3 purposes: -temporal locality exploitation (if any) -spatial locality epxloitation, via read-ahead -coalescing of small writes Cache design intertwined with RAID implementation (distinction bet. Normal and Degraded-Mode operation)
• #14: A summary of the prototype design, as presented so far …
• #15: Overview of our controller prototype, which is built using real hardware: - SCSI-layer at Host & its co-ordination with the controller’s firmware. - The controller’s firmware contains several layers, as described earlier.
• #16: In summary, we have designed and implemented two data protection features within a controller: -persistent checksums for integrity protection -versioning of storage volumes, to protect against human errors. From our experiments, on real hardware, we have found the overhead of EDC checking to be in the range of 12 to 20%, depending on the number of concurrent I/Os. Versioning adds a further overhead, in the range of 2.5 to 5%, depending on the number & size of writes.