Ceph Day Taipei - Accelerate Ceph via SPDK

Editor's Notes

• #12: Encryption is for maintaining data confidentiality and requires the use of a key (kept secret) in order to return to plaintext. Hashing is for validating the integrity of content by detecting all modification thereof via obvious changes to the hash output. CRC are good for preventing random errors in transmission but provide little protection from an intentional attack on your data RAID: provides parity function EC: Provides Encode /decode
• #13: Summary of DPDK as background (this is NOT a description of SPDK): If you look at a storage system, you have a wire that goes into the box. That wire has a NIC driver. The NIC driver comes from DPDK. It is a very tailored driver that runs in user space and does what is called “polling”. DPDK does not do an interrupt DPDK is very fast compared to traditional operating systems DPDK runs in user space. Not kernel space. That is relevant because kernel space is really close to the hardware and that is where the drivers run. You have extra privileges and you can mess up your system because you have all those privileges. Use space is a little more protected. You don’t have as many privileges. When you transition from user space to kernel space, there is a context switch (just like in an interrupt as described earlier) For every I/O operation you do, there are many more user space to kernel transitions than there are interrupts. You are doing the transition from user to kernel hundreds of thousands of times per second also. This is very painful in terms of consuming CPU resources when you look at this as a whole. A lot of what DPDK does is to get rid of those painful context switches. DPDK is also software that does not need “locks”. In a general purpose operating system, it has to be set up to handle anything at any time for any application that comes in over the wire. That means that there is a lot of synchronization that has to occur between cores and the threads. That synchronization is also very expensive. You get rid of that problem by creating software that is essentially lockless. This allows you to better scale your software – every core you add can increase your performance linearly. Adding a second core could double your performance, tripling your cores could triple your performance.   SPDK takes these concepts and applies them to storage with iSCSI and NVMe. If you take all these things together, you end up with a specific instance of a storage system. Looking at the data flow. Bits come in from the left, go through the NIC, the DPDK NIC driver is running in user space and is polling. It takes the bits and passes it to a TCP/IP stack. In this case, user-mode network services (UNS). The TCP/IP stack takes those bits and does a lot of work to make sure those bits were intended for you and properly formed. That goes into an iSCSI target. That speaks block-based language of SCSI. The user storage app could look at that SCSI request and decide how to service it. It could be a SCSI read request, a SCSI write request, or a SCSI inquiry request trying to find out what the device is capable of, etc. The storage app is where our customers come in. They have the opportunity to provide their value add differentiation services like in-line de-duplication, erasure code, compression, hashing, etc. After the user storage app completes its operation, it is going to want to persist the data somewhere. In this case, we use NVMe. NVMe is the Intel prescribed way of doing PCIe-based SSDs. NVMe is an open standard so we can create a driver that can be open sourced. From there, the data gets stored. WKB is currently under development. It will run on Linux or FreeBSD. Targeting release of WKB in 2015. WKB is also a good system-level vehicle for demonstrating ISA-L. This allows us to provide real-world performance numbers for ISA-L. Previously, all we could show is cycles per byte for the algorithm.
• #19: Not the easiest charts to read. For the left chart, the lines are latency, the bars are throughput. The key take away is that even at reasonably high queue depths (16) the SPDK stack delivers sub-millisecond latency at 500K IOPS using just two cores. On the right chart, the bars are the same, but the lines are looking at the number of cores consumed – this just highlights the differences in efficiency between similar levels of performance. The dark blue bar (indicating LIO + kernel TCP/IP) tracks along with the SPDK performance, but the line shows how many more cores are required to keep up. Even at a queue depth of 1, SPDK is twice as efficient as the kernel stack, and that advantage grows as queue depth (and thus overall throughput) increases. Latency is measured from the initiator side. This latency measurement denotes “avg. latency” metric computed by FIO which is the sum of submission + completion latencies. All the latency measurements are in milliseconds (msecs). LIO 2 core and SPDK 2 core data were carried out by restricting the system to only 2 cores. # echo 0 > /sys/devices/system/cpu/cpu{}/online
• #23: This is where it would be good to have actual data to update this foil. Our demo running at the performance of the right most bar for 4-NVMe devices. Configuration details”   SPDK NVMf Target Component Details Hardware Intel Xeon-D   Processor D-1567 No. of Sockets 1 No. of Cores 12 Cores/ 12 Threads per socket Memory 32G NIC Mellanox Connectx-4 EN Adapter (25Gbps) – Dual Port MTU 1500 OS Fedora 23 Linux Kernel 4.4.3-300.fc23.x86_64 NVMf Target Intel SPDK   Storage Array Component Details Storage Drives 4x Intel SSD 800GB DC P3700series     NVMf Initiator Component Details No. of Initiators Dell PowerEdge 730xd Processor Intel Xeon Processor E5-2699v3 (45M Cache, 2.30 GHz) No. of Sockets 2 sockets populated No. of Cores 18 Cores/ 18 Threads per socket Memory 132G NIC Mellanox Connectx-4 EN Adapter (25Gbps) – Dual Port MTU 1500 OS RHEL 7.2 Linux Kernel 4.5.0-rc3