NVMe_Infrastructure_final1.pdf
- 1. Architected for Performance NVM ExpressTM Infrastructure - Exploring Data Center PCIe® Topologies January 29, 2015 Jonmichael Hands – Product Marketing Manager, Intel Non-Volatile Memory Solutions Group Peter Onufryk – Sr. Director Product Development – PMC-Sierra Moderator: Ravi Chari – Storage Technologist – Oracle View recorded webcast at https://www.brighttalk.com/webcast/12367/141221
- 2. 2 Legal Disclaimer All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. No computer system can be absolutely secure. Intel does not assume any liability for lost or stolen data or systems or any damages resulting from such losses. Cost reduction scenarios described are intended as examples of how a given Intel- based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. *Other names and brands may be claimed as the property of others. Copyright © 2015 Intel Corporation. All rights reserved.
- 3. 3 Agenda - NVM ExpressTM(NVMeTM) Infrastructure • What is NVMe? • NVMe advantages over SATATM • NVMe driver ecosystem • PCIe® form factors, cables, and connectors • Link extension and port expansion for PCIe • PCIe Solid-State Drive Topologies • NVMe Management
- 4. 4 Whatis NVM ExpressTM is a standardized high performance software interface for PCI Express® Solid-State Drives Architected from the ground up for SSDs to be more efficient, scalable, and manageable NVMe is industry driven to be extensible for the needs of both the client and the data center ? If I had asked people what they wanted, they would have said faster horses - Henry Ford “ ”
- 5. 5 NVM ExpressTM Community Promoter Group Led by 13 elected companies NVM Express, Inc. Consists of more than 75 companies from across the industry Technical Workgroup Queuing interface, NVMe I/O and Admin command set Management Interface Workgroup Out-of-band management over PCIe® VDM and SMBus
- 6. 6 Deployment at scale Industry standard drivers, software, and management What NVM ExpressTM brings to the DATACENTER Works out of the box In standard operating systems Lower TCO Efficiency of protocol, increased storage density, lower system power
- 7. 7 NVM ExpressTM(NVMeTM) Advantages over SATATM PCIe® for scalable performance, flexible form factors, and industry stability Increased bandwidth: 1 GB/s per lane – 1-16 lanes per drive Directly attached to CPU, eliminate HBA cost and overhead NVMe provides lower latency and increased efficiency: lower CPU utilization, lower power, lower TCO Low power features from both PCIe and NVMe Security from Trusted Computing Group OPAL Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Configurations: Intel® S2600CP server, Intel® Xeon® E5-2690v2 x2, 64GB DDR3, Intel® SSD DC P3700 Series 400GB, LSI 9207-8i, Intel® SSD DC S3700, HGST 6GBps SAS
- 8. 8 NVMeTM Driver Ecosystem Native / in-box 6.5 | 7.0 SLES 11 SP3 SLES 12 Install NVMe driver ESXi 5.5 13 | 14 Windows 8.1 Linux NVMe driver is open source *Other names and brands may be claimed as the property of others.
- 9. 9 What do I need to start using an SSD? Software: NVMeTM driver Hardware: PCIe® infrastructure NVMe sits on top of PCIe
- 10. 10 Form Factors for PCI Express® Data Center Client SFF-8639 SATA Express AIC 2.5in SFF-8639 SATA ExpressTM M.2 Add in Card M.2 BGA HD SSD FF
- 11. 11 Drive Connectors SATA Signal Pins Power and Control Pins SATATM • Keyed only for SATA drives • Separate power and data Key Signal Pins (port A) Power and Control Pins Signal Pins (Dual port, B) SAS® • Backwards compatible with SATA • Dual port SFF-8639 • Supports SATA, SAS, and PCIe® x4 or two x2 • PCIe data, reference clock, and side band SAS / SATA Power and Control Pins SAS RefClk 0 & Lane 0 Lanes 1-3, SMBus, & Dual Port Enable Refclk 1, 3.3V Aux, & Resets
- 12. 12 SATA ExpressTM and SFF-8639 Comparison SATAe SFF-8639 SATA / SAS® SATA SATA / SAS PCI Express® x2 x4 or dual x2 Host Mux Yes No Ref Clock Optional Required EMI SRIS Shielding Height 7mm 15mm Max Performance 2 GB/s 4 GB/s Bottom Line Flexibility & Cost Performance Source: Seagate* (with permission) SFF-8639 designed for data center, SATAe designed for Client
- 13. 13 M.2 Form Factor Comparison Host Socket 2 Host Socket 3 Device w/ B&M Slots M.2 Socket 2 M.2 Socket 3 SATA Yes, Shared Yes, Shared PCIe® x2 PCIe x4 No Yes Comms Support Yes No Ref Clock Required Required Max Performance 2 GB/s 4 GB/s Bottom Line Flexibility Performance M.2 Socket 3 is the best option for Data Center PCIe SSDs
- 14. 14 Cabling Options for Data Center PCIe® SSD Topologies Reference Clock PCIe Reset SMBUS miniSAS HD cables lightly modified for PCIe are being used due to the robust connector and high volume manufacturing. Reference Clock
- 15. 15 Basic PCI Express® SSD Topology – 1 Connector • SFF-8639 Connector directly attached to board • Mostly used in small form factors such as compute node, blade, etc. 1
- 16. 16 Basic PCI Express® SSD Topology – 2 Connector 1 2 miniSAS HD Connector PCIe® Cable PCIe 3.0 x4 Enterprise SSD SFF-8639 Connector External Power
- 17. 17 Basic PCI Express® SSD Topology – 3 Connector Motherboard 1 miniSAS HD Connector PCIe® Cable miniSAS HD Connector 3 Backplane SSD Drive Carrier SFF-8639 Connector 2
- 18. 18 Port Expansion Devices - Switches Use Switches to expand number of PCIe® SSDs Switch PCIe 3.0 x8 link PCIe SSD x4 link Intel CPU PCIe SSD x4 link PCIe SSD x4 link PCIe SSD x4 link Port A Port B Port C Port D
- 19. 19 Link Extension Devices – Switches and Retimers Use Link Extension Devices for longer topologies Retimer PCIe 3.0 x4 link PCIe SSD x4 link Switch PCIe SSD x4 link Intel CPU x4 link Port A Port B PCIe® 3.0 x4 link
- 20. 20 PCI Express® (PCIe®) Switches and Retimers PCIe Switches • Use for link extension and/or port expansion • Hot-plug and error isolation • High performance peer-to-peer transfers • Extra software features Retimers • Mostly transparent to software • Retimers should be more common in PCIe 4.0 Link Extension Devices • Use when channel has > -20db loss: at 8GT/s PCIe 3.0 Retimer vs. Re-driver • Repeater: A Retimer or a Re-driver • Re-driver: Analog and not protocol aware Retimer: Physical Layer protocol aware, software transparent, Extension Device. Forms two separate electrical sub-links. Executes equalization procedure on each sub-link Recommend using only switches or retimers for link extension of PCIe
- 21. 21 High Function Switches NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD PCIe Switch NVMe SSD NVMe SSD PCIe Switch Host Host Inerconnect PCIe PCIe
- 22. 22 Complex PCI Express® Topology – 4 Connector PCIe x16 slot PCIe® Cable Cabled Add in card with Link Extension miniSAS HD for PCIe Backplane SSD Drive Carrier 1 2 3 4 SFF-8639 Connector
- 23. 23 Complex PCI Express® Topology – 5 Connector PCIe x16 slot PCIe Cable Cabled Add in card with Link Extension miniSAS HD for PCIe Backplane SSD Drive Carrier 1 2 4 SFF-8639 Connector PCIe® x16 Riser 3 5
- 24. 24 PCI Express® cabling for future topologies - OCuLink* Category OCuLink Standard Based PCI-SIG® PCIe® Lanes X4 Layout Smaller footprint Signal Integrity Similar on loss dominated channels PCIe 4.0 ready 16GT/s target Clock, power Supports SRIS and 3.3/5V power Production Availability Mid 2015 12.85mm 2.83mm Source: OCuLink internal cables and connectors
- 25. 25 OCuLink* Provides Flexible Data Center Topologies Board to board connections Cabled add in card Backplane SFF-8639 Connector PCIe® SSD
- 26. 26 NVMeTM Storage Device Management Server Caching Server Storage External Storage Root Complex PCIe/PCIe RAID NVMe NVMe NVMe x16 x4 Root Complex NVMe PCIe Switch NVMe NVMe NVMe x16 x4 Controller A Controller B Root Complex PCIe Switch x16 Root Complex PCIe Switch x16 SAS SAS NVMe NVMe NVMe NVMe SAS HDD Example Pre-boot Management Inventory, Power Budgeting, Configuration, Firmware Update Example Out-of-Band Management During System Operation Health Monitoring, Power/Thermal Management, Firmware Update, Configuration
- 27. 27 Driver vs. Out-of-Band Management NVM Subsystem NVMe SSD PCIe Port SMBus/I2C NVMe Driver Applications Management Controller Operating System Platform Management PCIe PCIe VDM SMBus/I2C
- 28. 28 Management Interface Protocol Layering Management Applications (e.g., Remote Console) SMBus/I2C PCIe MCTP over SMBus/I2C Binding MCTP over PCIe VDM Binding ManagementComponent Transport Protocol (MCTP) NVMe Management Interface ManagementController (BMC or HostProcessor) ManagementApplications (e.g., Remote Console) Physical Layer Transport Layer Protocol Layer Application Layer NVMe SSD
- 30. 30 NVMeTM Technical Overview • Supports deep queues of 64K commands per queue, up to 64K queues • Supports MSI-X and interrupt steering, enables even performance scaling • Streamlined & simple command set (13 required commands), optional features to address target segments • Built for the future, ready for next gen NVM
- 31. 31 Fully Exploiting Next Gen NVM With Next Gen NVM, the NVM is no longer the bottleneck App to SSD read latency for 4KB transfer at Queue Depth of 1 0 10 20 30 40 50 60 70 80 90 100 110 120 Future NVM PCIe x4 Gen3 NAND MLC PCIe x4 Gen3 ONFI3 NAND MLC SATA 3 ONFI3 NAND MLC SATA 3 ONFI2 us NVM Tread NVM xfer Misc SSD Link Xfer Platform + adapter Software
- 32. 32 NVMeTM Development History 2011 2012 2013 2014 • Host Memory Buffer • Replay Protected Area • Active/Idle Power and RTD3 • Temperature Thresholds • Namespace Management • Controller Memory Buffer • Live Firmware Update • Atomicity Enhancements NVMe 1.2 – Q4 2014 • Multi-Path IO • Namespace Sharing • Reservations • Autonomous Power Transition • Scatter Gather Lists NVMe 1.1 – Oct 2012 2015 • Queuing Interface • Command Set • End-to-End Protection • Security • PRPs NVMe 1.0 – Mar 2011