Troubleshooting_PCI_Express_Link_Training_and_Protocol_Issues_FROZEN.pdf

Copyright © 2015, PCI-SIG, All Rights Reserved 1
Troubleshooting PCI Express®
Link Training and Protocol Issues
Gordon Getty
Applications Engineer
Teledyne LeCroy

PCI-SIG Developers Conference
Disclaimer
Presentation Disclaimer: All opinions, judgments, recommendations,
etc. that are presented herein are the opinions of the presenter of the
material and do not necessarily reflect the opinions of the PCI-SIG®.

Agenda
 PCI Express® Layered Model
 Physical Layer Examples
 Power Management Examples
 Link Training Examples
 Equalization Examples
 M-PCIe™
 Data Link Layer Examples
 Flow Control Examples
 Transaction Layer Examples
 Completion Timeout
 PCIe® Storage Examples
 Different PCIe Storage Technologies
– NVMe
– PQI/SCSIe
– AHCI/SATA Express

PCI EXPRESS LAYERED
MODEL

PCI Express Layered Model
5
Data Link
Transaction
Physical
Application
Logical Sub Block
Electrical Sub Block
Software
– Ethernet, NVMe, SOP, AHCI, SATA…
Device Configuration and Control
Data Transfer to/from Memory
Management of Packets:
Flow Control and ACK/NAK Protocol
Electrical (Analog) Signalling
Link Training
Copyright © 2015, PCI-SIG, All Rights Reserved

What Can I See at Each Layer?
6
Data Link
Transaction
Physical
Application
Logical Sub Block
It is all inside the PCI Express Payload,
could include: NVMe commands,
Ethernet Frames etc.
TLPs: Config Rd/Wr, Mem Rd/Wr,
IO Rd/Wr, Messages
DLLPs: InitFC, UpdateFC, ACK, NAK
PM_Enter_L1 etc…..
Eye Diagrams, analog waveforms
TS1, TS2 all Ordered Sets

Physical
How do Two Devices Talk?
Data Link
Data Link
Transaction
Physical
Application Application
Transaction
Logical Sub Block
Logical Sub Block
RX
RX TX TX
Root Complex End Point

What about M-PCIe?
 “A PCI Express Link consists of a PCIe PHY as
defined in chapter 4 whereas the M-PCIe Link
consists of M-PHY, a physical layer specification
developed by MIPI Alliance.”

M-PCIe
Physical
Data Link
Data Link
Transaction
Physical
Application Application
Transaction
Link Training & Management
M-PHY
Link Training & Management
M-PHY

M-PCIe Trace Capture

M-PCIe Layers
 RRAP Protocol
 Link Training
 Flow Control Initialization and Transaction Layer

M-PCIe Form Factor
 M-PCIe may have different form factors
 M.2
 Embedded

M.2 Card Technology
13
 M.2 Card Format defines:
 Socket 2: SATA or x2 PCI Express fits modules with the “B”key for SSD,
cache
 Socket 3: x4 PCIe up to 4 GB/s fits modules with the “M” key for ultimate
performance SSD or cache
 PCIe SSD with both “B” and “M” key fit into both Socket 2 and 3
hosts using only two PCIe lanes in Socket 3 hosts
 Applications
 Notebooks, Ultrabooks & Desktops
 Tablets
 Servers
 Portable Gaming Devices
 Devices that require SSDs
 Portable Mobile Devices

Which Layer?
 Transaction layer problems
mostly due to legacy carry
over (at present)
 Data link layer problems can
be the most difficult to detect
in development
 Physical layer issues for
protocol are mostly related
to the LTSSM
14
Data Link
Transaction
Physical
Application
Logical Sub Block

“PCIe 1.0, 2.0, 3.0???”
 What is the difference?
 These are different revisions of the specification
 They do not necessarily mean a device can support higher
speeds
 The changes go beyond the speed capability and include ECNs
and other updates
 We tend to use Gen 1, Gen 2, Gen 3 synonymously with
2.5GT/s, 5GT/s and 8GT/s
 A PCIe 1.0a or 1.1 device supports 2.5GT/s
 A PCIe 2.0 device must support 2.5GT/s and can support 5GT/s
 A PCIe 3.0 device must support 2.5GT/s and can support up to
5GT/s and 8GT/s
 How does this apply to the layered model?

“PCIe 1.0, 2.0, 3.0???”
 The most significant change in specification
revisions was to add higher speed capabilities
 However, there are many other ECNs that were
approved after the release of a specification that
are rolled into newer releases
 For example, there was a 2.1 revision of the
specification that included ECNs proposed after
the release of the 2.0 specification
 The PCI Express 3.1 Base Specification was
released

“PCIe 1.0, 2.0, 3.0???”
 There are many changes in Configuration Space
that can results in differing test results
 The way PCI-SIG tests devices is not the same
for a PCIe 1.1 device and and PCIe 3.0 device
that only supports 2.5GT/s

PCI-SIG Compliance Testing
 How does this fit into the layered model?
 Compliance testing is split out into sections:
 Configuration Space
– Using PCIECV tool from PCI-SIG
 System BIOS Testing
– Using PCIEPT tool from PCI-SIG
 Link and Transaction Layer
– Transaction Layer testing
– Data Link Layer testing
– Testing Link Training
 Electrical
– Link Equalization/De-emphasis testing
– Transmitter Signal Quality
– Receiver Jitter Tolerance
Data Link
Transaction
Physical
Application
Logical Sub Block

PHYSICAL LAYER
EXAMPLES
Data Link
Transaction
Physical
Application
Logical Sub Block

My Device Does Not Show
in Device Manager
 Is the Link Active?
 Firstly do a snapshot trigger
 If the link is active, you will see Update_FC packets
and SKP ordered sets
 If the link is not active, you may see TS1 or TS2
ordered sets
 If the link is unstable, you may see Update_FC and
SKP ordered sets, with occasional TS1/TS2
sequences as the link goes to Recovery
 Bottom Line: If the link is not active, there is no
way the device manager will see the device

Is the Device Seen by the
System?

First Check the Electrical
Characteristics
 Is the Signal Integrity of the
Link good on both sides
 Signal Integrity problems can
show in a whole manner of
ways, from instable links to
devices just not showing up
22

Snapshot Trigger
TS1 and TS2
Ordered Sets
would indicate that
the link is not in L0

PCIe Link Training From the
Specification

PCIe Link Training Debug
 LTSSM state changes appear as 1000’s of
ordered sets
25

Example: Link is Not Running at
Maximum Speed
Do both sides advertise
5GT/s or 8GT/s – Was a
speed change
initiated?

Dynamic Equalization
 The PCI Express 3.0 Specification allows for
8.0GT/s data rate
 In order to establish a link reliably at 8GT/s, the
protocol allows for dynamic equalization of the
link during the speed change in the Recovery
state
 This is a 4 phase process and happens in the
Recovery.Equalization substate

Recovery Substate Machine
Recovery.Equalization
Substate

Potential Equalization Problems
 Link does not get to 8GT/s
 Link gets to 8GT/s but is unstable
 Equalization fails because of wrong coefficients
 Does the equalization process complete?
 Protocol Analyzer is the only way to get visibility

How Do I Debug Equalization?
 Use the EC field as a trigger – this will help navigate the equalization
 Compress the Training Sequences helps digest the process – there
are thousands of them!

POWER MANAGEMENT
EXAMPLES
Data Link
Transaction
Physical
Application
Logical Sub Block

L0s
 It can be challenging to debug L0s problems, it
is important to set things up properly
 The exit latency from L0s is very short
 Give the analyzer the best chance of locking by
disabling auto detection such as speed, link width and
polarity
 Make sure the analyzer is set up properly with L0s
turned off and ensure clean captures

ASPM – L0s
33

L1 and L1 Substates
 L1 is a more aggressive form of power
management than L0s, and the L1 Substates
ECN takes that further
 There are 2 mechanisms to enter L1 - ASPM L1
and PCIPM L1
 Each have a different entry mechanism
 PCIPM uses a config write from the RC to the
PMSCR register on the device to initiate the transition
to L1
 ASPM L1 the downstream component indicates the
desire to enter the L1 state, and sends
PM_Active_State_Request_L1 to the root, the root
acknowledges with PM_Request_ACK DLLP

ASPM
 Advanced post processing shows how many
times each state was entered
35

L1 Substates
 A device indicates its support of L1 substates in
the configuration space
 L1 PM Substates is considered enabled on a
Port when any combination of the ASPM L1.1
Enable, ASPM L1.2 Enable, PCI-PM L1.1
Enable and PCI-PM L1.2 Enable bits associated
with that Port are Set

L1 Substates – L1.1
Upstream Port Initiated Exit from L1.1
Downstream Port Initiated Exit from L1.1

L1 Substates – L1.2
Upstream Port Initiated Exit from L1.2
Downstream Port Initiated Exit from L1.2

How Does it Look on a
Protocol Analyzer?
Link Enters Electrical Idle
CLKREQ# Deasserted
CLKREQ# Asserted
Link Enters Recovery
L1 Entry Request

Failed Entry To L1

L1 Transition Sequence

DATA LINK LAYER
EXAMPLES
Data Link
Transaction
Physical
Application
Logical Sub Block

Data Link Layer Issues
 Some problems are related to flow control
mechanisms
 Most DLL FC problems will not be discovered by the
current compliance tests
 Many DLL FC problems are not discovered until after
a product is released
 Problems may only show up with specific combination
of Root and End Point that is not tested during
compliance testing
 Data Link Layer problems often result in performance
issues

PCI-SIG Developers Conference 44
Detecting Credit Errors
LOOK
HERE!

Data Link Layer Problems
 Flow control overflow
 Why is a packet getting a NAK?
 Set up trigger on NAK and look what happened
before

TRANSACTION
LAYER EXAMPLES
Data Link
Transaction
Physical
Application
Logical Sub Block

Completion Timeout Example
 This is a transaction layer problem, but how would it
exhibit itself on the link?
 System may have “Blue Screened”
 Need to look for a request, and then set a timer. If a
completion is not sent before the timer expires, then possible
completion timeout condition may be occurring
 Set the trigger close to the end of the buffer, to capture what
happened before the event
 The link may be still in L0 state, but TLP traffic has
stopped

PCI EXPRESS
STORAGE
Data Link
Transaction
Physical
Application
Logical Sub Block

PCIe Storage Devices
 Where does this fit into the layered model
 NVMe
 PQI/SOP
 AHCI/SATA Express
 How do I debug?
 “I don’t care about PCI Express!!!”
 Compliance Programs for Storage are outside of
the scope of PCI-SIG, however PCI-SIG is
impacted by form factor changes

PCI Express Storage
50
Data Link
Transaction
Physical
Application
Logical Sub Block
Software
– Ethernet, NVMe, SOP, AHCI, SATA…

PCIe Storage Devices
 There are several different technologies emerging for
PCI Express based storage
 These take advantage of the increasing speeds of Flash
Memory storage and the next generations of NAND
technology
 PCI Express provides a solid, scalable, tested platform
to take advantage of these higher speeds
 The principle of storage over PCI Express is the same
for all technologies but the software and implementations
are very different
 These technologies have history in legacy storage
technologies

How Do I Debug These Products?
 It may take a different way of thinking to use a
PCI Express analyzer for debugging a storage
problem
 The storage protocols reside within the PCI
Express payload data, and requires further
decoding
 It is possible that the only interface to these
devices is PCI Express
 New form factors are emerging also, SFF8639,
M.2 for example

PCIe Storage
 The underlying PCIe traffic has an impact on
performance of the Storage layers
 Queue balancing
 Possibility of multiple devices over one link
 Devices behind a switch
 Virtualized devices
 Built in scalability of technologies such as NVMe

NVMe Example
NVMe Commands on a
PCI Express Link

NVMe Example
The NVMe Commands
are actually PCI Express
Memory Reads and
Writes to specific
addresses

NVMe Example - Continued

Serial ATA Example
SATA Commands on a
PCI Express Link

Serial ATA Example
The ATA Commands are
actually PCI Express
Memory Reads and
Writes to specific
addresses

PCI Express Storage
 The packets or frames are all embedded into the
payload of the PCI Express transactions
 The storage technologies are typically quite
simple at the hardware level and leave software
to carry out most of the work
 A PCI Express analyzer with additional decodes
for storage protocols is required for visibility on
the bus
 Bear in mind that storage devices may or may
not reside on standard PCI Express (CEM) form
factors

Summary and Conclusions
 The easiest way to debug PCI Express problems is to
determine which layer is showing the problem
 Using the correct tool will make the job much easier
 Use a “Snapshot” trigger first of all to assess what is
going on
 Ensure lower level (electrical) layers are compliant and
within spec before debugging protocol problems
 Remember the PCI-SIG compliance testing is not a
substitute for thorough validation, it is a useful sanity
check
60

Troubleshooting_PCI_Express_Link_Training_and_Protocol_Issues_FROZEN.pdf

More Related Content

Similar to Troubleshooting_PCI_Express_Link_Training_and_Protocol_Issues_FROZEN.pdf (20)

Recently uploaded (20)

Troubleshooting_PCI_Express_Link_Training_and_Protocol_Issues_FROZEN.pdf