Case 01 - SI / Debug

PCIe Gen5 Link Training Debug Fault Tree

A structured debug case for intermittent downtraining or link training failure in an AI server platform, connecting SI margin, retimer configuration, reference clock quality, connector loss, lane mapping, rail noise, BMC events, and host logs.

Public-reference note

This is not an employer PCIe Gen5 incident. Its method is grounded in verified 10 Gbps+ FPGA SerDes, eDP 8.1 Gbps/lane, synchronized capture, single-variable comparison, and Link Margin experience.

Symptom

The visible failure is fabric behavior; the root cause may live outside SI.

The platform intermittently trains a high-speed link below the expected speed/width, or fails training during cold boot, warm reboot, thermal soak, or workload transition. The first mistake is to treat the issue as purely SI before checking power timing, firmware configuration, reset sequence, and log alignment.

Primary artifacts

Host log	Enumeration result, link speed/width, AER counters, retry timing, BIOS version.
BMC log	Rail state, thermal state, reset event, retimer power-good, timestamp correlation.
Lab capture	Refclk jitter, rail noise, eye/BER/compliance, connector inspection, retimer strap/config.

Fault Tree

Rank hypotheses by observability and blast radius.

Branch	Failure mechanism	Fast discriminator	Owner interface
Retimer	Wrong EEPROM image, strap mismatch, reset timing, firmware/config mismatch.	Compare known-good image, read status registers, force alternate link policy.	Board, firmware, vendor FA.
Refclk	Jitter, spread-spectrum interaction, mux settling, clock loss during reset release.	Scope clock during failing boot window, compare warm/cold reset, check jitter budget.	Board, SI, BIOS/platform firmware.
Channel	Connector/cable loss, via discontinuity, length mismatch, polarity or lane reversal issue.	Swap cable/module, inspect connector, run lower speed, compare lane error distribution.	SI, mechanical, supplier quality.
Power	Rail dip during training, sequencing race, retimer/endpoint brownout, PDN resonance.	Trigger scope on training window, correlate PMBus samples and BMC rail-good events.	Power, board, BMC firmware.
Thermal	Marginal retimer or connector behavior after soak, throttling policy interaction.	Run cold/warm/hot matrix, inspect thermal sensor timeline and airflow/cold-plate state.	Thermal, mechanical, validation.

Measurement plan

Capture the failing boot window, not only the steady-state link.

Use a synchronized timeline across host log, BMC log, oscilloscope trigger, retimer status dump, and thermal state. Record speed/width, error counters, rails, reset, refclk, and retimer firmware in one bundle.

Decision logic

Do not accept a fix without a mechanism.

A passing rerun is not closure. Closure requires the failure to move predictably with a controlled variable: cable swap, refclk source, retimer config, rail load, thermal condition, or reset timing.

Closure Criteria

The fix must survive productization, not just the lab bench.

Technical closure

Mechanism identified.

Root cause is tied to a measured signal, configuration, layout feature, component variance, or firmware sequence.

Validation closure

Regression matrix updated.

Cold/warm boot, thermal soak, workload transition, cable/module variation, and factory test coverage are updated.

Field closure

Service signal added.

Telemetry, fault code, log bundle, or inspection step helps avoid rediscovering the failure in future field and service workflows.