Executable proof artifact
Telemetry Debug Demo
A schema-first timeline that correlates workload markers, PMBus rail telemetry, BMC logs, PCIe link state, thermal signals, and validation closure across multiple failure cases.
Portfolio-grade demo
All events, measurements, and failure IDs are synthetic and public-safe. They illustrate a correlation and closure model; they are not employer logs, customer data, or proof of a production AI-server deployment.
Case Summary
Multiple platform failures, one correlation model.
Each case has a symptom, a shared correlation ID, evidence domains, and explicit validation closure.
closed
GPU load-step rail droop causes PCIe downtrain
GPU0 core rail droop exceeds validation threshold during load step.
Closure: Regression passes cold boot, warm reboot, load-step, and thermal soak coverage.
F-2026-001
12 events
BMC, PCIe, Power, Validation, Workload
closed
Retimer reset race causes Gen5 training retries
PCIe link retries during Gen5 training on warm reboot.
Closure: Warm reboot regression closes with stable Gen5 training over repeated cycles.
F-2026-002
8 events
BMC, PCIe, Validation, Workload
closed
Thermal sensor mismatch masks accelerator hotspot
Local VRM hotspot estimate exceeds derating threshold while inlet remains normal.
Closure: Thermal soak passes with updated sensor map, inspection step, and derating policy.
F-2026-003
8 events
BMC, Thermal, Validation, Workload
Timeline
Power, BMC, PCIe, thermal, and validation evidence aligned by timestamp.
| ts_ms | case | domain | source | correlation | message |
|---|---|---|---|---|---|
| 0 | case_power_pcie | Workload | host | - | System enters validation boot sequence. |
| 1180 | case_power_pcie | Power | pmbus | - | GPU0 core rail is stable before workload step. |
| 1800 | case_power_pcie | Workload | host | - | GPU stress workload transitions from idle to high-current phase. |
| 1806 | case_power_pcie | Power | pmbus | F-2026-001 | GPU0 core rail droop exceeds validation threshold during load step. |
| 1812 | case_power_pcie | BMC | bmc | F-2026-001 | BMC records PMBus undervoltage warning on GPU0_CORE. |
| 1826 | case_power_pcie | PCIe | host | F-2026-001 | PCIe link enters training retry window after rail droop. |
| 1872 | case_power_pcie | PCIe | host | F-2026-001 | Link downtrains after repeated training retries. |
| 1930 | case_power_pcie | Validation | lab | F-2026-001 | Scope capture confirms rail droop overlaps PCIe training retry window. |
| 2200 | case_power_pcie | Validation | lab | F-2026-001 | Mitigation selected: add bulk capacitance, tune compensation, extend OCP blanking, and guard retimer reset release. |
| 2920 | case_power_pcie | Power | pmbus | F-2026-001 | Retest rail droop stays within target after mitigation. |
| 2965 | case_power_pcie | PCIe | host | F-2026-001 | PCIe link trains at expected speed and width after mitigation. |
| 3100 | case_power_pcie | Validation | lab | F-2026-001 | Regression passes cold boot, warm reboot, load-step, and thermal soak coverage. |
| 4000 | case_retimer_refclk | Workload | host | - | Warm reboot validation begins for fabric stability check. |
| 4060 | case_retimer_refclk | BMC | bmc | F-2026-002 | BMC releases retimer reset shortly after reference clock lock indication. |
| 4074 | case_retimer_refclk | PCIe | host | F-2026-002 | PCIe link retries during Gen5 training on warm reboot. |
| 4108 | case_retimer_refclk | PCIe | host | F-2026-002 | Link falls back to Gen4 after retry budget is exhausted. |
| 4140 | case_retimer_refclk | Validation | lab | F-2026-002 | Capture shows retimer reset release is too close to refclk lock settle time. |
| 4300 | case_retimer_refclk | Validation | lab | F-2026-002 | Mitigation selected: add 25 ms guard time after refclk lock before retimer reset release. |
| 4620 | case_retimer_refclk | PCIe | host | F-2026-002 | PCIe link trains at Gen5 x16 after reset guard change. |
| 4700 | case_retimer_refclk | Validation | lab | F-2026-002 | Warm reboot regression closes with stable Gen5 training over repeated cycles. |
| 6000 | case_thermal_sensor | Workload | host | - | Thermal soak reaches steady workload phase. |
| 6120 | case_thermal_sensor | Thermal | bmc | - | Inlet temperature appears normal during thermal soak. |
| 6130 | case_thermal_sensor | Thermal | bmc | F-2026-003 | Local VRM hotspot estimate exceeds derating threshold while inlet remains normal. |
| 6134 | case_thermal_sensor | BMC | bmc | F-2026-003 | BMC derates GPU2 due to local hotspot policy. |
| 6150 | case_thermal_sensor | Validation | lab | F-2026-003 | Review finds inlet sensor cannot explain VRM hotspot; local sensor placement and cold-plate contact require inspection. |
| 6420 | case_thermal_sensor | Validation | lab | F-2026-003 | TIM contact adjusted and VRM zone threshold added to service checklist. |
| 6900 | case_thermal_sensor | Thermal | bmc | F-2026-003 | Retest hotspot temperature remains below derating threshold. |
| 7000 | case_thermal_sensor | Validation | lab | F-2026-003 | Thermal soak passes with updated sensor map, inspection step, and derating policy. |
Download
Validation report
Markdown report with case summaries and full timeline.
Machine-readable
Parsed JSON report
Structured report for future visualization, filters, or role-specific summaries.