# Telemetry Debug Validation Report

All events, measurements, and failure IDs are synthetic and public-safe. They illustrate a correlation and closure model; they are not employer logs, customer data, or proof of a production AI-server deployment.

- Events parsed: 28
- Cases parsed: 3
- Closed cases: 3
- Timeline window: 7000 ms

## Case Summary

### GPU load-step rail droop causes PCIe downtrain

- Case ID: `case_power_pcie`
- Correlation: `F-2026-001`
- Status: `closed`
- Domains: BMC, PCIe, Power, Validation, Workload
- First symptom: GPU0 core rail droop exceeds validation threshold during load step.
- Closure: Regression passes cold boot, warm reboot, load-step, and thermal soak coverage.

### Retimer reset race causes Gen5 training retries

- Case ID: `case_retimer_refclk`
- Correlation: `F-2026-002`
- Status: `closed`
- Domains: BMC, PCIe, Validation, Workload
- First symptom: PCIe link retries during Gen5 training on warm reboot.
- Closure: Warm reboot regression closes with stable Gen5 training over repeated cycles.

### Thermal sensor mismatch masks accelerator hotspot

- Case ID: `case_thermal_sensor`
- Correlation: `F-2026-003`
- Status: `closed`
- Domains: BMC, Thermal, Validation, Workload
- First symptom: Local VRM hotspot estimate exceeds derating threshold while inlet remains normal.
- Closure: Thermal soak passes with updated sensor map, inspection step, and derating policy.

## Timeline

| ts_ms | case | domain | source | correlation | message |
| ---: | --- | --- | --- | --- | --- |
| 0 | case_power_pcie | Workload | host | - | System enters validation boot sequence. |
| 1180 | case_power_pcie | Power | pmbus | - | GPU0 core rail is stable before workload step. |
| 1800 | case_power_pcie | Workload | host | - | GPU stress workload transitions from idle to high-current phase. |
| 1806 | case_power_pcie | Power | pmbus | F-2026-001 | GPU0 core rail droop exceeds validation threshold during load step. |
| 1812 | case_power_pcie | BMC | bmc | F-2026-001 | BMC records PMBus undervoltage warning on GPU0_CORE. |
| 1826 | case_power_pcie | PCIe | host | F-2026-001 | PCIe link enters training retry window after rail droop. |
| 1872 | case_power_pcie | PCIe | host | F-2026-001 | Link downtrains after repeated training retries. |
| 1930 | case_power_pcie | Validation | lab | F-2026-001 | Scope capture confirms rail droop overlaps PCIe training retry window. |
| 2200 | case_power_pcie | Validation | lab | F-2026-001 | Mitigation selected: add bulk capacitance, tune compensation, extend OCP blanking, and guard retimer reset release. |
| 2920 | case_power_pcie | Power | pmbus | F-2026-001 | Retest rail droop stays within target after mitigation. |
| 2965 | case_power_pcie | PCIe | host | F-2026-001 | PCIe link trains at expected speed and width after mitigation. |
| 3100 | case_power_pcie | Validation | lab | F-2026-001 | Regression passes cold boot, warm reboot, load-step, and thermal soak coverage. |
| 4000 | case_retimer_refclk | Workload | host | - | Warm reboot validation begins for fabric stability check. |
| 4060 | case_retimer_refclk | BMC | bmc | F-2026-002 | BMC releases retimer reset shortly after reference clock lock indication. |
| 4074 | case_retimer_refclk | PCIe | host | F-2026-002 | PCIe link retries during Gen5 training on warm reboot. |
| 4108 | case_retimer_refclk | PCIe | host | F-2026-002 | Link falls back to Gen4 after retry budget is exhausted. |
| 4140 | case_retimer_refclk | Validation | lab | F-2026-002 | Capture shows retimer reset release is too close to refclk lock settle time. |
| 4300 | case_retimer_refclk | Validation | lab | F-2026-002 | Mitigation selected: add 25 ms guard time after refclk lock before retimer reset release. |
| 4620 | case_retimer_refclk | PCIe | host | F-2026-002 | PCIe link trains at Gen5 x16 after reset guard change. |
| 4700 | case_retimer_refclk | Validation | lab | F-2026-002 | Warm reboot regression closes with stable Gen5 training over repeated cycles. |
| 6000 | case_thermal_sensor | Workload | host | - | Thermal soak reaches steady workload phase. |
| 6120 | case_thermal_sensor | Thermal | bmc | - | Inlet temperature appears normal during thermal soak. |
| 6130 | case_thermal_sensor | Thermal | bmc | F-2026-003 | Local VRM hotspot estimate exceeds derating threshold while inlet remains normal. |
| 6134 | case_thermal_sensor | BMC | bmc | F-2026-003 | BMC derates GPU2 due to local hotspot policy. |
| 6150 | case_thermal_sensor | Validation | lab | F-2026-003 | Review finds inlet sensor cannot explain VRM hotspot; local sensor placement and cold-plate contact require inspection. |
| 6420 | case_thermal_sensor | Validation | lab | F-2026-003 | TIM contact adjusted and VRM zone threshold added to service checklist. |
| 6900 | case_thermal_sensor | Thermal | bmc | F-2026-003 | Retest hotspot temperature remains below derating threshold. |
| 7000 | case_thermal_sensor | Validation | lab | F-2026-003 | Thermal soak passes with updated sensor map, inspection step, and derating policy. |