Skip to content

ci: add network diagnostics for test artifact downloads - #4795

Merged
gyohuangxin merged 1 commit into
ROCm:mainfrom
gyohuangxin:ci/aiter-wheel-network-diagnostics
Aug 18, 2026
Merged

ci: add network diagnostics for test artifact downloads#4795
gyohuangxin merged 1 commit into
ROCm:mainfrom
gyohuangxin:ci/aiter-wheel-network-diagnostics

Conversation

@gyohuangxin

@gyohuangxin gyohuangxin commented Aug 17, 2026

Copy link
Copy Markdown
Member

Summary

  • Add lightweight network preflight diagnostics before test artifact downloads on MI35X runners.
  • Cover Aiter Test shard-list download, Aiter wheel cache/artifact download, Aiter Test Triton wheel download, and Triton Test shard-list/Triton wheel downloads.
  • Print runner identity, hostname, DNS results, and curl -w timing for GitHub and Azure Blob endpoints.
  • Run speedtest-cli after relevant artifact download failures, with continue-on-error so diagnostics do not change CI semantics.

Validation

  • python3 -c "import yaml, pathlib; [yaml.safe_load(pathlib.Path(p).read_text()) for p in [\".github/workflows/aiter-test.yaml\", \".github/workflows/triton-test.yaml\"]]"
  • git diff --check

@gyohuangxin
gyohuangxin requested review from a team and a lite review from Copilot August 17, 2026 07:33
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4795 --add-label <label>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds CI-side network diagnostics to help troubleshoot intermittent Aiter wheel download failures on linux-aiter-mi35x-1 runners, without changing the overall job outcome semantics (diagnostic steps are continue-on-error).

Changes:

  • Adds a network preflight step (runner info, DNS checks, and curl -w timing) before Aiter wheel cache/artifact restore/download on linux-aiter-mi35x-1.
  • Adds a post-failure speedtest-cli step intended to run only after the Aiter wheel artifact download retry fails (and only on linux-aiter-mi35x-1).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/aiter-test.yaml Outdated
Comment on lines +591 to +593
- name: Speedtest after Aiter wheel download failure
if: ${{ failure() && matrix.runner == 'linux-aiter-mi35x-1' && steps.restore_aiter_wheel.outputs.cache-hit != 'true' && (steps.download_aiter_wheel.outcome == 'failure' || steps.retry_aiter_wheel.outcome == 'failure') }}
continue-on-error: true
Comment thread .github/workflows/aiter-test.yaml Outdated
Comment on lines +605 to +606
python3 -m pip install --user --disable-pip-version-check speedtest-cli || true
timeout 120 "$HOME/.local/bin/speedtest-cli" --simple --secure || true
@gyohuangxin
gyohuangxin force-pushed the ci/aiter-wheel-network-diagnostics branch from 34df926 to 9b5580d Compare August 17, 2026 07:42
@gyohuangxin gyohuangxin changed the title ci: add network diagnostics for Aiter wheel downloads ci: add network diagnostics for wheel downloads Aug 17, 2026
@gyohuangxin
gyohuangxin force-pushed the ci/aiter-wheel-network-diagnostics branch 2 times, most recently from c817887 to 3a1bd56 Compare August 17, 2026 07:51
@gyohuangxin
gyohuangxin force-pushed the ci/aiter-wheel-network-diagnostics branch from 3a1bd56 to 90d660f Compare August 17, 2026 08:54
@gyohuangxin gyohuangxin changed the title ci: add network diagnostics for wheel downloads ci: add network diagnostics for test artifact downloads Aug 17, 2026
@zufayu
zufayu requested a review from valarLip August 18, 2026 02:39
@gyohuangxin
gyohuangxin merged commit 7684e93 into ROCm:main Aug 18, 2026
65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants