[rascani]

nightly-espressif-idf.yml runs the component across every target, ESP-IDF
release and configuration that the per-PR jobs only sample. A prepare job
generates and packs the component into a file:// mirror and exports the
smoke, esp_nn:: mul, ESP-NN delegate and Cadence wake word models with
their kernel bundles, plus an all_ops bundle that registers every op of
every kernel library. Twelve build jobs (ESP-IDF 6.0.1, 6.1 and master,
four shards each) take that artifact and nothing else: they list the
idf_component_consumer configurations with idf-build-apps (each
sdkconfig.ci.* file for ESP32, -S3, -C3, -C5, -C6 and -P4, filtered by
.build-test-rules.yml), build their shard from the mirror with ESP-IDF's
own Python environment, check that every setting of the configuration
reached the build's sdkconfig, since ESP-IDF only warns about an unknown
symbol, and run each build on QEMU or esp-emu. Each job uploads its build
and emulator logs. The configurations are default, nolog (logging
compiled out), perf (ExecuTorch built without -Os, and the project for
speed), debug (debug assertions and full program verification),
wakeword, delegate, oplib, and all_ops on ESP32-S3 and -C6 with an 8 MB
flash layout, so every vendored kernel compiles under ESP-IDF's flags and
every registration resolves when the image links; plus a build-only
ESP32-P4 configuration for silicon before rev3, which esp-emu does not
run.

The consumer takes the model directory as ET_CONSUMER_MODELS_DIR, from
which CONFIG_ET_CONSUMER_MODEL picks the .pte and bundle. It has its own
sdkconfig.defaults instead of overriding SDKCONFIG_DEFAULTS. The script
builds each listed configuration itself with its own sdkconfig, because
idf-build-apps keeps one sdkconfig in the project directory, which the
next build overwrites before the emulator run needs it, and runs
idf-build-apps from a venv of its own so ESP-IDF's environment stays as
ESP-IDF installs it.

Jobs on macOS and Windows configure the two-model idf_component_smoke
project with the prepared component, once with its bundle and once in
automatic mode, and check that both select the same kernels. The
registry validation (compote component upload --dry-run --allow-existing)
runs here on the packed archive. setup.sh accepts an ESP-IDF branch such
as master. The workflow also runs for the ciflow/nightly label and on
pull requests that change what it runs, since workflow_dispatch only
works from the default branch.

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24711

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 00113ca with merge base 60a8b5f ():

NEW FAILURE - The following job has failed:

• trunk / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / test-arm-backend-vkml (test_pytest_models_vkml) / linux-job (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[rascani]

@claude review effort=max

[claude]

Claude finished @rascani's task in 50m 59s —— View job

---

Request Changes: this PR's own nightly run passed, but four things need another pass: the consumer's environment-variable side channel, the private --all-ops flag, the missing ciflow/nightly trigger and log upload, and configurations that never exercise ExecuTorch's own options.

Detailed review

PR Review: #24711

Summary

This PR adds a nightly workflow that generates and packs the component once, then builds and runs every idf_component_consumer sdkconfig.ci.* configuration. That covers six targets, split across four shards, on ESP-IDF 6.0.1, 6.1 and master. It also adds an all-ops image, a registry dry-run, and configure jobs on macOS and Windows.

The problems are:

• the consumer's new interface
• the --all-ops flag
• the triggers and the missing logs
• what the configurations actually test

How this was checked:

• Three independent reviewers covered all 19 files.
• I checked the findings against the code and against this PR's nightly run.
• I could not verify two things: what test-infra's continue-on-error input does, and how ESP-IDF handles unknown symbols in sdkconfig.ci.* files.

Code Quality

• The consumer's environment-variable interface exists for a build path the nightly never takes. idf_component_consumer/CMakeLists.txt:10-14 says the values come from the environment because idf-build-apps "passes no -D arguments", and that test_espressif_idf_nightly.sh "builds it that way". It doesn't. The script uses idf-build-apps only to list the apps (test_espressif_idf_nightly.sh:132). It builds them with idf.py, and already passes -DSDKCONFIG and -DSDKCONFIG_DEFAULTS (:158-160).

  • The side channel causes a regression. main/CMakeLists.txt:9 tests CONFIG_ET_CONSUMER_MODEL, which always has a value (default "smoke").
    • Now, a build with -DET_COMPONENT_VERSION but no ET_SMOKE_MODELS (and no ET_CONSUMER_MODELS_DIR in the environment) looks for /smoke.pte and /smoke_kernels and fails.
    • Before this PR, the same build produced the app without ExecuTorch that idf_component_smoke/CMakeLists.txt:14-15 documents.
  • CMake does not track $ENV{} reads. If either variable changes, an existing build directory keeps the old value.
  • Fix:
    • Pass -DET_COMPONENT_VERSION=… and -DET_CONSUMER_MODELS_DIR=… from build(), the way test_espressif_idf_consumer.sh:112 already passes the version.
    • In main/CMakeLists.txt, guard on the CMake variable ET_CONSUMER_MODELS_DIR. Keep CONFIG_ET_CONSUMER_MODEL to choose the model.
    • Remove both $ENV{} reads.
  • If the example really must build under idf-build-apps build, make the nightly build it that way so the interface actually gets tested.

  Fix this →

• --all-ops is a private on/off flag that still runs most of the per-PR script. test_espressif_idf_component.sh:23-26 silently ignores any other argument.

  • The comments are wrong. The header (:18-19, "it instead builds…") and :161 ("Skips the other checks") say the flag replaces the per-PR checks. In fact lines 48-108 all run first:
    • setup
    • three model exports
    • test_version_stamp.sh
    • component generation
    • manifest lint
    • SBOM validation
    • three kernel bundles (smoke_kernels, cpu_kernels, delegate_kernels), and the delegate-bundle check
  • It duplicates prepare. In this PR's run only the last 3½ of the job's 15 minutes built and ran the all-ops images. The recursive checkout and install_executorch took about 9½ minutes, and that preamble about 2¼. prepare already does the checkout, the install, the exports and the generation, and the rest repeats the per-PR job.
  • It tests a narrower setup than the rest of the workflow. It only runs ESP-IDF 6.1 (setup.sh's default), and it passes the generated component as ET_COMPONENT_DIR instead of resolving it through the component manager. The rest of this workflow tests three versions through the component manager.
  • The claim at :160 that "every vendored kernel must compile" already holds on every PR. Any build with a bundle compiles the whole _kernels static library (idf_component_template/CMakeLists.txt:49, project_include.cmake:361). What this job adds is linking and booting an image with every op registered.
  • Suggestion: make it one more matrix configuration and remove the flag, the job and the second component generation:
    • prepare makes an all_ops model and bundle.
    • sdkconfig.ci.all_ops sets 8 MB flash and a checked-in partition table.
    • .build-test-rules.yml limits it to esp32s3 and esp32c6.

  Fix this →

• prepare() and build() copy blocks from the per-PR scripts instead of sharing them. Each block below now exists in two or three places:

  Block	Nightly script	Per-PR copies
  QEMU library check	test_espressif_idf_nightly.sh:99-107	test_espressif_idf_consumer.sh:44-52
  Model exports and splits, with tolerances	:54-70	test_espressif_idf_component.sh:53-70, consumer.sh:58-64
  Generate, pack, mirror	:73-80	consumer.sh:66-76
  Fresh ESP-IDF env and source copy	:113-130	consumer.sh:88-99
  run_firmware	:191-217	consumer.sh:117-148

  The copied exports drop the comments that explain --repeat 2 and 1.2e-7. Any change to a tolerance, an export flag or the emulator call now has to be made in two or three places. A sourced helper used by all three scripts would keep them consistent.

  Fix this →

Infrastructure

• The ciflow/nightly label can't trigger this workflow, and the PR path filter doesn't match what it runs. There's no push: tags: - ciflow/nightly/* trigger, unlike nightly.yml:6-8 and the test-backend-*.yml workflows. So the label (.github/pytorch-probot.yml:12) does nothing here.

  • pull_request.paths (nightly-espressif-idf.yml:8-12) leaves out files the jobs execute:
    • test_espressif_idf_component.sh, which the all-ops job runs
    • examples/espressif/setup.sh, which the prepare, build and all-ops jobs run and which this PR changes
    • idf_component_smoke/**: the consumer includes its main/, and the macOS and Windows jobs configure it
    • build_idf_component.sh and idf_component_template/**, which make the component
    • esp_idf_mirror.py
  • The filter is also too broad. A change that only touches idf_component_consumer/** starts the all-ops, macos-15-xlarge and Windows jobs, and none of them use the consumer.
  • Suggestion: add the tag trigger and widen the paths. Or gate each job on _get-changed-files.yml, the way test-backend-cortex-m.yml:22-24 does.

  Fix this →

• The build matrix never uploads its logs.

  • build() writes build-*.log and emulator-*.log into ${work} inside the checkout (test_espressif_idf_nightly.sh:94,157,172). The build job has no upload-artifact (nightly-espressif-idf.yml:82-96).
  • When a nightly fails, all that remains is the 30-line tail printed at :163 and :178. That often misses the first CMake error or the start of a crash.
  • The per-PR jobs do upload their logs (test-espressif.yml:82,108).
  • Setting upload-artifact alone would upload the inputs again, because dir is RUNNER_ARTIFACT_DIR (yml:95). Move the inputs out first, then copy the logs in.
  • These runners are on AWS (meta-prod-aws-ue2 in this PR's run), so upload-artifact-to-s3: true with a name per matrix entry will work.

  Fix this →

• Question: does continue-on-error hide master failures instead of reporting them?

  • As far as I know, test-infra applies this input to the script step, not the job. Its description is "allow a job to pass when exec script step fails".
  • If so, a broken master shard shows as a passing job, and the failure is only visible inside the job's step list. That contradicts the comment at nightly-espressif-idf.yml:88-89, which says master "reports its status".
  • A job that calls a reusable workflow can't set continue-on-error itself. Nothing gates on a scheduled run, so simply letting master fail may give the clearer signal.
  • I couldn't read linux_job_v3.yml from here, and every master shard in this PR's run passed, so please confirm.

Testing

• The configurations barely exercise the component's own options.

  • sdkconfig.ci.os doesn't change how ExecuTorch code is compiled. The default CONFIG_ET_OPTIMIZE_SIZE=y already builds the runtime, kernels and delegate with -Os "regardless of the project's compiler optimization level" (build_idf_component.sh:573-580, idf_component_template/CMakeLists.txt:100-107).
  • os also runs the smoke model, which doesn't use esp-nn, the one ExecuTorch dependency that os does recompile.
  • No CI job, per-PR or nightly, builds any of these settings:
    • CONFIG_ET_OPTIMIZE_SIZE=n
    • CONFIG_ET_DEBUG_ASSERTS=y (no NDEBUG)
    • CONFIG_ET_PROGRAM_VERIFICATION=y
    • the non-default ET_LOG_LEVEL_* choices
  • Suggestion:
    • Replace os with perf: CONFIG_ET_OPTIMIZE_SIZE=n plus CONFIG_COMPILER_OPTIMIZATION_PERF=y. The Kconfig help says to turn ET_OPTIMIZE_SIZE off "when inference time matters more", and examples/espressif/project/sdkconfig.defaults:33 already sets CONFIG_COMPILER_OPTIMIZATION_PERF=y.
    • Add a debug configuration: CONFIG_ET_DEBUG_ASSERTS=y plus CONFIG_ET_PROGRAM_VERIFICATION=y.

  Fix this →

• Nothing checks that a configuration's settings took effect.

  • As far as I know, ESP-IDF only warns about an unknown symbol in a defaults file and carries on. So a typo, or a symbol renamed or missing in some ESP-IDF version, silently builds the default configuration under the configuration's name. For example: CONFIG_ET_ENABLE_LOGGING in nolog if the component renames it, or CONFIG_ESP32P4_SELECTS_REV_LESS_V3 if master drops it.
  • The model check has the same gap. The expected pass marker is read back from the generated sdkconfig (test_espressif_idf_nightly.sh:171). A misspelled CONFIG_ET_CONSUMER_MODEL in a future sdkconfig.ci.* would run smoke and still pass.
  • Suggestion: after each build (:166), check that every CONFIG_ line of sdkconfig.ci.${config} appears in the generated sdkconfig. An =n setting appears there as # CONFIG_X is not set.

  Fix this →

Recommendation

Request Changes

• The consumer's side-channel interface and the --all-ops flag are design problems to fix before the pattern spreads.
• Without the ciflow/nightly trigger and the uploaded logs, nightly failures are hard to reproduce and diagnose.
• The continue-on-error item is a question, not a blocker.
• The two Testing items are coverage gaps the new matrix could close cheaply.

Specific Comments

• examples/espressif/README.md:49-54, three inaccuracies:
  • "idf-build-apps builds it": idf-build-apps only lists the apps; idf.py builds them.
  • "runs each build on its emulator": p4_rev_less_v3 is built but never run.
  • "macOS and Windows jobs configure it": those jobs configure idf_component_smoke through ET_COMPONENT_DIR, not this consumer.
• .ci/scripts/test_espressif_idf_nightly.sh:48: the comment mentions esp-idf-sbom, which prepare never runs.
• .ci/scripts/test_espressif_idf_nightly.sh:113-118: the "fresh ESP-IDF Python environment" then has idf-build-apps pip-installed into it without ESP-IDF's constraints file. The builds may therefore not run on ESP-IDF's own package set. A separate venv just for find avoids this. Low confidence: I didn't check whether pip actually changed any packages.
• .ci/scripts/test_espressif_idf_nightly.sh:158 vs :197,211: ${idf_python_env} and ${IDF_PYTHON_ENV_PATH} name the same directory. Use one name.
• examples/espressif/setup.sh:72-77, two problems:
  • When IDF_PATH is not a git checkout, set -e now exits on git's own fatal: error at line 73 instead of printing the message at line 75.
  • A master checkout whose tip carries a vX.Y-dev tag is rejected.
  • Fix: keep both git lookups inside the if condition, which solves both.
• examples/espressif/setup.sh:24: the usage line still says --idf-version=<tag>, but the option now also takes a branch.

Fix the Specific Comments →