ci: load every plugin before shipping it, and report where the test time goes

[datlechin]

Two things CI never did: try to load a plugin it built, and say how long anything took.

Nothing ever loaded a plugin

CI built and signed 31 plugin bundles and never once tried to load one. "Bundle failed to load executable" is the failure this repo has shipped twice, both times because a witness table hard-referenced a PluginKit symbol that had been removed, and both times the first thing to find out was a user's install.

scripts/ci/verify-plugin-loads.sh dlopens each bundle's executable, which is exactly the dyld path that fails in that case. It takes milliseconds.

It runs in two places: the build job of macos-tests.yml right after AllPlugins compiles all 31, so every pull request checks them, and in build-plugin.yml before a bundle is published, because the registry serves those to users directly.

Verified against a real build. All 14 bundled plugins load:

✓ ClickHouseDriver  ✓ CSVExport  ✓ CSVImport  ✓ CSVInspectorPlugin  ✓ JSONExport
✓ JSONImport  ✓ MQLExport  ✓ MySQLDriver  ✓ PostgreSQLDriver  ✓ RedisDriver
✓ SQLExport  ✓ SQLImport  ✓ SQLiteDriver  ✓ XLSXExport
All 14 plugins load.

And it fails when one cannot. Corrupting a single executable:

✗ CSVExport: dlopen(.../CSVExport, 0x0006): [(no such file) (slice is not valid mach-o file)]
1 of 16 plugins will not load: CSVExport

dlerror prints every path it tried, which is a screenful, so the message keeps the reason and the distinct causes and drops the list. An empty directory is a failure too, not a silent pass, because "found nothing, therefore fine" is how a check like this stops working.

The plugins need DYLD_FRAMEWORK_PATH pointing at the products directory to resolve TableProPluginKit; without it every plugin would fail for the wrong reason.

Nothing measured how long tests take

grep -rn GITHUB_STEP_SUMMARY found nothing in the repo. Both .xcresult bundles were uploaded as artifacts and never read, so finding out which of 85 UI tests took ninety seconds meant downloading one and opening Xcode.

That is why a ten second XCUITest retry sat in every sample-database test, 30% of the UI job, until somebody went looking for it.

scripts/ci/summarize-xcresult.sh writes a duration table to $GITHUB_STEP_SUMMARY from the unit job and each UI shard. Run against the real bundle from run 32473179239, it surfaces exactly the hotspots that took manual digging to find:

85 cases in 36.5 min

| suite                     | cases | seconds |
| OpenQuicklyCommandUITests |     6 |   240.5 |
| QueryInsightsTabUITests   |     6 |   232.6 |
| QueryHistoryPanelUITests  |     5 |   131.8 |
| ResultTabPinUITests       |     1 |   103.0 |

It runs on failure as well, because a red run is when the durations matter most, and it never fails the job: a summary that can break a green run is worse than no summary. A missing or unreadable bundle exits 0 with a note.

Verification

actionlint clean across all six workflows, shellcheck -x --severity=warning clean on both new scripts. Both were exercised against real artifacts, in the passing and the failing direction.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.