Repository navigation
fix: harden API container bootstrap - #3203
Merged
Tal Zaccai (TalZaccai) merged 8 commits intoOct 22, 2026
Merged
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Default builds select the smoke runner, smoke imports fail, and Cosmos DB credentials bypass hosted filtering.
3 open issues
What changed in this PR
Hardens the production API container with non-root execution and dedicated managed-identity configuration.
Changes:
- Replaces Azure CLI bootstrap with validated, in-memory hosted configuration.
- Restricts runtime permissions and credential handling.
- Adds bootstrap regressions, container smoke checks, and deployment documentation.
| File | Description |
|---|---|
| ts/tools/scripts/testApiContainer.mjs | Builds and checks container images. |
| ts/packages/api/test/containerImage.spec.ts | Checks production image configuration. |
| ts/packages/api/test/containerBootstrap.spec.ts | Tests bootstrap validation and credential rejection. |
| ts/packages/api/src/typeAgentServer.ts | Accepts validated runtime configuration. |
| ts/packages/api/src/storageProviders/azureStorageProvider.ts | Uses the shared credential factory. |
| ts/packages/api/src/startup.ts | Gates startup on configuration loading. |
| ts/packages/api/src/index.ts | Integrates bootstrap before server startup. |
| ts/packages/api/src/containerBootstrap.ts | Implements dedicated hosted configuration loading. |
| ts/packages/api/README.md | Documents configuration and deployment requirements. |
| ts/packages/api/package.json | Restricts packaged runtime files. |
| ts/packages/api/container/start.sh | Adds fail-fast startup checks. |
| ts/packages/api/container/smoke.mjs | Checks runtime behavior with local mocks. |
| ts/packages/aiclient/test/auth.spec.ts | Tests credential selection. |
| ts/packages/aiclient/src/auth.ts | Adds managed-identity credential selection. |
| ts/Dockerfile | Separates build dependencies and configures non-root runtime. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
robgruen
approved these changes
Oct 22, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tal Zaccai (TalZaccai)
enabled auto-merge
October 22, 2026 05:55
Hillary Mutisya (hillary-mutisya)
added a commit
that referenced
this pull request
Oct 24, 2026
## Why this change macOS test jobs have been holding up PR validation for more than an hour, often after the corresponding Ubuntu and Windows jobs have finished. We reviewed five recent completed PRs to separate slow initial execution from time spent retrying failures. The table shows median **Test-step duration in minutes**, including retries, across each PR's configured Node versions. It does not include installation or build time. | PR / workflow run | macOS | Ubuntu | Windows | | --- | ---: | ---: | ---: | | [#3175](https://github.com/microsoft/TypeAgent/actions/runs/39969763926) | 93.03 | 19.92 | 25.33 | | [#3184](https://github.com/microsoft/TypeAgent/actions/runs/39912328526) | 81.53 | 23.09 | 20.75 | | [#3210](https://github.com/microsoft/TypeAgent/actions/runs/39911017633) | 71.80 | 16.62 | 25.73 | | [#3209](https://github.com/microsoft/TypeAgent/actions/runs/39856783535) | 90.22 | 22.43 | 33.19 | | [#3203](https://github.com/microsoft/TypeAgent/actions/runs/39726876699) | 72.02 | 18.82 | 28.16 | | Median across all tested jobs | **75.63** | **19.55** | **25.58** | | Jobs requiring a package retry | **12/12 (100%)** | **3/12 (25%)** | **1/12 (8.3%)** | This sample contains 36 executed test jobs. Some PRs had two Node versions rather than three. Two other recent PRs never reached testing and were excluded from the retry denominators. These are historical observations, not a controlled benchmark. Both initial execution and retries are expensive: - macOS initial passes took **38–92 minutes**, with a **61.85-minute median**, before retries. Every sampled macOS job retried, so the sample has no retry-free macOS control. - Memory-service retries reran **61–67 suites for only 1–6 failed files**, costing another **7–30 minutes**. In one job, dispatcher reran 146 suites after one file failed, taking 11.5 minutes. - Roughly 73% of macOS failed-case occurrences were timeouts or bounded waits. All recovered on the observed serial package retries. The logs suggest timing sensitivity but do not establish the underlying resource or runtime cause. - The slowdown extends beyond memory: in the separately investigated [run 39985375882](https://github.com/microsoft/TypeAgent/actions/runs/39985375882), dispatcher took 50–56 minutes and default-agent-provider took 34–37 minutes on the slow macOS jobs. Removing a few scaling tests would not address these delays. This PR keeps a smaller macOS compatibility suite blocking changes, moves full macOS coverage to nightly runs, and reduces unnecessary retry work. ## Changes 1. **Split blocking and nightly macOS coverage.** - PR, merge-queue and push jobs run an explicit selection of 13 package suites on macOS, across all three supported Node versions. - The selection covers file links, native SQLite, locking, memory publication/persistence/privacy, small Git history, session lifecycle, and representative cache merge/non-merge round trips. - All root script tests remain blocking, including aliased-directory healing and subprocess checks. - Full macOS correctness coverage runs nightly at **08:17 UTC** on the default branch, across all supported Node versions. Manual dispatch runs full coverage on all three operating systems. - Full Ubuntu/Windows blocking coverage is unchanged. The default local test profile also remains full. 2. **Retry failed files where the runner supports it.** - Recognized single-invocation Jest scripts retry the accumulated failed files serially, with caching disabled, up to twice. - Chained commands, lifecycle hooks and unsupported runners explicitly fall back to the complete package command so retries do not bypass setup or additional test phases. - Missing failure artifacts and invalid file selections fail closed rather than clearing failures. 3. **Fix test-phase accounting.** - Previously, a package failure short-circuited the root script-test command. A successful package retry could then report green without ever running those script tests. - Workspace tests/retries and root script tests now have independent outcomes. Root script tests run even after unrecovered package failures, and both phases must pass. - GitHub summaries report each phase separately; workflows retain test logs for 14 days. 4. **Apply consistent Jest settings.** - Browser, browser-extension and CLI configurations now honor the shared CI-only one-worker limit and retry cache setting. - CLI uses the shared failure reporter when reporting is enabled. 5. **Fix macOS aliased-path handling in generated-file healing.** - Resolve generated-file and scratch-manifest destinations from canonical roots. This avoids false containment failures when macOS temporary paths resolve from `/var` to `/private/var`. - Add an aliased-directory regression while preserving containment and regular-file checks. 6. **Separate memory correctness from scaling coverage.** - Keep a ten-commit history-equivalence case in normal CI. - Move the 100/1,000-commit variants to a dedicated regression command and a weekly/manual workflow on Ubuntu, Windows and macOS. - Keep bounded offline correctness scenarios in CI; the external-data/model-backed rehearsal was already excluded. ## Coverage tradeoff The blocking macOS selection is intentionally not the full correctness suite. Full Ubuntu/Windows coverage still blocks PRs, but macOS-only regressions outside the selection may first appear nightly. Maintainers should triage nightly failures and add missing platform-sensitive regressions to the blocking selection. Contract tests check the explicit selection and event-specific workflow configuration. This PR does not claim a measured macOS speedup yet. It also does not fix the separately identified MCP shutdown test that waits for the client's 60-second timeout. ## Validation - Full workspace build passed. - Compatibility profile: **13 suites / 276 tests passed** locally. - Combined profile, workflow, Node-version, Jest-configuration and retry checks: **78 tests passed**. - After the retry-orchestration refactor, all **56 focused retry tests passed**. The committed complexity, lint and test-debt gates passed without threshold changes. - Generated-file-healing tests: **32 passed**, with five existing Windows skips. - Local validation used Windows with Node 24.13.0, below the supported minimum. The broader root script suite had one installer-test failure because it requires a supported Node runtime; this PR does not change or suppress that test. - Actual macOS behavior and runtime still require CI validation. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
The production API container ran as root, established an Azure CLI session, and used the general
getKeysbootstrap to import broad shared configuration into plaintext. A compromised API or dependency could therefore access substantially more credentials and runtime authority than required.Mitigating steps
getKeysbootstrap, and persistent Azure CLI account state from the runtime image.TYPEAGENT_API_CONFIG_VAULTand least-privilege Key Vault secret-read plus Azure OpenAI RBAC for the managed identity.