Publish generator dev versions during RegenPreview so emitter deps are restorable in CI

[Copilot]

The regen-preview pipeline rewrites the Azure/mgmt *-emitter-package.json artifacts to point at host-only file: .tgz paths, so CI on the resulting azure-sdk-for-net PR always fails to restore those dependencies (example). This adds an opt-in path to publish the locally built generator packages to the registry configured in .npmrc and reference the published version instead.

Changes

• RegenPreview.psm1
  • Publish-GeneratorPackage: publishes a .tgz to a registry; treats an already-published version as success so re-runs are idempotent.
  • Update-EmitterPackageArtifact: shared helper to rewrite an emitter-package.json + lock file in either file: mode (default) or published-version mode.
  • Update-GeneratorPackage gains -PublishRegistry/-NpmrcPath: pins deps to the published version (not file:) and publishes the packed .tgz.
  • Update-AzureGenerator / Update-MgmtGenerator thread these through and use the shared helper.
• Submit-AzureSdkForNetPr.ps1: new -PublishGeneratorPackages switch that resolves the registry + authenticated .npmrc and drives publishing through the Azure/mgmt regeneration paths.
• publish.yml: new opt-in PublishGeneratorPackages pipeline parameter.

With the switch off (default), behavior is unchanged — deps stay pinned via file:. Publishing order (unbranded from the main stage → Azure → mgmt) ensures each package's dependencies are resolvable from the feed; the local RegenPreview.ps1 loop is untouched since it cleans up and never commits.

Result

The emitter artifact references a published version restorable by CI:

// azure-typespec-http-client-csharp-emitter-package.json (published mode)
"dependencies": {
  "@azure-typespec/http-client-csharp": "1.0.0-alpha.20250101.abc"  // was "file:/…/…-*.tgz"
}

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-csharp@11280

commit: 5881cf2

[jorgerangel-msft]

@copilot I don't think this should be a parameter. It should happen by default in the pipeline. Would it be cleaner if we split the submit-azuresdkfornetpr into different steps in the publish.yml? We already publish the unbranded generator to azure artifacts. We could add more steps to:

• Build the azure generator from the publish unbranded generator artifact
• Publish the azure generator
• Regen the libraries, both unbranded and branded

If any steps fail, fail the pipeline. These steps should only ever apply if the pipeline was triggered manually and a regen was requested

[Copilot]

Removed the PublishGeneratorPackages parameter. Publishing now happens by default whenever a regen is requested — it keys off -RegenerateAzureLibraries/-RegenerateMgmtLibraries, which are only set on a manual run, so it never publishes on scheduled/CI-to-main runs. Done in the latest commit.

On splitting Submit-AzureSdkForNetPr.ps1 into discrete publish.yml steps (build azure generator → publish → regen unbranded + branded, failing the pipeline on any step): that's a larger restructure since the script shares a lot of state (the cloned azure-sdk-for-net checkout, temp dirs, the built emitter/nuget artifacts) across those phases, so breaking it into separate pipeline steps means threading that state between steps. Happy to take it on as a follow-up — do you want it in this PR, or split out so this one stays focused on the publish-by-default fix?

[jorgerangel-msft]

@copilot we should do that in this pr, the script is getting too complicated and we need to ensure we are compliant and are using the correct publish templates. We should also not be publishing tgz artifacts and simply just publish using the next available version number following the existing version format for said pkgs

[Copilot]

Agreed on doing it here and moving the logic out of the script into compliant pipeline steps. I mapped the existing infra so we align before I rewrite the release path. Here's the plan:

Restructure publish.yml into discrete stages (each fails the pipeline on error), gated on a manual run + regen requested:

1. Build_Azure_Generator (build job) — consumes the build_artifacts_csharp unbranded generator artifact, builds/packs @azure-typespec/http-client-csharp (+ -mgmt when requested) and the Azure.Generator/mgmt nupkgs, and publishes them as a pipeline artifact (mirrors how the unbranded generator is packed today).
2. Publish_Azure_Generator (release/deployment job) — publishes that artifact via the existing compliant templates rather than my custom Publish-GeneratorPackage/npm publish *.tgz: /eng/tsp-core/pipelines/templates/release/ado-feed-release.yml (ADO feed) + esrp-release.yml, matching eng/tsp-core/pipelines/jobs/publish-npm.yml. This drops the tgz-file publishing.
3. Regen_Libraries — runs Submit-AzureSdkForNetPr.ps1 with the generators now resolved from the feed as published versions, regenerating both unbranded + branded. The Submit script shrinks to only the emitter-json rewrite (referencing the published version) + regen, since build/publish moved to stages 1–2.

Two decisions I want to confirm before I rewrite this, since they're compliance-sensitive and I can't test the pipeline end-to-end:

1. Publish target for the Azure/mgmt generator packages — should these go through the same ADO-feed + ESRP publish-npm.yml release path as the unbranded generator (public azure-sdk-for-js feed), or ADO feed only? That determines whether I reuse publish-npm.yml wholesale or just ado-feed-release.yml.

2. "Next available version, existing format" — today the Azure generator reuses the unbranded alpha stamp ($PackageVersion, i.e. X.Y.Z-alpha.<date>.<build>). To publish a real (non-tgz, non-file:) version I need the source of truth: should the Azure/mgmt packages keep their own X.Y.Z base (from the azure-sdk-for-net eng/packages/http-client-csharp package.json) with the alpha build stamp appended, and should "next available" be computed by querying the feed for the max published version and incrementing, or is reusing the current pipeline build stamp (unique per run) sufficient?

Once you confirm those two, I'll implement the full restructure in this PR.

[jorgerangel-msft]

@copilot

1 - it should only be published to the ado feed not the public feed.
2 - you should always query the ado feed to get the next available version for both the branded and mgmt emitters

[Copilot]

Both confirmed and implemented (latest commits):

1. ADO feed only — publishing is npm publish to the Azure Artifacts feed (.../azure-sdk/public/_packaging/azure-sdk-for-js/npm/registry/); no npmjs.org/ESRP path is involved. No .tgz is referenced via file: anymore — the emitter artifacts reference the published version.

2. Query the ADO feed for the next available version — added Get-NextGeneratorVersion, which queries the feed and returns the next available <base>-alpha.<yyyyMMdd>.<n> for each package independently. It's wired for both the branded (@azure-typespec/http-client-csharp) and mgmt (@azure-typespec/http-client-csharp-mgmt) emitters, with dependencies pinned to their exact published versions (Azure → published unbranded; mgmt → published Azure + unbranded). Verified the version logic with offline unit tests (never-published/E404, single-version npm output, per-day counter increment, base-version isolation).

One thing I'd like your call on to finish the publish.yml side: today it's a single job whose one AzureCLI@2 step runs Submit-AzureSdkForNetPr.ps1 end-to-end (clone → build → publish → regen → PR). Do you want me to split that into separate build → publish → regen steps within the same job (so each fails the pipeline independently while sharing the checkout/temp state on disk), or into separate stages with a dedicated 1ES release/deployment job for the publish? The step-in-same-job split is lower-risk given the shared filesystem state; the separate-stage split matches publish-npm.yml but needs the artifact + state threaded across stages. I'll wire whichever you prefer.

[jorgerangel-msft]

@copilot I expected to see publish.yml updated. Why are we making all of these edits in the ps1 scripts?

[Copilot]

Restructured in publish.yml (0d34d47). The CreateAzureSdkForNetPR job now drives the flow as three discrete, independently-failing steps instead of one monolithic script invocation:

1. Prepare azure-sdk-for-net checkout & regenerate unbranded emitter
2. Build & publish Azure/mgmt generators to ADO feed — only runs when a regen was requested (condition: or(RegenerateAzureLibraries, RegenerateMgmtLibraries)) and fails the pipeline on any build/publish error
3. Regenerate SDK libraries & create PR

The ps1 change is now just the plumbing that lets the pipeline orchestrate this: Submit-AzureSdkForNetPr.ps1 gained a -Phase parameter (Prepare | PublishGenerators | Regenerate, default All for local runs) plus shared -WorkingDirectory/-DebugFolder so each pipeline step operates on the same on-disk checkout. The reusable publish helpers (Publish-GeneratorPackage, Get-NextGeneratorVersion, Update-EmitterPackageArtifact) stay in RegenPreview.psm1 but are now invoked from the dedicated publish step rather than buried in the end-to-end run. Publishing is to the ADO feed only, using the next available <base>-alpha.<yyyyMMdd>.<n> version queried from the feed, and the emitter artifacts reference that published version (no file:/tgz paths).

[jorgerangel-msft]

@copilot publishing should be done via the shared publish template job/stage /eng/emitters/pipelines/templates/stages/emitter-stages.yml

[Copilot]

Done. Publishing no longer happens inside the ps1 scripts. I extracted the DevOps-feed publish steps out of emitter-stages.yml into a shared reusable template /eng/emitters/pipelines/templates/steps/publish-to-devops-feed.yml, refactored emitter-stages.yml to consume it (no behavior change), and publish.yml now uses that same shared template to publish the Azure/mgmt generator packages to the ADO feed.

The step that previously did "Build & publish generators" is now split: Submit-AzureSdkForNetPr.ps1 -Phase PublishGenerators only builds/packs the generators and stamps the next available feed version (pinning the emitter artifacts to it), and the publish is done by the shared template step immediately after. Publish-GeneratorPackage (the npm publish in RegenPreview.psm1) has been removed entirely.