tests: Add provisioned-environment E2E test tier - #141
Open
samir-gandhi wants to merge 4 commits into
Open
Conversation
Regression Test Results
✅ No regressions detectedAll export configurations produced compatible output. Generated by regression workflow • View run |
Regression Test Results
✅ No regressions detectedAll export configurations produced compatible output. Generated by regression workflow • View run |
Regression Test Results
✅ No regressions detectedAll export configurations produced compatible output. Generated by regression workflow • View run |
Regression Test Results
✅ No regressions detectedAll export configurations produced compatible output. Generated by regression workflow • View run |
Adds terraform-test-data/ (hand-authored Terraform fixtures per resource), tools/tf-regression-provision (creates/destroys a throwaway PingOne environment via Terraform itself, using org-admin credentials scoped to this run's lifecycle), and a `make e2e` entry point that applies fixtures into that environment and runs the existing base-vs-PR export/compare pipeline against it. Additive to tests/regression/, which continues comparing export output against a static environment unchanged. Closes #140.
Previously silently defaulted to NA, inheriting the same fallback used for the unrelated PINGCLI_PINGONE_REGION_CODE export credential elsewhere in this repo. The org-admin credential set should always be complete and specified on its own terms.
Adds regression-provisioned.yaml, a single-job workflow that runs the existing tests/regression-provisioned/run-local.sh on workflow_dispatch only (no schedule yet - real environment creation/deletion per run needs cost/quota sign-off first). Reuses the exact locally-proven script rather than decomposing into a multi-job pipeline, avoiding untested cross-job credential/state passing. Also fixes a report-persistence gap in run-local.sh found while wiring this up: reports were written to TMPDIR_LOCAL but never copied out before the cleanup trap deleted that directory, so the workflow's artifact-upload step would have found nothing. Added copy_reports(), mirroring the existing pattern in tests/regression/run-local.sh. Verified live end-to-end locally: provision, apply, all 5 matrix entries pass, reports land in regression-reports/, teardown confirmed via a follow-up API read (404).
samir-gandhi
force-pushed
the
feat/provisioned-e2e-test-tier
branch
from
August 12, 2026 17:43
0ca2efc to
f71304a
Compare
Regression Test Results
✅ No regressions detectedAll export configurations produced compatible output. Generated by regression workflow • View run |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
terraform-test-data/— hand-authored Terraform fixtures, starting with one representative resource (sso/population).pingone_environmentitself is a fixture resource interraform-test-data/root, so environment creation goes through the same Terraform/provider path as everything else.tools/tf-regression-provision— thin CLI wrapper aroundterraform apply/destroyplus output parsing. Holds no PingOne API logic itself; environment lifecycle and fixture provisioning are entirely expressed as Terraform, authenticated with a single org-admin credential scoped to this run.make e2e(backed bytests/regression-provisioned/run-local.sh), which provisions a throwaway environment, applies the fixtures, runs the existing base-vs-PR export/compare pipeline (tools/regression-compare,tests/regression/matrix.json) against it, and tears the environment down unconditionally on exit..github/workflows/regression-provisioned.yaml, a single-job workflow that runsmake e2eonworkflow_dispatchonly (noscheduleyet — real environment creation/deletion per run needs cost/quota sign-off first). Reuses the exact locally-proven script rather than a multi-job pipeline, avoiding untested cross-job credential/state passing..gitignorefor local/ephemeral Terraform artifacts (.terraform/, lock file, state,terraform.tfvars.json) and to stop the existing blanket*.tfignore (meant for generated export output) from swallowing the new hand-authoredterraform-test-data/**/*.tffixtures.This is additive to
tests/regression/, which is unchanged and continues comparing export output between binary versions against its existing static environment.Base branch is
pingone-core-resources, notmain. Thesso/populationfixture requirespingone_populationexport support, which only exists onpingone-core-resources;mainhas no SSO resource support yet. This PR was rebased ontopingone-core-resourcesafter that was discovered — the base/environment + DaVinci-only test run againstmainwas passing but not actually exercising SSO export coverage (the base binary silently succeeded with 0 resources for an unsupported type).Closes #140.
Test plan
go build ./...andgo vet ./tools/tf-regression-provision/...terraform validateandterraform fmt -check -recursiveonterraform-test-data/bash -n tests/regression-provisioned/run-local.shregression-provisioned.yamlgo mod tidy— nogo.mod/go.sumchanges (no new dependencies)make e2elive end-to-end against a real (non-prod) PingOne org, twice (once before rebase ontopingone-core-resources, once after): provisioned an environment, applied thesso/populationfixture, ran all 5 matrix entries fromtests/regression/matrix.json(base vs. PR export, 0 breaking / 0 acceptable diffs on each), then tore the environment down. Confirmed via a follow-up API read that the environment no longer exists after teardown, on every run.Notes for reviewers
PINGCLI_PINGONE_ORGADMIN_CLIENT_ID,PINGCLI_PINGONE_ORGADMIN_CLIENT_SECRET,PINGCLI_PINGONE_ORGADMIN_ENVIRONMENT_ID,PINGCLI_PINGONE_ORGADMIN_REGION_CODE,PINGCLI_PINGONE_ORGADMIN_LICENSE_ID. Live in a newregression-provisionGitHub Environment, separate from theregressionEnvironment used by the existing static-environment regression tier.terraform-test-data/currently covers only 1 of the ~23 currently-supported resource types; follow-up PRs will expand coverage in small, dependency-ordered batches (see the tracking issue).