Skip to content

feat(workflow): auto-detect committed baseline in agentops-pr templates#306

Merged
placerda merged 1 commit into
developfrom
feature/issue-155-pr-baseline-autodetect
Jun 14, 2026
Merged

feat(workflow): auto-detect committed baseline in agentops-pr templates#306
placerda merged 1 commit into
developfrom
feature/issue-155-pr-baseline-autodetect

Conversation

@placerda

Copy link
Copy Markdown
Contributor

Closes #155.

The PR-gate workflow templates emitted by �gentops workflow generate did not pass --baseline to �gentops eval run, even though docs/tutorial-baseline-comparison.md Section 4 promises:

The �gentops-pr.yml workflow shipped by �gentops workflow generate already supports this — drop a baseline file in your repo (e.g. .agentops/baseline/results.json) and add this step: ...

Both the GitHub Actions and Azure DevOps PR templates now wrap the eval invocation with a small bash guard that auto-detects .agentops/baseline/results.json:

BASELINE_ARG=""
if [ -f .agentops/baseline/results.json ]; then
  BASELINE_ARG="--baseline .agentops/baseline/results.json"
fi
agentops eval run --config <cfg> $BASELINE_ARG

When the file is present, the eval step compares against it. When absent, BASELINE_ARG stays empty and the bash invocation is byte-equivalent to the prior behaviour. Deploy templates (dev/qa/prod) are intentionally untouched.

Implementation

  • _eval_substitutions, _github_eval_substitutions, _ado_eval_substitutions now take a kind kwarg.
  • Only kind == 'pr' injects the baseline-detection bash.
  • Indentation is handled by two small helpers so the YAML stays valid for both the GH Actions run: | block and the ADO AzureCLI@2 inlineScript.

Tests

Added to tests/unit/test_cicd.py:

  • test_pr_template_autodetects_committed_baseline
  • test_deploy_templates_do_not_inject_baseline_autodetect
  • test_azure_devops_pr_template_autodetects_committed_baseline
  • test_azure_devops_deploy_templates_do_not_inject_baseline_autodetect

Validation

python -m pytest tests/unit -x -q — 946 passed, 1 skipped.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

The agentops-pr GitHub Actions and Azure DevOps templates emitted by
'agentops workflow generate' now wrap 'agentops eval run' with a small
bash guard that auto-detects a committed baseline file at
.agentops/baseline/results.json. When the file is present (e.g. dropped
into the repo per docs/tutorial-baseline-comparison.md Section 4), the
eval step compares against it via --baseline. When absent, BASELINE_ARG
stays empty and the bash invocation is byte-equivalent to the prior
behaviour, so existing PR pipelines do not change. Deploy templates
(dev/qa/prod) are intentionally untouched.

Implementation: _eval_substitutions, _github_eval_substitutions, and
_ado_eval_substitutions now accept a kind kwarg and only inject the
baseline-detection bash when kind == 'pr'. Two helper functions
(_github_baseline_autodetect_block, _ado_baseline_autodetect_block) keep
the indentation correct for both the GitHub Actions run block and the
ADO AzureCLI@2 inlineScript.

Tests added in tests/unit/test_cicd.py guard the PR / deploy split for
both GitHub Actions and Azure DevOps. Validation: 946 unit tests pass.

Closes #155

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@placerda placerda merged commit e2ad606 into develop Jun 14, 2026
8 checks passed
@placerda placerda deleted the feature/issue-155-pr-baseline-autodetect branch June 14, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant