Skip to content

test: add Gherkin evaluator tests using test-harness specs#1947

Open
aepfli wants to merge 1 commit intoopen-feature:mainfrom
open-feature-forking:feat/evaluator-gherkin-tests
Open

test: add Gherkin evaluator tests using test-harness specs#1947
aepfli wants to merge 1 commit intoopen-feature:mainfrom
open-feature-forking:feat/evaluator-gherkin-tests

Conversation

@aepfli
Copy link
Copy Markdown
Member

@aepfli aepfli commented Apr 16, 2026

Summary

  • Adds in-process Gherkin (BDD) tests for the JSON evaluator using godog and the shared test-harness/evaluator/ feature specs
  • Tests run directly against the evaluator without testcontainers, validating: flag evaluation, error handling, targeting, metadata, fractional bucketing (v2), semver, string operators, zero-values, evaluator refs, and no-default-variant behavior
  • 83 scenarios pass, @fractional-v1 excluded (flagd implements v2)
  • Adds test-evaluator-gherkin Makefile target

Related: open-feature/python-sdk-contrib#377 (equivalent Python testkit)
Part of: open-feature/flagd-testbed#366 (centralized evaluator testkit migration)
See also: open-feature/flagd-testbed#367 (error handling discussion)

Test plan

  • make test-evaluator-gherkin passes all 83 scenarios
  • Existing unit tests unaffected (go test -short ./core/pkg/evaluator/...)
  • CI passes

🤖 Generated with Claude Code

@aepfli aepfli requested review from a team as code owners April 16, 2026 06:54
@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 16, 2026

Deploy Preview for polite-licorice-3db33c canceled.

Name Link
🔨 Latest commit d43c66e
🔍 Latest deploy log https://app.netlify.com/projects/polite-licorice-3db33c/deploys/69e089461ac7c800088b5427

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces Gherkin-based integration tests for the evaluator component. It adds a new test file core/pkg/evaluator/gherkin_test.go utilizing the godog framework to execute feature-based tests against the evaluator's logic, including support for various data types, targeting keys, and metadata assertions. Additionally, the Makefile has been updated with a test-evaluator-gherkin target, and necessary dependencies have been added to core/go.mod. I have no feedback to provide.

Add in-process Gherkin tests for the JSON evaluator using godog and the
shared test-harness evaluator feature files. Tests validate flag evaluation,
error handling, targeting, metadata, fractional bucketing, semver, string
operators, zero-values, evaluator refs, and no-default-variant behavior
directly against the evaluator without testcontainers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Simon Schrottner <simon.schrottner@gmail.com>
@aepfli aepfli force-pushed the feat/evaluator-gherkin-tests branch from 1962bd7 to d43c66e Compare April 16, 2026 07:01
@sonarqubecloud
Copy link
Copy Markdown

@toddbaert
Copy link
Copy Markdown
Member

@aepfli Nice.

I don't think this is really a feat for end-users. I think let's make it a chore/test?

Also - should we run it in the CI?

@aepfli aepfli changed the title feat: add Gherkin evaluator tests using test-harness specs test: add Gherkin evaluator tests using test-harness specs Apr 20, 2026
@aepfli
Copy link
Copy Markdown
Member Author

aepfli commented Apr 20, 2026

I agree with the pr title, and changed it

Also - should we run it in the CI?

Maybe my knowledge is off, but we do not have the e2e tag, so this should execute with normal test execution, or am i wrong here? as it does not need docker, etc. it should be fast, and there is no reason to not run it during normal test execution.

@toddbaert
Copy link
Copy Markdown
Member

I agree with the pr title, and changed it

Also - should we run it in the CI?

Maybe my knowledge is off, but we do not have the e2e tag, so this should execute with normal test execution, or am i wrong here? as it does not need docker, etc. it should be fast, and there is no reason to not run it during normal test execution.

I don't think it's running. The submodule doesn't seem to be checked out in the CI and the coverage is exactly the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants