Skip to content

Enable PHP FFE parametric system tests#7003

Draft
leoromanovsky wants to merge 5 commits into
mainfrom
leo.romanovsky/pr-g-php-ffe-scaffold
Draft

Enable PHP FFE parametric system tests#7003
leoromanovsky wants to merge 5 commits into
mainfrom
leo.romanovsky/pr-g-php-ffe-scaffold

Conversation

@leoromanovsky
Copy link
Copy Markdown
Contributor

@leoromanovsky leoromanovsky commented May 21, 2026

Motivation

PHP Milestone 1 now has a runtime-backed implementation in DataDog/dd-trace-php#3906. System-tests should validate that PHP can consume live FFE_FLAGS Remote Config and evaluate through the libdatadog evaluator before PHP moves on to exposures and metrics.

Reference design doc: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0

Changes

  • Adds PHP parametric /ffe/start and /ffe/evaluate endpoints backed by DDTrace\FeatureFlags\Client.
  • Normalizes defaults, targeting keys, and scalar attributes into the PHP Datadog client request shape.
  • Preserves top-level empty JSON objects in parametric responses so canonical JSON fixture cases compare correctly.
  • Suppresses the M1 transitional warning inside PHP FFE endpoints so responses remain valid JSON.
  • Enables PHP for Test_Feature_Flag_Dynamic_Evaluation and Test_Parametric_FFE_Start in manifests/php.yml.
  • Keeps PHP weblog FFE, exposures, and flag-evaluation metrics marked missing_feature until their separate milestones are implemented.
  • Updates tests/ffe/README.md with the validated PHP 8.2 parametric runbook and Bob's local artifact build command.

Decisions

  • Activate the validated PHP 8.2 parametric path first; defer weblog activation because local Apache PHP 8.2 image builds are not reliable on this arm64 machine.
  • Use live Remote Config in the test path; no fake evaluator or fixture-only PHP product path.
  • Keep exposure and metric assertions disabled until PHP emits those signals through the same collector paths already proven with non-PHP baselines.

Verification

Using a local artifact built from DataDog/dd-trace-php#3906:

./tooling/bin/build-debug-artifact gnu-aarch64-8.2-nts /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr-g-php-ffe-scaffold/binaries

Local runs:

  • TEST_LIBRARY=php ./run.sh PARAMETRIC tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation -F tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation -vv: 25/25 passed.
  • focused test-json-config-flag.json: 1/1 passed.
  • focused test-case-of-7-empty-targeting-key.json: 1/1 passed.
  • TEST_LIBRARY=php ./run.sh PARAMETRIC tests/parametric/test_parametric_endpoints.py::Test_Parametric_FFE_Start -F tests/parametric/test_parametric_endpoints.py::Test_Parametric_FFE_Start -vv: 1/1 passed.
  • php -l utils/build/docker/php/parametric/server.php: passed.
  • ./format.sh: passed.
  • git diff --check: passed.

@github-actions
Copy link
Copy Markdown
Contributor

ghost commented May 21, 2026

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/AGENTS.md                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/php/common/ffe.php                                   @DataDog/apm-php @DataDog/system-tests-core
docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/php/common/rewrite-rules.conf                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/parametric/server.php                            @DataDog/apm-php @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

@leoromanovsky leoromanovsky changed the title Add PHP FFE system-test scaffold Enable PHP FFE parametric system tests May 22, 2026
@datadog-prod-us1-4
Copy link
Copy Markdown

datadog-prod-us1-4 Bot commented May 22, 2026

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 22 Pipeline jobs failed

Testing the test | System Tests (php, dev) / parametric / parametric (2)   View in Datadog   GitHub Actions

🔧 Fix in code (Fix with Cursor). 13 failed tests. Error: Failed to start FFE provider in tests/parametric/test_ffe/test_dynamic_evaluation.py:213.

🧪 10 Tests failed

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-boolean-false-assignment.json-library_env0, parametric-php] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Failed to start FFE provider
assert False

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f527e331760>
test_case_file = 'test-case-boolean-false-assignment.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f527ba7c9e0>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f527ba7ed20>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    @parametrize("test_case_file", ALL_TEST_CASE_FILES)
...
tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-boolean-one-of-matches.json-library_env0, parametric-php] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Failed to start FFE provider
assert False

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f527e3319d0>
test_case_file = 'test-case-boolean-one-of-matches.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f527b30bfb0>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f527bc08260>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    @parametrize("test_case_file", ALL_TEST_CASE_FILES)
...
View all 10 test failures

Testing the test | System Tests (php, prod) / parametric / parametric (1)   View in Datadog   GitHub Actions

🔧 Fix in code (Fix with Cursor). 16 tests failed due to failed assertion while starting the FFE provider.

🧪 10 Tests failed

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-boolean-false-assignment.json-library_env0, parametric-php] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Failed to start FFE provider
assert False

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fad42a686e0>
test_case_file = 'test-case-boolean-false-assignment.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fad0e443e90>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fad0de83d40>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    @parametrize("test_case_file", ALL_TEST_CASE_FILES)
...
tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-boolean-one-of-matches.json-library_env0, parametric-php] from system_tests_suite   View in Datadog (Fix with Cursor)
AssertionError: Failed to start FFE provider
assert False

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f09617ac6b0>
test_case_file = 'test-case-boolean-one-of-matches.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f092c4d7830>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f092c904f20>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    @parametrize("test_case_file", ALL_TEST_CASE_FILES)
...
View all 10 test failures

Testing the test | System Tests (java, prod) / parametric / parametric (1)   View in Datadog   GitHub Actions

🔧 Fix in code (Fix with Cursor). ValueError: Number (1) of metrics not available from test agent, got 0 in tests/parametric/test_otel_metrics.py:1329

View all 22 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ccc1276 | Docs | Datadog PR Page | Give us feedback!

…iners

Mirrors the existing DD_AGENT_HOST / DD_TRACE_AGENT_URL injection so
OTel-aware exporters in the system-under-test can reach the test agent's
OTLP/HTTP listener without each tracer needing to special-case the
parametric framework.

Concretely: the PHP FFE OTLP-metrics path (DataDog/dd-trace-php#3911)
resolves its endpoint per the OpenTelemetry spec —
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, then OTEL_EXPORTER_OTLP_ENDPOINT,
then localhost:4318. Without this injection, the parametric metric
test had no way to reach the test agent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant