Skip to content

[v25.2.x] schema_registry: enforce strict validation for missing required attri…#30559

Open
vbotbuildovich wants to merge 1 commit into
redpanda-data:v25.2.xfrom
vbotbuildovich:backport-pr-30525-v25.2.x-642
Open

[v25.2.x] schema_registry: enforce strict validation for missing required attri…#30559
vbotbuildovich wants to merge 1 commit into
redpanda-data:v25.2.xfrom
vbotbuildovich:backport-pr-30525-v25.2.x-642

Conversation

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

Backport of PR #30525
Fixes: #30558,

…butes

The `is_object_required_superset` function in the JSON schema validator contained a bug where `std::views::filter` was incorrectly dropping missing required fields during compatibility checks. This allowed schemas with missing required attributes to falsely pass validation.
This commit removes the faulty filter to strictly enforce superset validation. If a reader schema demands a required attribute that the writer schema lacks (and does not provide a default for), the validator now correctly identifies the incompatibility.
Additionally, updated the `test_json_compat_messages` test case to expect the newly exposed `required_attribute_added` error, which was previously masked by this bug.

(cherry picked from commit d7c5dd5)
@vbotbuildovich vbotbuildovich added this to the v25.2.x-next milestone May 20, 2026
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label May 20, 2026
@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

Retry command for Build#84755

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/datalake/disk_budget_test.py::DatalakeDiskUsageTest.test_idle_finish@{"cloud_storage_type":1,"concurrent_translations":4,"num_partitions":10}

@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

CI test results

test results on build#84755
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FAIL ClusterConfigLegacyDefaultTest test_legacy_default {"wipe_cache": false} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ClusterConfigLegacyDefaultTest&test_method=test_legacy_default
FLAKY(PASS) CompactionRecoveryTest test_index_recovery null integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab88-459e-9e02-37d35689c6af 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=CompactionRecoveryTest&test_method=test_index_recovery
FAIL ControlCharacterPermittedAfterUpgrade test_upgrade_from_pre_v23_2 {"initial_version": [22, 3, 11]} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ControlCharacterPermittedAfterUpgrade&test_method=test_upgrade_from_pre_v23_2
FAIL DatalakeDiskUsageTest test_idle_finish {"cloud_storage_type": 1, "concurrent_translations": 4, "num_partitions": 10} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab83-4a57-b228-706c7d3aa087 0/11 Test FAILS after retries.Significant increase in flaky rate(baseline=0.3929, p0=0.0001, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DatalakeDiskUsageTest&test_method=test_idle_finish
FAIL SchemaEvolutionE2ETests test_dropped_column_no_collision {"catalog_type": "nessie", "cloud_storage_type": 1, "produce_mode": "proto2", "query_engine": "spark"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_dropped_column_no_collision
FAIL SchemaEvolutionE2ETests test_dropped_column_no_collision {"catalog_type": "rest_hadoop", "cloud_storage_type": 1, "produce_mode": "proto2", "query_engine": "trino"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_dropped_column_no_collision
FAIL SchemaEvolutionE2ETests test_dropped_column_select_fails {"catalog_type": "nessie", "cloud_storage_type": 1, "produce_mode": "proto3", "query_engine": "trino"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_dropped_column_select_fails
FAIL SchemaEvolutionE2ETests test_illegal_schema_evolution {"catalog_type": "rest_hadoop", "cloud_storage_type": 1, "produce_mode": "proto3", "query_engine": "spark", "test_case": "drop column that appears in partition spec"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_illegal_schema_evolution
FAIL SchemaEvolutionE2ETests test_illegal_schema_evolution {"catalog_type": "rest_jdbc", "cloud_storage_type": 1, "produce_mode": "proto3", "query_engine": "spark", "test_case": "illegal promotion int->string"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_illegal_schema_evolution
FAIL SchemaEvolutionE2ETests test_illegal_schema_evolution {"catalog_type": "nessie", "cloud_storage_type": 1, "produce_mode": "avro", "query_engine": "trino", "test_case": "illegal promotion int->string"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_illegal_schema_evolution
FAIL SchemaEvolutionE2ETests test_legal_schema_evolution {"catalog_type": "rest_hadoop", "cloud_storage_type": 1, "produce_mode": "avro", "query_engine": "spark", "test_case": "add_column"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_legal_schema_evolution
FAIL SchemaEvolutionE2ETests test_legal_schema_evolution {"catalog_type": "rest_jdbc", "cloud_storage_type": 1, "produce_mode": "avro", "query_engine": "spark", "test_case": "drop_column"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_legal_schema_evolution
FAIL SchemaEvolutionE2ETests test_legal_schema_evolution {"catalog_type": "nessie", "cloud_storage_type": 1, "produce_mode": "proto2", "query_engine": "spark", "test_case": "promote_column"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_legal_schema_evolution
FAIL SchemaEvolutionE2ETests test_legal_schema_evolution {"catalog_type": "rest_hadoop", "cloud_storage_type": 1, "produce_mode": "proto2", "query_engine": "spark", "test_case": "reorder_columns"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_legal_schema_evolution
FAIL SchemaEvolutionE2ETests test_old_schema_writer {"catalog_type": "rest_jdbc", "cloud_storage_type": 1, "produce_mode": "proto2", "query_engine": "trino", "test_case": "promote_column"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_old_schema_writer
FAIL SchemaEvolutionE2ETests test_old_schema_writer {"catalog_type": "nessie", "cloud_storage_type": 1, "produce_mode": "proto3", "query_engine": "trino", "test_case": "reorder_columns"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_old_schema_writer
FAIL SchemaEvolutionE2ETests test_reorder_columns {"catalog_type": "rest_jdbc", "cloud_storage_type": 1, "produce_mode": "proto3", "query_engine": "trino"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaEvolutionE2ETests&test_method=test_reorder_columns
FAIL DeleteRecordsTest test_delete_records_segment_deletion {"cloud_storage_enabled": true, "truncate_point": "at_high_watermark"} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DeleteRecordsTest&test_method=test_delete_records_segment_deletion
FAIL DeleteRecordsTest test_delete_records_with_transactions {"cloud_storage_enabled": true} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DeleteRecordsTest&test_method=test_delete_records_with_transactions
FAIL FlexibleCompatTest test_api_versions_flexible null integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=FlexibleCompatTest&test_method=test_api_versions_flexible
FAIL WriteRejectTest test_refresh_disk_health null integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=WriteRejectTest&test_method=test_refresh_disk_health
FAIL NodeIdAssignmentUpgrade test_assign_after_upgrade null integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab8b-4c66-a4b5-ba6fcff590d2 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeIdAssignmentUpgrade&test_method=test_assign_after_upgrade
FLAKY(PASS) ShadowIndexingLocalRetentionTest test_shadow_indexing_non_default_local_retention {"cloud_storage_type": 2} integration https://buildkite.com/redpanda/redpanda/builds/84755#019e46c8-ab83-4b83-8afd-6765552c04a6 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0400, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1153, p1=0.2939, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowIndexingLocalRetentionTest&test_method=test_shadow_indexing_non_default_local_retention

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/redpanda kind/backport PRs targeting a stable branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants