[SPARK-57225][SS] Reject write operations on state data sources with clear error#56319
Open
shrirangmhalgi wants to merge 1 commit into
Open
[SPARK-57225][SS] Reject write operations on state data sources with clear error#56319shrirangmhalgi wants to merge 1 commit into
shrirangmhalgi wants to merge 1 commit into
Conversation
…clear error
StateDataSource (statestore) and StateMetadataSource (state-metadata) are read-only streaming checkpoint inspection sources. Previously, attempting to write via df.write.format("statestore").save() would fall through to the V1 write path and produce a confusing internal error.
This PR makes both sources implement CreatableRelationProvider to intercept write attempts early and throw a clear STDS_WRITE_UNSUPPORTED error: "The state data source '<sourceName>' is read-only and does not support write operations."
Changes:
- Add STDS_WRITE_UNSUPPORTED error class with sqlState 0A000
- StateDataSource implements CreatableRelationProvider, throws in createRelation
- StateMetadataSource implements CreatableRelationProvider, throws in createRelation
- Add regression tests in StateDataSourceNegativeTestSuite
Contributor
Author
|
@HeartSaVioR / @liviazhu Could you please review this PR which addresses SPARK-57225? It makes the state data sources ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Make
StateDataSourceandStateMetadataSourceexplicitly reject write operations with a clear error message (STDS_WRITE_UNSUPPORTED) instead of falling through to the V1 write path and producing a confusing internal error.Both sources now implement
CreatableRelationProviderand throw immediately increateRelation, producing: "The state data source 'statestore' is read-only and does not support write operations."Why are the changes needed?
StateDataSource(statestore) andStateMetadataSource(state-metadata) are read-only checkpoint inspection sources. Previously,df.write.format("statestore").save()fell through to the V1 write path and threwINTERNAL_ERROR: "does not allow create table as select"- a confusing message that doesn't tell the user the source is intentionally read-only.Does this PR introduce any user-facing change?
Yes. Write attempts on state data sources now produce a clear error:
Previously the error was a generic internal error.
How was this patch tested?
Added 2 tests in
StateDataSourceNegativeTestSuiteverifying thatdf.write.format("statestore").save()anddf.write.format("state-metadata").save()throwSTDS_WRITE_UNSUPPORTEDwith sqlState0A000.Was this patch authored or co-authored using generative AI tooling?
Yes. Using Claude-Opus 4.6