[GH-2640] Skip re-registration of functions in SedonaContext.create()#2642
Merged
[GH-2640] Skip re-registration of functions in SedonaContext.create()#2642
Conversation
When SedonaContext.create() is called a second time, all ~329 ST functions are re-registered. Spark's FunctionRegistry detects that each function already exists with a different builder closure and emits a WARN log for each one, producing hundreds of noisy warning lines. Guard both expression and aggregate function registration with functionExists() checks to skip functions that are already registered in the session's function registry. Closes #2640
jiayuasu
force-pushed
the
fix/2640-simplify-sedonacontext-warnings
branch
from
8553b05 to
61e54c9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes Simply the warning messages of SedonaContext.create() #2640What changes were proposed in this PR?
When
SedonaContext.create()is called a second time, all ~329 ST functions are re-registered. Spark'sFunctionRegistrydetects that each function already exists with a different builder closure and emits a WARN log for each one, producing hundreds of noisy "replaced a previously registered function" warning lines.This PR adds
functionExists()guards inAbstractCatalog.registerAll()before registering both expression functions and aggThis PR addsfunctionExists()guards inAbstractCatalog.registerAll()before registering both nThis PR addsfunctionExists()guards in `AbstractCatalog.ratch tested?Existing uExisting uExisting uExisting uExisting uExistinge Existing uExisting uExisting uExisting uExisting uExistinge Existing uExisting uExisting uExisting uExisting uExistinge Existing uExisting uExisting uExisting uExisting uExistinge anyExisting uExisting uExisting uEe the documentation.