Skip to content

Cannot use spark_event_logs_s3_uri in PySparkProcessor job #6253

Description

@swademb

PySDK Version

  • PySDK V2 (2.x)
  • PySDK V3 (3.x)

Describe the bug

Similar to #6252 - PySparkProcessor run no longer handles spark_event_logs_s3_uri due to changes in the ProcessingOutput interface - e.g. regression introduced in 3203e49

To reproduce

Provide any (valid) S3 URI for the spark log location.

Expected behavior

Successfully starts the job.

Screenshots or logs

ValidationError: 4 validation errors for ProcessingOutput
output_name
  Field required [type=missing, input_value={'source': '/opt/ml/proce...oad_mode': 'Continuous'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.13/v/missing
source
  Extra inputs are not permitted [type=extra_forbidden, input_value='/opt/ml/processing/spark-events/', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
destination
  Extra inputs are not permitted [type=extra_forbidden, input_value='s3://...', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
s3_upload_mode
  Extra inputs are not permitted [type=extra_forbidden, input_value='Continuous', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden

The cause of the error is here:

            output = ProcessingOutput(
                source=_SparkProcessorBase._spark_event_log_default_local_path,
                destination=spark_event_logs_s3_uri,
                s3_upload_mode="Continuous",
            )

System information

  • AWS SageMaker Studio 4.4.3
  • sagemaker-core: 2.20
  • Python 3.12.14
  • uname: Linux default 6.12.103-127.188.amzn2023.x86_64 SMP PREEMPT_DYNAMIC Tue Aug 25 15:42:53 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions