Skip to content

Bulter.py: Adds preprocess command for local preprocess#5266

Open
IvanBM18 wants to merge 4 commits intomasterfrom
feature/butler/local_preprocess
Open

Bulter.py: Adds preprocess command for local preprocess#5266
IvanBM18 wants to merge 4 commits intomasterfrom
feature/butler/local_preprocess

Conversation

@IvanBM18
Copy link
Copy Markdown
Collaborator

@IvanBM18 IvanBM18 commented May 5, 2026

Adds preprocess butler script

This command allows developers to generate the serialized and compressed uworker_input payload, upload it to real GCS, and get the signed download URL, exactly as it happens remotely. We can then use said url to trigger a task in any backend that we want:

  • In swarming trough a prpc request
  • In batch trough manually posting the task to the utask_main queue
    This accelerates local debugging of the tworker preprocessing phase without relying on remote execution queues.

Note: To use this command you need the Secret Manager Secret Accessor for Dev or setup a service account in your local that has those permissions.

Changes

  • Added the preprocess subcommand.
    • It interacts with the actual Datastore and GCS based on the provided configuration.
    • It fetches and populates the uworker_env with:
      • Job-specific environment variables from the Datastore.
      • Fuzzer-specific environment variables (for blackbox fuzzers).
      • Required logging metadata (CF_TASK_NAME, CF_TASK_ARGUMENT, CF_TASK_JOB_NAME, CF_TASK_ID) to ensure logs in the subsequent uworker_main step have the correct context.

Tests performed

Executed the following command in dev:

pipenv run python butler.py preprocess --fuzzer <fuzzer> --job <job> -c <config_dir>

Successfully creates and uploads the payload and returns a valid signed URL. This signed url was later used to trigger this swarming task trough prpc, here are the logs

@IvanBM18 IvanBM18 self-assigned this May 5, 2026
@IvanBM18 IvanBM18 requested a review from a team as a code owner May 5, 2026 04:01
@IvanBM18 IvanBM18 changed the title Bulter.py: Adds uworker_preprocess command for local preprocess Bulter.py: Adds preprocess command for local preprocess May 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant