Skip to content

Add automatic worker discovery to console#372

Merged
gabotechs merged 16 commits intodatafusion-contrib:mainfrom
EdsonPetry:edson.petry/console-worker-discovery
Mar 24, 2026
Merged

Add automatic worker discovery to console#372
gabotechs merged 16 commits intodatafusion-contrib:mainfrom
EdsonPetry:edson.petry/console-worker-discovery

Conversation

@EdsonPetry
Copy link
Copy Markdown
Contributor

@EdsonPetry EdsonPetry commented Mar 17, 2026

The console now always discovers workers via GetClusterWorkers RPC. This change enables the console to automatically discover workers over the network. Instrumentation now works like so:

// console/examples/console_worker.rs
let localhost_resolver = Arc::new(LocalhostWorkerResolver { ports: ..., });
let worker = Worker::default();

Server::builder()
    .add_service(worker.with_observability_service(resolver)) // users now pass in their WorkerResolver impl
    .add_service(worker.into_flight_server())
    .serve(SocketAddr::new(
        IpAddr::V4(Ipv4Addr::LOCALHOST),
        args.port.unwrap_or(DEFAULT_WORKER_PORT),
    )).await?;

Summary

  • Add a new GetClusterWorkers gRPC to the observability service that returns all known worker URLs by delegating to the existing WorkerResolver trait
  • Replace the console's manual --cluster-ports CLI flag with a --connect seed URL (defaults to port 9001)
  • Wire up the observability service in the benchmarks EC2 worker and all console examples with a user-provided WorkerResolver
  • Export DEFAULT_WORKER_PORT (9001) as the canonical default across the codebase
  • Add a console/README.md and update all example docs for the new auto-discovery workflow

@EdsonPetry EdsonPetry force-pushed the edson.petry/console-worker-discovery branch from 3af59fc to 6a34c46 Compare March 17, 2026 16:21
@EdsonPetry EdsonPetry changed the title Console automatically discovers workers Add automatic worker discovery to console Mar 17, 2026
@EdsonPetry EdsonPetry marked this pull request as ready for review March 17, 2026 19:34
Console now supports two modes:
- Auto-discovery (default): connects to a seed worker, calls
GetClusterWorkers to find all cluster workers, and re-polls every 5s for
topology changes
- Manual mode (--cluster-ports): unchanged behavior for local dev

New --connect flag specifies a seed URL; defaults to localhost:6789
…gument

Remove DEFAULT_WORKER_PORT from the library and make the console port a
required positional argument instead of silently defaulting to 9001.
@EdsonPetry EdsonPetry force-pushed the edson.petry/console-worker-discovery branch from 0ddbde1 to 8c87e4a Compare March 19, 2026 13:34
Copy link
Copy Markdown
Collaborator

@gabotechs gabotechs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 nice! no comments, this looks good

@gabotechs gabotechs merged commit 8b1c9ed into datafusion-contrib:main Mar 24, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants