feat(console): rewrite TUI with worker-centric dashboard and live metrics

[EdsonPetry]

Summary

• Rewrite the console TUI with a worker-centric dashboard featuring a cluster overview and per-worker detail views, replacing the previous global task view
• Extend the observability service to collect worker-level metrics including CPU usage, RSS memory, output rows, unique query IDs, and task counts via updated protobuf definitions
• Add CPU/RSS sparklines and sortable columns to the worker table, with queries-in-flight metric in the cluster metrics panel
• Add new examples: a cluster spawner/killer example and concurrent TPC-DS query runner with explain analyze support

Changes

Console TUI (console/)

• Split monolithic ui.rs into modular components: cluster.rs, worker.rs, header.rs, footer.rs, help.rs
• New input.rs for keyboard event handling and state.rs for shared UI state
• Worker table with sortable columns (Name, Stage, Tasks, Queries, CPU, Memory)
• Sparkline charts for CPU and RSS per worker
• Cluster-level metrics: throughput, active/total workers, completed queries, avg duration
• Help overlay (toggle with ?)

Observability (src/observability/)

• Extended protobuf schema with cpu_usage, rss_bytes, output_rows, query_ids, num_tasks fields
• Updated service to generate and collect the new per-worker metrics
• Worker now reports richer metrics through the progress API

Flight service (src/flight_service/)

• Changed observability_service() generator method on Worker to with_observability_service(), users now no longer need to wrap the ObservbilityServiceImpl that the old observability service generator created with a ObservabilityServiceServer.
  old:

    let worker = Worker::default();
   let observability_service = worker.observability_service();

    Server::builder()
        .add_service(ObservabilityServiceServer::new(observability_service))
        .add_service(worker.into_flight_server())
        .serve(SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), args.port))
        .await?;

new

    let worker = Worker::default();

    Server::builder()
        .add_service(worker.with_observability_service())
        .add_service(worker.into_flight_server())
        .serve(SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), args.port))
        .await?;

Examples

• cluster.rs: spawns and manages a local cluster of workers with observability
• tpcds_runner.rs: supports concurrent query execution and explain analyze

[gabotechs]

Looks good! I have not made a deep review over the code in the console/ folder, but I trust you on that. It looks great!

[EdsonPetry]

@gabotechs I didn't catch this before, but this code here isn't correct. I should use move here instead of async move because the body of the closure is never executed within the spawned thread. The spawned thread receives a future here that is goes unpolled. Also because this is a separate background thread that isn't in tokio's thread pool thus I believe calling thread::sleep is correct.

[EdsonPetry]

I'll push a fix for this in the PR #372

[gabotechs]

🤔 I don't understand why this should be incorrect. Can you push the fix as a separate PR so that we can review it in isolation?

[EdsonPetry]

Sounds good, the issue I was running into is that because the Future within that async closure was going unpolled the system metrics collection loop was never executing and thus the console was only displaying 0 values, this compiles fine because what the compiler sees is just thread::spawn(async move || { Future<...> }).

Opening a separate PR for this now.