Add resource agent, dashboard, and proxy resource snapshot reporting#82
Add resource agent, dashboard, and proxy resource snapshot reporting#82zhy1658858023 wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5745296314
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| it = self._items.get(instance_id) | ||
| if not it: | ||
| return False | ||
| it.last_seen_at = now |
There was a problem hiding this comment.
When the new resource agent/proxy_reporter is launched as a separate process, it can keep posting host snapshots after the actual Instance server or its heartbeat loop has stopped. Refreshing last_seen_at here makes InstancePool.list(include_dead=False) continue returning that dead instance, so select_instance can dispatch requests to an unavailable host:port instead of letting the TTL expire; resource updates should not renew serving liveness unless they are coupled to the Instance heartbeat.
Useful? React with 👍 / 👎.
| "{{\"index\":{},\"uuid\":\"{}\",\"name\":\"{}\",\"utilization_pct\":{},\"memory_used_mb\":{},\"memory_total_mb\":{},\"memory_free_mb\":{},\"temperature_c\":{},\"power_w\":{},\"health\":\"ok\"}}", | ||
| cols[0].parse::<u32>().unwrap_or(0), json_escape(cols[1]), json_escape(cols[2]), cols[3], cols[4], cols[5], cols[6], cols[7], cols[8].replace("[Not Supported]", "0") |
There was a problem hiding this comment.
On GPUs or drivers where nvidia-smi reports unsupported metrics as N/A or [Not Supported], these columns are interpolated directly into numeric JSON fields, so a value like "power_w":N/A makes /v1/resource/snapshot invalid JSON and causes both the dashboard and proxy_reporter to reject the snapshot. Parse each metric with a numeric fallback or emit null before formatting the JSON.
Useful? React with 👍 / 👎.
|
|
||
| def build_parser() -> argparse.ArgumentParser: | ||
| parser = argparse.ArgumentParser(description="Report Instance resource snapshots to Proxy control plane") | ||
| parser.add_argument("--agent-url", default="http://127.0.0.1:9201", help="local Rust resource agent base URL") |
There was a problem hiding this comment.
Running the new resource agent and proxy_reporter.py with their defaults does not work: the Rust agent binds to 127.0.0.1:9101, while the reporter queries http://127.0.0.1:9201. In that default setup the reporter just logs HTTP-unavailable warnings and no snapshots reach the Proxy unless users discover and override one side, so these defaults should be aligned.
Useful? React with 👍 / 👎.
|
|
||
| def build_parser() -> argparse.ArgumentParser: | ||
| p = argparse.ArgumentParser(description="CacheRoute Instance resource dashboard") | ||
| p.add_argument("--dashboard-listen", default="0.0.0.0:9202", help="dashboard listen address") |
There was a problem hiding this comment.
With the default --dashboard-listen this server binds to all interfaces while also exposing unauthenticated /api/agent/start and /api/agent/stop endpoints. On a shared or routable experiment host, anyone who can reach port 9202 can stop the managed resource agent or repeatedly start cargo run, disrupting resource reporting; defaulting to 127.0.0.1 or adding an explicit auth/unsafe flag would avoid exposing process control by default.
Useful? React with 👍 / 👎.
1 participant
Motivation
Description
instance/resource_agent/src/main.rsexposing/healthzand/v1/resource/snapshot, capturing CPU, memory, network, and GPU info and producing aresource_snapshot_v1JSON payload.instance/resource_agent/Cargo.tomlandCargo.lockfor the agent manifest.instance/resource_agent/proxy_reporter.pywhich periodically fetches the local agent snapshot and POSTs it to the Proxy control plane endpoint.instance/resource_dashboard/: a standard-library serverdashboard_server.py, a Tk desktop appdashboard_app.py, and static web UI assets instatic/(index.html,app.js,style.css) to visualize snapshots and manage the agent process.InstanceResourcetoproxy/resource/instance_pool.py, implementingreport_resource_snapshotand helpers to convert raw snapshots into structured fields.POST /v1/instance/resource_snapshotinproxy/resource/p_control_plane.py, and includeresourcein the/v1/instance/listresponse.Testing
cargo checkforinstance/resource_agentwhich completed successfully.python -m py_compile) which succeeded.Codex Task