Repository navigation
Conversation
The sync matched SMG router workers to jobs by URL and removed every worker that wasn't a ready dstack worker, so users couldn't register workers of their own. It also probed every worker on each sync and deregistered the ones that failed, so a flaky connection from the server made a worker flap even when the router, which reaches it within the cluster, had no problem with it. Probes ran one at a time, and only for the connection mode and runtime type of the workers already registered, which broke services that mix them or switch between them in a rolling deployment. Now each worker that dstack registers is labeled with its job ID (`dstack.ai/job-id`), and the sync works in terms of jobs: * A worker is probed only while some router lacks it. Once it's registered, its health is left to the router, which checks it anyway. A worker that is truly unreachable has its job terminated, which deregisters it. * A worker is removed once its job is no longer running. Workers that dstack can't map to a job are left alone, so users can register their own. * Workers registered before the label existed are mapped to jobs by address. The ones whose job has already stopped are no longer removed: the router marks them unhealthy, and a rolling deployment of the router drops them. * Unregistered workers are probed for HTTP, SGLang gRPC and vLLM gRPC at once, with no hints from the registered ones. * Routers and workers are processed in parallel, up to 8 replicas at a time. An unexpected error in one of them is logged and no longer aborts the sync for the others. Also: * `gather_async()` is added, and `gather_map_async()` is built on it. Both accept `max_concurrency`, and cancel the remaining awaitables when one fails unless `return_exceptions` is set. * Router responses are validated with pydantic. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sync matched SMG router workers to jobs by URL and removed every worker that wasn't a ready dstack worker, so users couldn't register workers of their own. It also probed every worker on each sync and deregistered the ones that failed, so a flaky connection from the server made a worker flap even when the router, which reaches it within the cluster, had no problem with it. Probes ran one at a time, and only for the connection mode and runtime type of the workers already registered, which broke services that mix them or switch between them in a rolling deployment.
Now each worker that dstack registers is labeled with its job ID (
dstack.ai/job-id), and the sync works in terms of jobs:Also:
gather_async()is added, andgather_map_async()is built on it. Both acceptmax_concurrency, and cancel the remaining awaitables when one fails unlessreturn_exceptionsis set.