From 679507ca418947df9784bfc1c5b1d150a548588e Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 10:30:56 -0400 Subject: [PATCH 1/7] docs: add cluster scaling guide for VM installs --- docs.json | 3 +- .../admin-console-configuration.mdx | 1 + enterprise/vm-install/scaling.mdx | 89 +++++++++++++++++++ 3 files changed, 92 insertions(+), 1 deletion(-) create mode 100644 enterprise/vm-install/scaling.mdx diff --git a/docs.json b/docs.json index d9a58c2d..eebaf5cf 100644 --- a/docs.json +++ b/docs.json @@ -539,7 +539,8 @@ { "group": "VM Install", "pages": [ - "enterprise/vm-install/admin-console-configuration" + "enterprise/vm-install/admin-console-configuration", + "enterprise/vm-install/scaling" ] }, { diff --git a/enterprise/vm-install/admin-console-configuration.mdx b/enterprise/vm-install/admin-console-configuration.mdx index 88cbcd08..a51a828b 100644 --- a/enterprise/vm-install/admin-console-configuration.mdx +++ b/enterprise/vm-install/admin-console-configuration.mdx @@ -220,6 +220,7 @@ See [External PostgreSQL](/enterprise/external-postgres) for version, encoding, | `Warm Runtime Count` | Number of ready sandboxes kept for faster conversation startup. Set to `0` for cold starts only. | | `Additional Host Path Mounts` | Host paths mounted into every sandbox, one per line as `host_path:container_path[:ro\|rw]`. | | `Enable /dev/kvm passthrough (QEMU/KVM)` | Makes host KVM acceleration available inside sandboxes. The node must expose `/dev/kvm`. | +| `Run sandboxes on dedicated nodes` | Confines sandboxes to machines added with the `sandbox` role, and keeps the application off those machines. Requires at least one `sandbox` machine already joined. See [Scaling the Cluster](/enterprise/vm-install/scaling). | `Idle Time` and `Deletion Time` control when idle and paused conversations are diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx new file mode 100644 index 00000000..688cd7a8 --- /dev/null +++ b/enterprise/vm-install/scaling.mdx @@ -0,0 +1,89 @@ +--- +title: Scaling the Cluster +description: Add machines to an OpenHands Enterprise VM deployment to increase capacity, and run sandboxes on dedicated machines. +icon: server +--- + +An OpenHands Enterprise VM deployment starts as a single machine that runs everything: the OpenHands application, its supporting services, and the sandboxes where conversations execute. Add machines when you need more capacity. + +## Machine Roles + +When you add a machine, you choose the role it takes. The role determines what runs on it and cannot be changed afterward. + +| Role | Runs | +|---|---| +| `app` | The OpenHands application and its supporting services. | +| `sandbox` | Sandboxes only. | + +## Recommended: Dedicated Sandbox Machines + +For production, run sandboxes on dedicated `sandbox` machines. + +Sandboxes are the most variable workload in a deployment. Each conversation gets one, their number rises and falls with user activity, and the code that runs inside them is arbitrary. When sandboxes share a machine with the OpenHands application, a burst of conversations competes for the same CPU and memory the application needs to serve requests. Separating them means sandbox demand cannot degrade or take down the application. + +Dedicated sandbox machines also give you a single dial for conversation capacity. When users need more concurrent conversations, add sandbox machines without touching the machines running the application. + +## Before You Begin + + + New machines must be able to reach the existing machines over your private network. Cluster traffic uses a specific set of ports, and some of it is UDP. If your environment restricts traffic between machines, open those ports first — a machine that cannot reach the others will appear to join successfully and then fail to run workloads. See [Replicated's port requirements](https://docs.replicated.com/enterprise/installing-embedded-requirements) for the list. + + +Adding machines requires the multi-node entitlement on your license. If `Add node` does not appear in the Admin Console, contact OpenHands Support. + +## Add a Machine + + + + In the Admin Console, select `Cluster Management`, then `Add node`. + + + Select `app` or `sandbox`. The role cannot be changed after the machine is added. + + + The Admin Console displays download, extraction, and join commands for the role you selected. Connect to the new machine and run them in order. Keep the Admin Console page open while you do this. + + + Return to `Cluster Management` and wait for the new machine's status to become `Ready`. + + + +## Add Sandbox Capacity + +Add one or more machines with the `sandbox` role, then confine sandboxes to them. + + + + Follow [Add a Machine](#add-a-machine) and select the `sandbox` role. Wait for its status to become `Ready`. + + + Open `Config`, find `Sandbox Configuration`, and enable `Run sandboxes on dedicated nodes`. Save and deploy the change. + + + + + Enable `Run sandboxes on dedicated nodes` only after at least one `sandbox` machine is `Ready`. Turn it on first and new conversations will have nowhere to run. A configuration check warns you if the setting is enabled while no sandbox machine exists. + + +Conversations that were already running stay on their original machine and are cleaned up normally as they go idle. Only new conversations move to the sandbox machines, so the transition needs no downtime. + +To add more conversation capacity later, add another `sandbox` machine. No configuration change is needed — the setting already routes sandboxes to every machine with that role. + +## Add Application Capacity + +Add machines with the `app` role to increase capacity for the OpenHands application itself. + +Machines with the `app` role also run the services that coordinate the cluster, which makes resilience step rather than climb: + +| `app` machines | Machine failures tolerated | +|---|---| +| 1 | 0 | +| 2 | 0 | +| 3 | 1 | + +Two `app` machines double capacity but tolerate no more failure than one, because the coordination services need more than half of them available. Three is the first count that survives losing a machine. If your goal is resilience rather than raw capacity, plan for three. + +## Related Guides + +- [Admin Console Configuration](/enterprise/vm-install/admin-console-configuration) +- [Conversations and Sandboxes](/enterprise/conversations-and-sandboxes) From 02d2b3789b6d48e563bf4c5429bc6151aee1b6cd Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 11:05:07 -0400 Subject: [PATCH 2/7] docs: tighten scaling guide wording --- enterprise/vm-install/scaling.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx index 688cd7a8..fe85fafc 100644 --- a/enterprise/vm-install/scaling.mdx +++ b/enterprise/vm-install/scaling.mdx @@ -19,9 +19,9 @@ When you add a machine, you choose the role it takes. The role determines what r For production, run sandboxes on dedicated `sandbox` machines. -Sandboxes are the most variable workload in a deployment. Each conversation gets one, their number rises and falls with user activity, and the code that runs inside them is arbitrary. When sandboxes share a machine with the OpenHands application, a burst of conversations competes for the same CPU and memory the application needs to serve requests. Separating them means sandbox demand cannot degrade or take down the application. +Sandboxes are the most variable workload in a deployment. When sandboxes share a machine with the OpenHands application, a burst of conversations competes for the same CPU and memory the application needs to serve requests. Separating them means sandbox demand cannot degrade or take down the application. -Dedicated sandbox machines also give you a single dial for conversation capacity. When users need more concurrent conversations, add sandbox machines without touching the machines running the application. +Dedicated sandbox machines also give you a dial for conversation capacity. ## Before You Begin @@ -41,7 +41,7 @@ Adding machines requires the multi-node entitlement on your license. If `Add nod Select `app` or `sandbox`. The role cannot be changed after the machine is added. - The Admin Console displays download, extraction, and join commands for the role you selected. Connect to the new machine and run them in order. Keep the Admin Console page open while you do this. + The Admin Console displays download, extraction, and join commands for the role you selected. Connect to the new machine and run them in order. Return to `Cluster Management` and wait for the new machine's status to become `Ready`. From eaadd1f89f815fa5a99f0bb1142c94442801d858 Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 11:12:44 -0400 Subject: [PATCH 3/7] docs: enumerate cluster ports and clarify sandbox setting order --- enterprise/vm-install/scaling.mdx | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx index fe85fafc..0f09a71c 100644 --- a/enterprise/vm-install/scaling.mdx +++ b/enterprise/vm-install/scaling.mdx @@ -26,10 +26,12 @@ Dedicated sandbox machines also give you a dial for conversation capacity. ## Before You Begin - New machines must be able to reach the existing machines over your private network. Cluster traffic uses a specific set of ports, and some of it is UDP. If your environment restricts traffic between machines, open those ports first — a machine that cannot reach the others will appear to join successfully and then fail to run workloads. See [Replicated's port requirements](https://docs.replicated.com/enterprise/installing-embedded-requirements) for the list. - + New machines must be able to reach the existing machines over your private network. If your environment restricts traffic between machines, open these ports first. A machine that cannot reach the others will appear to join successfully and then fail to run workloads. + + Open in both directions between all machines: `2380/TCP`, `4789/UDP`, `6443/TCP`, `9091/TCP`, `9443/TCP`, `10249/TCP`, `10250/TCP`, and `10256/TCP`. A joining machine also needs to reach `30000/TCP` and `50000/TCP` on the existing machines. -Adding machines requires the multi-node entitlement on your license. If `Add node` does not appear in the Admin Console, contact OpenHands Support. + Note that `4789` is UDP. Opening the whole set as TCP leaves the machines unable to exchange traffic. + ## Add a Machine @@ -62,7 +64,7 @@ Add one or more machines with the `sandbox` role, then confine sandboxes to them - Enable `Run sandboxes on dedicated nodes` only after at least one `sandbox` machine is `Ready`. Turn it on first and new conversations will have nowhere to run. A configuration check warns you if the setting is enabled while no sandbox machine exists. + You can enable `Run sandboxes on dedicated nodes` before adding a `sandbox` machine, but new conversations cannot start until one is `Ready`. A configuration check warns you if the setting is enabled while no sandbox machine exists. Conversations that were already running stay on their original machine and are cleaned up normally as they go idle. Only new conversations move to the sandbox machines, so the transition needs no downtime. From 7cb42d8e5c3d61de83b7acfe3bace922f7a1b5b4 Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 11:13:47 -0400 Subject: [PATCH 4/7] docs: drop HA guidance from scaling guide --- enterprise/vm-install/scaling.mdx | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx index 0f09a71c..ea616512 100644 --- a/enterprise/vm-install/scaling.mdx +++ b/enterprise/vm-install/scaling.mdx @@ -69,22 +69,12 @@ Add one or more machines with the `sandbox` role, then confine sandboxes to them Conversations that were already running stay on their original machine and are cleaned up normally as they go idle. Only new conversations move to the sandbox machines, so the transition needs no downtime. -To add more conversation capacity later, add another `sandbox` machine. No configuration change is needed — the setting already routes sandboxes to every machine with that role. +To add more conversation capacity later, add another `sandbox` machine. ## Add Application Capacity Add machines with the `app` role to increase capacity for the OpenHands application itself. -Machines with the `app` role also run the services that coordinate the cluster, which makes resilience step rather than climb: - -| `app` machines | Machine failures tolerated | -|---|---| -| 1 | 0 | -| 2 | 0 | -| 3 | 1 | - -Two `app` machines double capacity but tolerate no more failure than one, because the coordination services need more than half of them available. Three is the first count that survives losing a machine. If your goal is resilience rather than raw capacity, plan for three. - ## Related Guides - [Admin Console Configuration](/enterprise/vm-install/admin-console-configuration) From 2d0c18619966ddbd12d803848b475d81f178cba8 Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 11:15:14 -0400 Subject: [PATCH 5/7] docs: bullet the cluster port list --- enterprise/vm-install/scaling.mdx | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx index ea616512..6a33ef20 100644 --- a/enterprise/vm-install/scaling.mdx +++ b/enterprise/vm-install/scaling.mdx @@ -28,7 +28,18 @@ Dedicated sandbox machines also give you a dial for conversation capacity. New machines must be able to reach the existing machines over your private network. If your environment restricts traffic between machines, open these ports first. A machine that cannot reach the others will appear to join successfully and then fail to run workloads. - Open in both directions between all machines: `2380/TCP`, `4789/UDP`, `6443/TCP`, `9091/TCP`, `9443/TCP`, `10249/TCP`, `10250/TCP`, and `10256/TCP`. A joining machine also needs to reach `30000/TCP` and `50000/TCP` on the existing machines. + Open in both directions between all machines: + + - `2380/TCP` + - `4789/UDP` + - `6443/TCP` + - `9091/TCP` + - `9443/TCP` + - `10249/TCP` + - `10250/TCP` + - `10256/TCP` + + A joining machine also needs to reach `30000/TCP` and `50000/TCP` on the existing machines. Note that `4789` is UDP. Opening the whole set as TCP leaves the machines unable to exchange traffic. From 12b7b457da0dfe3a464141d560613d858f3b0432 Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 11:15:37 -0400 Subject: [PATCH 6/7] docs: trim UDP note --- enterprise/vm-install/scaling.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx index 6a33ef20..bb64cdc2 100644 --- a/enterprise/vm-install/scaling.mdx +++ b/enterprise/vm-install/scaling.mdx @@ -41,7 +41,7 @@ Dedicated sandbox machines also give you a dial for conversation capacity. A joining machine also needs to reach `30000/TCP` and `50000/TCP` on the existing machines. - Note that `4789` is UDP. Opening the whole set as TCP leaves the machines unable to exchange traffic. + Note that `4789` is UDP. ## Add a Machine From 92eef47d100df395dde7f9ef65a19090c59cbb2f Mon Sep 17 00:00:00 2001 From: Joe Laverty Date: Fri, 21 Aug 2026 11:25:47 -0400 Subject: [PATCH 7/7] docs: warn against selecting both node roles --- enterprise/vm-install/scaling.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/enterprise/vm-install/scaling.mdx b/enterprise/vm-install/scaling.mdx index bb64cdc2..8ba7ef3b 100644 --- a/enterprise/vm-install/scaling.mdx +++ b/enterprise/vm-install/scaling.mdx @@ -61,6 +61,10 @@ Dedicated sandbox machines also give you a dial for conversation capacity. + + You can select both `app` and `sandbox`, but this is not recommended. A machine with both roles runs the application and sandboxes together, which gives up the separation you are adding the machine for. When adding a sandbox machine, make sure `app` is unchecked. + + ## Add Sandbox Capacity Add one or more machines with the `sandbox` role, then confine sandboxes to them.