diff --git a/aws-healthomics/POWER.md b/aws-healthomics/POWER.md index 16f66ed..be07604 100644 --- a/aws-healthomics/POWER.md +++ b/aws-healthomics/POWER.md @@ -16,6 +16,7 @@ Whenever you are asked to perform a task related to any of the following scenari - Creating a workflow from a remote Git repository URL (GitHub, GitLab, Bitbucket) -> use `./steering/git-integration.md` (takes precedence over workflow-development.md) - Creating a new WDL, Nextflow or CWL workflow from local files -> use `./steering/workflow-development.md` +- Configuring a task to automatically fall back across GPU types or to CPU using `omicsResourceFallbackOrder` -> use `./steering/resource-fallback.md` - Running a deployed HealthOmics workflow -> use `./steering/running-a-workflow.md` - Submitting, monitoring, or managing batch runs (multiple samples/runs at once) -> use `./steering/batch-runs.md` - Onboarding an existing WDL workflow ensuring compatibility with HealthOmics -> use `./steering/migration-guide-for-wdl.md` diff --git a/aws-healthomics/steering/resource-fallback.md b/aws-healthomics/steering/resource-fallback.md new file mode 100644 index 0000000..b5a8f9c --- /dev/null +++ b/aws-healthomics/steering/resource-fallback.md @@ -0,0 +1,199 @@ +# SOP: Custom Compute Fallback (`omicsResourceFallbackOrder`) + +## Purpose + +This SOP defines how you, the agent, configure a HealthOmics WDL task to automatically fall +back across an ordered list of resource profiles (for example GPU type A -> GPU type B -> CPU) +when the preferred resource isn't available, instead of the task failing outright. For general +task resource directives (`acceleratorType`, `cpu`, `memory`), see the +[Workflow Development SOP](./workflow-development.md#task-parameters). + +## When to Use This + +Use `omicsResourceFallbackOrder` when the user wants a task to: +- Try one GPU type, then fall back to a different GPU type if the first is out of capacity. +- Try a GPU, then fall back to a CPU-only profile if no GPU is available. +- Reduce `ALL_PROFILES_INSTANCE_RESERVATION_FAILED`/capacity-related task failures for + GPU-dependent tasks without the customer polling and resubmitting manually. + +Do NOT reach for this directive for OOM or transient 5xx retries — those are already handled +by the standard `maxRetries` directive and operate independently (see +[Interaction with Other Retries](#interaction-with-other-retries) below). + +## Procedure: Adding `omicsResourceFallbackOrder` to a Task + +### Step 1. Replace the individual resource directives + +- `omicsResourceFallbackOrder` REPLACES the task's top-level `acceleratorType`, + `acceleratorCount`, `cpu`, and `memory` directives. You MUST NOT set any of these at the + top level in a task that also sets `omicsResourceFallbackOrder`. +- Task-level fields that are NOT resource-profile-specific (`docker`, `maxRetries`, + `preemptible`, etc.) stay at the top level as usual and apply the same way regardless of + which profile is active. + +### Step 2. Write the profile list + +- `omicsResourceFallbackOrder` is a WDL array of map literals in the `runtime {}` block. Each + map is one profile to try, in order. +- You MUST use quoted (string) keys in every profile — e.g. `{"acceleratorType": "nvidia-l4"}`. + Unquoted/bareword keys are invalid and fail workflow validation. +- Each profile MAY set any of these 5 fields; all are optional per profile, but a profile + MUST NOT be empty (`{}`): + + | Field | Type | Default when omitted | Max | + |---|---|---|---| + | `acceleratorType` | String | Field absent -> CPU profile | n/a — must be one of the 7 supported values below, or absent | + | `acceleratorCount` | Integer | Absent when `acceleratorType` is also absent | 4 | + | `cpu` | Integer or Float | 1 vCPU, or a GPU-instance default if a GPU profile omits it | 192 | + | `memory` | String (e.g. `"32 GiB"`) | 1 GiB, or a GPU-instance default if a GPU profile omits it | 1536 GiB | + | `omicsResourceWaitTimeoutInMin` | Integer | System default wait window | none (recommend >= 20 min; lower produces a warning) | + +- `acceleratorType`, when set, MUST be one of the 7 supported values: `nvidia-tesla-t4`, + `nvidia-tesla-t4-a10g`, `nvidia-tesla-a10g`, `nvidia-t4-a10g-l4`, `nvidia-l4-a10g`, + `nvidia-l4`, `nvidia-l40s`. Omit the field entirely to signal a CPU-only profile — do NOT + set it to `""` or `"cpu"`; both are rejected. +- `acceleratorCount` and `acceleratorType` MUST be set together — a profile cannot have one + without the other. +- The list MUST contain at least 1 and at most 10 profiles. +- At most one CPU profile (a profile that omits `acceleratorType`) is allowed, and if + present it MUST be the last entry in the list — HealthOmics exhausts all GPU options + before falling back to CPU. +- A field omitted from a profile takes its documented default, NOT a value copied from an + earlier profile in the list — defaults do not carry over between profiles. + +### Step 3. Handle the resource-type signal in the command + +- HealthOmics sets the `AWS_HEALTHOMICS_RESOURCE_TYPE` environment variable in the container + to the active profile's accelerator type (e.g. `nvidia-l40s`), or `cpu` for a CPU-only + profile. Use it to branch the command when the GPU and CPU code paths differ. +- The container image (`docker` directive) MUST support every code path referenced across + all profiles in the list, since the same image runs regardless of which profile is active. + +### Step 4. Set `docker` and `maxRetries` normally + +- `docker` and `maxRetries` are shared, not per-profile — set them once at the top level as + you would for any other task. See the [Workflow Development SOP](./workflow-development.md#task-parameters). + +## Example + +```wdl +task align { + command <<< + if [ "$AWS_HEALTHOMICS_RESOURCE_TYPE" = "cpu" ]; then + sentieon bwa mem -t 32 ~{reference} ~{fastq} + else + pbrun fq2bam --ref ~{reference} --in-fq ~{fastq} + fi + >>> + + runtime { + docker: "my-registry/align-multi-arch:latest" # must support both GPU and CPU code paths + maxRetries: 2 # OOM retry with 2 attempts + + omicsResourceFallbackOrder: [ + {"acceleratorType": "nvidia-l40s", "acceleratorCount": 1, "cpu": 8, "memory": "32 GiB", "omicsResourceWaitTimeoutInMin": 45}, + {"acceleratorType": "nvidia-l4", "acceleratorCount": 1, "cpu": 8, "memory": "32 GiB"}, + {"cpu": 32, "memory": "128 GiB"} + ] + } +} +``` + +This task tries `nvidia-l40s` first (waiting up to 45 minutes for capacity), then `nvidia-l4` +(default wait window), then falls back to a CPU-only profile (32 vCPUs, 128 GiB) if neither +GPU is available. + +## Interaction with Other Retries + +- OOM and service-error (5xx) retries via `maxRetries` are unaffected by + `omicsResourceFallbackOrder` — they retry within the currently active profile and never + advance to the next profile in the fallback order. A retry that exhausts `maxRetries` fails + the task as usual. +- If every profile in the fallback list is exhausted with no instance reservation, the task + fails with failure reason `ALL_PROFILES_INSTANCE_RESERVATION_FAILED`. This is terminal — + HealthOmics does NOT retry for this failure reason even if `maxRetries` is configured. If a + user hits this, suggest raising `omicsResourceWaitTimeoutInMin` on the relevant profile(s) + rather than relying on `maxRetries`. + +## Validation Rules Enforced at Workflow Creation and Task Run Time + +Most rules below cause outright rejection. The exceptions — below-minimum wait time, +duplicate profiles, and an `acceleratorType` unsupported in the run's region — only produce a +warning (the profile is skipped at scheduling time for the region case). + +1. Cannot mix `omicsResourceFallbackOrder` with the top-level `acceleratorType`, + `acceleratorCount`, `cpu`, `memory`, or `omicsResourceWaitTimeoutInMin` directives. +2. Must be a list (array), not a single map. +3. Must contain at least 1 profile. +4. Must not contain more than 10 profiles. +5. Every profile key MUST be a quoted string. +6. No empty profiles (`{}`). +7. `acceleratorType` and `acceleratorCount` must be set together, or both omitted. +8. `acceleratorType`, if set, must be one of the 7 supported values (or omitted for CPU). +9. `omicsResourceWaitTimeoutInMin` below 20 minutes produces a warning, not a rejection. +10. Unrecognized fields in a profile are rejected. +11. Field values must be the correct type (e.g. `cpu` must be numeric, not a string). +12. At most one CPU profile is allowed in the list. +13. A CPU profile, if present, must be the last profile in the list. +14. Duplicate profiles are allowed but produce a warning (they typically provide no benefit). +15. `cpu` (max 192), `memory` (max 1536 GiB), and `acceleratorCount` (max 4) are each + rejected above their maximum. +16. An `acceleratorType` unsupported in the workflow's AWS region produces a warning, not a + rejection — that profile is skipped at task scheduling time, so confirm a later profile in + the list doesn't also depend on an unsupported type. + +## If Workflow Creation or a Run Fails + +- IF the workflow fails to create with a validation error referencing + `omicsResourceFallbackOrder`, check the profile list against the Validation Rules above — + the most common causes are unquoted keys (rule 5), mixing with top-level resource directives + (rule 1), or a CPU profile that isn't last (rule 13). +- IF a task fails with `ALL_PROFILES_INSTANCE_RESERVATION_FAILED`, this means every profile in + the list ran out its wait window without securing capacity — see + [Interaction with Other Retries](#interaction-with-other-retries). For other run failures, + see the [Troubleshooting SOP](./troubleshooting.md). + +## Observability + +HealthOmics emits customer-facing CloudWatch log events while a task moves through the +fallback list. Point the user to these when they ask why a task took a while to start or which +profile it landed on: +- `RESOURCE_PROFILE_STARTED` — emitted when the task begins searching for capacity on a + profile. +- `RESOURCE_PROFILE_ADVANCED` — emitted when a profile's wait window is exhausted and the task + advances to the next profile; the message includes which profile was unavailable, how long + it waited, and which profile it's advancing to. + +## Timeout Best Practices + +- The default wait window when `omicsResourceWaitTimeoutInMin` is omitted is 20 minutes. +- A value as low as 1 minute is accepted, but only allows roughly one capacity-search cycle + before advancing — this is USUALLY too aggressive for anything but a deliberately fast + first-choice profile. +- RECOMMEND >= 20 minutes so the profile gets a fair chance before advancing (see rule 9 + above). Shortening trades faster fallback for lower odds of securing that profile — this + applies regardless of accelerator type, since there's no signal for current capacity. +- The sum of every profile's wait window SHOULD stay well under the run's overall timeout + (default maximum 7 days) — a long fallback list with generous per-profile timeouts can + otherwise consume most of the run's time budget before a task even starts. + +## Region Behavior + +- IF a profile's `acceleratorType` has no capacity fleet in the workflow's AWS region, that + profile is skipped almost immediately rather than waiting out its + `omicsResourceWaitTimeoutInMin` window — HealthOmics only enters the timed capacity-search + loop once it confirms the type is deployed in the region. +- This is NOT an error — it's expected behavior for a fallback list that includes an + accelerator type unsupported in the current region (see validation rule 16). The task + advances to the next profile in the list as usual. +- Tell the user to check [Regional Capabilities SOP](./regional-capabilities.md) if they see a + profile skip unexpectedly fast — the accelerator type they listed may not be deployed in + their run's region. + +## Related Documentation +- For general task resource directives (`acceleratorType`, `cpu`, `memory` outside of a + fallback list), see [Workflow Development SOP](./workflow-development.md#task-parameters). +- For the supported GPU accelerator types and per-region availability, see + [Regional Capabilities SOP](./regional-capabilities.md). +- For diagnosing workflow creation issues and run failures generally, see the + [Troubleshooting SOP](./troubleshooting.md).