feat(cli): VPC support for the AWS installer

[kylehounslow]

VPC support for the AWS CLI installer

Closes opensearch-project#331.

Adds an opt-in VPC deployment path to the aws/cli-installer. When you pass --vpc-id (with --subnet-ids and --security-group-ids), the OpenSearch domain, the OSIS ingestion pipeline, and the EC2 demo instance are all placed inside the selected VPC on private endpoints. Omitting --vpc-id leaves the current public-endpoint path unchanged.

What changed

• New flags --vpc-id, --subnet-ids, --security-group-ids (comma-separated), plus a Network topology step in interactive advanced mode.
• OpenSearch domain: created with VPCOptions across the given subnets; zone-aware when more than one subnet (AZ) is provided, rounding the data-node count up to a multiple of the AZ count.
• OSIS pipeline: attached to the same VPC (first two subnets) so it reaches the private domain endpoint; the ingestion endpoint becomes VPC-private.
• EC2 demo: launches into the selected subnet and security groups so the otel-demo workload sits in the same network as the domain and pipeline.
• Pre-flight VPC validation: before any resource is created, validateVpcTopology() calls the EC2 Describe APIs to confirm the VPC exists in the region, every subnet and security group belongs to it, and the subnets sit in distinct AZs (the zone-awareness rule the domain config derives from the subnet count). Wrong network inputs fail in seconds with a clear message instead of minutes into domain or pipeline creation, and leave nothing half-built.
• Pre-deploy diagram shows the boundary: the ASCII architecture diagram rendered right before the deploy confirmation now marks the boxes that run inside the VPC ([vpc] on the EC2 demo, the OSI ingest endpoint, and the OpenSearch domain) and adds a network-topology header with the VPC/subnet/SG IDs. Prometheus, the Connected Data Source, and the UI stay untagged since they're regional services reached over the AWS-internal path. The public render is unchanged.

Reachability model (why you can run this from anywhere)

A VPC-private domain's own endpoint isn't reachable from outside the VPC, but the managed OpenSearch UI (Application) endpoint is public and proxies to the domain over the AWS-internal path. So for VPC domains the installer:

1. sets the caller's IAM principal as the domain master (MasterUserARN, no internal user DB), so SigV4 is authorized on a fresh domain with no in-VPC bootstrap;
2. runs authorize-vpc-endpoint-access for the application.opensearchservice.amazonaws.com service so the UI can reach the domain through its VPC endpoint (without this the UI returns No Living connections); and
3. performs FGAC role mapping and UI setup through that reachable Application endpoint.

No bastion or VPN is required.

Reliability: creation ordering / race conditions

A fresh deploy is now deterministic. Fixed four ways it could intermittently fail or silently not flow data:

1. Domain readiness gates on Processing clearing, not just the endpoint URL appearing. The URL is published while the cluster is still initializing, which raced the security-API calls that run immediately after.
2. FGAC role mapping (both the direct basic-auth path and the VPC OpenSearch-UI path) retries transient 5xx / connection / No Living connections errors, and fails hard if it never lands. Previously a silent miss left the pipeline ACTIVE but unable to write.
3. OSIS CreatePipeline and AddDirectQueryDataSource retry on IAM role-not-propagated errors instead of relying on a fixed sleep.
4. The CLI validates up front that an OpenSearch backend is selected. Advanced mode with only VPC flags used to leave the backend unset, skip domain creation, and fail deep in pipeline creation with an empty host.

Architecture

General VPC deployment (the example shows two private subnets across AZs; one or three also work):

flowchart TB
  caller["Operator / CI<br/>runs the CLI; its IAM principal<br/>is the VPC domain master"]

  subgraph aws["AWS Account / Region"]
    direction TB

    subgraph clientrow[" "]
      direction LR
      user["Browser (dashboards)"]

      subgraph regional["Managed services (regional, outside the VPC)"]
        direction LR
        app["OpenSearch UI (Application)<br/>public endpoint"]
        cds["Connected Data Source<br/>OpenSearch to APS direct query"]
        amp["Amazon Managed Prometheus<br/>workspace (remote_write)"]
      end
    end

    subgraph vpc["VPC 10.0.0.0/16"]
      direction TB

      sg["Security group<br/>self-referencing intra-VPC allow;<br/>egress to NAT"]

      subgraph pub["Public subnet"]
        nat["NAT gateway</br>(when otel-demo enabled)"]
      end

      subgraph azA["Private subnet - AZ a"]
        domA["OpenSearch node + ENI<br/>(zone-aware)"]
        osiA["OSIS pipeline + ENI"]
        apps["Existing apps / pods<br/>(EKS / ECS workloads)<br/>emit OTLP :4317/:4318"]
      end

      subgraph azB["Private subnet - AZ b"]
        domB["OpenSearch node + ENI<br/>(zone-aware)"]
        osiB["OSIS pipeline + ENI"]
        ec2["EC2 demo instance (optional)<br/>OTel Collector + OTel Demo<br/>(OTLP :4317/:4318)"]
      end
    end
  end

  %% Telemetry data path: one OSIS pipeline, ENIs in both subnets;
  %% metrics egress drawn once from the ENI nearest APS to avoid a full-width edge
  apps e0@-->|"OTLP over SigV4 (osis:Ingest)"| osiA
  ec2 e1@-->|"OTLP over SigV4 (osis:Ingest)"| osiB
  osiA e2@-->|"logs / traces / service-map<br/>es:ESHttp*"| domA
  osiB e3@--> domB
  osiA e4@-->|"metrics</br>(to AMP via remote_write)"| amp

  %% Query / UI path
  user e5@-->|"HTTPS (AWS sign-in)"| app
  app e6@-->|"PPL query: logs / traces<br/>on the OpenSearch nodes"| domA
  app e7@-->|"metrics query via<br/>connected data source"| cds
  cds e8@-->|"PromQL direct query"| amp

  %% Control / setup path
  caller e9@-->|"create + FGAC role mapping<br/>via the UI (SigV4)"| app
  caller e10@-->|"authorize-vpc-endpoint-access<br/>for the UI service"| domA
  ec2 e11@-->|"image + repo pulls"| nat

  %% Animate the live telemetry data path (fast) and the query/UI path (slow)
  e0@{ animation: fast }
  e1@{ animation: fast }
  e2@{ animation: fast }
  e3@{ animation: fast }
  e4@{ animation: fast }
  e5@{ animation: slow }
  e6@{ animation: slow }
  e7@{ animation: slow }
  e8@{ animation: slow }

  %% Edge colors carry meaning: green telemetry, blue query/UI, muted control/setup
  linkStyle 0,1,2,3,4 stroke:#a3e635,stroke-width:2px
  linkStyle 5,6,7,8 stroke:#60a5fa,stroke-width:2px
  linkStyle 9,10,11 stroke:#a3a3a3,stroke-width:1.5px,stroke-dasharray:4 4

  classDef ext fill:#eef,stroke:#557,color:#113;
  classDef net fill:#efe,stroke:#575,color:#131;
  classDef byoc fill:#fff,stroke:#888,stroke-width:1.5px,stroke-dasharray:5 4,color:#333;
  class app,amp,cds ext;
  class sg,nat net;
  class apps byoc;
  style clientrow fill:none,stroke:none;

Loading

End-to-end verification

The change was deployed and verified end to end on a real VPC (2 private subnets across AZs, self-referencing security group, NAT egress). Domain, pipeline, and demo were all VPC-private; verification was driven entirely from outside the VPC.

Deployed

• OpenSearch domain: VPC-private endpoint, zone-aware across two AZs, IAM-principal master (no internal password). Provisioned in ~11 min.
• OSIS pipeline: VPC-attached (endpoint interfaces in both private subnets); the ingest endpoint is VPC-private.
• APS workspace + Connected Data Source: OpenSearch to Prometheus direct query.
• OpenSearch UI (Application): public endpoint; FGAC roles (all_access, security_manager), the workspace, index patterns, and dashboards were all set up through it against the VPC-private domain.
• EC2 demo: in a private subnet with no public IP, running the OTel Collector + OTel Demo (load generator driving traffic), exporting OTLP to the VPC-private OSIS endpoint over SigV4.

Data flowing (CloudWatch AWS/OSIS, ~10 min window):

Stage	Metric	Value
OTLP received	otlp.traces.successRequests.count	75
Logs to OpenSearch	otel-logs-pipeline.opensearch.recordsIn.count	4,628
Traces processed	otel-traces-pipeline.recordsProcessed.count	13,343
Metrics to Prometheus	otel-metrics-pipeline.prometheus.sinkMetricsSucceeded.count	18,454
Service-map to OpenSearch	service-map-pipeline.opensearch.recordsIn.count	66

Queryable in the domain via the OpenSearch UI (SigV4, through the public Application endpoint that proxies to the VPC-private domain):

Index	_count
logs-otel-v1*	5,416
otel-v1-apm-span*	12,825
otel-v2-apm-service-map*	249

All test resources were torn down afterward with destroy.

One operational note the live run surfaced: a VPC OSIS ingest endpoint terminates TLS immediately but does not serve requests for ~10-15 min after the pipeline reports ACTIVE (connection closes with no HTTP response; requestsReceived stays 0). Once warm it behaves identically to a public pipeline. This is AWS-side ingestion warmup, not a security-group or configuration issue: a public pipeline on the same host returned auth responses throughout, and the VPC endpoint's DNS/TLS resolved correctly the whole time.

Testing

node --test test/unit.test.mjs: 64 tests pass. New coverage: config-level VPC option validation, live VPC topology validation (each EC2 Describe branch, via injected accessors), the retry helper, the role-not-propagated and transient-HTTP classifiers, the backend-required guard, FGAC principal selection, and the architecture diagram's VPC annotations (including a regression check that box widths match the public render). Existing pipeline/EC2 coverage unchanged.