Skip to content

STRUCTURE.md

Latest commit

 

History

History
232 lines (181 loc) · 10.6 KB

STRUCTURE.md

File metadata and controls

232 lines (181 loc) · 10.6 KB
Raw

Repository Structure & Naming Convention

This page defines how code in cloudemu is named and where it goes. The goal is that a service has the same name in every layer, files are where you expect them, and new code has an obvious place. Read it before adding a service, a resource or a sub-surface.

For the reasons behind the three-layer design, see architecture.md. This page only covers naming and placement.

1. The three layers

Every service spans up to three layers, in three parallel trees:

services/<capability>/           # portable API + driver interface (cross-cloud abstraction)
providers/<cloud>/<service>/     # in-memory mock for one cloud
server/<cloud>/<service>/        # SDK-compat wire handler for one cloud
  • <cloud> is one of aws, azure, gcp.
  • Not every service has all three layers. A product-specific mock (e.g. bedrock) may have no cross-cloud services/ abstraction, and some services have no wire handler yet. The naming rules below still apply to whichever layers exist.

Sibling modules under contrib/

Large opt-in add-ons live in their own Go modules under contrib/, so their dependencies stay out of the core cloudemu module:

contrib/realengine/       # opt-in real engines, no Docker (embedded-postgres, miniredis, subprocess)
contrib/dockerengine/     # opt-in real engines, Docker required (mysql, compute, containers, azure functions)
contrib/server/           # the `cloudemu-server` binary (real engines wired onto the wire servers)
contrib/testcontainers/   # a Testcontainers Go module

A real-engine implementation goes in the matching contrib/<realengine|dockerengine>/<capability>/ subpackage and satisfies the config.<X>Engine interface from config/engine.go.

2. Naming (the rule)

cloudemu is an SDK-compat emulator, so pointing a real SDK or CLI at it should feel natural. The provider and wire layers keep the name that cloud's own SDK uses, and the shared abstraction uses a generic capability name.

2.1 services/<capability>: generic capability name

The shared abstraction is named for the capability, never a product: storage, compute, networking, loadbalancer, iam, database, relationaldb, dns, monitoring, logging, secrets, messagequeue, eventbus, serverless, cache, containerregistry, notification.

Product-specific services with no cross-cloud abstraction keep their product name everywhere (bedrock, vertexai, databricks, bigtable, ecs, sagemaker, …).

2.2 providers/<cloud>/<service> and server/<cloud>/<service>: short service name, no cloud prefix

The mock and wire layers use the service's own short name, without a cloud prefix. That is the real SDK/CLI name where a service clearly has one (s3, ec2, elbv2), and a short capability name otherwise (cache, dns, monitor, sql). Azure's SDK calls these armredis/armdns/armmonitor, but the short form reads better and has no prefix:

Capability AWS Azure GCP
Storage s3 blobstorage gcs
Compute ec2 virtualmachines compute
Networking vpc vnet vpc
IAM iam iam iam
Load balancer elbv2 loadbalancer loadbalancer
Database (NoSQL) dynamodb cosmosdb firestore
DNS route53 dns clouddns
Cache elasticache cache memorystore
Monitoring cloudwatch monitor monitoring

Two hard rules:

  • No redundant cloud prefix. The parent directory already names the cloud, so an aws/azure/gcp prefix on the leaf adds nothing. Drop it: awsiam → iam, azurecache → cache, azuredns → dns, azureiam → iam, azurelb → loadbalancer, azuremonitor → monitor, azuresql → sql, azureai → ai, azuresearch → search, gcpiam → iam, gcplb → loadbalancer, gcpvpc → vpc.
  • Provider and wire names match within a cloud. For a given cloud, the providers/<cloud>/X and server/<cloud>/X directories have the same name X, so the two layers of a service line up by name (e.g. providers/azure/blobstorage ↔ server/azure/blobstorage).

Names don't have to be identical across clouds at the provider/wire layer. s3 / blobstorage / gcs each read naturally for their own cloud, and that is the intent. The cross-cloud name lives only in services/.

Exceptions to the "provider and wire share a name" rule:

  • AWS providers/aws/vpc ↔ server/aws/ec2. AWS's own SDK puts VPC operations under the EC2 service, so the wire handler lives in ec2, while the mock keeps the clearer vpc name. This is intentional.
  • services/azureai / services/azuresearch keep azure in their name, while their providers dropped it (providers/azure/ai, providers/azure/search). They are product-specific services (Azure AI, Azure AI Search) with no cross-cloud abstraction, so per §2.1 the product name, including the azure that is part of it, is kept at the services/ layer.

3. File naming (the rule)

  • snake_case, full words, no abbreviations, in every layer. natgw.go → nat_gateway.go, eip.go → elastic_ip.go, eni.go → network_interface.go, igw.go → internet_gateway.go, trafficmirror.go → traffic_mirror.go.
  • A feature uses the same filename in all three layers, so one grep (or one filename) finds the interface, the mock and the wire handler for that feature.
Feature Filename (all layers)
NAT gateway nat_gateway.go
Internet gateway internet_gateway.go
Elastic IP / public IP elastic_ip.go
Network interface network_interface.go
Traffic mirroring traffic_mirror.go

Test files sit next to what they test: <feature>_test.go.

4. Per-directory layout (the template)

services/<capability>/

driver/driver.go        # the driver interface(s), always in a driver/ subpackage
<capability>.go         # portable API + do()/pipeline wiring
<capability>_test.go
<feature>.go            # optional: one file per portable-API feature area

providers/<cloud>/<service>/

<service>.go            # store + core CRUD
<service>_test.go
<feature>.go            # one file per feature (same filename as the wire layer's)
clone.go                # optional: copy-on-write helpers
tags.go                 # optional: tagging, if the service is tagged

server/<cloud>/<service>/

handler.go              # routing / dispatch (Matches + ServeHTTP)
types.go                # wire DTOs + mapping to/from the driver
<feature>.go            # one wire file per feature (same filename as the provider's)

The driver/ subpackage rule

Every portable-API service in services/ puts its interface in a driver/ subpackage. These are not portable-API services, so they have no driver/:

  • services/kubernetes: a self-contained data-plane engine with its own HTTP surface.
  • services/resourcediscovery: a cross-service engine that consumes other drivers.
  • services/cost, services/pricing, services/scope: cross-cutting utilities, not a cloud capability (pricing is the rate-table lookup that feeds the cost estimate model).

A new portable-API service must have driver/.

Cross-cutting behaviors (not capabilities) live under features/ instead: recorder, metrics, ratelimit, inject, chaos, topology, plus vcr (wire record/replay), timetravel (named state save/rewind/fork) and quota (per-service quota registry). These have no driver/ either.

5. Sub-surfaces → same-named subdirectories

When a service grows a large or self-contained sub-surface, move it into a subdirectory with the same name in each layer instead of keeping many flat files. The example below is only an illustration. No service is laid out this way today (see the note on flat exceptions below); traffic mirroring is currently the flat providers/aws/vpc/traffic_mirror.go / server/aws/ec2/traffic_mirror.go:

services/networking/traffic_mirror/        # interface fragment
providers/aws/networking/traffic_mirror/   # mock
server/aws/networking/traffic_mirror/      # wire

Rule of thumb: a sub-surface with more than about 5 files, or a self-contained sub-API, gets its own same-named subdirectory in the layers it touches. Smaller sub-surfaces stay as flat <feature>.go files. Either way the file or directory name is the same in every layer, so each feature has one home.

This rule is for new services; existing coupled mocks stay flat

Subdirectories are for new services designed that way from the start, where each sub-surface owns its own state. They are not retrofitted onto existing large single-Mock providers such as providers/aws/vpc and providers/azure/databricks, which stay flat on purpose. In Go a directory is a package, so moving a sub-surface out would mean splitting one Mock (vpc: 256 methods, 57 fields under a single mutex) into sub-packages. That needs a shared internal helper package, callbacks between surfaces, and a change to the single-lock concurrency model. The behavior risk isn't worth it, since the per-feature snake_case files (§3) already make the code easy to navigate. For these services, keep one package with one <feature>.go per sub-surface.

6. Where new code goes

A new resource on an existing service: add <feature>.go (same filename) to the provider mock and the wire handler, and, if it is part of the portable API, to the driver/ interface and the services/<capability> wrapper. Add <feature>_test.go next to each.

A new service: create services/<capability>/driver/driver.go + services/<capability>/<capability>.go, then providers/<cloud>/<service>/ and server/<cloud>/<service>/ for each cloud you implement, all following §2 to §4. Wire it into the provider factory (providers/<cloud>/<cloud>.go) and SetMonitoring() if it emits metrics.

7. Migration policy (for existing drift)

The rules above are the target. Older code that predates them is brought in line a little at a time, not all at once:

  • One service per PR. A rename is a mechanical git mv plus import fixes and a factory rename, with no behavior change in that PR.
  • Start with the clearest wins (drop the cloud prefixes in §2.2; the snake_case filename fixes in §3).
  • Keep each PR small so it rebases cleanly on top of other work in progress. Almost every branch edits the factory and wire files, so a repo-wide rename would conflict with everything.

The Structure CI job fails on a new cloud-prefixed directory, and an advisory step flags a feature whose filename differs between the provider and wire layers.