[aparnagoy]

feat: support Data Cloud (D360) dataspace-scoped metadata types

Summary

Adds first-class SDR support for Data Cloud dataspace-scoped metadata types.
These types do not fit any existing adapter/transformer because their on-disk
representation is unlike every other metadata type:

• Each component is a pair of generic .json files (there is no -meta.xml):
  • ‹name›.json — the payload (the component definition).
  • ‹name›.meta.json — a sidecar carrying relationship metadata
    (componentType, componentName, dataspaceName, retrieveWith, dependsOn).
• Components that belong to a dataspace are nested under
  d360/dataspace/‹dataspace›/‹typeDir›/, and their fullName is
  dataspace-scoped: ‹dataspace›.‹name› (e.g. ‹ComponentType›:‹dataspace›.‹name›),
  where ‹dataspace› is the path segment immediately above the type directory.
• Components that exist outside any dataspace (e.g. Data Lake Objects, Data
  Stream bundles) live directly under d360/‹typeDir›/ and keep their bare ‹name›.
  They can still be requested for one dataspace as ‹dataspace›.‹name› (see below).
• The d360/… path is preserved verbatim in both directions.

Registered types

Type	Directory	In package.xml
CalculatedInsight	calculatedInsights	yes
DataModelObject	dataModelObjects	yes
DataLakeObject	dataLakeObjects	yes
DataStreamBundle	dataStreamBundles	yes
DataConnection	dataConnections	yes
DataspaceMember	dataspaceMembers	no — deployed as part of its Data Lake Object
DataModelObjectMapping	dataModelObjectMappings	no — deployed as part of its Data Lake Object
SecondaryIndex	secondaryIndexes	no — deployed as part of its Data Lake Object

Directory names are the plural folder names the Metadata API writes, like every
other source-format folder (classes/, objects/).

The registry, resolver, converter, and deploy-closure logic are taught to handle
that shape end-to-end, in both retrieve (metadata → source) and deploy
(source → metadata) directions.

The runtime pieces are type-agnostic: they derive everything from the path
shape and the sidecar convention, so onboarding an additional dataspace-scoped
type is a registry change (a type entry plus a directory mapping), never a
code change. This keeps the design open for extension and closed for
modification — new types plug in without touching the adapter, transformer, or
closure logic.

Design

Three new pieces of behavior, wired through the existing strategy indirection so
no core class is modified. All three share the single adapter/transformer
strategy id d360:

Layer	Component	Responsibility
registry	metadataRegistry.json	Register each type with the shared d360 adapter + transformer strategies; index the json suffix so bare .json files are recognized as metadata; strict directory names disambiguate one type's ‹typeDir› from another's.
resolve	D360SourceAdapter	Build the ‹dataspace›.‹name› fullName from the path; treat the .meta.json sidecar as part of its sibling payload component (never a component of its own).
convert	D360MetadataTransformer	Preserve the full d360/… path in both directions and co-write the sidecar alongside the payload.
collections	expandD360ComponentSet	Build the minimal deploy closure by walking each component's sidecar retrieveWith set.
collections	resolveDataspaceQualifiedComponents	Resolve a ‹dataspace›.‹name› deploy request for a component stored outside any dataspace to its local bare-named source.
collections	ComponentSet.addWithoutManifestEntry + registry excludeFromManifest	Send a component's files in the payload without giving it its own package.xml entry.

Only the requested components go in package.xml

The Metadata API deploys retrieveWith peers as part of the component that
pulls them in; some peer types (DataspaceMember, DataModelObjectMapping,
SecondaryIndex) are rejected as manifest entries of their own. So the closure
adds peers with addWithoutManifestEntry: their files travel in the zip, while
package.xml lists just what the user asked for — the same as on retrieve. The
registry flag excludeFromManifest keeps those peer-only types out of the
manifest in every other code path too.

Deploying for one dataspace: ‹dataspace›.‹name›

Retrieve accepts DataLakeObject:default.myDlo__dll because the org resolves the
name. Locally that DLO is stored once, as myDlo__dll, so the same name on deploy
used to match no source. Now:

1. ComponentSetBuilder resolves the request to the local myDlo__dll
   (resolveDataspaceQualifiedComponents).
2. package.xml keeps the requested name, default.myDlo__dll.
3. The closure walks only the default dataspace: peers in other dataspaces are
   left out, including entries that name another dataspace. Requesting the bare
   name still walks every dataspace.
4. The DLO's sidecar is sent with retrieveWith trimmed to the default peers
   (and peers outside every dataspace), so the org isn't asked for peers the
   deploy doesn't carry. The local file is left as is.

One sidecar for every dataspace

A component outside any dataspace is stored once, but it can be retrieved through
several dataspaces, each returning its own retrieveWith. Retrieving
default.myDlo__dll and then other.myDlo__dll must not lose the default peers:

• The payload (‹name›.json) is overwritten with the latest definition.
• The sidecar (‹name›.meta.json) is merged with the one already in the
  project: the new retrieveWith replaces the entries of the dataspaces it covers,
  and entries of other dataspaces are kept. Everything else comes from the new sidecar.
• With no local sidecar, or nothing to keep, the retrieved file is written as is.

So the sidecar can list peers from several dataspaces, and deploy (above) sends
only the requested dataspace's share of it.

Matching retrieveWith entries

Each entry is { componentType, name, dataspaceName? }. Peers are matched by
type and name against their sidecars, because different types can share a
name (a DLO and its DataspaceMember are both named after the DLO). When an
entry names a dataspace, the peer in that dataspace matches, and so does one
stored outside every dataspace. A Data Stream bundle, for example, names its
DLO with the dataspace it uses it in, while the DLO itself is stored bare.

Why the json suffix index matters

Resolution gates every file through isMetadata(), which calls
registry.getTypeBySuffix(extName(path)). A suffix that is not present in the
global suffixes index is silently skipped, so bare .json files were never
considered metadata. Indexing json is what lets these components resolve at
all; resolveType then disambiguates the concrete type per-directory via the
strict folder name (each type's registered directoryName), so a single shared
json suffix mapping serves every dataspace-scoped type.

Sidecar handling and the name-collision guard

baseName() splits on the first ., so ‹name›.json and ‹name›.meta.json
both reduce to ‹name›. D360SourceAdapter.populate() therefore must
exclude *.meta.json; otherwise the sidecar would resolve to the same
dataspace-scoped name as the payload and collide. The sidecar is not lost — the
transformer writes it next to the payload, merged with the local copy on retrieve
(see above).

retrieveWith vs dependsOn

The sidecar declares two distinct relationship lists:

• retrieveWith — the components that travel together with this one as a
  single deployable unit. This is what the deploy closure walks.
• dependsOn — informational metadata about what a component references
  (the components it points at). It is not used to build the closure.

End-to-end flow (across repos)

The stack is always user → CLI → SDR → Core (server). The CLI is the command
entry point, SDR is the library that resolves/converts/packages the components
and computes the deploy closure, and Core owns the on-disk/wire contract (it
emits the two-file layout and the server-populated retrieveWith on retrieve,
and validates/applies the unit on deploy). SDR carries no per-type logic — the
d360 strategy is selected purely from the registry.

Layer	Repo	Responsibility
CLI (plugin)	plugin-deploy-retrieve	Builds the component set; on deploy, expands the retrieveWith closure before handing off to SDR.
SDR (library)	source-deploy-retrieve (this PR)	Registry + D360SourceAdapter + D360MetadataTransformer that resolve/convert the two-file layout both directions, and expandD360ComponentSet which computes the deploy closure.
Core (server)	Data Cloud platform	On retrieve emits the d360/… payload + .meta.json sidecar and populates each component's retrieveWith. On deploy validates and applies the unit.

What crosses the SDR boundary

Call	Input into SDR	What SDR does	Returns	To whom
retrieve	requested ComponentSet + org connection	issues the Metadata API retrieve, extracts the returned zip, resolves it (D360SourceAdapter), converts metadata → source (D360MetadataTransformer.toSourceFormat), co-writes the .meta.json sidecar, merging its retrieveWith with the local copy	RetrieveResult + FileResponse[]	CLI (writes files, prints)
deploy closure	full project ComponentSet + requested ComponentSet (+ registry)	indexes d360 members by sidecar type + componentName, walks retrieveWith transitively (limited to the requested dataspace, if any), trims shared sidecars to that dataspace	expanded ComponentSet (requested in the manifest + closure as files)	CLI (feeds it into the deploy)
deploy	expanded ComponentSet + org connection	resolves, converts source → metadata (D360MetadataTransformer.toMetadataFormat), packages the zip, submits the Metadata API deploy, polls	DeployResult	CLI → user

sf project retrieve start — metadata → source

sequenceDiagram
    actor User
    participant CLI as CLI (plugin-deploy-retrieve)
    participant SDR as SDR (source-deploy-retrieve)
    participant Core as Core (server)
    User->>CLI: sf project retrieve start --metadata ‹Type›:‹dataspace›.‹name›
    CLI->>SDR: ComponentSetBuilder.build(requested) → componentSet.retrieve(org)
    SDR->>Core: Metadata API retrieve request
    Core-->>SDR: unpackaged.zip (payload + .meta.json sidecar,<br/>server-populated retrieveWith)
    SDR->>SDR: resolve (D360SourceAdapter) + convert to source (D360MetadataTransformer.toSourceFormat)
    SDR-->>CLI: RetrieveResult + FileResponse[]
    CLI-->>User: writes force-app/main/default/d360/dataspace/…<br/>(payload + sidecar), prints result

Loading

Call stack inside SDR (retrieve):

componentSet.retrieve(...)
└─ MetadataApiRetrieve.start() → poll checkStatus() → post()
   ├─ extract unpackaged.zip into a ZipTreeContainer
   ├─ MetadataResolver.getComponentsFromPath (over the zip tree)
   │  ├─ RegistryAccess.getTypeBySuffix('json')          # suffix now indexed
   │  ├─ RegistryAccess.resolveTypeFromStrictFolder      # per-type directoryName
   │  └─ SourceAdapterFactory.getAdapter('d360') → D360SourceAdapter
   │     └─ populate(): skip *.meta.json, name = ‹dataspace›.‹name›
   └─ MetadataConverter.convert(components, 'source', …)
      └─ ComponentConverter → MetadataTransformerFactory.getTransformer → D360MetadataTransformer
         └─ toSourceFormat → getWriteInfos: payload + .meta.json sidecar
            ├─ getD360Destination: join(main/default, trimUntil(path, 'd360'))
            └─ mergeSidecars: keep local retrieveWith entries of other dataspaces
# returns → RetrieveResult { fileProperties, success, … } + FileResponse[]

sf project deploy start — source → metadata

sequenceDiagram
    actor User
    participant CLI as CLI (plugin-deploy-retrieve)
    participant SDR as SDR (source-deploy-retrieve)
    participant Core as Core (server)
    User->>CLI: sf project deploy start --source-dir … | --metadata ‹Type›:‹dataspace›.‹name›
    CLI->>SDR: ComponentSetBuilder.build(requested)
    SDR->>SDR: resolveDataspaceQualifiedComponents (‹dataspace›.‹name› → local ‹name›)
    SDR-->>CLI: requested ComponentSet
    CLI->>CLI: hasD360Components(requested)?
    CLI->>SDR: ComponentSetBuilder.build(whole project) → full
    CLI->>SDR: expandD360ComponentSet(full, requested, registry)
    SDR-->>CLI: requested + transitive retrieveWith closure
    CLI->>SDR: componentSet.deploy(org)  (MetadataApiDeploy)
    SDR->>Core: Metadata API deploy (payload + sidecar under d360/…)
    Core-->>SDR: DeployResult
    SDR-->>CLI: DeployResult
    CLI-->>User: prints result

Loading

Call stack inside SDR (deploy):

# 0) requested set
ComponentSetBuilder.build(...)
└─ resolveDataspaceQualifiedComponents      # ‹dataspace›.‹name› → local ‹name›, manifest keeps the requested name

# 1) closure (CLI calls the exported SDR function)
expandD360ComponentSet(full, requested, registry)
├─ index d360 members by sidecar componentType + componentName
├─ seed worklist with requested members (one walk per requested dataspace)
├─ while worklist: collect each retrieveWith peer (files only), recurse
└─ withSidecarFor: a component walked only for some dataspaces gets its sidecar
   trimmed to them (served from memory; the local file is unchanged)
# returns → expanded ComponentSet (requested + closure)

# 2) deploy (CLI hands the expanded set back to SDR)
componentSet.deploy(...)
└─ MetadataApiDeploy.start()
   └─ MetadataConverter.convert(components, 'metadata', { type: 'zip' })
      └─ ComponentConverter → MetadataTransformerFactory.getTransformer → D360MetadataTransformer
         └─ toMetadataFormat → getWriteInfos: payload + .meta.json sidecar
            └─ getD360Destination: trimUntil(path, 'd360') at the package root
   └─ POST zip to Core Metadata API deploy → poll
# returns → DeployResult

Call stack (resolution & conversion detail)

Resolution (retrieve):

MetadataResolver.getComponentsFromPathRecursive
└─ MetadataResolver.resolveComponent
   ├─ RegistryAccess.getTypeBySuffix('json')            # gate: suffix now indexed
   ├─ RegistryAccess.resolveTypeFromStrictFolder        # per-type directoryName
   └─ SourceAdapterFactory.getAdapter('d360')
      └─ D360SourceAdapter.getComponent
         └─ (MixedContentSourceAdapter) → D360SourceAdapter.populate
            ├─ skip when path endsWith '.meta.json'
            └─ calculateD360Name  →  '‹dataspace›.‹name›'

Conversion (both directions):

MetadataConverter.convert
└─ ComponentConverter (transform stream)
   └─ MetadataTransformerFactory.getTransformer → D360MetadataTransformer
      ├─ toSourceFormat / toMetadataFormat
      └─ getWriteInfos
         ├─ walkContent()                       # payload files
         ├─ derive '‹name›.meta.json' sidecar    # co-written when present
         └─ getD360Destination                   # trimUntil(path, 'd360') under target root

Deploy closure:

expandD360ComponentSet(full, requested)
├─ index d360 members by sidecar componentType + componentName
├─ seed worklist with requested members (one walk per requested dataspace)
└─ while worklist: for each retrieveWith entry
   ├─ match peers by type + name (+ dataspaceName when given, else the walked dataspace;
   │                              a peer outside every dataspace always qualifies)
   └─ addWithoutManifestEntry(peer), recurse

Testing

• yarn build (compile + lint) — green.
• yarn test (unit suite) — green.
• New unit tests (test/collections/d360Dependencies.test.ts, componentSet.test.ts) cover:
  • telling apart components of different types that share a name;
  • pulling peers from every dataspace, only the requested dataspace, or only the dataspace a reference names;
  • listing only the requested component (bare or ‹dataspace›.‹name›) in package.xml;
  • resolving ‹dataspace›.‹name› to the local component;
  • a Data Stream bundle pulling its Data Lake Object and that DLO's closure, also when
    the bundle names the DLO with a dataspace and the DLO is stored outside every dataspace;
  • skipping entries that name another dataspace than the requested one;
  • sending a sidecar trimmed to the requested dataspace, and unchanged for a bare-name request.
• New unit tests (test/convert/transformers/d360MetadataTransformer.test.ts) cover
  merging a retrieved sidecar into the local one (entries of other dataspaces kept,
  those of a covered dataspace replaced), writing it as is when there's no local copy,
  and reading from a tree that can't read synchronously, as a retrieve zip can't.
  • excludeFromManifest / addWithoutManifestEntry.
• Manual end-to-end:
  • sf project retrieve start writes the payload + .meta.json sidecar for a
    retrieved component and its retrieveWith peers under force-app/main/default/d360/…,
    and the reverse deploy direction reproduces the same layout in the mdapi package.
  • sf project deploy start --metadata "DataLakeObject:default.‹name›" deploys the
    DLO with only its default peers, with package.xml listing default.‹name›.

[salesforce-cla]

Thanks for the contribution! Unfortunately we can't verify the commit author(s): Aparna Goyal <r***@A***.com>. One possible solution is to add that email to your GitHub account. Alternatively you can change your commits to another email and force push the change. After getting your commits associated with your GitHub account, sign the Salesforce Inc. Contributor License Agreement and this Pull Request will be revalidated.