Skip to content

Commit 700bfc1

Browse files
naman-contentstackclaude
andcommitted
fix(bulk-operations): chunk CS Assets bulk delete/move to the 100-item API cap (DX-9732)
CS Assets bulk delete/move sent the entire asset array in one POST, so any request over 100 items failed with HTTP 422 "Assets cannot exceed the max limit of 100." — blocking bulk deletes of >100 assets entirely. - CSAssetsAdapter.bulkDeleteAssets/bulkMoveAssets now split the payload into <=100-item batches (CS_ASSETS_BULK_MUTATE_MAX_ITEMS) dispatched with bounded concurrency (makeConcurrentCall) and per-batch retry (429/5xx; 4xx not retried). - Continue-on-partial-failure: failed batches are collected, not rethrown, since each request commits independently. Aggregate carries job_ids, batch counts, and per-batch failures (with the failed uids). - postJson throws a typed CsAssetsPostError carrying HTTP status (no error-string parsing). - Runner reports partial outcomes, writes failed uids to a {"uids":[...]} file in the bulk-operation log folder for re-run via --asset-uids-file, and states that a submitted delete job is submission (not completion) since delete is async. Tests: adapter chunking/partial-failure/uid-capture; runner partial-failure file write. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent a3bead5 commit 700bfc1

10 files changed

Lines changed: 370 additions & 124 deletions

File tree

.talismanrc

Lines changed: 2 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -1,100 +1,4 @@
11
fileignoreconfig:
2-
- filename: pnpm-lock.yaml
3-
checksum: 2e1116588e19b058f0d6a5efa28f54c6a3af11b0f0d8c442ed9866a90b77b713
4-
- filename: packages/contentstack-clone/test/lib/helpers/command-helpers.test.ts
5-
checksum: 32d385a677a76faa54cb3ddfe92fb1f1f53cfab878a78e2792c823a363956bd2
6-
- filename: packages/contentstack-clone/test/commands/stack-clone.test.ts
7-
checksum: 4547ba010cc3c9954de49e2b2bef45288dd070c4ad000b6ae3e9a8c9bb1ad296
8-
- filename: packages/contentstack-asset-management/test/unit/import/workspaces.test.ts
9-
checksum: 67c11d312ba6838659c0b2cefd59d706d4b3c1a9807ed551dcb6d94f4d1d60a3
10-
- filename: packages/contentstack-clone/test/lib/util/clone-handler.branch.test.ts
11-
checksum: db8c090aaada21d3952c5bc382228a0a11cebb63926512a5e2581bdaf5f4c2f6
12-
- filename: packages/contentstack-asset-management/test/unit/import/spaces.test.ts
13-
checksum: d2363470664a6cdeaf59c305fa6307704e48b73f92b7d471433c17528daa5450
14-
- filename: packages/contentstack-asset-management/test/unit/utils/detect-asset-management-export.test.ts
15-
checksum: 1917c51ebf4ba7f94cc440c27341d840911b5c315e250a451732c2d37518db64
16-
- filename: packages/contentstack-bootstrap/test/bootstrap-integration.test.js
17-
checksum: 67579b79fc5c1b4feead8f537e7bdfd14fac58aa93c127f590505e47d26e0eb0
18-
- filename: packages/contentstack-import/test/unit/import/modules/index.test.ts
19-
checksum: 27dd2c5f60f9f4c70f2ebf4c08cf8bcbe2d5e4dc7fe52a53972af3fdb57e0e63
20-
- filename: packages/contentstack-bulk-operations/test/unit/base-bulk-command.test.ts
21-
checksum: 522b91a2d4bb6a73ebe5ab92c0111e74d567c4360470beefd6a7ca8d30a6e311
22-
- filename: packages/contentstack-content-type/tests/core/command.test.ts
23-
checksum: 81da32fc47083a3421f5f1d9c04149ab313cb238d6416d3182f8b6d64dc8827d
24-
- filename: packages/contentstack-content-type/tests/core/contentstack/client.test.ts
25-
checksum: da15b04aa0a7ab0e48f70b847e68c05372943279e51c22f8539b01bb48033183
26-
- filename: packages/contentstack-import/test/unit/import/modules/marketplace-apps.test.ts
27-
checksum: 17b3e6e0f7ce12ddabd688996c72d04dba5245fd300a23b541cf7bdfbbf7ef1a
28-
- filename: packages/contentstack-asset-management/test/unit/import/asset-types.test.ts
29-
checksum: 76aad963f5977059f7632d69a6fb36004229f3455b1618c372579dfa3cbc2d62
30-
- filename: packages/contentstack-asset-management/test/unit/import/assets.test.ts
31-
checksum: f5f0494c9263ebcf7eabff806ff41bd7beed0cb6a0f9a631dd70ffa01524a304
32-
- filename: packages/contentstack-import/test/unit/utils/common-helper.test.ts
33-
checksum: 7a10c4625d666cf705cd9869b74b3bc2dcec94926cbb5f719e4f8dc4f053b2c7
34-
- filename: packages/contentstack-content-type/tests/utils/index.test.ts
35-
checksum: 8510fedf5eb6078c27d6a0d65d8bff4aa2d41b545246aed220d1a6a643188ad8
36-
- filename: packages/contentstack-import/test/unit/utils/interactive.test.ts
37-
checksum: dad20e229e02e847b5e939fdaf987b1a6c9ff4e1529b6d96a51130f7293285a9
38-
- filename: packages/contentstack-asset-management/test/unit/import/base.test.ts
39-
checksum: 53168bda5251300d8dfd0effd470070a5733c37b77abacc846d91e7086c8c022
40-
- filename: packages/contentstack-asset-management/test/unit/import/fields.test.ts
41-
checksum: 9e97001a6c92d213a9f55a1458bf39c0996952467e0b4aabd954ddd8ad1805b9
42-
- filename: packages/contentstack-export-to-csv/test/unit/commands/export-to-csv.test.ts
43-
checksum: cc82f4b0c50783dd42d8d57ba2af1e5c9aa3857399c5abf39bc5dc3fac105658
44-
- filename: packages/contentstack-bulk-operations/test/unit/utils/bulk-operation-log-handler.test.ts
45-
checksum: 8698f77bbb828df10ddc3a94726076a262fbf47d2468233c7e10e66cac42f97d
46-
- filename: packages/contentstack-import/test/unit/utils/login-handler.test.ts
47-
checksum: a60f46b4d79ccdbeecfed27ffffbc97a44d6753f36ee74b68b725c8b20bbb2eb
48-
- filename: packages/contentstack-export-to-csv/test/unit/utils/error-handler.test.ts
49-
checksum: cef8f515ab6c0441f67748693473bf1499c2d54945aab1b89baaad37be8af2d9
50-
- filename: packages/contentstack-export-to-csv/test/unit/utils/teams-export.functional.test.ts
51-
checksum: bcf70dbb201a0a1755f0b8f59550aae95a3e1d03700959eee7e8d415f9647746
52-
- filename: packages/contentstack-export/test/unit/utils/marketplace-app-helper.test.ts
53-
checksum: 5648d90b0f7dc33a5b020324db5218707861ebd7201802f7fef00f1c03c5ce82
54-
- filename: packages/contentstack-export/test/unit/export/module-exporter.test.ts
55-
checksum: 485bbd175cc95aff6c1c3c6b2904f92c3d4ea1272a056158a901da3eff9692d6
56-
- filename: packages/contentstack-import-setup/test/unit/common-helper.test.ts
57-
checksum: ccbd8209d342bf651c2c38326cba13bc3e5c2af4200d7110fdd7c3de4824d21d
58-
- filename: packages/contentstack-export/test/unit/utils/common-helper.test.ts
59-
checksum: a4bfba19846e1075821521a202d856203f02a6bc0e5ec4f51b5265801cb4ddd9
60-
- filename: packages/contentstack-import/test/unit/import/modules/assets.test.ts
61-
checksum: 259d5c325ec7de08d0f45099df9d3eb7c97374691c1c4d281221e112b9097191
62-
- filename: packages/contentstack-export/test/unit/export/modules/marketplace-apps.test.ts
63-
checksum: 07eef22a3c471e8da12b1e7e51d45b76410725c4c160a0f35432d9aac5916974
64-
- filename: packages/contentstack-export/test/unit/utils/interactive.test.ts
65-
checksum: 0ea47db305f576c8c9da8cb34ae915f6b01992208ff1c4380cb7ffd93f538e5d
66-
- filename: packages/contentstack-bulk-operations/test/unit/utils/revert-retry-handler.test.ts
67-
checksum: c34a4bdea186f4ce48171d90b8ab0f1c94162e441acb23e3a0e00cb9f9f9255f
68-
- filename: packages/contentstack-bulk-operations/test/unit/utils/config-builder.test.ts
69-
checksum: e5891257999e9e940956ca77f5a7834be35d259edccd41c06f77736194ad7454
70-
- filename: packages/contentstack-migrate-rte/test/commands/json-migration.test.js
71-
checksum: 0d18641a32f7cf33d013a5a7c447c1067ba77022513b42304a727f873c741e99
72-
- filename: packages/contentstack-seed/test/commands/cm/stacks/seed.test.ts
73-
checksum: e469a60f09a9a5fd3e2ce80dd6df8bf4f2c82acad1afebde00f458afa26d36c9
74-
- filename: packages/contentstack-migrate-rte/test/utils/index.js
75-
checksum: f67235e607a446e5b2dd624046966a1ac5af7bce128aa080ec3c420c48b8bddc
76-
- filename: packages/contentstack-query-export/test/unit/common-helper.test.ts
77-
checksum: a0aa399a2ec15760fe25f7e952d64792b55d3c38d2281858a48e64f0f9081344
78-
- filename: packages/contentstack-seed/test/seed/contentstack/client.test.ts
79-
checksum: 63bd3325e2717e5653f12e4afa362938e58eeaad5fbe07c1d41e2bd15e3451bd
80-
- filename: packages/contentstack-import/test/unit/commands/cm/stacks/import.test.ts
81-
checksum: d241111877a997f00702eaf171c1fde53a075a9c8f7971a8e6b1577c1c83279d
82-
- filename: packages/contentstack-import/test/unit/import/modules/base-class.test.ts
83-
checksum: 40c504d508d40b6416ccedb5a323365dd97f313b1dcc885e4fec3639e807b383
84-
- filename: packages/contentstack-import/test/unit/import/modules/content-types.test.ts
85-
checksum: 662cb42217c12af2b4dad5d1355858095c5df062e42c89e09b2680fdbee80699
86-
- filename: packages/contentstack-import/test/unit/import/modules/entries.test.ts
87-
checksum: a7311e3077fee52bfd23c46d51ebe9ce58d202dd735514a44ec230615f4cd917
88-
- filename: packages/contentstack-import/test/unit/import/modules/global-fields.test.ts
89-
checksum: 2b4e0d5d11f825d21d45569f681acd886c5f84717873b26998b4dab2e91bb439
90-
- filename: packages/contentstack-seed/test/seed/interactive.test.ts
91-
checksum: 8364557eb69b9a297281f868420c13c5b61b26b79bc2ceb0a9b548cfb0365d6d
92-
- filename: packages/contentstack-variants/test/unit/export/variant-entries.test.ts
93-
checksum: dd7018cd5cc8b5fab73004b2968afed7ce33abec0b20e1b17f15f2be23b117db
94-
- filename: packages/contentstack-import-setup/test/unit/login-handler.test.ts
95-
checksum: 2df43be75a96e1a3a00dd256628e191909aa9d7f192b672b56cb3772e074958c
96-
- filename: packages/contentstack-export/test/unit/utils/export-config-handler.test.ts
97-
checksum: f612661a8b6784b20ea62794b35192b340a34d8424a54a1a2cffec1878ab7528
98-
- filename: packages/contentstack-import/test/unit/utils/marketplace-app-helper.test.ts
99-
checksum: de694e861560c4c242100eaf17a6d8e230247b99f94e4b428e55ef065084c10c
2+
- filename: packages/contentstack-asset-management/src/utils/cs-assets-api-adapter.ts
3+
checksum: 86f16cef82b891d0946871a5b48726bb2592dc8c16de4c878baf39e5a0a505f7
1004
version: ""

packages/contentstack-asset-management/src/constants/index.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ export const FALLBACK_AM_API_CONCURRENCY = 5;
66
export const DEFAULT_AM_API_CONCURRENCY = FALLBACK_AM_API_CONCURRENCY;
77
export const FALLBACK_AM_API_PAGE_SIZE = 100;
88
export const FALLBACK_AM_API_FETCH_CONCURRENCY = 5;
9+
/** Max assets/uids the CS Assets bulk delete/move endpoints accept per request. */
10+
export const CS_ASSETS_BULK_MUTATE_MAX_ITEMS = 100;
911

1012
/** Fallback strip lists when import options omit `fieldsImportInvalidKeys` / `assetTypesImportInvalidKeys`. */
1113
export const FALLBACK_FIELDS_IMPORT_INVALID_KEYS = [

packages/contentstack-asset-management/src/types/cs-assets-api.ts

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,18 +127,45 @@ export type BulkDeleteAssetItem = { uid: string; locale: string };
127127

128128
export type BulkDeleteAssetsPayload = { assets: BulkDeleteAssetItem[] };
129129

130+
/** One failed batch when a bulk mutate is split across multiple ≤100-item requests. */
131+
export type BulkMutateFailure = {
132+
batchIndex: number;
133+
count: number;
134+
status?: number;
135+
error: string;
136+
/** Asset uids in the failed batch, so callers can re-run just the failures. */
137+
uids: string[];
138+
};
139+
140+
/** Raw response of a single bulk-mutate request (one ≤100-item POST) as returned by the API. */
141+
export type CsAssetsMutateBatchResponse = { notice?: string; job_id?: string };
142+
143+
/** Aggregate result of a bulk delete, combining every dispatched ≤100-item batch. */
130144
export type BulkDeleteAssetsResponse = {
145+
/** First notice returned; a human-facing message, safe to surface as the summary line. */
131146
notice?: string;
132-
job_id?: string;
147+
/** One submitted job id, for a short summary line; `job_ids` holds all of them (batches
148+
* run concurrently, so this is whichever completed first — not a stable "first batch"). */
149+
primaryJobId?: string;
150+
notices?: string[];
151+
job_ids?: string[];
152+
failures?: BulkMutateFailure[];
153+
batchesTotal?: number;
154+
batchesSucceeded?: number;
133155
};
134156

135157
export type BulkMoveAssetsPayload = {
136158
asset_uids: string[];
137159
target_folder_uid: string;
138160
};
139161

162+
/** Aggregate result of a bulk move (sync; no job ids). */
140163
export type BulkMoveAssetsResponse = {
141164
notice?: string;
165+
notices?: string[];
166+
failures?: BulkMutateFailure[];
167+
batchesTotal?: number;
168+
batchesSucceeded?: number;
142169
};
143170

144171
/**

packages/contentstack-asset-management/src/utils/cs-assets-api-adapter.ts

Lines changed: 93 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,12 @@ import chunk from 'lodash/chunk';
44
import { HttpClient, log, authenticationHandler, handleAndLogError } from '@contentstack/cli-utilities';
55

66
import { withRetry, RetryableHttpError, isRetryableStatus, parseRetryAfterMs } from './retry';
7-
import { FALLBACK_AM_API_FETCH_CONCURRENCY, FALLBACK_AM_API_PAGE_SIZE } from '../constants/index';
7+
import {
8+
CS_ASSETS_BULK_MUTATE_MAX_ITEMS,
9+
FALLBACK_AM_API_CONCURRENCY,
10+
FALLBACK_AM_API_FETCH_CONCURRENCY,
11+
FALLBACK_AM_API_PAGE_SIZE,
12+
} from '../constants/index';
813

914
import type {
1015
CSAssetsAPIConfig,
@@ -13,6 +18,8 @@ import type {
1318
BulkDeleteAssetsResponse,
1419
BulkMoveAssetsPayload,
1520
BulkMoveAssetsResponse,
21+
BulkMutateFailure,
22+
CsAssetsMutateBatchResponse,
1623
CreateAssetMetadata,
1724
CreateAssetTypePayload,
1825
CreateFieldPayload,
@@ -91,6 +98,21 @@ export type CustomPromiseHandlerInput = {
9198

9299
export type CustomPromiseHandler = (input: CustomPromiseHandlerInput) => Promise<any>;
93100

101+
/**
102+
* Error thrown by {@link CSAssetsAdapter.postJson} for a failed POST. Carries the HTTP
103+
* `status` (undefined for network/transport failures) so callers can classify failures
104+
* without parsing the message string.
105+
*/
106+
export class CsAssetsPostError extends Error {
107+
constructor(
108+
message: string,
109+
public readonly status?: number,
110+
) {
111+
super(message);
112+
this.name = 'CsAssetsPostError';
113+
}
114+
}
115+
94116
export class CSAssetsAdapter implements ICSAssetsAdapter {
95117
private readonly config: CSAssetsAPIConfig;
96118
private readonly apiClient: HttpClient;
@@ -607,10 +629,11 @@ export class CSAssetsAdapter implements ICSAssetsAdapter {
607629
}
608630
const text = await response.text().catch(() => '');
609631
const bodySnippet = this.formatResponseBodyForError(text);
610-
throw new Error(
632+
throw new CsAssetsPostError(
611633
`CS Assets API POST failed: status ${response.status} path ${path}${
612634
bodySnippet ? `\nResponse: ${bodySnippet}` : ''
613635
}`,
636+
response.status,
614637
);
615638
}
616639
return response.json() as Promise<T>;
@@ -627,12 +650,17 @@ export class CSAssetsAdapter implements ICSAssetsAdapter {
627650
: await doPost();
628651
} catch (error) {
629652
if (error instanceof RetryableHttpError) {
630-
throw new Error(`CS Assets API POST failed: path ${path} (status ${error.status ?? 'network'}) - ${error.message}`);
653+
throw new CsAssetsPostError(
654+
`CS Assets API POST failed: path ${path} (status ${error.status ?? 'network'}) - ${error.message}`,
655+
error.status,
656+
);
631657
}
632-
if (error instanceof Error && error.message.includes('CS Assets API POST failed')) {
658+
if (error instanceof CsAssetsPostError) {
633659
throw error;
634660
}
635-
throw new Error(`CS Assets API POST failed: path ${path} - ${error instanceof Error ? error.message : String(error)}`);
661+
throw new CsAssetsPostError(
662+
`CS Assets API POST failed: path ${path} - ${error instanceof Error ? error.message : String(error)}`,
663+
);
636664
}
637665
}
638666

@@ -737,18 +765,76 @@ export class CSAssetsAdapter implements ICSAssetsAdapter {
737765
payload: BulkDeleteAssetsPayload,
738766
): Promise<BulkDeleteAssetsResponse> {
739767
const path = `/api/spaces/${encodeURIComponent(spaceUid)}/assets/bulk/delete?workspace=${encodeURIComponent(workspaceUid)}`;
740-
return this.postJson<BulkDeleteAssetsResponse>(path, payload, { space_key: spaceUid });
768+
const bodies = chunk(payload.assets, CS_ASSETS_BULK_MUTATE_MAX_ITEMS).map((assets) => ({ assets }));
769+
const { notices, jobIds, failures, batchesTotal } = await this.dispatchBulkMutateBatches(spaceUid, path, bodies);
770+
return {
771+
notice: notices[0],
772+
primaryJobId: jobIds[0],
773+
notices,
774+
job_ids: jobIds,
775+
failures,
776+
batchesTotal,
777+
batchesSucceeded: batchesTotal - failures.length,
778+
};
741779
}
742780

743781
/**
744782
* POST /api/spaces/{spaceUid}/assets/bulk-move — move assets into a folder.
783+
* Split into ≤{@link CS_ASSETS_BULK_MUTATE_MAX_ITEMS}-item requests (same cap as delete).
745784
*/
746785
async bulkMoveAssets(
747786
spaceUid: string,
748787
workspaceUid: string = 'main',
749788
payload: BulkMoveAssetsPayload,
750789
): Promise<BulkMoveAssetsResponse> {
751790
const path = `/api/spaces/${encodeURIComponent(spaceUid)}/assets/bulk-move?workspace=${encodeURIComponent(workspaceUid)}`;
752-
return this.postJson<BulkMoveAssetsResponse>(path, payload, { space_key: spaceUid });
791+
const bodies = chunk(payload.asset_uids, CS_ASSETS_BULK_MUTATE_MAX_ITEMS).map((asset_uids) => ({
792+
asset_uids,
793+
target_folder_uid: payload.target_folder_uid,
794+
}));
795+
const { notices, failures, batchesTotal } = await this.dispatchBulkMutateBatches(spaceUid, path, bodies);
796+
return {
797+
notice: notices[0],
798+
notices,
799+
failures,
800+
batchesTotal,
801+
batchesSucceeded: batchesTotal - failures.length,
802+
};
803+
}
804+
805+
/**
806+
* Dispatch pre-chunked bulk-mutate request bodies (each already ≤100 items) through
807+
* {@link postJson} with bounded concurrency via {@link makeConcurrentCall}. A batch
808+
* failure is collected — never rethrown — because the CS Assets bulk endpoints commit
809+
* each request independently, so earlier batches are already applied server-side and
810+
* callers must be able to report partial outcomes. `postJson` retries transient
811+
* (429/5xx) failures; 4xx like the 422 item-cap are not retried.
812+
*/
813+
private async dispatchBulkMutateBatches(
814+
spaceUid: string,
815+
path: string,
816+
bodies: unknown[],
817+
): Promise<{ notices: string[]; jobIds: string[]; failures: BulkMutateFailure[]; batchesTotal: number }> {
818+
const notices: string[] = [];
819+
const jobIds: string[] = [];
820+
const failures: BulkMutateFailure[] = [];
821+
const apiBatches = chunk(bodies, FALLBACK_AM_API_CONCURRENCY);
822+
823+
await this.makeConcurrentCall({ module: `bulk-mutate ${path}`, apiBatches }, async ({ element, batchIndex, index }) => {
824+
const globalIndex = batchIndex * FALLBACK_AM_API_CONCURRENCY + index;
825+
const body = element as { assets?: { uid: string }[]; asset_uids?: string[] };
826+
const uids = body.assets ? body.assets.map((a) => a.uid) : (body.asset_uids ?? []);
827+
try {
828+
const r = await this.postJson<CsAssetsMutateBatchResponse>(path, body, { space_key: spaceUid }, { retry: true });
829+
if (typeof r.notice === 'string') notices.push(r.notice);
830+
if (typeof r.job_id === 'string') jobIds.push(r.job_id);
831+
} catch (e) {
832+
const status = e instanceof CsAssetsPostError ? e.status : undefined;
833+
const message = e instanceof Error ? e.message : String(e);
834+
failures.push({ batchIndex: globalIndex, count: uids.length, status, error: message, uids });
835+
}
836+
});
837+
838+
return { notices, jobIds, failures, batchesTotal: bodies.length };
753839
}
754840
}

0 commit comments

Comments
 (0)