diff --git a/CHANGELOG.md b/CHANGELOG.md index 650ee39b..7818b50f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,9 +18,16 @@ This changelog follows the principles of [Keep a Changelog](https://keepachangel - Collections: Added `allowedDatasetTypes` field to the [Collection](./src/collections/domain/models/Collection.ts) model. This field is optional and only populated the feature is enabled on the installation and configured on the collection. - Collections: Added theme information when retrieving a collection using `getCollection`. - Collections: Added `setDefaultContributorRole` use case. +- Datasets: `listDatasetTreeNode` use case and repository method backing `GET /datasets/{id}/versions/{versionId}/tree` for paginated, lazy listing of folders/files inside a dataset version. Returns `FileTreePage` with folder-first ordering, opaque keyset cursors, and per-file `downloadUrl`. The per-file `access` marker is one of `retentionExpired`, `restricted`, `embargoed`, or `public` (resolved in that order), and folder `counts` carry the matching mutually exclusive `restricted`/`embargoed`/`retentionExpired` buckets. +- Datasets: `iterateDatasetTreeNode` async generator that walks the cursor chain so callers can consume one folder's children without driving pagination by hand. +- Core: re-export `DataverseApiAuthMechanism` from the public surface so consumers of the standalone reusable bundles (e.g. `dv-tree-view`, `dv-uploader`) can import it without reaching into `core/...`. +- Files: export `DirectUploadClient` and `DirectUploadClientConfig` from the public files surface so consumers can construct their own client with custom timeouts / retry counts without reaching into the SDK's `infra/` path. ### Changed +- **BREAKING**: `DirectUploadClient` constructor signature changed from `(filesRepository, maxMultipartRetries = 5)` to `(filesRepository, config: DirectUploadClientConfig = {})`. `config` now holds `maxMultipartRetries` (default 5) and the new `fileUploadTimeoutMs` (default 60000). Existing TypeScript consumers passing the second argument as a number will need to migrate to `{ maxMultipartRetries: N }`; a bare number passed by plain-JS callers is still honored at runtime as `maxMultipartRetries`. +- Files: `DirectUploadClient` now reads the `tagging` field from the upload-destination response so operators on storage that doesn't accept S3 tags can opt out per-driver via `dataverse.files..disable-tagging=true`. The default behaviour is unchanged: when the server omits the field the client still sends `x-amz-tagging: dv-state=temp` (the same tag that earlier SDK versions hard-coded), so the new SDK is backwards-compatible with Dataverse releases that predate the response field. A server that explicitly returns an empty `tagging` value tells the client to skip the header entirely. Multipart uploads never send `x-amz-tagging` on part uploads — the part URLs are not signed for the header; the matching Dataverse server applies the temporary tag itself when it initiates the multipart upload. + ### Fixed ### Removed @@ -52,7 +59,7 @@ This changelog follows the principles of [Keep a Changelog](https://keepachangel ### Changed -- Add pagination query parameters to Dataset Version Summeries and File Version Summaries use cases. +- Add pagination query parameters to Dataset Version Summaries and File Version Summaries use cases. - Templates: Rename `CreateDatasetTemplateDTO` to `CreateTemplateDTO`. - Templates: Rename `createDatasetTemplate` repository method to `createTemplate`. - Templates: Rename `getDatasetTemplates` repository method to `getTemplatesByCollectionId`. diff --git a/docs/useCases.md b/docs/useCases.md index 79b7e853..06dfbf05 100644 --- a/docs/useCases.md +++ b/docs/useCases.md @@ -63,6 +63,8 @@ The different use cases currently available in the package are classified below, - [Get Dataset Available Dataset Types](#get-dataset-available-dataset-types) - [Get Dataset Available Dataset Type](#get-dataset-available-dataset-type) - [Get Dataset Upload Limits](#get-dataset-upload-limits) + - [List a Folder of a Dataset Version (Tree View)](#list-a-folder-of-a-dataset-version-tree-view) + - [Iterate a Folder of a Dataset Version (Tree View)](#iterate-a-folder-of-a-dataset-version-tree-view) - [Datasets write use cases](#datasets-write-use-cases) - [Create a Dataset](#create-a-dataset) - [Update a Dataset](#update-a-dataset) @@ -1912,6 +1914,81 @@ _See [use case](../src/datasets/domain/useCases/GetDatasetUploadLimits.ts) imple If the backend does not define any quota limits for the dataset, the returned object can be empty (`{}`). +#### List a Folder of a Dataset Version (Tree View) + +Returns a [FileTreePage](../src/datasets/domain/models/FileTreePage.ts) for the immediate children (folders and files) inside a folder of a dataset version, intended for lazy tree-view UIs that fetch each folder's children on demand. + +Folders come first, then files. Both are name-sorted (case-insensitive); files break ties on data file id for stability. The page carries an opaque `nextCursor` token; clients echo it back to fetch the next page and never construct one themselves. + +##### Example call: + +```typescript +import { listDatasetTreeNode, FileTreePage } from '@iqss/dataverse-client-javascript' + +/* ... */ + +const datasetId = 'doi:10.77777/FK2/AAAAAA' + +listDatasetTreeNode + .execute({ + datasetId, + datasetVersionId: '1.0', + path: 'data/raw', + limit: 100 + }) + .then((page: FileTreePage) => { + /* ... */ + }) + +/* ... */ +``` + +_See [use case](../src/datasets/domain/useCases/ListDatasetTreeNode.ts) implementation_. + +`datasetId` can be a numeric id or a persistent identifier string. `datasetVersionId` is optional and defaults to `DatasetNotNumberedVersion.LATEST`. + +Other optional parameters: `cursor` (opaque, from a previous response), `include` (`'all' | 'folders' | 'files'`, default `'all'`), `order` (`'NameAZ' | 'NameZA'`, default `'NameAZ'`), `includeDeaccessioned` (default `false`), and `originals` (when `true`, ingested tabular files are reported in their original-upload form: the per-file `downloadUrl` carries `?format=original`, and `checksum` and `size` reflect the saved original instead of the converted TSV). + +The `path` value is normalized by the server with the same rules applied to folder names at upload time (slash/backslash runs collapse, leading dots/dashes/spaces are stripped); paths emitted by the endpoint itself always round-trip, and a path that normalizes to nothing (for example `..`) is rejected with a `400` error. + +For published, non-deaccessioned versions the underlying API emits an `ETag` (derived from the request inputs plus the current date) and `Cache-Control: private, no-cache` headers. A released version's file list is frozen, but the response is still time-dependent — the per-file `access` marker flips when an embargo lapses or a retention period expires — so the contract is revalidate-always (`no-cache`) with a date-scoped validator: passing the ETag back in `If-None-Match` yields a body-less `304 Not Modified` while it still matches. The `private` directive keeps responses out of shared proxy caches because the route is auth-required. Drafts and deaccessioned versions emit no caching headers. + +#### Iterate a Folder of a Dataset Version (Tree View) + +Returns an async generator over [FileTreeNode](../src/datasets/domain/models/FileTreeNode.ts) values for one folder, walking the cursor chain so callers can consume the children without driving pagination by hand. + +##### Example call: + +```typescript +import { + iterateDatasetTreeNode, + FileTreeNode, + isFileTreeFileNode +} from '@iqss/dataverse-client-javascript' + +/* ... */ + +const datasetId = 'doi:10.77777/FK2/AAAAAA' + +for await (const node of iterateDatasetTreeNode.execute({ + datasetId, + datasetVersionId: '1.0', + path: 'data/raw' +})) { + if (isFileTreeFileNode(node)) { + /* ... */ + } else { + /* node is a folder ... */ + } +} + +/* ... */ +``` + +_See [use case](../src/datasets/domain/useCases/IterateDatasetTreeNode.ts) implementation_. + +The generator stops after yielding everything in the requested folder; it does **not** descend into subfolders. Pass each subfolder's `path` back through `iterateDatasetTreeNode` if you want a recursive walk. + ## Files ### Files read use cases diff --git a/src/core/index.ts b/src/core/index.ts index ddd54a8a..e8c17b25 100644 --- a/src/core/index.ts +++ b/src/core/index.ts @@ -1,6 +1,6 @@ export { ReadError } from './domain/repositories/ReadError' export { WriteError } from './domain/repositories/WriteError' -export { ApiConfig } from './infra/repositories/ApiConfig' +export { ApiConfig, DataverseApiAuthMechanism } from './infra/repositories/ApiConfig' export { DvObjectOwnerNode, DvObjectType } from './domain/models/DvObjectOwnerNode' export { PublicationStatus } from './domain/models/PublicationStatus' export { StorageDriver } from './domain/models/StorageDriver' diff --git a/src/datasets/domain/models/FileTreeNode.ts b/src/datasets/domain/models/FileTreeNode.ts new file mode 100644 index 00000000..c0510ace --- /dev/null +++ b/src/datasets/domain/models/FileTreeNode.ts @@ -0,0 +1,41 @@ +export enum FileTreeNodeType { + FOLDER = 'folder', + FILE = 'file' +} + +export interface FileTreeFolderNode { + type: FileTreeNodeType.FOLDER + name: string + path: string + counts?: { + files: number + folders: number + bytes: number + restricted: number + embargoed: number + retentionExpired: number + } +} + +export interface FileTreeFileNode { + type: FileTreeNodeType.FILE + id: number + name: string + path: string + size: number + contentType?: string + access?: 'public' | 'restricted' | 'embargoed' | 'retentionExpired' + checksum?: { + type: string + value: string + } + downloadUrl: string +} + +export type FileTreeNode = FileTreeFolderNode | FileTreeFileNode + +export const isFileTreeFolderNode = (node: FileTreeNode): node is FileTreeFolderNode => + node.type === FileTreeNodeType.FOLDER + +export const isFileTreeFileNode = (node: FileTreeNode): node is FileTreeFileNode => + node.type === FileTreeNodeType.FILE diff --git a/src/datasets/domain/models/FileTreePage.ts b/src/datasets/domain/models/FileTreePage.ts new file mode 100644 index 00000000..77c139e3 --- /dev/null +++ b/src/datasets/domain/models/FileTreePage.ts @@ -0,0 +1,22 @@ +import { FileTreeNode } from './FileTreeNode' + +export enum FileTreeInclude { + ALL = 'all', + FOLDERS = 'folders', + FILES = 'files' +} + +export enum FileTreeOrder { + NAME_AZ = 'NameAZ', + NAME_ZA = 'NameZA' +} + +export interface FileTreePage { + path: string + items: FileTreeNode[] + nextCursor: string | null + limit: number + order: FileTreeOrder + include: FileTreeInclude + approximateCount?: number +} diff --git a/src/datasets/domain/repositories/IDatasetsRepository.ts b/src/datasets/domain/repositories/IDatasetsRepository.ts index a50ad2ff..0dea979f 100644 --- a/src/datasets/domain/repositories/IDatasetsRepository.ts +++ b/src/datasets/domain/repositories/IDatasetsRepository.ts @@ -19,9 +19,22 @@ import { DatasetTypeDTO } from '../dtos/DatasetTypeDTO' import { StorageDriver } from '../../../core/domain/models/StorageDriver' import { DatasetUploadLimits } from '../models/DatasetUploadLimits' import { DatasetReview } from '../models/DatasetReview' +import { FileTreePage, FileTreeInclude, FileTreeOrder } from '../models/FileTreePage' import { ExportedDatasetMetadata } from '../models/ExportedDatasetMetadata' import { DatasetNotNumberedVersion } from '../models/DatasetNotNumberedVersion' +export interface ListDatasetTreeNodeParams { + datasetId: number | string + datasetVersionId?: string + path?: string + limit?: number + cursor?: string + include?: FileTreeInclude + order?: FileTreeOrder + includeDeaccessioned?: boolean + originals?: boolean +} + export interface IDatasetsRepository { getDataset( datasetId: number | string, @@ -113,4 +126,5 @@ export interface IDatasetsRepository { getDatasetStorageDriver(datasetId: number | string): Promise getDatasetUploadLimits(datasetId: number | string): Promise getDatasetReviews(datasetId: number | string): Promise + listDatasetTreeNode(params: ListDatasetTreeNodeParams): Promise } diff --git a/src/datasets/domain/useCases/IterateDatasetTreeNode.ts b/src/datasets/domain/useCases/IterateDatasetTreeNode.ts new file mode 100644 index 00000000..46faf380 --- /dev/null +++ b/src/datasets/domain/useCases/IterateDatasetTreeNode.ts @@ -0,0 +1,26 @@ +import { IDatasetsRepository, ListDatasetTreeNodeParams } from '../repositories/IDatasetsRepository' +import { FileTreeNode } from '../models/FileTreeNode' + +export class IterateDatasetTreeNode { + constructor(private readonly datasetsRepository: IDatasetsRepository) {} + + async *execute(params: ListDatasetTreeNodeParams): AsyncGenerator { + let cursor = params.cursor + do { + const page = await this.datasetsRepository.listDatasetTreeNode({ + ...params, + cursor + }) + for (const item of page.items) { + yield item + } + const nextCursor = page.nextCursor ?? undefined + if (nextCursor !== undefined && nextCursor === cursor) { + throw new Error( + `Dataset tree pagination cursor did not advance ("${nextCursor}"); aborting iteration` + ) + } + cursor = nextCursor + } while (cursor !== undefined) + } +} diff --git a/src/datasets/domain/useCases/ListDatasetTreeNode.ts b/src/datasets/domain/useCases/ListDatasetTreeNode.ts new file mode 100644 index 00000000..90d148e7 --- /dev/null +++ b/src/datasets/domain/useCases/ListDatasetTreeNode.ts @@ -0,0 +1,11 @@ +import { UseCase } from '../../../core/domain/useCases/UseCase' +import { IDatasetsRepository, ListDatasetTreeNodeParams } from '../repositories/IDatasetsRepository' +import { FileTreePage } from '../models/FileTreePage' + +export class ListDatasetTreeNode implements UseCase { + constructor(private readonly datasetsRepository: IDatasetsRepository) {} + + async execute(params: ListDatasetTreeNodeParams): Promise { + return this.datasetsRepository.listDatasetTreeNode(params) + } +} diff --git a/src/datasets/index.ts b/src/datasets/index.ts index de7a32b4..f2052278 100644 --- a/src/datasets/index.ts +++ b/src/datasets/index.ts @@ -36,6 +36,8 @@ import { UpdateDatasetLicense } from './domain/useCases/UpdateDatasetLicense' import { GetDatasetStorageDriver } from './domain/useCases/GetDatasetStorageDriver' import { GetDatasetUploadLimits } from './domain/useCases/GetDatasetUploadLimits' import { GetDatasetReviews } from './domain/useCases/GetDatasetReviews' +import { ListDatasetTreeNode } from './domain/useCases/ListDatasetTreeNode' +import { IterateDatasetTreeNode } from './domain/useCases/IterateDatasetTreeNode' import { ExportDatasetMetadata } from './domain/useCases/ExportDatasetMetadata' const datasetsRepository = new DatasetsRepository() @@ -89,6 +91,8 @@ const updateDatasetLicense = new UpdateDatasetLicense(datasetsRepository) const getDatasetStorageDriver = new GetDatasetStorageDriver(datasetsRepository) const getDatasetUploadLimits = new GetDatasetUploadLimits(datasetsRepository) const getDatasetReviews = new GetDatasetReviews(datasetsRepository) +const listDatasetTreeNode = new ListDatasetTreeNode(datasetsRepository) +const iterateDatasetTreeNode = new IterateDatasetTreeNode(datasetsRepository) const exportDatasetMetadata = new ExportDatasetMetadata(datasetsRepository) export { @@ -124,6 +128,8 @@ export { getDatasetStorageDriver, getDatasetUploadLimits, getDatasetReviews, + listDatasetTreeNode, + iterateDatasetTreeNode, exportDatasetMetadata } export { DatasetNotNumberedVersion } from './domain/models/DatasetNotNumberedVersion' @@ -170,3 +176,13 @@ export { DatasetReviewRubricMetadataBlock, DatasetReviewRubricMetadataField } from './domain/models/DatasetReview' +export { + FileTreeNode, + FileTreeFolderNode, + FileTreeFileNode, + FileTreeNodeType, + isFileTreeFolderNode, + isFileTreeFileNode +} from './domain/models/FileTreeNode' +export { FileTreePage, FileTreeInclude, FileTreeOrder } from './domain/models/FileTreePage' +export { ListDatasetTreeNodeParams } from './domain/repositories/IDatasetsRepository' diff --git a/src/datasets/infra/repositories/DatasetsRepository.ts b/src/datasets/infra/repositories/DatasetsRepository.ts index 1eba3fa4..f497ee0b 100644 --- a/src/datasets/infra/repositories/DatasetsRepository.ts +++ b/src/datasets/infra/repositories/DatasetsRepository.ts @@ -1,5 +1,11 @@ import { ApiRepository } from '../../../core/infra/repositories/ApiRepository' -import { IDatasetsRepository } from '../../domain/repositories/IDatasetsRepository' +import { + IDatasetsRepository, + ListDatasetTreeNodeParams +} from '../../domain/repositories/IDatasetsRepository' +import { DatasetNotNumberedVersion } from '../../domain/models/DatasetNotNumberedVersion' +import { FileTreePage } from '../../domain/models/FileTreePage' +import { transformTreeResponseToFileTreePage } from './transformers/fileTreeTransformers' import { Dataset, VersionUpdateType } from '../../domain/models/Dataset' import { transformVersionResponseToDataset, @@ -26,7 +32,6 @@ import { transformDatasetLinkedCollectionsResponseToDatasetLinkedCollection } fr import { FormattedCitation } from '../../domain/models/FormattedCitation' import { DatasetType } from '../../domain/models/DatasetType' import { TermsOfAccess } from '../../domain/models/Dataset' -import { DatasetNotNumberedVersion } from '../../domain/models/DatasetNotNumberedVersion' import { transformTermsOfAccessToUpdatePayload } from './transformers/termsOfAccessTransformers' import { DatasetLicenseUpdateRequest } from '../../domain/dtos/DatasetLicenseUpdateRequest' import { DatasetTypeDTO } from '../../domain/dtos/DatasetTypeDTO' @@ -568,4 +573,33 @@ export class DatasetsRepository extends ApiRepository implements IDatasetsReposi throw error }) } + + public async listDatasetTreeNode(params: ListDatasetTreeNodeParams): Promise { + const versionId = params.datasetVersionId ?? DatasetNotNumberedVersion.LATEST + const queryParams: Record = {} + if (params.path !== undefined) queryParams.path = params.path + if (params.limit !== undefined) queryParams.limit = params.limit + if (params.cursor !== undefined) queryParams.cursor = params.cursor + if (params.include !== undefined) queryParams.include = params.include + if (params.order !== undefined) queryParams.order = params.order + if (params.includeDeaccessioned !== undefined) { + queryParams.includeDeaccessioned = params.includeDeaccessioned + } + if (params.originals !== undefined) { + queryParams.originals = params.originals + } + return this.doGet( + this.buildApiEndpoint( + this.datasetsResourceName, + `versions/${versionId}/tree`, + params.datasetId + ), + true, + queryParams + ) + .then((response) => transformTreeResponseToFileTreePage(response)) + .catch((error) => { + throw error + }) + } } diff --git a/src/datasets/infra/repositories/transformers/fileTreeTransformers.ts b/src/datasets/infra/repositories/transformers/fileTreeTransformers.ts new file mode 100644 index 00000000..1abfcd1e --- /dev/null +++ b/src/datasets/infra/repositories/transformers/fileTreeTransformers.ts @@ -0,0 +1,96 @@ +import { AxiosResponse } from 'axios' +import { FileTreeInclude, FileTreeOrder, FileTreePage } from '../../../domain/models/FileTreePage' +import { + FileTreeFileNode, + FileTreeFolderNode, + FileTreeNode, + FileTreeNodeType +} from '../../../domain/models/FileTreeNode' + +interface FolderItemPayload { + type: 'folder' + name: string + path: string + counts?: { + files: number + folders: number + bytes: number + restricted: number + embargoed: number + retentionExpired: number + } +} + +interface FileItemPayload { + type: 'file' + id: number + name: string + path: string + size: number + contentType?: string + access?: 'public' | 'restricted' | 'embargoed' | 'retentionExpired' + checksum?: { type: string; value: string } + downloadUrl: string +} + +type ItemPayload = FolderItemPayload | FileItemPayload + +interface TreeResponsePayload { + path: string + items: ItemPayload[] + nextCursor: string | null + limit: number + order: string + include: string + approximateCount?: number +} + +export const transformTreeResponseToFileTreePage = (response: AxiosResponse): FileTreePage => { + const payload = response.data.data as TreeResponsePayload + return { + path: payload.path, + items: payload.items.map(transformItem), + nextCursor: payload.nextCursor, + limit: payload.limit, + order: parseOrder(payload.order), + include: parseInclude(payload.include), + approximateCount: payload.approximateCount + } +} + +const transformItem = (item: any): FileTreeNode => { + if (item?.type === 'folder') return transformFolder(item as FolderItemPayload) + if (item?.type === 'file') return transformFile(item as FileItemPayload) + throw new Error(`Unknown dataset tree node type "${String(item?.type)}"`) +} + +const transformFolder = (item: FolderItemPayload): FileTreeFolderNode => ({ + type: FileTreeNodeType.FOLDER, + name: item.name, + path: item.path, + counts: item.counts +}) + +const transformFile = (item: FileItemPayload): FileTreeFileNode => ({ + type: FileTreeNodeType.FILE, + id: item.id, + name: item.name, + path: item.path, + size: item.size, + contentType: item.contentType, + access: item.access, + checksum: item.checksum, + downloadUrl: item.downloadUrl +}) + +const parseOrder = (value: string): FileTreeOrder => { + return (Object.values(FileTreeOrder) as string[]).includes(value) + ? (value as FileTreeOrder) + : FileTreeOrder.NAME_AZ +} + +const parseInclude = (value: string): FileTreeInclude => { + return (Object.values(FileTreeInclude) as string[]).includes(value) + ? (value as FileTreeInclude) + : FileTreeInclude.ALL +} diff --git a/src/files/domain/models/FileUploadDestination.ts b/src/files/domain/models/FileUploadDestination.ts index 4bb42c2a..4d6edf13 100644 --- a/src/files/domain/models/FileUploadDestination.ts +++ b/src/files/domain/models/FileUploadDestination.ts @@ -4,4 +4,5 @@ export interface FileUploadDestination { partSize: number abortEndpoint?: string completeEndpoint?: string + tagging?: string } diff --git a/src/files/index.ts b/src/files/index.ts index f49a0ea3..cc80ce0f 100644 --- a/src/files/index.ts +++ b/src/files/index.ts @@ -104,3 +104,4 @@ export { FileMetadataChange, FileVersionSummarySubset } from './domain/models/FileVersionSummaryInfo' +export { DirectUploadClient, DirectUploadClientConfig } from './infra/clients/DirectUploadClient' diff --git a/src/files/infra/clients/DirectUploadClient.ts b/src/files/infra/clients/DirectUploadClient.ts index 8dfd1a9b..eed5bf71 100644 --- a/src/files/infra/clients/DirectUploadClient.ts +++ b/src/files/infra/clients/DirectUploadClient.ts @@ -15,15 +15,22 @@ import { MultipartAbortError } from './errors/MultipartAbortError' import { FileUploadCancelError } from './errors/FileUploadCancelError' import { ApiConstants } from '../../../core/infra/repositories/ApiConstants' +export interface DirectUploadClientConfig { + maxMultipartRetries?: number + fileUploadTimeoutMs?: number +} + export class DirectUploadClient implements IDirectUploadClient { private filesRepository: IFilesRepository private maxMultipartRetries: number + private readonly fileUploadTimeoutMs: number - private readonly fileUploadTimeoutMs: number = 60_000 - - constructor(filesRepository: IFilesRepository, maxMultipartRetries = 5) { + constructor(filesRepository: IFilesRepository, config: DirectUploadClientConfig = {}) { + const normalized: DirectUploadClientConfig = + typeof config === 'number' ? { maxMultipartRetries: config } : config this.filesRepository = filesRepository - this.maxMultipartRetries = maxMultipartRetries + this.maxMultipartRetries = normalized.maxMultipartRetries ?? 5 + this.fileUploadTimeoutMs = normalized.fileUploadTimeoutMs ?? 60_000 } public async uploadFile( @@ -59,11 +66,15 @@ export class DirectUploadClient implements IDirectUploadClient { ): Promise { try { const arrayBuffer = await file.arrayBuffer() + const headers: Record = { + 'Content-Type': 'application/octet-stream' + } + const tag = destination.tagging ?? 'dv-state=temp' + if (tag !== '') { + headers['x-amz-tagging'] = tag + } await axios.put(destination.urls[0], arrayBuffer, { - headers: { - 'Content-Type': 'application/octet-stream', - 'x-amz-tagging': 'dv-state=temp' - }, + headers, timeout: this.fileUploadTimeoutMs, signal: abortController.signal, onUploadProgress: (progressEvent) => @@ -115,6 +126,7 @@ export class DirectUploadClient implements IDirectUploadClient { eTags[`${index + 1}`] = eTag } catch (error) { if (axios.isCancel(error)) { + limitConcurrency.clearQueue() await this.abortMultipartUpload(file.name, datasetId, destination.abortEndpoint as string) throw new FileUploadCancelError(file.name, datasetId) } @@ -123,6 +135,7 @@ export class DirectUploadClient implements IDirectUploadClient { await new Promise((resolve) => setTimeout(resolve, backoffDelay)) await uploadPart(destinationUrl, index, retries + 1) } else { + limitConcurrency.clearQueue() await this.abortMultipartUpload(file.name, datasetId, destination.abortEndpoint as string) const errorMessage = diff --git a/src/files/infra/repositories/transformers/fileUploadDestinationsTransformers.ts b/src/files/infra/repositories/transformers/fileUploadDestinationsTransformers.ts index 55a35757..f1a195dc 100644 --- a/src/files/infra/repositories/transformers/fileUploadDestinationsTransformers.ts +++ b/src/files/infra/repositories/transformers/fileUploadDestinationsTransformers.ts @@ -5,6 +5,7 @@ export interface FileSingleUploadDestinationPayload { url: string partSize: number storageIdentifier: string + tagging?: string } export interface FileMultipartUploadDestinationPayload { @@ -13,6 +14,7 @@ export interface FileMultipartUploadDestinationPayload { storageIdentifier: string complete?: string abort?: string + tagging?: string } export const transformUploadDestinationsResponseToUploadDestination = ( @@ -24,7 +26,8 @@ export const transformUploadDestinationsResponseToUploadDestination = ( return { urls: [fileUploadDestinationsPayload.url], partSize: fileUploadDestinationsPayload.partSize, - storageId: fileUploadDestinationsPayload.storageIdentifier + storageId: fileUploadDestinationsPayload.storageIdentifier, + tagging: fileUploadDestinationsPayload.tagging } } else { return transformMultipartUploadDestinationsPayloadToMultipartUploadDestinationModel( @@ -45,6 +48,7 @@ export const transformMultipartUploadDestinationsPayloadToMultipartUploadDestina partSize: fileUploadDestinationsPayload.partSize, storageId: fileUploadDestinationsPayload.storageIdentifier, abortEndpoint: fileUploadDestinationsPayload.abort?.substring(4), - completeEndpoint: fileUploadDestinationsPayload.complete?.substring(4) + completeEndpoint: fileUploadDestinationsPayload.complete?.substring(4), + tagging: fileUploadDestinationsPayload.tagging } } diff --git a/test/environment/setup.ts b/test/environment/setup.ts index bc46cc42..c312162e 100644 --- a/test/environment/setup.ts +++ b/test/environment/setup.ts @@ -2,6 +2,10 @@ import * as fs from 'fs' import { DockerComposeEnvironment, Wait } from 'testcontainers' import axios from 'axios' import { TestConstants } from '../testHelpers/TestConstants' +import { + DATASET_TREE_ENDPOINT_AVAILABLE_ENV_VAR, + isDatasetTreeEndpointAvailableViaApi +} from '../testHelpers/datasets/datasetTreeHelper' const COMPOSE_FILE = 'docker-compose.yml' @@ -17,6 +21,7 @@ const API_KEY_USER_PASSWORD = 'admin1' export default async function setupTestEnvironment(): Promise { await setupContainers(SKIP_CONTAINERS) //Set skipContainers to true to skip container setup and run tests against an already running instance await setupApiKey() + await detectDatasetTreeEndpoint() } async function setupContainers(skipContainers?: boolean): Promise { @@ -49,3 +54,13 @@ async function setupApiKey(): Promise { }) console.log('API key obtained') } + +async function detectDatasetTreeEndpoint(): Promise { + const available = await isDatasetTreeEndpointAvailableViaApi().catch(() => false) + process.env[DATASET_TREE_ENDPOINT_AVAILABLE_ENV_VAR] = String(available) + console.log( + available + ? 'Dataset tree endpoint available; tree integration tests will run' + : 'Dataset tree endpoint missing on this Dataverse; tree integration tests will be skipped' + ) +} diff --git a/test/functional/collections/UpdateCollectionFeaturedItems.test.ts b/test/functional/collections/UpdateCollectionFeaturedItems.test.ts index 87d3ebec..6f7e162c 100644 --- a/test/functional/collections/UpdateCollectionFeaturedItems.test.ts +++ b/test/functional/collections/UpdateCollectionFeaturedItems.test.ts @@ -27,6 +27,7 @@ import { FeaturedItemType } from '../../../src/collections/domain/models/FeaturedItem' import { uploadFileViaApi } from '../../testHelpers/files/filesHelper' +import { normalizeHtml } from '../../testHelpers/html/htmlNormalizer' import { deletePublishedDatasetViaApi, publishDatasetViaApi, @@ -165,7 +166,9 @@ describe('execute', () => { expect(secondItemResponse.imageFileUrl).toBeUndefined() expect(secondItemResponse.imageFileName).toBeUndefined() - expect(thirdItemResponse.content).toEqual(EXPECTED_CONTENT_FIELD_WITH_ALL_TAGS) + expect(normalizeHtml(thirdItemResponse.content)).toEqual( + normalizeHtml(EXPECTED_CONTENT_FIELD_WITH_ALL_TAGS) + ) expect(thirdItemResponse.displayOrder).toBe(newFeaturedItems[2].displayOrder) expect(thirdItemResponse.imageFileName).toEqual('featured-item-test-image-3.png') expect(thirdItemResponse.imageFileUrl).toContain( diff --git a/test/integration/datasets/DatasetTreeNode.test.ts b/test/integration/datasets/DatasetTreeNode.test.ts new file mode 100644 index 00000000..ec6d2d04 --- /dev/null +++ b/test/integration/datasets/DatasetTreeNode.test.ts @@ -0,0 +1,250 @@ +import { ApiConfig, DataverseApiAuthMechanism } from '../../../src' +import { ReadError } from '../../../src/core/domain/repositories/ReadError' +import { + CreatedDatasetIdentifiers, + DatasetNotNumberedVersion, + FileTreeFileNode, + FileTreeFolderNode, + FileTreeInclude, + FileTreeNode, + FileTreeNodeType, + FileTreeOrder, + createDataset, + isFileTreeFileNode, + isFileTreeFolderNode, + iterateDatasetTreeNode, + listDatasetTreeNode +} from '../../../src/datasets' +import { TestConstants } from '../../testHelpers/TestConstants' +import { + createCollectionViaApi, + deleteCollectionViaApi +} from '../../testHelpers/collections/collectionHelper' +import { deleteUnpublishedDatasetViaApi } from '../../testHelpers/datasets/datasetHelper' +import { + createDatasetTreeFixtureViaApi, + datasetTreeEndpointIsAvailable +} from '../../testHelpers/datasets/datasetTreeHelper' + +const describeTree = datasetTreeEndpointIsAvailable() ? describe : describe.skip + +describeTree('Dataset tree node listing', () => { + const testCollectionAlias = 'datasetTreeTestCollection' + let testDatasetIds: CreatedDatasetIdentifiers + + const names = (items: FileTreeNode[]): string[] => items.map((item) => item.name) + + beforeAll(async () => { + ApiConfig.init( + TestConstants.TEST_API_URL, + DataverseApiAuthMechanism.API_KEY, + process.env.TEST_API_KEY + ) + await createCollectionViaApi(testCollectionAlias) + try { + testDatasetIds = await createDataset.execute( + TestConstants.TEST_NEW_DATASET_DTO, + testCollectionAlias + ) + } catch (error) { + throw new Error('Tests beforeAll(): Error while creating test dataset') + } + await createDatasetTreeFixtureViaApi(testDatasetIds.numericId) + }) + + afterAll(async () => { + await deleteUnpublishedDatasetViaApi(testDatasetIds.numericId) + await deleteCollectionViaApi(testCollectionAlias) + }) + + describe('listDatasetTreeNode', () => { + test('should list the immediate children of the dataset root, folders before files', async () => { + const actual = await listDatasetTreeNode.execute({ datasetId: testDatasetIds.numericId }) + + expect(actual.path).toBe('') + expect(actual.limit).toBeGreaterThan(0) + expect(actual.order).toBe(FileTreeOrder.NAME_AZ) + expect(actual.include).toBe(FileTreeInclude.ALL) + expect(actual.nextCursor).toBeNull() + expect(actual.approximateCount).toBe(3) + expect(names(actual.items)).toEqual(['data', 'docs', 'root.txt']) + + const dataFolder = actual.items[0] as FileTreeFolderNode + expect(dataFolder.type).toBe(FileTreeNodeType.FOLDER) + expect(dataFolder.path).toBe('data') + expect(dataFolder.counts?.files).toBe(3) + expect(dataFolder.counts?.folders).toBe(1) + expect(dataFolder.counts?.bytes).toBeGreaterThan(0) + expect(dataFolder.counts?.restricted).toBe(0) + expect(dataFolder.counts?.embargoed).toBe(0) + expect(dataFolder.counts?.retentionExpired).toBe(0) + + const rootFile = actual.items[2] as FileTreeFileNode + expect(rootFile.type).toBe(FileTreeNodeType.FILE) + expect(rootFile.path).toBe('root.txt') + expect(rootFile.size).toBeGreaterThan(0) + expect(rootFile.contentType).toBe('text/plain') + expect(rootFile.access).toBe('public') + expect(rootFile.checksum?.value).toEqual(expect.any(String)) + expect(rootFile.downloadUrl).toBe(`/api/access/datafile/${rootFile.id}`) + }) + + test('should narrow the type of each returned node through the exported type guards', async () => { + const actual = await listDatasetTreeNode.execute({ datasetId: testDatasetIds.numericId }) + + expect(actual.items.filter(isFileTreeFolderNode).map((folder) => folder.name)).toEqual([ + 'data', + 'docs' + ]) + expect(actual.items.filter(isFileTreeFileNode).map((file) => file.name)).toEqual(['root.txt']) + }) + + test('should list only the immediate children of a nested path', async () => { + const actual = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + path: 'data' + }) + + expect(actual.path).toBe('data') + expect(names(actual.items)).toEqual(['sub', 'a.txt', 'b.txt']) + expect((actual.items[0] as FileTreeFolderNode).path).toBe('data/sub') + expect((actual.items[1] as FileTreeFileNode).path).toBe('data/a.txt') + }) + + test('should accept a persistent identifier and an explicit version as the dataset coordinates', async () => { + const actual = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.persistentId, + datasetVersionId: DatasetNotNumberedVersion.DRAFT + }) + + expect(names(actual.items)).toEqual(['data', 'docs', 'root.txt']) + }) + + test('should return only folders when the include filter asks for folders', async () => { + const actual = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + include: FileTreeInclude.FOLDERS + }) + + expect(actual.include).toBe(FileTreeInclude.FOLDERS) + expect(names(actual.items)).toEqual(['data', 'docs']) + expect(actual.items.every(isFileTreeFolderNode)).toBe(true) + }) + + test('should return only files when the include filter asks for files', async () => { + const actual = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + include: FileTreeInclude.FILES + }) + + expect(actual.include).toBe(FileTreeInclude.FILES) + expect(names(actual.items)).toEqual(['root.txt']) + expect(actual.items.every(isFileTreeFileNode)).toBe(true) + }) + + test('should reverse the name ordering within each node type when descending order is requested', async () => { + const actual = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + order: FileTreeOrder.NAME_ZA + }) + + expect(actual.order).toBe(FileTreeOrder.NAME_ZA) + expect(names(actual.items)).toEqual(['docs', 'data', 'root.txt']) + }) + + test('should point the download url at the original form when originals are requested', async () => { + const actual = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + include: FileTreeInclude.FILES, + originals: true + }) + + const rootFile = actual.items[0] as FileTreeFileNode + expect(rootFile.downloadUrl).toBe(`/api/access/datafile/${rootFile.id}?format=original`) + }) + + test('should page through the listing with the server-issued cursor', async () => { + const firstPage = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + limit: 2 + }) + + expect(firstPage.limit).toBe(2) + expect(names(firstPage.items)).toEqual(['data', 'docs']) + expect(firstPage.nextCursor).toEqual(expect.any(String)) + + const secondPage = await listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + limit: 2, + cursor: firstPage.nextCursor as string + }) + + expect(names(secondPage.items)).toEqual(['root.txt']) + expect(secondPage.nextCursor).toBeNull() + }) + + test('should throw a ReadError when the cursor was not issued by the server', async () => { + await expect( + listDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + cursor: 'not-a-real-cursor' + }) + ).rejects.toThrow(ReadError) + }) + + test('should throw a ReadError when the dataset does not exist', async () => { + await expect( + listDatasetTreeNode.execute({ datasetId: TestConstants.TEST_DUMMY_PERSISTENT_ID }) + ).rejects.toThrow(ReadError) + }) + }) + + describe('iterateDatasetTreeNode', () => { + const collect = async (generator: AsyncGenerator): Promise => { + const collected: FileTreeNode[] = [] + for await (const node of generator) { + collected.push(node) + } + return collected + } + + test('should yield every child of a path, walking the cursor across pages', async () => { + const actual = await collect( + iterateDatasetTreeNode.execute({ datasetId: testDatasetIds.numericId, limit: 1 }) + ) + + expect(names(actual)).toEqual(['data', 'docs', 'root.txt']) + }) + + test('should yield the same nodes as a single page when the whole listing fits in one', async () => { + const iterated = await collect( + iterateDatasetTreeNode.execute({ datasetId: testDatasetIds.numericId }) + ) + const listed = await listDatasetTreeNode.execute({ datasetId: testDatasetIds.numericId }) + + expect(iterated).toEqual(listed.items) + }) + + test('should walk a nested path with the include and order options applied to every page', async () => { + const actual = await collect( + iterateDatasetTreeNode.execute({ + datasetId: testDatasetIds.numericId, + path: 'data', + include: FileTreeInclude.FILES, + order: FileTreeOrder.NAME_ZA, + limit: 1 + }) + ) + + expect(names(actual)).toEqual(['b.txt', 'a.txt']) + }) + + test('should surface a ReadError from the first page instead of yielding nodes', async () => { + const generator = iterateDatasetTreeNode.execute({ + datasetId: TestConstants.TEST_DUMMY_PERSISTENT_ID + }) + + await expect(generator.next()).rejects.toThrow(ReadError) + }) + }) +}) diff --git a/test/integration/files/DirectUpload.test.ts b/test/integration/files/DirectUpload.test.ts index a3931ab8..b92607f0 100644 --- a/test/integration/files/DirectUpload.test.ts +++ b/test/integration/files/DirectUpload.test.ts @@ -10,6 +10,8 @@ import { import { DataverseApiAuthMechanism } from '../../../src/core/infra/repositories/ApiConfig' import { FilesRepository } from '../../../src/files/infra/repositories/FilesRepository' import { DirectUploadClient } from '../../../src/files/infra/clients/DirectUploadClient' +import { DirectUploadClientConfig } from '../../../src/files' +import { FileUploadDestination } from '../../../src/files/domain/models/FileUploadDestination' import { TestConstants } from '../../testHelpers/TestConstants' import { createCollectionViaApi, @@ -20,10 +22,15 @@ import { deleteUnpublishedDatasetViaApi } from '../../testHelpers/datasets/datas import axios from 'axios' import { createMultipartFileBlob, - createSinglepartFileBlob + createSinglepartFileBlob, + getObjectTagsFromBucket } from '../../testHelpers/files/filesHelper' import { FileUploadCancelError } from '../../../src/files/infra/clients/errors/FileUploadCancelError' +import { FileUploadError } from '../../../src/files/infra/clients/errors/FileUploadError' +import { FilePartUploadError } from '../../../src/files/infra/clients/errors/FilePartUploadError' import * as crypto from 'crypto' +import * as http from 'http' +import { AddressInfo } from 'net' describe('Direct Upload', () => { const testCollectionAlias = 'directUploadTestCollection' @@ -33,6 +40,7 @@ describe('Direct Upload', () => { let testDatset4Ids: CreatedDatasetIdentifiers let testDataset5Ids: CreatedDatasetIdentifiers let testDataset6Ids: CreatedDatasetIdentifiers + let testDataset7Ids: CreatedDatasetIdentifiers const filesRepositorySut = new FilesRepository() const directUploadSut: DirectUploadClient = new DirectUploadClient(filesRepositorySut) @@ -79,6 +87,10 @@ describe('Direct Upload', () => { TestConstants.TEST_NEW_DATASET_DTO, testCollectionAlias ) + testDataset7Ids = await createDataset.execute( + TestConstants.TEST_NEW_DATASET_DTO, + testCollectionAlias + ) } catch (error) { throw new Error('Tests beforeAll(): Error while creating test dataset') } @@ -93,6 +105,7 @@ describe('Direct Upload', () => { await deleteUnpublishedDatasetViaApi(testDatset4Ids.numericId) await deleteUnpublishedDatasetViaApi(testDataset5Ids.numericId) await deleteUnpublishedDatasetViaApi(testDataset6Ids.numericId) + await deleteUnpublishedDatasetViaApi(testDataset7Ids.numericId) await deleteCollectionViaApi(testCollectionAlias) }) @@ -664,6 +677,143 @@ describe('Direct Upload', () => { ) }) + describe('Server-driven S3 tagging', () => { + test('should tag the uploaded object with dv-state=temp when the server omits a tagging value', async () => { + const destination: FileUploadDestination = { + ...(await createTestFileUploadDestination(singlepartFile, testDataset7Ids.numericId)), + tagging: undefined + } + + await directUploadSut.uploadFile( + testDataset7Ids.numericId, + singlepartFile, + jest.fn(), + new AbortController(), + destination + ) + + expect(await getObjectTagsFromBucket(destination.urls[0])).toEqual({ 'dv-state': 'temp' }) + }) + + test('should tag the uploaded object with the tagging value the server returned', async () => { + const destination: FileUploadDestination = { + ...(await createTestFileUploadDestination(singlepartFile, testDataset7Ids.numericId)), + tagging: 'dv-state=temp&sdk-integration-test=true' + } + + await directUploadSut.uploadFile( + testDataset7Ids.numericId, + singlepartFile, + jest.fn(), + new AbortController(), + destination + ) + + expect(await getObjectTagsFromBucket(destination.urls[0])).toEqual({ + 'dv-state': 'temp', + 'sdk-integration-test': 'true' + }) + }) + + test('should store the object untagged when the server returns an empty tagging value', async () => { + const destination: FileUploadDestination = { + ...(await createTestFileUploadDestination(singlepartFile, testDataset7Ids.numericId)), + tagging: '' + } + + await directUploadSut.uploadFile( + testDataset7Ids.numericId, + singlepartFile, + jest.fn(), + new AbortController(), + destination + ) + + expect(await getObjectTagsFromBucket(destination.urls[0])).toEqual({}) + }) + }) + + describe('DirectUploadClientConfig', () => { + const allowQueuedPartsToRun = async (): Promise => { + await new Promise((resolve) => setTimeout(resolve, 1500)) + } + + test('should give up on a single-part upload once the configured timeout elapses', async () => { + const destination = await createTestFileUploadDestination( + singlepartFile, + testDataset7Ids.numericId + ) + const config: DirectUploadClientConfig = { fileUploadTimeoutMs: 1 } + const sut = new DirectUploadClient(filesRepositorySut, config) + + await expect( + sut.uploadFile( + testDataset7Ids.numericId, + singlepartFile, + jest.fn(), + new AbortController(), + destination + ) + ).rejects.toThrow( + new FileUploadError( + singlepartFile.name, + testDataset7Ids.numericId, + 'timeout of 1ms exceeded' + ) + ) + }) + + test('should retry a failing part exactly as many times as configured, then abort the upload', async () => { + const partRequestPaths: string[] = [] + const failingPartStore = http.createServer((request, response) => { + partRequestPaths.push(request.url as string) + request.resume() + request.on('end', () => { + response.writeHead(500).end('part upload rejected by test server') + }) + }) + await new Promise((resolve) => failingPartStore.listen(0, '127.0.0.1', resolve)) + const failingPartStoreUrl = `http://127.0.0.1:${ + (failingPartStore.address() as AddressInfo).port + }` + + try { + const serverDestination = await createTestFileUploadDestination( + multipartFile, + testDataset7Ids.numericId + ) + expect(serverDestination.urls.length).toBeGreaterThan(1) + + const destination: FileUploadDestination = { + ...serverDestination, + urls: serverDestination.urls.map( + (_, index) => `${failingPartStoreUrl}/part-${index + 1}` + ), + partSize: 100 + } + + const config: DirectUploadClientConfig = { maxMultipartRetries: 1 } + const sut = new DirectUploadClient(filesRepositorySut, config) + + await expect( + sut.uploadFile( + testDataset7Ids.numericId, + singlepartFile, + jest.fn(), + new AbortController(), + destination + ) + ).rejects.toThrow(FilePartUploadError) + + await allowQueuedPartsToRun() + + expect(partRequestPaths).toEqual(['/part-1', '/part-1']) + } finally { + await new Promise((resolve) => failingPartStore.close(() => resolve())) + } + }) + }) + const createTestFileUploadDestination = async (file: File, testDatasetId: number) => { const filesRepository = new FilesRepository() const destination = await filesRepository.getFileUploadDestination(testDatasetId, file) diff --git a/test/testHelpers/datasets/datasetTreeHelper.ts b/test/testHelpers/datasets/datasetTreeHelper.ts new file mode 100644 index 00000000..837e49dd --- /dev/null +++ b/test/testHelpers/datasets/datasetTreeHelper.ts @@ -0,0 +1,56 @@ +import axios from 'axios' +import { randomUUID } from 'node:crypto' +import { Blob } from '@web-std/file' +import { TestConstants } from '../TestConstants' + +export const DATASET_TREE_ENDPOINT_AVAILABLE_ENV_VAR = 'TEST_DATASET_TREE_ENDPOINT_AVAILABLE' + +const MISSING_ENDPOINT_MESSAGE_FRAGMENT = 'API endpoint does not exist' + +const DATAVERSE_API_REQUEST_HEADERS = { + headers: { 'Content-Type': 'application/json', 'X-Dataverse-Key': process.env.TEST_API_KEY } +} + +export const isDatasetTreeEndpointAvailableViaApi = async (): Promise => { + const response = await axios.get( + `${TestConstants.TEST_API_URL}/datasets/0/versions/:latest/tree`, + { + ...DATAVERSE_API_REQUEST_HEADERS, + validateStatus: () => true + } + ) + const message = typeof response.data?.message === 'string' ? response.data.message : '' + return !message.includes(MISSING_ENDPOINT_MESSAGE_FRAGMENT) +} + +export const datasetTreeEndpointIsAvailable = (): boolean => + process.env[DATASET_TREE_ENDPOINT_AVAILABLE_ENV_VAR] === 'true' + +export const uploadTreeFixtureFileViaApi = async ( + datasetId: number, + label: string, + directoryLabel?: string +): Promise => { + const formData = new FormData() + const content = `tree fixture ${directoryLabel ?? ''}/${label} ${randomUUID()}` + formData.append('file', new Blob([content]), label) + formData.append( + 'jsonData', + JSON.stringify(directoryLabel === undefined ? { label } : { label, directoryLabel }) + ) + + await axios.post(`${TestConstants.TEST_API_URL}/datasets/${datasetId}/add`, formData, { + headers: { + 'Content-Type': 'multipart/form-data', + 'X-Dataverse-Key': process.env.TEST_API_KEY + } + }) +} + +export const createDatasetTreeFixtureViaApi = async (datasetId: number): Promise => { + await uploadTreeFixtureFileViaApi(datasetId, 'root.txt') + await uploadTreeFixtureFileViaApi(datasetId, 'a.txt', 'data') + await uploadTreeFixtureFileViaApi(datasetId, 'b.txt', 'data') + await uploadTreeFixtureFileViaApi(datasetId, 'c.txt', 'data/sub') + await uploadTreeFixtureFileViaApi(datasetId, 'readme.md', 'docs') +} diff --git a/test/testHelpers/files/filesHelper.ts b/test/testHelpers/files/filesHelper.ts index a4fce9ed..11a97b88 100644 --- a/test/testHelpers/files/filesHelper.ts +++ b/test/testHelpers/files/filesHelper.ts @@ -271,3 +271,18 @@ export const singlepartFileExistsInBucket = async (fileUrl: string): Promise> => { + const separator = fileUrl.includes('?') ? '&' : '?' + const response = await axios.get(`${fileUrl}${separator}tagging=`, { + responseType: 'text' + }) + const tags: Record = {} + const tagPattern = /([^<]*)<\/Key>([^<]*)<\/Value><\/Tag>/g + let match = tagPattern.exec(response.data) + while (match !== null) { + tags[match[1]] = match[2] + match = tagPattern.exec(response.data) + } + return tags +} diff --git a/test/testHelpers/html/htmlNormalizer.ts b/test/testHelpers/html/htmlNormalizer.ts new file mode 100644 index 00000000..3e307caf --- /dev/null +++ b/test/testHelpers/html/htmlNormalizer.ts @@ -0,0 +1,145 @@ +const WHITESPACE_SENSITIVE_TAGS = new Set(['pre', 'textarea']) + +const BLOCK_TAGS = new Set([ + 'address', + 'article', + 'aside', + 'blockquote', + 'body', + 'br', + 'div', + 'dd', + 'dl', + 'dt', + 'fieldset', + 'figcaption', + 'figure', + 'footer', + 'form', + 'h1', + 'h2', + 'h3', + 'h4', + 'h5', + 'h6', + 'head', + 'header', + 'hr', + 'html', + 'li', + 'main', + 'nav', + 'ol', + 'p', + 'pre', + 'section', + 'table', + 'tbody', + 'td', + 'tfoot', + 'th', + 'thead', + 'tr', + 'ul' +]) + +const TAG_PATTERN = /^<\s*(\/?)\s*([a-zA-Z][\w:-]*)([\s\S]*?)(\/?)\s*>$/ +const ATTRIBUTE_PATTERN = /([\w:-]+)(?:\s*=\s*("[^"]*"|'[^']*'|[^\s"'>]+))?/g + +interface ParsedTag { + closing: boolean + name: string + selfClosing: boolean +} + +const parseTag = (token: string): ParsedTag | undefined => { + const match = TAG_PATTERN.exec(token) + if (match === null) { + return undefined + } + return { + closing: match[1] === '/', + name: match[2].toLowerCase(), + selfClosing: match[4] === '/' + } +} + +const normalizeTag = (token: string): string => { + const match = TAG_PATTERN.exec(token) + if (match === null) { + return token + } + const [, closing, name, attributeSource, selfClosing] = match + const attributes = Array.from(attributeSource.matchAll(ATTRIBUTE_PATTERN)) + .map(([, attributeName, attributeValue]) => + attributeValue === undefined + ? attributeName.toLowerCase() + : `${attributeName.toLowerCase()}=${normalizeAttributeValue(attributeValue)}` + ) + .sort() + const renderedAttributes = attributes.length === 0 ? '' : ` ${attributes.join(' ')}` + return `<${closing}${name.toLowerCase()}${renderedAttributes}${selfClosing}>` +} + +const normalizeAttributeValue = (value: string): string => { + const unquoted = + (value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'")) + ? value.slice(1, -1) + : value + return `"${unquoted}"` +} + +const isBlockBoundary = (token: string | undefined): boolean => { + if (token === undefined) { + return true + } + const tag = parseTag(token) + return tag !== undefined && BLOCK_TAGS.has(tag.name) +} + +export const normalizeHtml = (html: string): string => { + const tokens = html.split(/(<[^>]*>)/).filter((token) => token !== '') + const normalized: string[] = [] + let whitespaceSensitiveDepth = 0 + + tokens.forEach((token, index) => { + const tag = token.startsWith('<') ? parseTag(token) : undefined + + if (tag !== undefined) { + if (tag.closing && WHITESPACE_SENSITIVE_TAGS.has(tag.name)) { + whitespaceSensitiveDepth = Math.max(0, whitespaceSensitiveDepth - 1) + } + normalized.push(normalizeTag(token)) + if (!tag.closing && !tag.selfClosing && WHITESPACE_SENSITIVE_TAGS.has(tag.name)) { + whitespaceSensitiveDepth += 1 + } + return + } + + if (token.startsWith('<') || whitespaceSensitiveDepth > 0) { + normalized.push(token) + return + } + + const previousToken = tokens[index - 1] + const nextToken = tokens[index + 1] + + if (token.trim() === '') { + if (!isBlockBoundary(previousToken) && !isBlockBoundary(nextToken)) { + normalized.push(' ') + } + return + } + + let text = token.replace(/\s+/g, ' ') + if (isBlockBoundary(previousToken)) { + text = text.replace(/^ /, '') + } + if (isBlockBoundary(nextToken)) { + text = text.replace(/ $/, '') + } + normalized.push(text) + }) + + return normalized.join('') +} diff --git a/test/unit/datasets/IterateDatasetTreeNode.test.ts b/test/unit/datasets/IterateDatasetTreeNode.test.ts new file mode 100644 index 00000000..e348d303 --- /dev/null +++ b/test/unit/datasets/IterateDatasetTreeNode.test.ts @@ -0,0 +1,93 @@ +import { IterateDatasetTreeNode } from '../../../src/datasets/domain/useCases/IterateDatasetTreeNode' +import { + IDatasetsRepository, + ListDatasetTreeNodeParams +} from '../../../src/datasets/domain/repositories/IDatasetsRepository' +import { + FileTreeInclude, + FileTreeOrder, + FileTreePage +} from '../../../src/datasets/domain/models/FileTreePage' +import { FileTreeNodeType } from '../../../src/datasets/domain/models/FileTreeNode' + +const page = (overrides: Partial): FileTreePage => ({ + path: '', + items: [], + nextCursor: null, + limit: 100, + order: FileTreeOrder.NAME_AZ, + include: FileTreeInclude.ALL, + ...overrides +}) + +describe('IterateDatasetTreeNode (unit)', () => { + test('iterates a single page', async () => { + const file = { + type: FileTreeNodeType.FILE, + id: 1, + name: 'a.txt', + path: 'a.txt', + size: 100, + downloadUrl: '/api/access/datafile/1' + } + const repo: IDatasetsRepository = {} as IDatasetsRepository + repo.listDatasetTreeNode = jest.fn().mockResolvedValue(page({ items: [file] })) + + const sut = new IterateDatasetTreeNode(repo) + const collected: (typeof file)[] = [] + for await (const node of sut.execute({ datasetId: 1 })) { + collected.push(node as typeof file) + } + expect(collected.map((n) => n.id)).toEqual([1]) + }) + + test('walks the cursor chain until exhausted', async () => { + const fileFor = (id: number) => ({ + type: FileTreeNodeType.FILE, + id, + name: `f${id}.txt`, + path: `f${id}.txt`, + size: 100, + downloadUrl: `/api/access/datafile/${id}` + }) + const pages: FileTreePage[] = [ + page({ items: [fileFor(1), fileFor(2)], nextCursor: 'c2' }), + page({ items: [fileFor(3)], nextCursor: 'c3' }), + page({ items: [fileFor(4)] }) + ] + const calls: ListDatasetTreeNodeParams[] = [] + const repo: IDatasetsRepository = {} as IDatasetsRepository + repo.listDatasetTreeNode = jest.fn().mockImplementation((params: ListDatasetTreeNodeParams) => { + calls.push(params) + const idx = calls.length - 1 + return Promise.resolve(pages[idx]) + }) + + const sut = new IterateDatasetTreeNode(repo) + const ids: number[] = [] + for await (const node of sut.execute({ datasetId: 1 })) { + if (node.type === FileTreeNodeType.FILE) { + ids.push(node.id) + } + } + expect(ids).toEqual([1, 2, 3, 4]) + expect(calls.map((c) => c.cursor)).toEqual([undefined, 'c2', 'c3']) + }) + + test('throws instead of looping forever when the cursor does not advance', async () => { + const repo: IDatasetsRepository = {} as IDatasetsRepository + repo.listDatasetTreeNode = jest + .fn() + .mockResolvedValueOnce(page({ nextCursor: 'stuck' })) + .mockResolvedValue(page({ nextCursor: 'stuck' })) + + const sut = new IterateDatasetTreeNode(repo) + const iterate = async () => { + for await (const node of sut.execute({ datasetId: 1 })) { + void node + } + } + await expect(iterate()).rejects.toThrow('cursor did not advance') + expect(repo.listDatasetTreeNode).toHaveBeenCalledTimes(2) + }) +}) diff --git a/test/unit/datasets/ListDatasetTreeNode.test.ts b/test/unit/datasets/ListDatasetTreeNode.test.ts new file mode 100644 index 00000000..5b7dc836 --- /dev/null +++ b/test/unit/datasets/ListDatasetTreeNode.test.ts @@ -0,0 +1,61 @@ +import { ListDatasetTreeNode } from '../../../src/datasets/domain/useCases/ListDatasetTreeNode' +import { IDatasetsRepository } from '../../../src/datasets/domain/repositories/IDatasetsRepository' +import { + FileTreeInclude, + FileTreeOrder, + FileTreePage +} from '../../../src/datasets/domain/models/FileTreePage' +import { FileTreeNodeType } from '../../../src/datasets/domain/models/FileTreeNode' +import { ReadError } from '../../../src/core/domain/repositories/ReadError' + +describe('ListDatasetTreeNode (unit)', () => { + const testPage: FileTreePage = { + path: 'data', + items: [ + { + type: FileTreeNodeType.FOLDER, + name: 'sub', + path: 'data/sub', + counts: { + files: 1, + folders: 0, + bytes: 1024, + restricted: 0, + embargoed: 0, + retentionExpired: 0 + } + }, + { + type: FileTreeNodeType.FILE, + id: 7, + name: 'a.txt', + path: 'data/a.txt', + size: 1024, + downloadUrl: '/api/access/datafile/7' + } + ], + nextCursor: null, + limit: 100, + order: FileTreeOrder.NAME_AZ, + include: FileTreeInclude.ALL, + approximateCount: 2 + } + + test('returns the page produced by the repository', async () => { + const repo: IDatasetsRepository = {} as IDatasetsRepository + repo.listDatasetTreeNode = jest.fn().mockResolvedValue(testPage) + + const sut = new ListDatasetTreeNode(repo) + const result = await sut.execute({ datasetId: 1, path: 'data' }) + expect(result).toEqual(testPage) + expect(repo.listDatasetTreeNode).toHaveBeenCalledWith({ datasetId: 1, path: 'data' }) + }) + + test('propagates ReadError', async () => { + const repo: IDatasetsRepository = {} as IDatasetsRepository + repo.listDatasetTreeNode = jest.fn().mockRejectedValue(new ReadError('[400] bad cursor')) + + const sut = new ListDatasetTreeNode(repo) + await expect(sut.execute({ datasetId: 1 })).rejects.toThrow(ReadError) + }) +}) diff --git a/test/unit/datasets/fileTreeTransformers.test.ts b/test/unit/datasets/fileTreeTransformers.test.ts new file mode 100644 index 00000000..fbcbfd4a --- /dev/null +++ b/test/unit/datasets/fileTreeTransformers.test.ts @@ -0,0 +1,127 @@ +import { AxiosResponse } from 'axios' +import { transformTreeResponseToFileTreePage } from '../../../src/datasets/infra/repositories/transformers/fileTreeTransformers' +import { FileTreeInclude, FileTreeOrder } from '../../../src/datasets/domain/models/FileTreePage' +import { + FileTreeNodeType, + isFileTreeFolderNode, + isFileTreeFileNode +} from '../../../src/datasets/domain/models/FileTreeNode' + +const buildResponse = (data: unknown): AxiosResponse => + ({ + data: { data }, + status: 200, + statusText: 'OK', + headers: {}, + config: {} as never + } as AxiosResponse) + +describe('transformTreeResponseToFileTreePage', () => { + test('maps folder and file payloads to typed FileTreeNodes', () => { + const response = buildResponse({ + path: 'data', + items: [ + { + type: 'folder', + name: 'raw', + path: 'data/raw', + counts: { + files: 3, + folders: 0, + bytes: 4096, + restricted: 0, + embargoed: 0, + retentionExpired: 0 + } + }, + { + type: 'file', + id: 42, + name: 'a.csv', + path: 'data/a.csv', + size: 1024, + contentType: 'text/csv', + access: 'public', + checksum: { type: 'MD5', value: 'abc' }, + downloadUrl: '/api/access/datafile/42' + } + ], + nextCursor: 'eyJ', + limit: 100, + order: 'NameAZ', + include: 'all', + approximateCount: 2 + }) + + const page = transformTreeResponseToFileTreePage(response) + expect(page.path).toBe('data') + expect(page.items).toHaveLength(2) + expect(page.nextCursor).toBe('eyJ') + expect(page.limit).toBe(100) + expect(page.order).toBe(FileTreeOrder.NAME_AZ) + expect(page.include).toBe(FileTreeInclude.ALL) + expect(page.approximateCount).toBe(2) + + const folder = page.items[0] + if (!isFileTreeFolderNode(folder)) { + throw new Error('expected folder') + } + expect(folder.name).toBe('raw') + expect(folder.counts).toEqual({ + files: 3, + folders: 0, + bytes: 4096, + restricted: 0, + embargoed: 0, + retentionExpired: 0 + }) + + const file = page.items[1] + if (!isFileTreeFileNode(file)) { + throw new Error('expected file') + } + expect(file.id).toBe(42) + expect(file.size).toBe(1024) + expect(file.access).toBe('public') + expect(file.checksum).toEqual({ type: 'MD5', value: 'abc' }) + }) + + test('falls back to defaults when order/include are unrecognized', () => { + const response = buildResponse({ + path: '', + items: [], + nextCursor: null, + limit: 100, + order: 'WhateverElse', + include: 'something' + }) + const page = transformTreeResponseToFileTreePage(response) + expect(page.order).toBe(FileTreeOrder.NAME_AZ) + expect(page.include).toBe(FileTreeInclude.ALL) + }) + + test('parses non-default order/include echoed by the server', () => { + const response = buildResponse({ + path: 'docs', + items: [ + { + type: 'file', + id: 1, + name: 'README.md', + path: 'docs/README.md', + size: 200, + downloadUrl: '/api/access/datafile/1' + } + ], + nextCursor: null, + limit: 100, + order: 'NameZA', + include: 'files' + }) + + const page = transformTreeResponseToFileTreePage(response) + expect(page.order).toBe(FileTreeOrder.NAME_ZA) + expect(page.include).toBe(FileTreeInclude.FILES) + expect(page.items[0].type).toBe(FileTreeNodeType.FILE) + }) +}) diff --git a/test/unit/files/DirectUploadClient.test.ts b/test/unit/files/DirectUploadClient.test.ts index 38f1921a..1560c135 100644 --- a/test/unit/files/DirectUploadClient.test.ts +++ b/test/unit/files/DirectUploadClient.test.ts @@ -19,6 +19,17 @@ import { TestConstants } from '../../testHelpers/TestConstants' import { DataverseApiAuthMechanism } from '../../../src/core/infra/repositories/ApiConfig' import { FileUploadDestination } from '../../../src/files/domain/models/FileUploadDestination' +describe('constructor', () => { + test('honors a legacy numeric second argument as maxMultipartRetries', () => { + const filesRepositoryStub: IFilesRepository = {} as IFilesRepository + const sut = new DirectUploadClient( + filesRepositoryStub, + 1 as unknown as ConstructorParameters[1] + ) + expect((sut as unknown as { maxMultipartRetries: number }).maxMultipartRetries).toBe(1) + }) +}) + describe('uploadFile', () => { beforeEach(() => { ApiConfig.init( @@ -87,6 +98,110 @@ describe('uploadFile', () => { expect(actual).toEqual(testDestination.storageId) }) + + test('should include S3 tagging header when upload destination provides tagging', async () => { + const filesRepositoryStub: IFilesRepository = {} as IFilesRepository + const testDestination: FileUploadDestination = { + ...createSingleFileUploadDestinationModel(), + tagging: 'dv-state=temp' + } + filesRepositoryStub.getFileUploadDestination = jest.fn().mockResolvedValue(testDestination) + + const axiosPutSpy = jest.spyOn(axios, 'put').mockResolvedValue(undefined) + + const sut = new DirectUploadClient(filesRepositoryStub) + + const progressMock = jest.fn() + const abortController = new AbortController() + + await sut.uploadFile(1, testFile, progressMock, abortController) + + expect(axiosPutSpy).toHaveBeenCalledWith( + testDestination.urls[0], + expect.anything(), + expect.objectContaining({ + headers: expect.objectContaining({ + 'x-amz-tagging': 'dv-state=temp' + }) + }) + ) + }) + + test('should default to dv-state=temp tagging when upload destination omits tagging', async () => { + const filesRepositoryStub: IFilesRepository = {} as IFilesRepository + const testDestination: FileUploadDestination = createSingleFileUploadDestinationModel() + filesRepositoryStub.getFileUploadDestination = jest.fn().mockResolvedValue(testDestination) + + const axiosPutSpy = jest.spyOn(axios, 'put').mockResolvedValue(undefined) + + const sut = new DirectUploadClient(filesRepositoryStub) + + const progressMock = jest.fn() + const abortController = new AbortController() + + await sut.uploadFile(1, testFile, progressMock, abortController) + + expect(axiosPutSpy).toHaveBeenCalledWith( + testDestination.urls[0], + expect.anything(), + expect.objectContaining({ + headers: expect.objectContaining({ + 'x-amz-tagging': 'dv-state=temp' + }) + }) + ) + }) + + test('should omit the S3 tagging header when upload destination explicitly returns empty tagging', async () => { + const filesRepositoryStub: IFilesRepository = {} as IFilesRepository + const testDestination: FileUploadDestination = { + ...createSingleFileUploadDestinationModel(), + tagging: '' + } + filesRepositoryStub.getFileUploadDestination = jest.fn().mockResolvedValue(testDestination) + + const axiosPutSpy = jest.spyOn(axios, 'put').mockResolvedValue(undefined) + + const sut = new DirectUploadClient(filesRepositoryStub) + + const progressMock = jest.fn() + const abortController = new AbortController() + + await sut.uploadFile(1, testFile, progressMock, abortController) + + expect(axiosPutSpy).toHaveBeenCalledWith( + testDestination.urls[0], + expect.anything(), + expect.objectContaining({ + headers: expect.not.objectContaining({ + 'x-amz-tagging': expect.anything() + }) + }) + ) + }) + + test('should use configured file upload timeout', async () => { + const filesRepositoryStub: IFilesRepository = {} as IFilesRepository + const testDestination: FileUploadDestination = createSingleFileUploadDestinationModel() + filesRepositoryStub.getFileUploadDestination = jest.fn().mockResolvedValue(testDestination) + + const axiosPutSpy = jest.spyOn(axios, 'put').mockResolvedValue(undefined) + + const sut = new DirectUploadClient(filesRepositoryStub, { fileUploadTimeoutMs: 30_000 }) + + const progressMock = jest.fn() + const abortController = new AbortController() + + await sut.uploadFile(1, testFile, progressMock, abortController) + + expect(axiosPutSpy).toHaveBeenCalledWith( + testDestination.urls[0], + expect.anything(), + expect.objectContaining({ + timeout: 30_000 + }) + ) + }) }) describe('Multiple parts file', () => { @@ -113,7 +228,7 @@ describe('uploadFile', () => { jest.spyOn(axios, 'delete').mockResolvedValue(undefined) jest.spyOn(axios, 'put').mockRejectedValue('error') - const sut = new DirectUploadClient(filesRepositoryStub, 1) + const sut = new DirectUploadClient(filesRepositoryStub, { maxMultipartRetries: 1 }) const progressMock = jest.fn() const abortController = new AbortController() @@ -143,7 +258,7 @@ describe('uploadFile', () => { const progressMock = jest.fn() const abortController = new AbortController() - const sut = new DirectUploadClient(filesRepositoryStub, 1) + const sut = new DirectUploadClient(filesRepositoryStub, { maxMultipartRetries: 1 }) await expect(sut.uploadFile(1, testFile, progressMock, abortController)).rejects.toThrow( MultipartAbortError @@ -165,7 +280,7 @@ describe('uploadFile', () => { const progressMock = jest.fn() const abortController = new AbortController() - const sut = new DirectUploadClient(filesRepositoryStub, 1) + const sut = new DirectUploadClient(filesRepositoryStub, { maxMultipartRetries: 1 }) await expect(sut.uploadFile(1, testFile, progressMock, abortController)).rejects.toThrow( MultipartCompletionError ) @@ -187,7 +302,7 @@ describe('uploadFile', () => { .mockResolvedValueOnce(successfulPartResponse) .mockResolvedValueOnce(undefined) - const sut = new DirectUploadClient(filesRepositoryStub, 1) + const sut = new DirectUploadClient(filesRepositoryStub, { maxMultipartRetries: 1 }) const progressMock = jest.fn() const abortController = new AbortController() diff --git a/test/unit/files/FilesRepository.test.ts b/test/unit/files/FilesRepository.test.ts index aa33e7e6..5896e72d 100644 --- a/test/unit/files/FilesRepository.test.ts +++ b/test/unit/files/FilesRepository.test.ts @@ -149,6 +149,48 @@ describe('FilesRepository', () => { expect(actual).toEqual(testMultipleFileUploadDestination) }) + test('should return destination with tagging when single response includes tagging', async () => { + const tagging = 'dv-state=temp' + jest.spyOn(axios, 'get').mockResolvedValue({ + data: { + status: 'OK', + data: { + ...createSingleFileUploadDestinationPayload(), + tagging + } + } + }) + jest.spyOn(fs, 'statSync').mockReturnValue({ size: testFileSize } as fs.Stats) + + const actual = await sut.getFileUploadDestination(testDatasetId, singlepartFile) + + expect(actual).toEqual({ + ...testSingleFileUploadDestination, + tagging + }) + }) + + test('should return destination with tagging when multipart response includes tagging', async () => { + const tagging = 'dv-state=temp' + jest.spyOn(axios, 'get').mockResolvedValue({ + data: { + status: 'OK', + data: { + ...createMultipartFileUploadDestinationPayload(), + tagging + } + } + }) + jest.spyOn(fs, 'statSync').mockReturnValue({ size: testFileSize } as fs.Stats) + + const actual = await sut.getFileUploadDestination(testDatasetId, multipartFile) + + expect(actual).toEqual({ + ...testMultipleFileUploadDestination, + tagging + }) + }) + test('should return error on repository read error', async () => { jest.spyOn(axios, 'get').mockRejectedValue(TestConstants.TEST_ERROR_RESPONSE) jest.spyOn(fs, 'statSync').mockReturnValue({ size: testFileSize } as fs.Stats) diff --git a/test/unit/testHelpers/htmlNormalizer.test.ts b/test/unit/testHelpers/htmlNormalizer.test.ts new file mode 100644 index 00000000..5b877662 --- /dev/null +++ b/test/unit/testHelpers/htmlNormalizer.test.ts @@ -0,0 +1,77 @@ +import { normalizeHtml } from '../../testHelpers/html/htmlNormalizer' +import { + CONTENT_FIELD_WITH_ALL_TAGS, + EXPECTED_CONTENT_FIELD_WITH_ALL_TAGS +} from '../../testHelpers/collections/collectionHelper' + +describe('normalizeHtml', () => { + describe('differences the server may introduce', () => { + test('should ignore the order of attributes', () => { + expect( + normalizeHtml('t') + ).toEqual( + normalizeHtml('t') + ) + }) + + test('should ignore indentation introduced between block elements', () => { + expect(normalizeHtml('
  • Item

')).toEqual( + normalizeHtml('
    \n
  • \n

    Item

    \n
  • \n
') + ) + }) + + test('should ignore indentation around the content of a block element', () => { + expect(normalizeHtml('

Item

')).toEqual(normalizeHtml('

\n Item\n

')) + }) + + test('should ignore the case of tag and attribute names', () => { + expect(normalizeHtml('

t

')).toEqual(normalizeHtml('

t

')) + }) + + test('should treat the sent and pretty-printed forms of the featured item fixture as equal', () => { + expect(normalizeHtml(CONTENT_FIELD_WITH_ALL_TAGS)).toEqual( + normalizeHtml(EXPECTED_CONTENT_FIELD_WITH_ALL_TAGS) + ) + }) + }) + + describe('differences that must still be detected', () => { + test('should not ignore differing text content', () => { + expect(normalizeHtml('

Item

')).not.toEqual(normalizeHtml('

Other

')) + }) + + test('should not ignore differing attribute values', () => { + expect(normalizeHtml('t')).not.toEqual( + normalizeHtml('t') + ) + }) + + test('should not ignore a dropped attribute', () => { + expect(normalizeHtml('t')).not.toEqual( + normalizeHtml('t') + ) + }) + + test('should not ignore differing structure', () => { + expect(normalizeHtml('
  • a
  • b
')).not.toEqual( + normalizeHtml('
  • a
') + ) + }) + + test('should not ignore a changed tag', () => { + expect(normalizeHtml('t')).not.toEqual(normalizeHtml('t')) + }) + + test('should preserve whitespace inside a preformatted block', () => { + expect(normalizeHtml('
  indented\n  lines
')).not.toEqual( + normalizeHtml('
indented lines
') + ) + }) + + test('should preserve whitespace that separates inline elements', () => { + expect(normalizeHtml('

a b

')).not.toEqual( + normalizeHtml('

ab

') + ) + }) + }) +})