We are new to Dataverse. And some related configurations:
- using OSN (open storage network) s3 bucket as our backend storage
- direct upload is not supported currently as the CORS policy was not implemented on the bucket from OSN
Recently, we upgraded to 6.11 from 6.10.1.
And noticed two issues:
- One thing we noticed was that, when trying to publishing a dataset which includes a file ~160MB, we got the error message:
Once the file is removed from the dataset, publishing was fine.
- Another issue was with direct-download disabled, when downloading this file (from the draft since it failed to publish ), the downloaded file is truncated at ~8MB. However, with direct-download enabled, the downloaded file is complete.
What steps does it take to reproduce the issue?
On our test dataverse instance, with the same S3 backend storage (Open Storage Network) and upload-redirect=false:
-
Upload a ~160 MB file to a draft dataset through the web UI. Dataverse writes it to S3 as a multipart object — ETag ends in -20, PartsCount: 20, part 1 is ~ 8MB
-
Publish the dataset. Publishing fails, the dataset is locked with FileValidationFailed and and /api/admin/validate/dataset/files/{id} returns:
Checksum Validation Unexpected Exception:
software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 412,
Request ID: tx00000dc10857e1119c449-…) (SDK Attempt Count: 1)
- Outside Dataverse, download that file and re-upload the identical bytes to the same key with a single PUT (aws s3api put-object), so the object is no longer multipart. Nothing about the file changes — the MD5 is identical before and after, and it still matches datafile.checksumvalue.
- Publish again. It succeeds.
The only variable between steps 2 and 4 is whether the file carries multipart part boundaries. And it looks like the reason it fails is that the ETag returned for a partNumber request is the part's ETag, not the object's. The SDK carries the first part's ETag forward as If-Match on the next part, and it never matches. Reproduced by hand against the storage, with no Dataverse involved:
$ aws s3api head-object --bucket B --key K --query ETag --output text
"44c9c29827fb957f4e20c100604564dc-20" # object
$ aws s3api get-object --bucket B --key K --part-number 1 /dev/null \
--query '[ContentLength,PartsCount,ETag]' --output text
8388608 20 "21253d656a4ae7fcd84024696eefb724" # part 1
$ aws s3api head-object --bucket B --key K --part-number 2 \
--if-match '"21253d656a4ae7fcd84024696eefb724"'
An error occurred (412) when calling the HeadObject operation: Precondition Failed
Additionally, datasets published before we upgraded to 6.11 contain files of similar size that are single-part objects. For example, a 116 MB file uploaded in April returns:
$ aws s3api head-object --bucket B --key K --query '[ETag,ContentLength]' --output text
"dd57f7377b99dfaed6926f42b9f80e74" 116061051
$ aws s3api get-object --bucket B --key K --part-number 1 /dev/null \
--query '[ContentLength,PartsCount]' --output text
116061051 None
Possible cause:
And one thing we noticed changed from the v6.10 to v6.11 was the addition of the following in the file S3AccessIO.java:
// Always enable multipart upload. It is only used when necessary
s3CB.multipartEnabled(true);
And the doc mentioned that
Enables automatic conversion of GET, PUT and COPY methods to their equivalent multipart operation.
So enabling it appears to have enabled multipart download as well, which matches the truncated file we experienced.
We are in the process of getting CORS enabled to do direct uploading, it may be able to prevent this error.
-
When does this issue occur?
Publishing datasets
Downloading datasets
-
Which page(s) does it occurs on?
Dataset draft publishing page
-
What happens?
This seems to have something to do with the multipart upload/download
-
To whom does it occur (all users, curators, superusers)?
curators, or all users if downloading
Which version of Dataverse are you using?
v6.11
Screenshots:
No matter the issue, screenshots are always welcome.
To add a screenshot, please use one of the following formats and/or methods described here:
We are new to Dataverse. And some related configurations:
Recently, we upgraded to 6.11 from 6.10.1.
And noticed two issues:
Once the file is removed from the dataset, publishing was fine.
What steps does it take to reproduce the issue?
On our test dataverse instance, with the same S3 backend storage (Open Storage Network) and upload-redirect=false:
Upload a ~160 MB file to a draft dataset through the web UI. Dataverse writes it to S3 as a multipart object — ETag ends in -20, PartsCount: 20, part 1 is ~ 8MB
Publish the dataset. Publishing fails, the dataset is locked with FileValidationFailed and and /api/admin/validate/dataset/files/{id} returns:
The only variable between steps 2 and 4 is whether the file carries multipart part boundaries. And it looks like the reason it fails is that the ETag returned for a partNumber request is the part's ETag, not the object's. The SDK carries the first part's ETag forward as If-Match on the next part, and it never matches. Reproduced by hand against the storage, with no Dataverse involved:
Additionally, datasets published before we upgraded to 6.11 contain files of similar size that are single-part objects. For example, a 116 MB file uploaded in April returns:
Possible cause:
And one thing we noticed changed from the v6.10 to v6.11 was the addition of the following in the file S3AccessIO.java:
And the doc mentioned that
We are in the process of getting CORS enabled to do direct uploading, it may be able to prevent this error.
When does this issue occur?
Publishing datasets
Downloading datasets
Which page(s) does it occurs on?
Dataset draft publishing page
What happens?
This seems to have something to do with the multipart upload/download
To whom does it occur (all users, curators, superusers)?
curators, or all users if downloading
Which version of Dataverse are you using?
v6.11
Screenshots:
No matter the issue, screenshots are always welcome.
To add a screenshot, please use one of the following formats and/or methods described here: