Skip to content

Error during CRC check when uploading an open + unlinked file #34987

Description

@rolftimmermans

Context

The google-cloud-storage library attempts to reopen any File that is attempted to be uploaded. See this line:
https://github.com/shubhangi-google/google-cloud-ruby/blob/042468dcccc83a802ae4993e76ebea203827d3cb/google-cloud-storage/lib/google/cloud/storage/file/verifier.rb#L84

This is a problem when the file no longer exists.

When would that actually happen? The puma web server notably unlinks files immediately after storing, while keeping the file descriptor open:
https://github.com/puma/puma/blob/v8.0.2/lib/puma/client.rb#L507-L511

Although this trips people up, it is very much intentional and in fact recommended by the Ruby documentation.
puma/puma#3597
aws/aws-sdk-ruby#3163
aws/aws-sdk-ruby#3164
https://docs.ruby-lang.org/en/3.4/Tempfile.html#class-Tempfile-label-Unlink+after+creation

It would be tremendously helpful if the google-cloud-storage client would not assume that a File or Tempfile is still linked when it is being uploaded.

Environment details

  • OS: Alpine Linux, Debian Linux
  • Ruby version: 3.4
  • Gem name and version: google-cloud-storage (1.62.0)

Steps to reproduce

  1. Use Puma web server
  2. Upload files above configured memory limit
  3. Observe that google-cloud-storage fails to compute digest

Code example

This is a minimal example that mimics Puma's behaviour but it is a much simpler reproduction.

source "https://rubygems.org"

gem "google-cloud-storage", "1.62.0"
require "google/cloud/storage"
require "tempfile"

storage = Google::Cloud::Storage.anonymous
bucket = storage.bucket("unused", skip_lookup: true)

Tempfile.create("gcs-repro", binmode: true) do |file|
  file.write("still readable")
  file.rewind

  path = file.path
  File.unlink(path)

  puts "Path exists: #{File.exist?(path)}"
  puts "Contents: #{file.read.inspect}"
  file.rewind

  bucket.create_file(file, "test")
end

Full backtrace

3.4-slim: Pulling from library/ruby
b16cb8fde57a: Pulling fs layer
ead741ce7b27: Pulling fs layer
59f54fbcd984: Pulling fs layer
b160435c969d: Pulling fs layer
52a2f7711503: Pulling fs layer
b16cb8fde57a: Download complete
ead741ce7b27: Download complete
b160435c969d: Download complete
4371fa0b2e33: Download complete
79dd7f77afe3: Download complete
52a2f7711503: Download complete
59f54fbcd984: Download complete
59f54fbcd984: Pull complete
ead741ce7b27: Pull complete
b160435c969d: Pull complete
b16cb8fde57a: Pull complete
52a2f7711503: Pull complete
Digest: sha256:614edae6a80eb2a7cf1984f03a6d814f523a48355e8f96deb5ddd25faa86353e
Status: Downloaded newer image for ruby:3.4-slim
/usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'File#initialize': No such file or directory @ rb_sysopen - /tmp/gcs-repro20260721-29-v1priq (Errno::ENOENT)
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'IO.open'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'Google::Cloud::Storage::File::Verifier._digest_for'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:59:in 'Google::Cloud::Storage::File::Verifier.crc32c_for'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:3507:in 'Google::Cloud::Storage::Bucket#crc32c_for'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:2007:in 'Google::Cloud::Storage::Bucket#create_file'
	from -e:1:in 'block in <main>'
	from /usr/local/lib/ruby/3.4.0/tempfile.rb:576:in 'Tempfile.create_with_filename'
	from /usr/local/lib/ruby/3.4.0/tempfile.rb:562:in 'Tempfile.create'
	from -e:1:in '<main>'
path exists: false
contents: still readable

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions