tar source plugin fails to download long urls

[harrysarson]

The following element file:

kind: import

sources:
- kind: tar
  # works
  # url: curl:download/curl-8.14.1.tar.gz#000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
  # fails
  url: curl:download/curl-8.14.1.tar.gz#0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
  ref: 6766ada7101d292b42b8b15681120acd68effa4a9660935853cf6d61f0d984d4

(with the curl alias defined in the project.conf as curl: https://curl.se/)

Fails to download:

$ bst source fetch harry-curl.bst

...

10:40:55 00:00:00 1525c594    fetch:harry-curl.bst BUG     Fetch

    An unhandled exception occured:
    
    Traceback (most recent call last):
      File "<...>/buildstream/src/buildstream/_scheduler/jobs/job.py", line 350, in child_action
        result = self.child_process()  # pylint: disable=assignment-from-no-return
                 ^^^^^^^^^^^^^^^^^^^^
      File "<...>/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 81, in child_process
        return self._action_cb(self._element)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<...>/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 75, in _fetch_not_original
        element._fetch(fetch_original=False)
      File "<...>/buildstream/src/buildstream/element.py", line 2252, in _fetch
        self.__sources.fetch()
      File "<...>/buildstream/src/buildstream/_elementsources.py", line 221, in fetch
        self.fetch_sources()
      File "<...>/buildstream/src/buildstream/_elementsources.py", line 250, in fetch_sources
        self._fetch_source(source)
      File "<...>/buildstream/src/buildstream/_elementsources.py", line 431, in _fetch_source
        source._fetch()
      File "<...>/buildstream/src/buildstream/source.py", line 1421, in _fetch
        self.__do_fetch()
      File "<...>/buildstream/src/buildstream/source.py", line 1927, in __do_fetch
        new_source.fetch(**kwargs)
      File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 341, in fetch
        sha256 = self._ensure_mirror(
                 ^^^^^^^^^^^^^^^^^^^^
      File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 399, in _ensure_mirror
        os.makedirs(self._mirror_dir, exist_ok=True)
      File "<frozen os>", line 225, in makedirs
    OSError: [Errno 36] File name too long: '<...>/.cache/buildstream/sources/tar/curl_download_curl_8_14_1_tar_gz_0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'


Fetch failure on element: harry-curl.bst

I believe buildstream computes the mirror directory here

buildstream/src/buildstream/downloadablefilesource.py

Line 277 in 6558f96

self._mirror_dir = os.path.join(self.get_mirror_directory(), utils.url_directory_name(self.original_url))

using utils.url_directory_name which can generate filenames longer than 255 characters.

This URL is a ridiculous one but in an private project I have a file I need to download (with a very long url) where I cannot control the url I download from.

[juergbi]

We may be able to use directory file descriptors to mitigate this issue.

[nanonyme]

I think certain other plugins store files by hash to avoid this kind of issues.