Unable to install CUDA with build-conda-env

[rafaelrpinto]

Trying ./data-science-stack build-conda-env on a fresh ubuntu 20.04 gcp vm causes an error updating the repo:

###NV### Tue Aug 31 07:54:08 UTC 2021 #### START Installing CUDA
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   190  100   190    0     0   3958      0 --:--:-- --:--:-- --:--:--  3958
Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.ua1MwW1AJb/gpg.1.sh --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
gpg: requesting key from 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub'
gpg: key F60F4B3D7FA2AF80: "cudatools <cudatools@nvidia.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Hit:2 http://europe-west4.gce.archive.ubuntu.com/ubuntu focal InRelease
Hit:3 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease
Hit:4 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu focal InRelease
Hit:5 http://europe-west4.gce.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease
Hit:7 https://download.docker.com/linux/ubuntu focal InRelease
Hit:8 http://europe-west4.gce.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:9 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease
Hit:10 http://security.ubuntu.com/ubuntu focal-security InRelease
Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Get:11 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release [697 B]
Get:12 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release.gpg [836 B]
Ign:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Ign:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Ign:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Err:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
  404  Not Found [IP: 152.199.20.126 443]
Fetched 836 B in 2s (492 B/s)
Reading package lists...
E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/by-hash/SHA256/751939d95516afc289908a19e447f0acc1506367f72ed356431a2b1a469cc8ca  404  Not Found [IP: 152.199.20.126 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.

[rafaelrpinto]

Fixed by preinstalling CUDA following these instructions:

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=deb_local

[rafaelrpinto]

Dummy question: Do we really need to install CUDA on the OS too as we already install it on conda?

- nvidia::cudatoolkit=11.2.72