Skip to content

[AN-503] Update to Dataproc 2.2.X #4839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LizBaldo merged 39 commits into from
May 6, 2025
Merged

[AN-503] Update to Dataproc 2.2.X #4839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
65c5444
Generate new dataproc image with opsagent installed for logging
LizBaldo Apr 16, 2025
dea02ec
update docker compose
LizBaldo Apr 23, 2025
7e53309
point to new dataproc 2.2 image
LizBaldo Apr 23, 2025
8e07d19
set up loggin receivers
LizBaldo Apr 23, 2025
5a98acd
empty ops agent config file for debugging purposes
LizBaldo Apr 24, 2025
5873b01
arg typo!
LizBaldo Apr 24, 2025
764709f
fix docker compose install and ops agent config format
LizBaldo Apr 24, 2025
de472f1
point to new dataproc with docker compose installed
LizBaldo Apr 24, 2025
e6083a1
Merge branch 'develop' into AN-503-update-to-dataproc-2.2.x
LizBaldo Apr 24, 2025
df20494
try to docker restart to get around proxy issues
LizBaldo Apr 24, 2025
cda8ee6
try to restart docker just before pulling
LizBaldo Apr 24, 2025
f44e998
try gcr mirror on proxy image
LizBaldo Apr 25, 2025
5d8581e
comment out part of init action that do not work so I can ssh into th…
LizBaldo Apr 29, 2025
8882b07
fix typo
LizBaldo Apr 29, 2025
63b6b78
bypass nb extension issue for now
LizBaldo Apr 29, 2025
4f153c3
continue bypassing network connectivuty issues
LizBaldo Apr 30, 2025
d069eb6
match ipc mode with previous settings
LizBaldo Apr 30, 2025
52bfe37
Merge branch 'develop' into AN-503-update-to-dataproc-2.2.x
LizBaldo Apr 30, 2025
ef8ac93
start from blank init action
LizBaldo Apr 30, 2025
d759fee
cache new hail and aou images compatible with dataproc 2.2.x
LizBaldo May 1, 2025
1efedaa
specify InternalIpOnly flag to False for dataproc
LizBaldo May 1, 2025
a04ec7e
Merge branch 'develop' into AN-503-update-to-dataproc-2.2.x
LizBaldo May 1, 2025
64b6d9b
update with cached hail 234 image
LizBaldo May 2, 2025
6a06cb5
Merge branch 'AN-503-update-to-dataproc-2.2.x' of github.com:DataBios…
LizBaldo May 2, 2025
a998322
point to new gce image that has the latest aou as well
LizBaldo May 2, 2025
591321c
cleanup
LizBaldo May 2, 2025
41f7e89
update default gatk image version used
LizBaldo May 5, 2025
4b01b25
cache both aou 2.2.13 and 2.2.16 to help them transition smoothly
LizBaldo May 5, 2025
67f9f48
use dataproc 2.1 for aou and 2.2 for terra
LizBaldo May 5, 2025
9741e77
we also need to point aou to the legacy init script
LizBaldo May 5, 2025
ae01725
easier to also set up the legacy init scrupt for the gce cluster config
LizBaldo May 5, 2025
47104ef
fix legacy init script name typo
LizBaldo May 5, 2025
c239e54
addressing comment from Lucy and adding link to aou prod version
LizBaldo May 5, 2025
32fe664
remove old aou image from dataproc 2.2
LizBaldo May 6, 2025
89bfb68
remove new aou image from dataproc 2.1
LizBaldo May 6, 2025
e589062
test with smaller dataproc 2 image
LizBaldo May 6, 2025
7009a1e
forgot to point to dataproc 2.1 version
LizBaldo May 6, 2025
f6d5f96
cleanup, time to test!
LizBaldo May 6, 2025
7c9abd7
fix formatting
LizBaldo May 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions automation/src/test/resources/reference.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
leonardo {
rImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-r:2.2.7"
pythonImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-python:1.1.6"
hailImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-hail:1.1.12"
hailImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-hail:1.1.14"
gatkImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-gatk:2.3.9"
aouImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-aou:2.2.15"
aouImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-aou:2.2.16"
baseImageUrl = "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-base:1.1.4"
gcrWelderUri = "us.gcr.io/broad-dsp-gcr-public/welder-server"
dockerHubWelderUri = "broadinstitute/welder-server"
Expand Down
Loading
Loading