Skip to content

chore:Migrate gsutil usage to gcloud storage #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bhandarivijay-png wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore:Migrate gsutil usage to gcloud storage #159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,15 +267,15 @@ it.
Create a Cloud Storage bucket for storing test data and Dataflow staging location.

```shell script
gsutil mb -p ${PROJECT_ID} -l ${REGION_ID} "gs://${TEMP_GCS_BUCKET}"
gcloud storage buckets create "gs://${TEMP_GCS_BUCKET}" --project=${PROJECT_ID} --location=${REGION_ID}
```

### Copy test data to Cloud Storage

You can use your own file datasets or copy the included demo dataset (`userdata.avro` or `userdata.parquet`).

```shell script
gsutil cp userdata.avro gs://${TEMP_GCS_BUCKET}
gcloud storage cp userdata.avro gs://${TEMP_GCS_BUCKET}
```

## Compile modules
Expand Down Expand Up @@ -381,7 +381,7 @@ The sample & identify pipeline outputs the Avro schema (or converted for Parquet

```shell script
mkdir -p dlp_report/ && rm dlp_report/*.json
gsutil -m cp "gs://${TEMP_GCS_BUCKET}/dlp_report/*.json" dlp_report/
gcloud storage cp "gs://${TEMP_GCS_BUCKET}/dlp_report/*.json" dlp_report/
```

List all the column names that have been identified.
Expand Down
8 changes: 4 additions & 4 deletions sample_identify_and_tag.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ page in the Cloud Console.

Create a Cloud Storage bucket for storing test data and Dataflow staging location:

gsutil mb -p ${PROJECT_ID} -l ${REGION_ID} "gs://${TEMP_GCS_BUCKET}"
gcloud storage buckets create "gs://${TEMP_GCS_BUCKET}" --project="${PROJECT_ID}" --location="${REGION_ID}"

### Create Inspection Tag tempalte

Expand Down Expand Up @@ -195,7 +195,7 @@ You can use your own file datasets or copy the included demonstration dataset (`

1. Copy the sample dataset to Cloud Storage for staging into Cloud SQL:

gsutil cp contacts5k.sql.gz gs://${TEMP_GCS_BUCKET}
gcloud storage cp contacts5k.sql.gz gs://${TEMP_GCS_BUCKET}

1. Create a new Database in the Cloud SQL instance:

Expand All @@ -208,7 +208,7 @@ You can use your own file datasets or copy the included demonstration dataset (`

export SQL_SERVICE_ACCOUNT=$(gcloud sql instances describe "${SQL_INSTANCE}" --project="${PROJECT_ID}" | grep serviceAccountEmailAddress: | sed "s/serviceAccountEmailAddress: //g")

gsutil iam ch "serviceAccount:${SQL_SERVICE_ACCOUNT}:objectAdmin" gs://${TEMP_GCS_BUCKET}
gcloud storage buckets add-iam-policy-binding gs://${TEMP_GCS_BUCKET} --member="serviceAccount:${SQL_SERVICE_ACCOUNT}" --role="roles/storage.objectAdmin"

Learn more about [Importing data into Cloud SQL](https://cloud.google.com/sql/docs/mysql/import-export/importing#mysqldump).

Expand Down Expand Up @@ -368,7 +368,7 @@ sensitive information.
1. Retrieve the report to your local machine:

mkdir -p auto_dlp_report/ && rm auto_dlp_report/*.json
gsutil -m cp "gs://${TEMP_GCS_BUCKET}/auto_dlp_report/*.json" auto_dlp_report/
gcloud storage cp "gs://${TEMP_GCS_BUCKET}/auto_dlp_report/*.json" auto_dlp_report/

1. List all the column names that have been identified:

Expand Down