feat: export datasets to SciCat

[boucherjeanphi]

Add possibility to export a data connector from Renku to SciCat. The export flow is very similar to Zenodo.

Solution

1. SciCat OAuth integration
2. Exchange access token for SciCat token (needed to call the SciCat API)
3. Create a new SciCat dataset through a SciCat API call
4. Start a job that uploads files to the previously created dataset using scicat cli

Changes

• add scicat in ProviderKind, same adapter as generic_oidc
• fix issue in generate_code_verifier
• remove conditions on ProviderKind that prevent setting oidc_issuer_url for ProviderKind != generic_oidc
• add ScicatAPIClient class for SciCat API calls
• Update scicat cli to accept a dataset id arg (--pid)
• add scicat in DepositProvider / DepositSource
• update /deposits API routes for SciCat
• add _create_scicat_configmap_manifest and _create_scicat_upload_job_manifest in components/renku_data_services/data_connectors/core.py
• add alembic migration script

/deploy renku=jp/scicat-export renku-ui=jp/scicat-export extra-values=dataService.dataDeposits.scicat.url=https://discovery-qa.psi.ch,dataService.dataDeposits.scicat.apiUrl=https://dacat-qa.psi.ch/api/v3,dataService.dataDeposits.scicat.image=ghcr.io/swissdatasciencecenter/scicat-cli:latest,dataService.dataDeposits.scicat.testEnv=true

[RenkuBot]

You can access the deployment of this PR at https://renku-ci-ds-1432.dev.renku.ch

[olevski]

This looks really good Jean-Philippe. Just a few minor comments.

[coveralls]

Coverage Report for CI Build 36536868119

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage decreased (-0.1%) to 85.235%

Details

• Coverage decreased (-0.1%) from the base build.
• Patch coverage: 80 uncovered changes across 4 files (152 of 232 lines covered, 65.52%).
• 11 coverage regressions across 7 files.

Uncovered Changes

File	Changed	Covered	%
components/renku_data_services/data_connectors/blueprints.py	45	6	13.33%
components/renku_data_services/data_connectors/core.py	42	3	7.14%
components/renku_data_services/connected_services/core.py	5	4	80.0%
components/renku_data_services/connected_services/utils.py	1	0	0.0%
Total (12 files)	232	152	65.52%

Coverage Regressions

11 previously-covered lines in 7 files lost coverage.

File	Lines Losing Coverage	Coverage
components/renku_data_services/data_connectors/core.py	3	40.28%
components/renku_data_services/crc/core.py	2	79.34%
components/renku_data_services/notebooks/crs.py	2	88.97%
components/renku_data_services/k8s/watcher/core.py	1	80.7%
components/renku_data_services/connected_services/core.py	1	51.9%
components/renku_data_services/capacity_reservation/core.py	1	26.92%
components/renku_data_services/base_api/pagination.py	1	88.16%

---

Coverage Stats

Relevant Lines:	34501
Covered Lines:	29407
Line Coverage:	85.24%
Coverage Strength:	1.48 hits per line

---

💛 - Coveralls

[olevski]

One small change - you can address it in a followup pr if you wish.

[RenkuBot]

Tearing down the temporary RenkuLab deployment for this PR.