From c131c78be83966d6645dc651e01789c66ff810b9 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 19 May 2026 12:13:03 -0400 Subject: [PATCH 1/9] feat: create README file --- CONTRIBUTING.md | 81 +------- README.md | 53 +---- dataconnect/readme/README-v1.0.0.md | 302 ++++++++++++++++++++++++++++ 3 files changed, 313 insertions(+), 123 deletions(-) create mode 100644 dataconnect/readme/README-v1.0.0.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f5f83a3..1056613 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,80 +1 @@ -# Contributing - -## Installation - -This project requires Artifactory credentials. For local development, one can -obtain these from their jfrog user profile -[here](https://mdsol.jfrog.io/ui/admin/artifactory/user_profile) - -### Installation (Docker - recommended) - -To setup your python docker container, download the [engineering helper scripts](https://github.com/mdsol/engineering-helper-scripts/blob/6484ac03490c0562cee26aca89bfbed11e5124e2/.travis.yml#L6) into a -directory parallel to this one and run - -`PACKAGE_GROUPS="dev,test" ../engineering-helper-scripts/docker/build_container.sh .12factor/Dockerfile --build-arg PACKAGE_GROUPS` - -Once your container is built, you can then use `docker compose up` to get -terminal access and attach to the container. - -By using Docker compose, any work you do inside the container will be copied -into your local machine. This is especially useful when the poetry lock file -needs to be regenerated by running `poetry lock` inside the container. - -NOTE: As poetry is not built into the final layer you'll need to build an -image with the python_packages as a target in order to update the lock file. - -`PACKAGE_GROUPS="dev,test" ../engineering-helper-scripts/docker/build_container.sh .12factor/Dockerfile --build-arg PACKAGE_GROUPS --target=python_packages` - -NOTE2: Make sure `.dockerignore` is updated to just include the files/folders you need in your build context. - -### Installation (Local) - -In order to install this package, you will first need to install poetry. -See [here](https://python-poetry.org/docs/#installation) for instructions. - -You can configure poetry via: - -WARNING: Do not check your config file into version control or build it into a Docker image. Doing so will compromise your API KEY. - -```bash -poetry config http-basic.mdsol -``` - -Once these are in place, you can install the package inside a local -virtual environment via: - -```bash -poetry install -``` - -## Testing - -This project uses [pytest](https://docs.pytest.org/en/7.1.x/contents.html) -for automated testing - -You can run the test suite within the projects virtual environment via: - -```bash -poetry run pytest -``` - -Or activate a shell and run the test inside the environment: - -```bash -poetry shell -pytest -``` - -You can also distribute the execution of the test suite on multiple CPUs: - -```bash -pytest -n -``` - -See [pytest's documentation](https://docs.pytest.org/en/latest/contents.html) -for a more detailed guide. - -## Branching - -Please branch out of main, and open a PR. Merges require at least one reviewer. -All tests must pass in travis before merge. +**Public contribution is currently not allowed.** diff --git a/README.md b/README.md index 8577f70..5689688 100644 --- a/README.md +++ b/README.md @@ -1,50 +1,17 @@ -# dataconnect-library-python +# Medidata DataConnect Python library -Python SDK for the [Medidata DataConnect](https://github.com/mdsol/dataconnect-library-r) service. +## Overview ---- +This Python library is built by Medidata to give technical users of Medidata Clinical Data Studio and Medidata Data Connect, including clinical programmers, data scientists, and statisticians, access to relevant data directly within their existing Python environment. You can use it with core Python functionality and other Python libraries in your workflow. -## Transport note +**Note**: This Python library is supported for use in Python environments only. Compatibility with non-Python IDEs is not guaranteed or supported. -The DataConnect service uses **Apache Arrow Flight** (gRPC binary protocol), and **not** a plain REST/HTTP API. `pyarrow.flight` is the primary transport -dependency. +# Versioned Documentation ---- +This directory contains documentation snapshots for each release of the `dataconnect` Python library. -## Installation +## Available Versions -```bash -pip install dataconnect # core (pyarrow) -pip install dataconnect[pandas] # + pandas for .to_pandas() on results -``` - -Requires **Python ≥ 3.13**. - ---- - -## Quick start - -```python -from uuid import UUID - -from dataconnect import DataConnectClient - - -with DataConnectClient.connect( - host="dataconnect.example.com", - port=443, - token="your-bearer-token", -) as client: - - studies = client.get_studies(search_study_name="ACME") - - pagination = client.get_datasets(study_environment_uuid=UUID("cec9f2a7-07ba-4fa8-bfcf-34fbc5d56793")) - datasets = pagination.items - -``` -## Development - -```bash -pip install -e ".[dev]" -pytest -``` +| Version | Documentation | Current version | +|---------|---------------------------------------------|-----------------| +|1.1.0 | [README-v1.0.0.md](readme/README-v1.0.0.md) | Yes | diff --git a/dataconnect/readme/README-v1.0.0.md b/dataconnect/readme/README-v1.0.0.md new file mode 100644 index 0000000..9982d67 --- /dev/null +++ b/dataconnect/readme/README-v1.0.0.md @@ -0,0 +1,302 @@ +# Dataconnect Python Library v1.0.0 + +The Dataconnect Python library provides a Python client for connecting to Medidata Dataconnect and retrieving relevant data programmatically. +To use this library, you must have a valid iMedidata account and access to required building blocks in the Medidata Platform. For details, see the Medidata [Knowledge Hub](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). + +## Table of Contents + +- [Environment Setup and Requirements](#environment-setup-and-requirements) + - [System Requirements](#system-requirements) + - [Authentication and Connectivity](#authentication-and-connectivity) +- [Installation](#installation) + - [Option 1: Install from source (recommended for this repository)](#option-1-install-from-source-recommended-for-this-repository) + - [Option 2: Install using Poetry](#option-2-install-using-poetry) +- [Quick Start](#quick-start) +- [Features](#features) +- [Public API Reference](#public-api-reference) +- [Error Handling](#error-handling) +- [Data Models](#data-models) + +## Environment Setup and Requirements + +### System Requirements + +| Requirement | Version / Notes | +|-------------------|--------------------------| +| Python | 3.13 | +| Operating systems | macOS, Linux, Windows | +| Core dependencies | pyarrow 19.x, pandas 2.x | + +### Authentication and Connectivity + +- You need a valid Dataconnect access token. +- Default connection settings used by the client: + - Host: `enodia-gateway.platform.imedidata.com` + - Port: `443` + - TLS: enabled + +## Installation + +Choose one of the following approaches. + +### Option 1: Install from source (recommended for this repository) + +```bash +git clone +cd Dataconnect-library-python +python3.13 -m venv .venv +source .venv/bin/activate +pip install -e . +``` + +### Option 2: Install using Poetry + +```bash +git clone +cd Dataconnect-library-python +poetry install +poetry shell +``` + +## Quick Start + +```python +from uuid import UUID + +from Dataconnect import DataconnectClient + +with DataconnectClient.connect(token="your-bearer-token") as client: + studies = client.get_studies() + print(f"Found {len(studies)} studies") + + if studies and studies[0].environments: + study_environment_uuid = studies[0].environments[0].uuid + datasets_data = client.get_datasets(study_environment_uuid=study_environment_uuid, page=1, page_size=10) + print(f"Found {datasets_data.total_records} datasets") + + if datasets_data.items: + dataset_uuid = UUID(datasets_data.items[0].dataset_uuid) + df = client.fetch_data(dataset_uuid, first_n_rows=100) + print(df.head()) +``` + +## Features + +- Connect to Dataconnect with secure Arrow Flight transport. +- Get studies available to the authenticated user. +- Get datasets for a study environment with pagination and name filtering. +- Get versions for a dataset. +- Fetch dataset records as a pandas DataFrame. + +## Public API Reference + +The main public entry point is `Dataconnect.DataconnectClient`. + +### `DataconnectClient.connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` + +#### Description +Creates a connected client using the default Arrow Flight transport. + +#### Usage +```python +from Dataconnect import DataconnectClient + +client = DataconnectClient.connect(token="") +try: + studies = client.get_studies() +finally: + client.close() +``` + +#### Arguments +- `host` (`str`): Dataconnect host. +- `port` (`int`): Server port. +- `use_tls` (`bool`): Enable TLS. +- `token` (`str`): Bearer token for authorization. + +#### Output +- Returns: `DataconnectClient` + +#### Data Validation +- Raises `AuthenticationError`, `AuthorizationError`, `NotFoundError`, `ServerError`, or `ValidationError` when the server/transport returns an error. + +--- + +### `get_studies(search_study_name=None)` + +#### Description +Lists studies the authenticated user can access, optionally filtered by full or partial study name. + +#### Usage +```python +studies = client.get_studies(search_study_name="") +for study in studies: + print(study.uuid, study.name) +``` + +#### Arguments +- `search_study_name` (`str | None`): Optional name filter. + +#### Output +- Returns: `list[Study]` + +#### Data Validation +- Raises `DataconnectError` subclasses for server/transport failures. + +--- + +### `get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` + +#### Description +Retrieves datasets for a specific study environment and returns paginated results. + +#### Usage +```python +from uuid import UUID + +response = client.get_datasets( + study_environment_uuid=UUID(""), + search_dataset_name="", + page=1, + page_size=25, +) + +print(response.total_records) +for dataset in response.items: + print(dataset.dataset_uuid, dataset.dataset_name) +``` + +#### Arguments +- `study_environment_uuid` (`UUID`): Required study environment UUID. +- `search_dataset_name` (`str`): Full or partial dataset name filter. +- `page` (`int`): Page number for paginated results (>=1). +- `page_size` (`int`): Number of results per page (>=1). + +#### Output +- Returns: `PaginatedResponse[Dataset]` + +#### Data Validation +- Raises `ValidationError` for invalid UUID/page/page_size. +- Raises other `DataconnectError` subclasses for service failures. + +--- + +### `get_dataset_versions(dataset_uuid)` + +#### Description +Retrieves all available versions for a dataset, sorted in descending version order. + +#### Usage +```python +from uuid import UUID + +versions = client.get_dataset_versions(UUID("")) +for version in versions: + print(version.dataset_name, version.dataset_version) +``` + +#### Arguments +- `dataset_uuid` (`UUID`): Required dataset UUID. + +#### Output +- Returns: `list[DatasetVersion]` + +#### Data Validation +- Raises `ValidationError` for invalid UUID. +- Raises other `DataconnectError` subclasses for service failures. + +--- + +### `fetch_data(dataset_uuid, first_n_rows=None)` + +#### Description +Fetches dataset rows into a pandas DataFrame. + +#### Usage +```python +from uuid import UUID + +df = client.fetch_data(UUID(""), first_n_rows=100) +print(df.shape) +``` + +#### Arguments +- `dataset_uuid` (`UUID`): Required dataset UUID. +- `first_n_rows` (`int | None`): Optional positive row limit. + +#### Output +- Returns: `pandas.DataFrame` + +#### Data Validation +- Raises `ValidationError` for invalid `dataset_uuid` or non-positive `first_n_rows`. +- Raises other `DataconnectError` subclasses for service failures. + +### `close()` + +Closes the underlying transport connection. + +| Item | Details | +|---|---| +| Description | Releases network resources used by the client. | +| Parameters | None | +| Returns | `None` | +| Error handling | May raise `DataconnectError` subclasses if close fails at transport level. | + +Usage example: + +```python +client = DataconnectClient.connect(token="") +try: + pass +finally: + client.close() +``` + +## Error Handling + +All public errors inherit from `DataconnectError`. + +| Exception | Typical meaning | +|---|---| +| `AuthenticationError` | Invalid or missing credentials/token. | +| `AuthorizationError` | Authenticated but not allowed to access requested resource. | +| `NotFoundError` | Requested study/dataset/resource does not exist. | +| `ServerError` | Unexpected server-side failure. | +| `ValidationError` | Invalid inputs or malformed/invalid response payloads. | + +Example: + +```python +from Dataconnect import ( + AuthenticationError, + AuthorizationError, + DataconnectError, + NotFoundError, + ValidationError, +) + +try: + studies = client.get_studies() +except AuthenticationError as exc: + print("Authentication failed:", exc) +except AuthorizationError as exc: + print("Not authorized:", exc) +except NotFoundError as exc: + print("Resource not found:", exc) +except ValidationError as exc: + print("Invalid request:", exc) +except DataconnectError as exc: + print("Dataconnect request failed:", exc) +``` + +## Data Models + +| Model | Fields | +|---|---| +| `StudyEnvironment` | `uuid`, `name` | +| `Study` | `uuid`, `name`, `environments` | +| `Dataset` | `dataset_uuid`, `study_uuid`, `study_env_uuid`, `dataset_name` | +| `DatasetVersion` | `study_uuid`, `study_environment_uuid`, `dataset_uuid`, `dataset_name`, `dataset_version` | +| `Pagination` | `page`, `page_size`, `total_pages` | +| `PaginatedResponse[T]` | `total_records`, `pagination`, `items` | + From 40de6c74a8957dbbd012aa75af42bcab7ab5de12 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 19 May 2026 12:28:38 -0400 Subject: [PATCH 2/9] feat: convert arguments description to table --- dataconnect/readme/README-v1.0.0.md | 36 ++++++++++++++++++----------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/dataconnect/readme/README-v1.0.0.md b/dataconnect/readme/README-v1.0.0.md index 9982d67..ad9b740 100644 --- a/dataconnect/readme/README-v1.0.0.md +++ b/dataconnect/readme/README-v1.0.0.md @@ -109,10 +109,12 @@ finally: ``` #### Arguments -- `host` (`str`): Dataconnect host. -- `port` (`int`): Server port. -- `use_tls` (`bool`): Enable TLS. -- `token` (`str`): Bearer token for authorization. +| Argument | Type | Description | +|---|---|---| +| `host` | `str` | Dataconnect host. | +| `port` | `int` | Server port. | +| `use_tls` | `bool` | Enable TLS. | +| `token` | `str` | Bearer token for authorization. | #### Output - Returns: `DataconnectClient` @@ -135,7 +137,9 @@ for study in studies: ``` #### Arguments -- `search_study_name` (`str | None`): Optional name filter. +| Argument | Type | Description | +|---|---|---| +| `search_study_name` | `str | None` | Optional full or partial study name filter. | #### Output - Returns: `list[Study]` @@ -167,10 +171,12 @@ for dataset in response.items: ``` #### Arguments -- `study_environment_uuid` (`UUID`): Required study environment UUID. -- `search_dataset_name` (`str`): Full or partial dataset name filter. -- `page` (`int`): Page number for paginated results (>=1). -- `page_size` (`int`): Number of results per page (>=1). +| Argument | Type | Description | +|---|---|---| +| `study_environment_uuid` | `UUID` | Required study environment UUID. | +| `search_dataset_name` | `str` | Full or partial dataset name filter. | +| `page` | `int` | Page number for paginated results (>=1). | +| `page_size` | `int` | Number of results per page (>=1). | #### Output - Returns: `PaginatedResponse[Dataset]` @@ -196,7 +202,9 @@ for version in versions: ``` #### Arguments -- `dataset_uuid` (`UUID`): Required dataset UUID. +| Argument | Type | Description | +|---|---|---| +| `dataset_uuid` | `UUID` | Required dataset UUID. | #### Output - Returns: `list[DatasetVersion]` @@ -221,8 +229,10 @@ print(df.shape) ``` #### Arguments -- `dataset_uuid` (`UUID`): Required dataset UUID. -- `first_n_rows` (`int | None`): Optional positive row limit. +| Argument | Type | Description | +|---|---|---| +| `dataset_uuid` | `UUID` | Required dataset UUID. | +| `first_n_rows` | `int | None` | Optional positive row limit. | #### Output - Returns: `pandas.DataFrame` @@ -239,7 +249,7 @@ Closes the underlying transport connection. |---|---| | Description | Releases network resources used by the client. | | Parameters | None | -| Returns | `None` | +| Returns | None | | Error handling | May raise `DataconnectError` subclasses if close fails at transport level. | Usage example: From 4481e6474d92c5a5ce8f478155e9580f85cc2c9c Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 19 May 2026 12:56:56 -0400 Subject: [PATCH 3/9] feat: updated formatting --- dataconnect/readme/README-v1.0.0.md | 77 +++++++++++++++++------------ 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/dataconnect/readme/README-v1.0.0.md b/dataconnect/readme/README-v1.0.0.md index ad9b740..49e58c5 100644 --- a/dataconnect/readme/README-v1.0.0.md +++ b/dataconnect/readme/README-v1.0.0.md @@ -92,12 +92,15 @@ with DataconnectClient.connect(token="your-bearer-token") as client: The main public entry point is `Dataconnect.DataconnectClient`. -### `DataconnectClient.connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` +### connect() #### Description Creates a connected client using the default Arrow Flight transport. #### Usage +`connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` + +#### Example ```python from Dataconnect import DataconnectClient @@ -111,10 +114,10 @@ finally: #### Arguments | Argument | Type | Description | |---|---|---| -| `host` | `str` | Dataconnect host. | -| `port` | `int` | Server port. | -| `use_tls` | `bool` | Enable TLS. | -| `token` | `str` | Bearer token for authorization. | +| host | str | Dataconnect host. | +| port | int | Server port. | +| use_tls | bool | Enable TLS. | +| token | str | Bearer token for authorization. | #### Output - Returns: `DataconnectClient` @@ -124,12 +127,15 @@ finally: --- -### `get_studies(search_study_name=None)` +### get_studies() #### Description Lists studies the authenticated user can access, optionally filtered by full or partial study name. #### Usage +`get_studies(search_study_name=None)` + +#### Example ```python studies = client.get_studies(search_study_name="") for study in studies: @@ -139,7 +145,7 @@ for study in studies: #### Arguments | Argument | Type | Description | |---|---|---| -| `search_study_name` | `str | None` | Optional full or partial study name filter. | +| search_study_name | str or None | Optional full or partial study name filter. | #### Output - Returns: `list[Study]` @@ -149,12 +155,15 @@ for study in studies: --- -### `get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` +### get_datasets() #### Description Retrieves datasets for a specific study environment and returns paginated results. #### Usage +`get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` + +#### Example ```python from uuid import UUID @@ -173,10 +182,10 @@ for dataset in response.items: #### Arguments | Argument | Type | Description | |---|---|---| -| `study_environment_uuid` | `UUID` | Required study environment UUID. | -| `search_dataset_name` | `str` | Full or partial dataset name filter. | -| `page` | `int` | Page number for paginated results (>=1). | -| `page_size` | `int` | Number of results per page (>=1). | +| study_environment_uuid | UUID | Required study environment UUID. | +| search_dataset_name | str | Full or partial dataset name filter. | +| page | int | Page number for paginated results (>=1). | +| page_size | int | Number of results per page (>=1). | #### Output - Returns: `PaginatedResponse[Dataset]` @@ -187,12 +196,15 @@ for dataset in response.items: --- -### `get_dataset_versions(dataset_uuid)` +### get_dataset_versions() #### Description Retrieves all available versions for a dataset, sorted in descending version order. #### Usage +`get_dataset_versions(dataset_uuid)` + +#### Example ```python from uuid import UUID @@ -204,7 +216,7 @@ for version in versions: #### Arguments | Argument | Type | Description | |---|---|---| -| `dataset_uuid` | `UUID` | Required dataset UUID. | +| dataset_uuid | UUID | Required dataset UUID. | #### Output - Returns: `list[DatasetVersion]` @@ -215,12 +227,15 @@ for version in versions: --- -### `fetch_data(dataset_uuid, first_n_rows=None)` +### fetch_data(dataset_uuid, first_n_rows=None) #### Description Fetches dataset rows into a pandas DataFrame. -#### Usage +### Usage +`fetch_data(dataset_uuid, first_n_rows=None)` + +#### Example ```python from uuid import UUID @@ -231,8 +246,8 @@ print(df.shape) #### Arguments | Argument | Type | Description | |---|---|---| -| `dataset_uuid` | `UUID` | Required dataset UUID. | -| `first_n_rows` | `int | None` | Optional positive row limit. | +| dataset_uuid | UUID | Required dataset UUID. | +| first_n_rows | int or None | Optional positive row limit. | #### Output - Returns: `pandas.DataFrame` @@ -241,7 +256,7 @@ print(df.shape) - Raises `ValidationError` for invalid `dataset_uuid` or non-positive `first_n_rows`. - Raises other `DataconnectError` subclasses for service failures. -### `close()` +### close() Closes the underlying transport connection. @@ -252,7 +267,7 @@ Closes the underlying transport connection. | Returns | None | | Error handling | May raise `DataconnectError` subclasses if close fails at transport level. | -Usage example: +Example example: ```python client = DataconnectClient.connect(token="") @@ -268,11 +283,11 @@ All public errors inherit from `DataconnectError`. | Exception | Typical meaning | |---|---| -| `AuthenticationError` | Invalid or missing credentials/token. | -| `AuthorizationError` | Authenticated but not allowed to access requested resource. | -| `NotFoundError` | Requested study/dataset/resource does not exist. | -| `ServerError` | Unexpected server-side failure. | -| `ValidationError` | Invalid inputs or malformed/invalid response payloads. | +| AuthenticationError | Invalid or missing credentials/token. | +| AuthorizationError | Authenticated but not allowed to access requested resource. | +| NotFoundError | Requested study/dataset/resource does not exist. | +| ServerError | Unexpected server-side failure. | +| ValidationError | Invalid inputs or malformed/invalid response payloads. | Example: @@ -303,10 +318,10 @@ except DataconnectError as exc: | Model | Fields | |---|---| -| `StudyEnvironment` | `uuid`, `name` | -| `Study` | `uuid`, `name`, `environments` | -| `Dataset` | `dataset_uuid`, `study_uuid`, `study_env_uuid`, `dataset_name` | -| `DatasetVersion` | `study_uuid`, `study_environment_uuid`, `dataset_uuid`, `dataset_name`, `dataset_version` | -| `Pagination` | `page`, `page_size`, `total_pages` | -| `PaginatedResponse[T]` | `total_records`, `pagination`, `items` | +| StudyEnvironment | `uuid`, `name` | +| Study | `uuid`, `name`, `environments` | +| Dataset | `dataset_uuid`, `study_uuid`, `study_env_uuid`, `dataset_name` | +| DatasetVersion | `study_uuid`, `study_environment_uuid`, `dataset_uuid`, `dataset_name`, `dataset_version` | +| Pagination | `page`, `page_size`, `total_pages` | +| PaginatedResponse[T] | `total_records`, `pagination`, `items` | From 9e4fceee6d4ab6dfda1c1910e2f211a9239143f8 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 19 May 2026 13:02:32 -0400 Subject: [PATCH 4/9] feat: added functions to table of contents --- dataconnect/readme/README-v1.0.0.md | 46 ++++++++++++++++------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/dataconnect/readme/README-v1.0.0.md b/dataconnect/readme/README-v1.0.0.md index 49e58c5..ce90346 100644 --- a/dataconnect/readme/README-v1.0.0.md +++ b/dataconnect/readme/README-v1.0.0.md @@ -14,6 +14,12 @@ To use this library, you must have a valid iMedidata account and access to requi - [Quick Start](#quick-start) - [Features](#features) - [Public API Reference](#public-api-reference) + - [connect()](#connect) + - [get_studies()](#get_studies) + - [get_datasets()](#get_datasets) + - [get_dataset_versions()](#get_dataset_versions) + - [fetch_data()](#fetch_data) + - [close()](#close) - [Error Handling](#error-handling) - [Data Models](#data-models) @@ -114,10 +120,10 @@ finally: #### Arguments | Argument | Type | Description | |---|---|---| -| host | str | Dataconnect host. | -| port | int | Server port. | -| use_tls | bool | Enable TLS. | -| token | str | Bearer token for authorization. | +| host | str | Dataconnect host | +| port | int | Server port | +| use_tls | bool | Enable TLS | +| token | str | Bearer token for authorization | #### Output - Returns: `DataconnectClient` @@ -145,7 +151,7 @@ for study in studies: #### Arguments | Argument | Type | Description | |---|---|---| -| search_study_name | str or None | Optional full or partial study name filter. | +| search_study_name | str or None | Optional full or partial study name filter | #### Output - Returns: `list[Study]` @@ -182,10 +188,10 @@ for dataset in response.items: #### Arguments | Argument | Type | Description | |---|---|---| -| study_environment_uuid | UUID | Required study environment UUID. | -| search_dataset_name | str | Full or partial dataset name filter. | -| page | int | Page number for paginated results (>=1). | -| page_size | int | Number of results per page (>=1). | +| study_environment_uuid | UUID | Required study environment UUID | +| search_dataset_name | str | Full or partial dataset name filter | +| page | int | Page number for paginated results (>=1) | +| page_size | int | Number of results per page (>=1) | #### Output - Returns: `PaginatedResponse[Dataset]` @@ -216,7 +222,7 @@ for version in versions: #### Arguments | Argument | Type | Description | |---|---|---| -| dataset_uuid | UUID | Required dataset UUID. | +| dataset_uuid | UUID | Required dataset UUID | #### Output - Returns: `list[DatasetVersion]` @@ -227,7 +233,7 @@ for version in versions: --- -### fetch_data(dataset_uuid, first_n_rows=None) +### fetch_data() #### Description Fetches dataset rows into a pandas DataFrame. @@ -246,8 +252,8 @@ print(df.shape) #### Arguments | Argument | Type | Description | |---|---|---| -| dataset_uuid | UUID | Required dataset UUID. | -| first_n_rows | int or None | Optional positive row limit. | +| dataset_uuid | UUID | Required dataset UUID | +| first_n_rows | int or None | Optional positive row limit | #### Output - Returns: `pandas.DataFrame` @@ -262,10 +268,10 @@ Closes the underlying transport connection. | Item | Details | |---|---| -| Description | Releases network resources used by the client. | +| Description | Releases network resources used by the client | | Parameters | None | | Returns | None | -| Error handling | May raise `DataconnectError` subclasses if close fails at transport level. | +| Error handling | May raise `DataconnectError` subclasses if close fails at transport level | Example example: @@ -283,11 +289,11 @@ All public errors inherit from `DataconnectError`. | Exception | Typical meaning | |---|---| -| AuthenticationError | Invalid or missing credentials/token. | -| AuthorizationError | Authenticated but not allowed to access requested resource. | -| NotFoundError | Requested study/dataset/resource does not exist. | -| ServerError | Unexpected server-side failure. | -| ValidationError | Invalid inputs or malformed/invalid response payloads. | +| AuthenticationError | Invalid or missing credentials/token | +| AuthorizationError | Authenticated but not allowed to access requested resource | +| NotFoundError | Requested study/dataset/resource does not exist | +| ServerError | Unexpected server-side failure | +| ValidationError | Invalid inputs or malformed/invalid response payloads | Example: From 97042f998b1b54b6ef4e3d99f1c74f01c6c19212 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 19 May 2026 14:50:46 -0400 Subject: [PATCH 5/9] feat: aligned with R lib docs --- dataconnect/readme/README-v1.0.0.md | 283 ++++++++-------------------- readme/README-v1.0.0.md | 283 ++++++++++++++++++++++++++++ vignettes/pythonLibrary_setup.md | 44 +++++ vignettes/pythonLibrary_usage.md | 118 ++++++++++++ 4 files changed, 527 insertions(+), 201 deletions(-) create mode 100644 readme/README-v1.0.0.md create mode 100644 vignettes/pythonLibrary_setup.md create mode 100644 vignettes/pythonLibrary_usage.md diff --git a/dataconnect/readme/README-v1.0.0.md b/dataconnect/readme/README-v1.0.0.md index ce90346..5bf6f0e 100644 --- a/dataconnect/readme/README-v1.0.0.md +++ b/dataconnect/readme/README-v1.0.0.md @@ -12,16 +12,19 @@ To use this library, you must have a valid iMedidata account and access to requi - [Option 1: Install from source (recommended for this repository)](#option-1-install-from-source-recommended-for-this-repository) - [Option 2: Install using Poetry](#option-2-install-using-poetry) - [Quick Start](#quick-start) -- [Features](#features) -- [Public API Reference](#public-api-reference) +- [Functions](#functions) - [connect()](#connect) - [get_studies()](#get_studies) - [get_datasets()](#get_datasets) - [get_dataset_versions()](#get_dataset_versions) - [fetch_data()](#fetch_data) - [close()](#close) -- [Error Handling](#error-handling) -- [Data Models](#data-models) +- [Errors](#errors) +- [Reporting known issues ](#reporting-known-issues) +- [Backend](#backend) +- [Versions](#versions) +- [Licensing](#licensing) + ## Environment Setup and Requirements @@ -35,129 +38,58 @@ To use this library, you must have a valid iMedidata account and access to requi ### Authentication and Connectivity -- You need a valid Dataconnect access token. -- Default connection settings used by the client: - - Host: `enodia-gateway.platform.imedidata.com` - - Port: `443` - - TLS: enabled - -## Installation - -Choose one of the following approaches. - -### Option 1: Install from source (recommended for this repository) +* **Retrieving data:** You must have a user token to establish a connection between your Python environment and Medidata Data Connect. You can generate this token through Data Connect’s Developer Center. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). Medidata recommends that you save the token in a separate file and input it into the below initiation function. -```bash -git clone -cd Dataconnect-library-python -python3.13 -m venv .venv -source .venv/bin/activate -pip install -e . -``` +* **Publish data:** You must have a project token to publish a dataset from your Python environment to Medidata Data Connect. You can generate this token through Data Connect > Transformations, by creating a Custom Code project. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/generate_custom_code_projects.html). -### Option 2: Install using Poetry +# Installation -```bash -git clone -cd Dataconnect-library-python -poetry install -poetry shell -``` +To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-r/blob/main/vignettes/dataconnect/readme/vignettes/pythonLibrary_usage.md). ## Quick Start -```python -from uuid import UUID - -from Dataconnect import DataconnectClient - -with DataconnectClient.connect(token="your-bearer-token") as client: - studies = client.get_studies() - print(f"Found {len(studies)} studies") - - if studies and studies[0].environments: - study_environment_uuid = studies[0].environments[0].uuid - datasets_data = client.get_datasets(study_environment_uuid=study_environment_uuid, page=1, page_size=10) - print(f"Found {datasets_data.total_records} datasets") - - if datasets_data.items: - dataset_uuid = UUID(datasets_data.items[0].dataset_uuid) - df = client.fetch_data(dataset_uuid, first_n_rows=100) - print(df.head()) -``` +For end-to-end examples, see the [Python usage guide](../../vignettes/pythonLibrary_usage.md). -## Features +## Functions -- Connect to Dataconnect with secure Arrow Flight transport. -- Get studies available to the authenticated user. -- Get datasets for a study environment with pagination and name filtering. -- Get versions for a dataset. -- Fetch dataset records as a pandas DataFrame. - -## Public API Reference - -The main public entry point is `Dataconnect.DataconnectClient`. +The main public entry point is `DataconnectClient`. ### connect() #### Description -Creates a connected client using the default Arrow Flight transport. +Creates a connected client. #### Usage `connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` -#### Example -```python -from Dataconnect import DataconnectClient - -client = DataconnectClient.connect(token="") -try: - studies = client.get_studies() -finally: - client.close() -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| host | str | Dataconnect host | -| port | int | Server port | -| use_tls | bool | Enable TLS | -| token | str | Bearer token for authorization | +| host | str | Server host. Default host="enodia-gateway.platform.imedidata.com" | +| port | int | Server port. Default port="443" | +| use_tls | bool | Denotes whether to use TLS. Default use_tls = True | +| token | str | Authentication token, this is the user authentication token generated from the Developer Center in Medidata Data Connect | #### Output -- Returns: `DataconnectClient` - -#### Data Validation -- Raises `AuthenticationError`, `AuthorizationError`, `NotFoundError`, `ServerError`, or `ValidationError` when the server/transport returns an error. +DataconnectClient object. This enables you to interact with Medidata Data Connect data in Python environment. --- ### get_studies() #### Description -Lists studies the authenticated user can access, optionally filtered by full or partial study name. +Retrieves a list of studies where the user has permission to manage custom code projects. Use the optional study name search parameter to filter results. #### Usage `get_studies(search_study_name=None)` -#### Example -```python -studies = client.get_studies(search_study_name="") -for study in studies: - print(study.uuid, study.name) -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| search_study_name | str or None | Optional full or partial study name filter | +| search_study_name | str or None | Optional. The approximate name of the study | #### Output -- Returns: `list[Study]` - -#### Data Validation -- Raises `DataconnectError` subclasses for server/transport failures. +Returns a list of studies. Each study includes `name`, `uuid`, and a list of `environments`. Each environment includes `name` and `uuid`. --- @@ -169,68 +101,35 @@ Retrieves datasets for a specific study environment and returns paginated result #### Usage `get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` -#### Example -```python -from uuid import UUID - -response = client.get_datasets( - study_environment_uuid=UUID(""), - search_dataset_name="", - page=1, - page_size=25, -) - -print(response.total_records) -for dataset in response.items: - print(dataset.dataset_uuid, dataset.dataset_name) -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| study_environment_uuid | UUID | Required study environment UUID | -| search_dataset_name | str | Full or partial dataset name filter | -| page | int | Page number for paginated results (>=1) | -| page_size | int | Number of results per page (>=1) | +| study_environment_uuid | UUID | Unique iMedidata study environment identifier. You can find this in iMedidata’s Developer Info details | +| search_dataset_name | str | Optional. The approximate name of the dataset | +| page | int | Optional. Page number for paginated results. Default: 1 | +| page_size | int | Optional. Number of results per page. Default: 50 | #### Output - Returns: `PaginatedResponse[Dataset]` -#### Data Validation -- Raises `ValidationError` for invalid UUID/page/page_size. -- Raises other `DataconnectError` subclasses for service failures. - --- ### get_dataset_versions() #### Description -Retrieves all available versions for a dataset, sorted in descending version order. +Retrieves all available versions for a dataset. #### Usage `get_dataset_versions(dataset_uuid)` -#### Example -```python -from uuid import UUID - -versions = client.get_dataset_versions(UUID("")) -for version in versions: - print(version.dataset_name, version.dataset_version) -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| dataset_uuid | UUID | Required dataset UUID | +| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() function | #### Output - Returns: `list[DatasetVersion]` -#### Data Validation -- Raises `ValidationError` for invalid UUID. -- Raises other `DataconnectError` subclasses for service failures. - --- ### fetch_data() @@ -241,27 +140,15 @@ Fetches dataset rows into a pandas DataFrame. ### Usage `fetch_data(dataset_uuid, first_n_rows=None)` -#### Example -```python -from uuid import UUID - -df = client.fetch_data(UUID(""), first_n_rows=100) -print(df.shape) -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| dataset_uuid | UUID | Required dataset UUID | +| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() and dataset_versions() functions | | first_n_rows | int or None | Optional positive row limit | #### Output - Returns: `pandas.DataFrame` -#### Data Validation -- Raises `ValidationError` for invalid `dataset_uuid` or non-positive `first_n_rows`. -- Raises other `DataconnectError` subclasses for service failures. - ### close() Closes the underlying transport connection. @@ -273,61 +160,55 @@ Closes the underlying transport connection. | Returns | None | | Error handling | May raise `DataconnectError` subclasses if close fails at transport level | -Example example: - -```python -client = DataconnectClient.connect(token="") -try: - pass -finally: - client.close() -``` - -## Error Handling - -All public errors inherit from `DataconnectError`. - -| Exception | Typical meaning | -|---|---| -| AuthenticationError | Invalid or missing credentials/token | -| AuthorizationError | Authenticated but not allowed to access requested resource | -| NotFoundError | Requested study/dataset/resource does not exist | -| ServerError | Unexpected server-side failure | -| ValidationError | Invalid inputs or malformed/invalid response payloads | - -Example: - -```python -from Dataconnect import ( - AuthenticationError, - AuthorizationError, - DataconnectError, - NotFoundError, - ValidationError, -) - -try: - studies = client.get_studies() -except AuthenticationError as exc: - print("Authentication failed:", exc) -except AuthorizationError as exc: - print("Not authorized:", exc) -except NotFoundError as exc: - print("Resource not found:", exc) -except ValidationError as exc: - print("Invalid request:", exc) -except DataconnectError as exc: - print("Dataconnect request failed:", exc) -``` - -## Data Models - -| Model | Fields | -|---|---| -| StudyEnvironment | `uuid`, `name` | -| Study | `uuid`, `name`, `environments` | -| Dataset | `dataset_uuid`, `study_uuid`, `study_env_uuid`, `dataset_name` | -| DatasetVersion | `study_uuid`, `study_environment_uuid`, `dataset_uuid`, `dataset_name`, `dataset_version` | -| Pagination | `page`, `page_size`, `total_pages` | -| PaginatedResponse[T] | `total_records`, `pagination`, `items` | - +## Errors + +R Library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. + +| Error Code | Type | Scenario| +| :--- | :--- |:---| +| AUTHZ_001 | Authorization | Authorization service check failed | +| VAL_002 | Validation - Page Number | Page number is not a positive integer +| VAL_003 | Validation - Page Size | Page size is out of range [1, 100] +| VAL_004 | Validation - Study Parameter | Invalid study uuid +| VAL_005 | Validation - Study Environment Parameter | Missing or invalid study environment uuid +| VAL_006 | Validation - Dataset Parameter | Invalid dataset uuid +| VAL_007 | Validation - Configuration Error | Required input parameters are missing or invalid in configuration +| VAL_008 | Validation - Project Token | Invalid project token +| VAL_009 | Validation - Unsupported Data Type | Unsupported data types. +| VAL_010 | Validation - Unsupported Data Type | Unsupported datetime formats. +| VAL_011 | Validation - Pagination | Pagination is out of range +| VAL_012 | Validation - Concurrency | Project actively being published +| VAL_013 | Validation - Formatting Error | Data validation failed. One or more records contain formatting errors. +| RES_002 | Resource Exceptions - Study Environment | No authorized Study Environments found for the authenticated user +| RES_003 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. +| RES_004 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. +| RES_005 | Resource Exceptions - Study Group | Study Group not found for the Dataset's Study Environment. +| RES_006 | Resource Exceptions - Study | Study Group not found for the Dataset's Study Environment. +| RES_007 | Resource Exceptions - Client Division | Client Division not found for the Dataset's Study Environment. +| RES_008 | Resource Exceptions - Custom Code Project | Transformation Project is not found. +| INT_001 | Internal Application Exception | Something went wrong on our end. + +# Reporting known issues + +If you believe you have found an issue, please contact Medidata Support by submitting a ticket to Medidata Support. All issue reports should include a minimal reproducible example to ensure our team can diagnose the issue. + +Additionally, all known issues are available [here](https://learn.medidata.com/en-US/bundle/current-issues/page/current_known_issues_for_data_connect.html). + +# Backend + +This library uses the Arrow open source library and the Iceberg open table format to enable data interoperability across platforms. + +* [Apache arrow](https://arrow.apache.org/docs/r/): This library uses Arrow’s highly efficient format [pyarrow](https://arrow.apache.org/cookbook/py/flight.html) to transfer massive datasets over the network, allowing users to access & interact with remote datasets. + +* [Apache Iceberg](https://iceberg.apache.org/): This is the open table format underlying Medidata Data Connect's structured data management to support high-performance and reliable data analytics and storage. + +# Licensing + +BY DOWNLOADING THIS FILE (“DOWNLOAD”) YOU AGREE TO THE FOLLOWING TERMS: +MEDIDATA SOLUTIONS, INC. AND ITS AFFILIATES (COLLECTIVELY “MEDIDATA”) GRANT A FREE OF CHARGE, NON-EXCLUSIVE AND NON-TRANSFERABLE RIGHT TO USE THE DOWNLOAD. USE OF THIS DOWNLOAD IS PERMITTED FOR INTERNAL BUSINESS PURPOSES ONLY. + +THIS DOWNLOAD IS MADE AVAILABLE ON AN "AS IS" BASIS WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, ORAL OR WRITTEN, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. + +MEDIDATA SHALL HAVE NO LIABILITY FOR DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES, INCLUDING, WITHOUT LIMITATION, CLAIMS FOR LOST PROFITS, BUSINESS INTERRUPTION AND LOSS OF DATA THAT IN ANY WAY RELATE TO THIS DOWNLOAD, WHETHER OR NOT MEDIDATA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND NOTWITHSTANDING THE FAILURE OF THE ESSENTIAL PURPOSE OF ANY REMEDY. + +YOUR USE OF THIS DOWNLOAD SHALL BE AT YOUR SOLE RISK. NO SUPPORT OF ANY KIND OF THE DOWNLOAD IS PROVIDED BY MEDIDATA. diff --git a/readme/README-v1.0.0.md b/readme/README-v1.0.0.md new file mode 100644 index 0000000..0b85511 --- /dev/null +++ b/readme/README-v1.0.0.md @@ -0,0 +1,283 @@ +# Dataconnect Python Library v1.0.0 + +The Dataconnect Python library provides a Python client for connecting to Medidata Dataconnect and retrieving relevant data programmatically. +To use this library, you must have a valid iMedidata account and access to required building blocks in the Medidata Platform. For details, see the Medidata [Knowledge Hub](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). + +## Table of Contents + +- [Environment Setup and Requirements](#environment-setup-and-requirements) + - [System Requirements](#system-requirements) + - [Authentication and Connectivity](#authentication-and-connectivity) +- [Installation](#installation) + - [Option 1: Install from source (recommended for this repository)](#option-1-install-from-source-recommended-for-this-repository) + - [Option 2: Install using Poetry](#option-2-install-using-poetry) +- [Quick Start](#quick-start) +- [Functions](#functions) + - [connect()](#connect) + - [get_studies()](#get_studies) + - [get_datasets()](#get_datasets) + - [get_dataset_versions()](#get_dataset_versions) + - [fetch_data()](#fetch_data) + - [close()](#close) +- [Error Handling](#error-handling) +- [Data Models](#data-models) + +## Environment Setup and Requirements + +### System Requirements + +| Requirement | Version / Notes | +|-------------------|--------------------------| +| Python | 3.13 | +| Operating systems | macOS, Linux, Windows | +| Core dependencies | pyarrow 19.x, pandas 2.x | + +### Authentication and Connectivity + +* **Retrieving data:** You must have a user token to establish a connection between your Python environment and Medidata Data Connect. You can generate this token through Data Connect’s Developer Center. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). Medidata recommends that you save the token in a separate file and input it into the below initiation function. + +* **Publish data:** You must have a project token to publish a dataset from your Python environment to Medidata Data Connect. You can generate this token through Data Connect > Transformations, by creating a Custom Code project. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/generate_custom_code_projects.html). + +## Installation + +# Installation + +To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-r/blob/main/vignettes/dataconnect/readme/vignettes/pythonLibrary_usage.md). + +## Functions + +The main public entry point is `DataconnectClient`. + +### connect() + +#### Description +Creates a connected client using the default Arrow Flight transport. + +#### Usage +`connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` + +#### Example +```python +from Dataconnect import DataconnectClient + +client = DataconnectClient.connect(token="") +try: + studies = client.get_studies() +finally: + client.close() +``` + +#### Arguments +| Argument | Type | Description | +|---|---|---| +| host | str | Dataconnect host | +| port | int | Server port | +| use_tls | bool | Enable TLS | +| token | str | Authentication token, this is the user authentication token generated from the Developer Center in Medidata Data Connect | + +#### Output +- Returns: `DataconnectClient` + +#### Data Validation +- Raises `AuthenticationError`, `AuthorizationError`, `NotFoundError`, `ServerError`, or `ValidationError` when the server/transport returns an error. + +--- + +### get_studies() + +#### Description +Lists studies the authenticated user can access, optionally filtered by full or partial study name. + +#### Usage +`get_studies(search_study_name=None)` + +#### Example +```python +studies = client.get_studies(search_study_name="") +for study in studies: + print(study.uuid, study.name) +``` + +#### Arguments +| Argument | Type | Description | +|---|---|---| +| search_study_name | str or None | Optional full or partial study name filter | + +#### Output +- Returns: `list[Study]` + +#### Data Validation +- Raises `DataconnectError` subclasses for server/transport failures. + +--- + +### get_datasets() + +#### Description +Retrieves datasets for a specific study environment and returns paginated results. + +#### Usage +`get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` + +#### Example +```python +from uuid import UUID + +response = client.get_datasets( + study_environment_uuid=UUID(""), + search_dataset_name="", + page=1, + page_size=25, +) + +print(response.total_records) +for dataset in response.items: + print(dataset.dataset_uuid, dataset.dataset_name) +``` + +#### Arguments +| Argument | Type | Description | +|---|---|---| +| study_environment_uuid | UUID | Unique iMedidata study environment identifier. You can find this in iMedidata’s Developer Info details | +| search_dataset_name | str | Full or partial dataset name filter | +| page | int | Page number for paginated results (>=1) | +| page_size | int | Number of results per page (>=1) | + +#### Output +- Returns: `PaginatedResponse[Dataset]` + +#### Data Validation +- Raises `ValidationError` for invalid UUID/page/page_size. +- Raises other `DataconnectError` subclasses for service failures. + +--- + +### get_dataset_versions() + +#### Description +Retrieves all available versions for a dataset, sorted in descending version order. + +#### Usage +`get_dataset_versions(dataset_uuid)` + +#### Example +```python +from uuid import UUID + +versions = client.get_dataset_versions(UUID("")) +for version in versions: + print(version.dataset_name, version.dataset_version) +``` + +#### Arguments +| Argument | Type | Description | +|---|---|---| +| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() and dataset_versions() functions | + +#### Output +- Returns: `list[DatasetVersion]` + +#### Data Validation +- Raises `ValidationError` for invalid UUID. +- Raises other `DataconnectError` subclasses for service failures. + +--- + +### fetch_data() + +#### Description +Fetches dataset rows into a pandas DataFrame. + +### Usage +`fetch_data(dataset_uuid, first_n_rows=None)` + +#### Example +```python +from uuid import UUID + +df = client.fetch_data(UUID(""), first_n_rows=100) +print(df.shape) +``` + +#### Arguments +| Argument | Type | Description | +|---|---|---| +| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() and dataset_versions() functions | +| first_n_rows | int or None | Optional positive row limit | + +#### Output +- Returns: `pandas.DataFrame` + +#### Data Validation +- Raises `ValidationError` for invalid `dataset_uuid` or non-positive `first_n_rows`. +- Raises other `DataconnectError` subclasses for service failures. + +### close() + +Closes the underlying transport connection. + +| Item | Details | +|---|---| +| Description | Releases network resources used by the client | +| Parameters | None | +| Returns | None | +| Error handling | May raise `DataconnectError` subclasses if close fails at transport level | + +Example example: + +```python +client = DataconnectClient.connect(token="") +try: + pass +finally: + client.close() +``` + +## Error Handling + +All public errors inherit from `DataconnectError`. + +| Exception | Typical meaning | +|---|---| +| AuthenticationError | Invalid or missing credentials/token | +| AuthorizationError | Authenticated but not allowed to access requested resource | +| NotFoundError | Requested study/dataset/resource does not exist | +| ServerError | Unexpected server-side failure | +| ValidationError | Invalid inputs or malformed/invalid response payloads | + +Example: + +```python +from Dataconnect import ( + AuthenticationError, + AuthorizationError, + DataconnectError, + NotFoundError, + ValidationError, +) + +try: + studies = client.get_studies() +except AuthenticationError as exc: + print("Authentication failed:", exc) +except AuthorizationError as exc: + print("Not authorized:", exc) +except NotFoundError as exc: + print("Resource not found:", exc) +except ValidationError as exc: + print("Invalid request:", exc) +except DataconnectError as exc: + print("Dataconnect request failed:", exc) +``` + +## Data Models + +| Model | Fields | +|---|---| +| StudyEnvironment | `uuid`, `name` | +| Study | `uuid`, `name`, `environments` | +| Dataset | `dataset_uuid`, `study_uuid`, `study_env_uuid`, `dataset_name` | +| DatasetVersion | `study_uuid`, `study_environment_uuid`, `dataset_uuid`, `dataset_name`, `dataset_version` | +| Pagination | `page`, `page_size`, `total_pages` | +| PaginatedResponse[T] | `total_records`, `pagination`, `items` | + diff --git a/vignettes/pythonLibrary_setup.md b/vignettes/pythonLibrary_setup.md new file mode 100644 index 0000000..1096479 --- /dev/null +++ b/vignettes/pythonLibrary_setup.md @@ -0,0 +1,44 @@ +# Data Connect Python Library - Setup + +Choose one of the following approaches. + +### Option 1: Install from source (recommended for this repository) + +```bash +git clone +cd Dataconnect-library-python +python3.13 -m venv .venv +source .venv/bin/activate +pip install -e . +``` + +### Option 2: Install using Poetry + +```bash +git clone +cd Dataconnect-library-python +poetry install +poetry shell +``` + +## Quick Start + +```python +from uuid import UUID + +from Dataconnect import DataconnectClient + +with DataconnectClient.connect(token="your-bearer-token") as client: + studies = client.get_studies() + print(f"Found {len(studies)} studies") + + if studies and studies[0].environments: + study_environment_uuid = studies[0].environments[0].uuid + datasets_data = client.get_datasets(study_environment_uuid=study_environment_uuid, page=1, page_size=10) + print(f"Found {datasets_data.total_records} datasets") + + if datasets_data.items: + dataset_uuid = UUID(datasets_data.items[0].dataset_uuid) + df = client.fetch_data(dataset_uuid, first_n_rows=100) + print(df.head()) +``` diff --git a/vignettes/pythonLibrary_usage.md b/vignettes/pythonLibrary_usage.md new file mode 100644 index 0000000..f57ae7a --- /dev/null +++ b/vignettes/pythonLibrary_usage.md @@ -0,0 +1,118 @@ +# DataConnect Python Usage Guide + +This guide contains runnable examples for all public `DataConnectClient` functions. + +## Prerequisites + +- Python 3.13 +- Installed package dependencies +- A valid Data Connect user token +- A valid project token + +## connect() + +```python +from dataconnect import DataConnectClient + +client = DataConnectClient.connect(token="") +try: + studies = client.get_studies() + print(f"Found {len(studies)} studies") +finally: + client.close() +``` + +## get_studies() + +```python +from dataconnect import DataConnectClient + +with DataConnectClient.connect(token="") as client: + studies = client.get_studies(search_study_name="") + for study in studies: + print(study.uuid, study.name) +``` + +## get_datasets() + +```python +from dataconnect import DataConnectClient + +with DataConnectClient.connect(token="") as client: + response = client.get_datasets( + study_environment_uuid="", + search_dataset_name="", + page=1, + page_size=25, + ) + + print(response.total_records) + for dataset in response.items: + print(dataset.dataset_uuid, dataset.dataset_name) +``` + +## get_dataset_versions() + +```python +from uuid import UUID + +from dataconnect import DataConnectClient + +with DataConnectClient.connect(token="") as client: + versions = client.get_dataset_versions(UUID("")) + for version in versions: + print(version.dataset_name, version.dataset_version) +``` + +## fetch_data() + +```python +from uuid import UUID + +from dataconnect import DataConnectClient + +with DataConnectClient.connect(token="") as client: + df = client.fetch_data(UUID(""), first_n_rows=100) + print(df.shape) + print(df.head()) +``` + +## close() + +```python +from dataconnect import DataConnectClient + +client = DataConnectClient.connect(token="") +try: + pass +finally: + client.close() +``` + +## Error Handling Example + +```python +from dataconnect import ( + AuthenticationError, + AuthorizationError, + DataConnectClient, + DataConnectError, + NotFoundError, + ValidationError, +) + +try: + with DataConnectClient.connect(token="") as client: + studies = client.get_studies() + print(studies) +except AuthenticationError as exc: + print("Authentication failed:", exc) +except AuthorizationError as exc: + print("Not authorized:", exc) +except NotFoundError as exc: + print("Resource not found:", exc) +except ValidationError as exc: + print("Invalid request:", exc) +except DataConnectError as exc: + print("DataConnect request failed:", exc) +``` From 1f961eda63fcf024227d51d1ffaeb81e3ee3f0f1 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 19 May 2026 14:55:22 -0400 Subject: [PATCH 6/9] feat: clean up --- dataconnect/readme/README-v1.0.0.md | 214 --------------------------- readme/README-v1.0.0.md | 218 ++++++++++------------------ 2 files changed, 74 insertions(+), 358 deletions(-) delete mode 100644 dataconnect/readme/README-v1.0.0.md diff --git a/dataconnect/readme/README-v1.0.0.md b/dataconnect/readme/README-v1.0.0.md deleted file mode 100644 index 5bf6f0e..0000000 --- a/dataconnect/readme/README-v1.0.0.md +++ /dev/null @@ -1,214 +0,0 @@ -# Dataconnect Python Library v1.0.0 - -The Dataconnect Python library provides a Python client for connecting to Medidata Dataconnect and retrieving relevant data programmatically. -To use this library, you must have a valid iMedidata account and access to required building blocks in the Medidata Platform. For details, see the Medidata [Knowledge Hub](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). - -## Table of Contents - -- [Environment Setup and Requirements](#environment-setup-and-requirements) - - [System Requirements](#system-requirements) - - [Authentication and Connectivity](#authentication-and-connectivity) -- [Installation](#installation) - - [Option 1: Install from source (recommended for this repository)](#option-1-install-from-source-recommended-for-this-repository) - - [Option 2: Install using Poetry](#option-2-install-using-poetry) -- [Quick Start](#quick-start) -- [Functions](#functions) - - [connect()](#connect) - - [get_studies()](#get_studies) - - [get_datasets()](#get_datasets) - - [get_dataset_versions()](#get_dataset_versions) - - [fetch_data()](#fetch_data) - - [close()](#close) -- [Errors](#errors) -- [Reporting known issues ](#reporting-known-issues) -- [Backend](#backend) -- [Versions](#versions) -- [Licensing](#licensing) - - -## Environment Setup and Requirements - -### System Requirements - -| Requirement | Version / Notes | -|-------------------|--------------------------| -| Python | 3.13 | -| Operating systems | macOS, Linux, Windows | -| Core dependencies | pyarrow 19.x, pandas 2.x | - -### Authentication and Connectivity - -* **Retrieving data:** You must have a user token to establish a connection between your Python environment and Medidata Data Connect. You can generate this token through Data Connect’s Developer Center. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). Medidata recommends that you save the token in a separate file and input it into the below initiation function. - -* **Publish data:** You must have a project token to publish a dataset from your Python environment to Medidata Data Connect. You can generate this token through Data Connect > Transformations, by creating a Custom Code project. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/generate_custom_code_projects.html). - -# Installation - -To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-r/blob/main/vignettes/dataconnect/readme/vignettes/pythonLibrary_usage.md). - -## Quick Start - -For end-to-end examples, see the [Python usage guide](../../vignettes/pythonLibrary_usage.md). - -## Functions - -The main public entry point is `DataconnectClient`. - -### connect() - -#### Description -Creates a connected client. - -#### Usage -`connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` - -#### Arguments -| Argument | Type | Description | -|---|---|---| -| host | str | Server host. Default host="enodia-gateway.platform.imedidata.com" | -| port | int | Server port. Default port="443" | -| use_tls | bool | Denotes whether to use TLS. Default use_tls = True | -| token | str | Authentication token, this is the user authentication token generated from the Developer Center in Medidata Data Connect | - -#### Output -DataconnectClient object. This enables you to interact with Medidata Data Connect data in Python environment. - ---- - -### get_studies() - -#### Description -Retrieves a list of studies where the user has permission to manage custom code projects. Use the optional study name search parameter to filter results. - -#### Usage -`get_studies(search_study_name=None)` - -#### Arguments -| Argument | Type | Description | -|---|---|---| -| search_study_name | str or None | Optional. The approximate name of the study | - -#### Output -Returns a list of studies. Each study includes `name`, `uuid`, and a list of `environments`. Each environment includes `name` and `uuid`. - ---- - -### get_datasets() - -#### Description -Retrieves datasets for a specific study environment and returns paginated results. - -#### Usage -`get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` - -#### Arguments -| Argument | Type | Description | -|---|---|---| -| study_environment_uuid | UUID | Unique iMedidata study environment identifier. You can find this in iMedidata’s Developer Info details | -| search_dataset_name | str | Optional. The approximate name of the dataset | -| page | int | Optional. Page number for paginated results. Default: 1 | -| page_size | int | Optional. Number of results per page. Default: 50 | - -#### Output -- Returns: `PaginatedResponse[Dataset]` - ---- - -### get_dataset_versions() - -#### Description -Retrieves all available versions for a dataset. - -#### Usage -`get_dataset_versions(dataset_uuid)` - -#### Arguments -| Argument | Type | Description | -|---|---|---| -| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() function | - -#### Output -- Returns: `list[DatasetVersion]` - ---- - -### fetch_data() - -#### Description -Fetches dataset rows into a pandas DataFrame. - -### Usage -`fetch_data(dataset_uuid, first_n_rows=None)` - -#### Arguments -| Argument | Type | Description | -|---|---|---| -| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() and dataset_versions() functions | -| first_n_rows | int or None | Optional positive row limit | - -#### Output -- Returns: `pandas.DataFrame` - -### close() - -Closes the underlying transport connection. - -| Item | Details | -|---|---| -| Description | Releases network resources used by the client | -| Parameters | None | -| Returns | None | -| Error handling | May raise `DataconnectError` subclasses if close fails at transport level | - -## Errors - -R Library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. - -| Error Code | Type | Scenario| -| :--- | :--- |:---| -| AUTHZ_001 | Authorization | Authorization service check failed | -| VAL_002 | Validation - Page Number | Page number is not a positive integer -| VAL_003 | Validation - Page Size | Page size is out of range [1, 100] -| VAL_004 | Validation - Study Parameter | Invalid study uuid -| VAL_005 | Validation - Study Environment Parameter | Missing or invalid study environment uuid -| VAL_006 | Validation - Dataset Parameter | Invalid dataset uuid -| VAL_007 | Validation - Configuration Error | Required input parameters are missing or invalid in configuration -| VAL_008 | Validation - Project Token | Invalid project token -| VAL_009 | Validation - Unsupported Data Type | Unsupported data types. -| VAL_010 | Validation - Unsupported Data Type | Unsupported datetime formats. -| VAL_011 | Validation - Pagination | Pagination is out of range -| VAL_012 | Validation - Concurrency | Project actively being published -| VAL_013 | Validation - Formatting Error | Data validation failed. One or more records contain formatting errors. -| RES_002 | Resource Exceptions - Study Environment | No authorized Study Environments found for the authenticated user -| RES_003 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. -| RES_004 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. -| RES_005 | Resource Exceptions - Study Group | Study Group not found for the Dataset's Study Environment. -| RES_006 | Resource Exceptions - Study | Study Group not found for the Dataset's Study Environment. -| RES_007 | Resource Exceptions - Client Division | Client Division not found for the Dataset's Study Environment. -| RES_008 | Resource Exceptions - Custom Code Project | Transformation Project is not found. -| INT_001 | Internal Application Exception | Something went wrong on our end. - -# Reporting known issues - -If you believe you have found an issue, please contact Medidata Support by submitting a ticket to Medidata Support. All issue reports should include a minimal reproducible example to ensure our team can diagnose the issue. - -Additionally, all known issues are available [here](https://learn.medidata.com/en-US/bundle/current-issues/page/current_known_issues_for_data_connect.html). - -# Backend - -This library uses the Arrow open source library and the Iceberg open table format to enable data interoperability across platforms. - -* [Apache arrow](https://arrow.apache.org/docs/r/): This library uses Arrow’s highly efficient format [pyarrow](https://arrow.apache.org/cookbook/py/flight.html) to transfer massive datasets over the network, allowing users to access & interact with remote datasets. - -* [Apache Iceberg](https://iceberg.apache.org/): This is the open table format underlying Medidata Data Connect's structured data management to support high-performance and reliable data analytics and storage. - -# Licensing - -BY DOWNLOADING THIS FILE (“DOWNLOAD”) YOU AGREE TO THE FOLLOWING TERMS: -MEDIDATA SOLUTIONS, INC. AND ITS AFFILIATES (COLLECTIVELY “MEDIDATA”) GRANT A FREE OF CHARGE, NON-EXCLUSIVE AND NON-TRANSFERABLE RIGHT TO USE THE DOWNLOAD. USE OF THIS DOWNLOAD IS PERMITTED FOR INTERNAL BUSINESS PURPOSES ONLY. - -THIS DOWNLOAD IS MADE AVAILABLE ON AN "AS IS" BASIS WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, ORAL OR WRITTEN, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. - -MEDIDATA SHALL HAVE NO LIABILITY FOR DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES, INCLUDING, WITHOUT LIMITATION, CLAIMS FOR LOST PROFITS, BUSINESS INTERRUPTION AND LOSS OF DATA THAT IN ANY WAY RELATE TO THIS DOWNLOAD, WHETHER OR NOT MEDIDATA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND NOTWITHSTANDING THE FAILURE OF THE ESSENTIAL PURPOSE OF ANY REMEDY. - -YOUR USE OF THIS DOWNLOAD SHALL BE AT YOUR SOLE RISK. NO SUPPORT OF ANY KIND OF THE DOWNLOAD IS PROVIDED BY MEDIDATA. diff --git a/readme/README-v1.0.0.md b/readme/README-v1.0.0.md index 0b85511..6dc8cb6 100644 --- a/readme/README-v1.0.0.md +++ b/readme/README-v1.0.0.md @@ -19,8 +19,11 @@ To use this library, you must have a valid iMedidata account and access to requi - [get_dataset_versions()](#get_dataset_versions) - [fetch_data()](#fetch_data) - [close()](#close) -- [Error Handling](#error-handling) -- [Data Models](#data-models) +- [Errors](#errors) +- [Reporting known issues ](#reporting-known-issues) +- [Backend](#backend) +- [Versions](#versions) +- [Licensing](#licensing) ## Environment Setup and Requirements @@ -38,12 +41,14 @@ To use this library, you must have a valid iMedidata account and access to requi * **Publish data:** You must have a project token to publish a dataset from your Python environment to Medidata Data Connect. You can generate this token through Data Connect > Transformations, by creating a Custom Code project. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/generate_custom_code_projects.html). -## Installation - # Installation To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-r/blob/main/vignettes/dataconnect/readme/vignettes/pythonLibrary_usage.md). +## Quick Start + +For end-to-end examples, see the [Usage Guide](../../vignettes/pythonLibrary_usage.md). + ## Functions The main public entry point is `DataconnectClient`. @@ -51,63 +56,39 @@ The main public entry point is `DataconnectClient`. ### connect() #### Description -Creates a connected client using the default Arrow Flight transport. +Creates a connected client. #### Usage `connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` -#### Example -```python -from Dataconnect import DataconnectClient - -client = DataconnectClient.connect(token="") -try: - studies = client.get_studies() -finally: - client.close() -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| host | str | Dataconnect host | -| port | int | Server port | -| use_tls | bool | Enable TLS | +| host | str | Server host. Default host="enodia-gateway.platform.imedidata.com" | +| port | int | Server port. Default port="443" | +| use_tls | bool | Denotes whether to use TLS. Default use_tls = True | | token | str | Authentication token, this is the user authentication token generated from the Developer Center in Medidata Data Connect | #### Output -- Returns: `DataconnectClient` - -#### Data Validation -- Raises `AuthenticationError`, `AuthorizationError`, `NotFoundError`, `ServerError`, or `ValidationError` when the server/transport returns an error. +DataconnectClient object. This enables you to interact with Medidata Data Connect data in Python environment. --- ### get_studies() #### Description -Lists studies the authenticated user can access, optionally filtered by full or partial study name. +Retrieves a list of studies where the user has permission to manage custom code projects. Use the optional study name search parameter to filter results. #### Usage `get_studies(search_study_name=None)` -#### Example -```python -studies = client.get_studies(search_study_name="") -for study in studies: - print(study.uuid, study.name) -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| search_study_name | str or None | Optional full or partial study name filter | +| search_study_name | str or None | Optional. The approximate name of the study | #### Output -- Returns: `list[Study]` - -#### Data Validation -- Raises `DataconnectError` subclasses for server/transport failures. +Returns a list of studies. Each study includes `name`, `uuid`, and a list of `environments`. Each environment includes `name` and `uuid`. --- @@ -119,68 +100,35 @@ Retrieves datasets for a specific study environment and returns paginated result #### Usage `get_datasets(study_environment_uuid, search_dataset_name="", page=1, page_size=50)` -#### Example -```python -from uuid import UUID - -response = client.get_datasets( - study_environment_uuid=UUID(""), - search_dataset_name="", - page=1, - page_size=25, -) - -print(response.total_records) -for dataset in response.items: - print(dataset.dataset_uuid, dataset.dataset_name) -``` - #### Arguments | Argument | Type | Description | |---|---|---| | study_environment_uuid | UUID | Unique iMedidata study environment identifier. You can find this in iMedidata’s Developer Info details | -| search_dataset_name | str | Full or partial dataset name filter | -| page | int | Page number for paginated results (>=1) | -| page_size | int | Number of results per page (>=1) | +| search_dataset_name | str | Optional. The approximate name of the dataset | +| page | int | Optional. Page number for paginated results. Default: 1 | +| page_size | int | Optional. Number of results per page. Default: 50 | #### Output - Returns: `PaginatedResponse[Dataset]` -#### Data Validation -- Raises `ValidationError` for invalid UUID/page/page_size. -- Raises other `DataconnectError` subclasses for service failures. - --- ### get_dataset_versions() #### Description -Retrieves all available versions for a dataset, sorted in descending version order. +Retrieves all available versions for a dataset. #### Usage `get_dataset_versions(dataset_uuid)` -#### Example -```python -from uuid import UUID - -versions = client.get_dataset_versions(UUID("")) -for version in versions: - print(version.dataset_name, version.dataset_version) -``` - #### Arguments | Argument | Type | Description | |---|---|---| -| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() and dataset_versions() functions | +| dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() function | #### Output - Returns: `list[DatasetVersion]` -#### Data Validation -- Raises `ValidationError` for invalid UUID. -- Raises other `DataconnectError` subclasses for service failures. - --- ### fetch_data() @@ -191,14 +139,6 @@ Fetches dataset rows into a pandas DataFrame. ### Usage `fetch_data(dataset_uuid, first_n_rows=None)` -#### Example -```python -from uuid import UUID - -df = client.fetch_data(UUID(""), first_n_rows=100) -print(df.shape) -``` - #### Arguments | Argument | Type | Description | |---|---|---| @@ -208,10 +148,6 @@ print(df.shape) #### Output - Returns: `pandas.DataFrame` -#### Data Validation -- Raises `ValidationError` for invalid `dataset_uuid` or non-positive `first_n_rows`. -- Raises other `DataconnectError` subclasses for service failures. - ### close() Closes the underlying transport connection. @@ -223,61 +159,55 @@ Closes the underlying transport connection. | Returns | None | | Error handling | May raise `DataconnectError` subclasses if close fails at transport level | -Example example: - -```python -client = DataconnectClient.connect(token="") -try: - pass -finally: - client.close() -``` - -## Error Handling - -All public errors inherit from `DataconnectError`. - -| Exception | Typical meaning | -|---|---| -| AuthenticationError | Invalid or missing credentials/token | -| AuthorizationError | Authenticated but not allowed to access requested resource | -| NotFoundError | Requested study/dataset/resource does not exist | -| ServerError | Unexpected server-side failure | -| ValidationError | Invalid inputs or malformed/invalid response payloads | - -Example: - -```python -from Dataconnect import ( - AuthenticationError, - AuthorizationError, - DataconnectError, - NotFoundError, - ValidationError, -) - -try: - studies = client.get_studies() -except AuthenticationError as exc: - print("Authentication failed:", exc) -except AuthorizationError as exc: - print("Not authorized:", exc) -except NotFoundError as exc: - print("Resource not found:", exc) -except ValidationError as exc: - print("Invalid request:", exc) -except DataconnectError as exc: - print("Dataconnect request failed:", exc) -``` - -## Data Models - -| Model | Fields | -|---|---| -| StudyEnvironment | `uuid`, `name` | -| Study | `uuid`, `name`, `environments` | -| Dataset | `dataset_uuid`, `study_uuid`, `study_env_uuid`, `dataset_name` | -| DatasetVersion | `study_uuid`, `study_environment_uuid`, `dataset_uuid`, `dataset_name`, `dataset_version` | -| Pagination | `page`, `page_size`, `total_pages` | -| PaginatedResponse[T] | `total_records`, `pagination`, `items` | - +## Errors + +R Library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. + +| Error Code | Type | Scenario| +| :--- | :--- |:---| +| AUTHZ_001 | Authorization | Authorization service check failed | +| VAL_002 | Validation - Page Number | Page number is not a positive integer +| VAL_003 | Validation - Page Size | Page size is out of range [1, 100] +| VAL_004 | Validation - Study Parameter | Invalid study uuid +| VAL_005 | Validation - Study Environment Parameter | Missing or invalid study environment uuid +| VAL_006 | Validation - Dataset Parameter | Invalid dataset uuid +| VAL_007 | Validation - Configuration Error | Required input parameters are missing or invalid in configuration +| VAL_008 | Validation - Project Token | Invalid project token +| VAL_009 | Validation - Unsupported Data Type | Unsupported data types. +| VAL_010 | Validation - Unsupported Data Type | Unsupported datetime formats. +| VAL_011 | Validation - Pagination | Pagination is out of range +| VAL_012 | Validation - Concurrency | Project actively being published +| VAL_013 | Validation - Formatting Error | Data validation failed. One or more records contain formatting errors. +| RES_002 | Resource Exceptions - Study Environment | No authorized Study Environments found for the authenticated user +| RES_003 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. +| RES_004 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. +| RES_005 | Resource Exceptions - Study Group | Study Group not found for the Dataset's Study Environment. +| RES_006 | Resource Exceptions - Study | Study Group not found for the Dataset's Study Environment. +| RES_007 | Resource Exceptions - Client Division | Client Division not found for the Dataset's Study Environment. +| RES_008 | Resource Exceptions - Custom Code Project | Transformation Project is not found. +| INT_001 | Internal Application Exception | Something went wrong on our end. + +# Reporting known issues + +If you believe you have found an issue, please contact Medidata Support by submitting a ticket to Medidata Support. All issue reports should include a minimal reproducible example to ensure our team can diagnose the issue. + +Additionally, all known issues are available [here](https://learn.medidata.com/en-US/bundle/current-issues/page/current_known_issues_for_data_connect.html). + +# Backend + +This library uses the Arrow open source library and the Iceberg open table format to enable data interoperability across platforms. + +* [Apache arrow](https://arrow.apache.org/docs/r/): This library uses Arrow’s highly efficient format [pyarrow](https://arrow.apache.org/cookbook/py/flight.html) to transfer massive datasets over the network, allowing users to access & interact with remote datasets. + +* [Apache Iceberg](https://iceberg.apache.org/): This is the open table format underlying Medidata Data Connect's structured data management to support high-performance and reliable data analytics and storage. + +# Licensing + +BY DOWNLOADING THIS FILE (“DOWNLOAD”) YOU AGREE TO THE FOLLOWING TERMS: +MEDIDATA SOLUTIONS, INC. AND ITS AFFILIATES (COLLECTIVELY “MEDIDATA”) GRANT A FREE OF CHARGE, NON-EXCLUSIVE AND NON-TRANSFERABLE RIGHT TO USE THE DOWNLOAD. USE OF THIS DOWNLOAD IS PERMITTED FOR INTERNAL BUSINESS PURPOSES ONLY. + +THIS DOWNLOAD IS MADE AVAILABLE ON AN "AS IS" BASIS WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, ORAL OR WRITTEN, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. + +MEDIDATA SHALL HAVE NO LIABILITY FOR DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES, INCLUDING, WITHOUT LIMITATION, CLAIMS FOR LOST PROFITS, BUSINESS INTERRUPTION AND LOSS OF DATA THAT IN ANY WAY RELATE TO THIS DOWNLOAD, WHETHER OR NOT MEDIDATA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND NOTWITHSTANDING THE FAILURE OF THE ESSENTIAL PURPOSE OF ANY REMEDY. + +YOUR USE OF THIS DOWNLOAD SHALL BE AT YOUR SOLE RISK. NO SUPPORT OF ANY KIND OF THE DOWNLOAD IS PROVIDED BY MEDIDATA. From 97a363f834e1dbd63c42679c828d469e07b381c2 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Thu, 21 May 2026 15:48:37 -0400 Subject: [PATCH 7/9] feat: updated Output section --- readme/README-v1.0.0.md | 60 +++++++--------- vignettes/pythonLibrary_usage.md | 118 ------------------------------- 2 files changed, 27 insertions(+), 151 deletions(-) delete mode 100644 vignettes/pythonLibrary_usage.md diff --git a/readme/README-v1.0.0.md b/readme/README-v1.0.0.md index 6dc8cb6..b02cbca 100644 --- a/readme/README-v1.0.0.md +++ b/readme/README-v1.0.0.md @@ -1,6 +1,6 @@ -# Dataconnect Python Library v1.0.0 +# DataConnect Python Library v1.0.0 -The Dataconnect Python library provides a Python client for connecting to Medidata Dataconnect and retrieving relevant data programmatically. +The DataConnect Python library provides a Python client for connecting to Medidata DataConnect and retrieving relevant data programmatically. To use this library, you must have a valid iMedidata account and access to required building blocks in the Medidata Platform. For details, see the Medidata [Knowledge Hub](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). ## Table of Contents @@ -25,29 +25,19 @@ To use this library, you must have a valid iMedidata account and access to requi - [Versions](#versions) - [Licensing](#licensing) -## Environment Setup and Requirements +## Installation -### System Requirements +To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-r/blob/main/vignettes/dataconnect/readme/vignettes/pythonLibrary_setup.md). -| Requirement | Version / Notes | -|-------------------|--------------------------| -| Python | 3.13 | -| Operating systems | macOS, Linux, Windows | -| Core dependencies | pyarrow 19.x, pandas 2.x | - -### Authentication and Connectivity - -* **Retrieving data:** You must have a user token to establish a connection between your Python environment and Medidata Data Connect. You can generate this token through Data Connect’s Developer Center. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). Medidata recommends that you save the token in a separate file and input it into the below initiation function. - -* **Publish data:** You must have a project token to publish a dataset from your Python environment to Medidata Data Connect. You can generate this token through Data Connect > Transformations, by creating a Custom Code project. For details, see [here](https://learn.medidata.com/en-US/bundle/data-connect/page/generate_custom_code_projects.html). +## Quick Start -# Installation +For end-to-end examples, see the [Usage Guide](). -To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-r/blob/main/vignettes/dataconnect/readme/vignettes/pythonLibrary_usage.md). +### Authentication and Connectivity -## Quick Start +* **Retrieving data:** You must have a user token to establish a connection between your Python environment and Medidata Data Connect. You can generate this token through Data Connect’s Developer Center. For details, visit the [knowledge hub](https://learn.medidata.com/en-US/bundle/data-connect/page/developer_center.html). Medidata recommends that you save the token in a local file and input it into the below initiation function. -For end-to-end examples, see the [Usage Guide](../../vignettes/pythonLibrary_usage.md). +* **Publish data:** You must have a project token to publish a dataset from your Python environment to Medidata Data Connect. You can generate this token through Data Connect > Transformations, by creating a Custom Code project. For details, visit the [knowledge hub](https://learn.medidata.com/en-US/bundle/data-connect/page/generate_custom_code_projects.html). ## Functions @@ -59,7 +49,7 @@ The main public entry point is `DataconnectClient`. Creates a connected client. #### Usage -`connect(host="enodia-gateway.platform.imedidata.com", port=443, use_tls=True, token="")` +`connect(token="")` #### Arguments | Argument | Type | Description | @@ -88,7 +78,7 @@ Retrieves a list of studies where the user has permission to manage custom code | search_study_name | str or None | Optional. The approximate name of the study | #### Output -Returns a list of studies. Each study includes `name`, `uuid`, and a list of `environments`. Each environment includes `name` and `uuid`. +Returns a list containing `total_records` (total studies available) and a `studies` array. Each study includes `name`, `uuid`, and an environments array. Each environment includes `name` and `uuid`. --- @@ -109,7 +99,7 @@ Retrieves datasets for a specific study environment and returns paginated result | page_size | int | Optional. Number of results per page. Default: 50 | #### Output -- Returns: `PaginatedResponse[Dataset]` +Returns a list containing `total_records` (total datasets available across all pages), `pagination` and `datasets` array. --- @@ -127,7 +117,7 @@ Retrieves all available versions for a dataset. | dataset_uuid | UUID | Unique iMedidata dataset identifier. This is available in the output of datasets() function | #### Output -- Returns: `list[DatasetVersion]` +Returns all available versions of the dataset. --- @@ -136,7 +126,7 @@ Retrieves all available versions for a dataset. #### Description Fetches dataset rows into a pandas DataFrame. -### Usage +#### Usage `fetch_data(dataset_uuid, first_n_rows=None)` #### Arguments @@ -146,22 +136,26 @@ Fetches dataset rows into a pandas DataFrame. | first_n_rows | int or None | Optional positive row limit | #### Output -- Returns: `pandas.DataFrame` +Returns data from a specific dataset. + +--- ### close() +#### Description Closes the underlying transport connection. -| Item | Details | -|---|---| -| Description | Releases network resources used by the client | -| Parameters | None | -| Returns | None | -| Error handling | May raise `DataconnectError` subclasses if close fails at transport level | +#### Usage +`close()` -## Errors +#### Arguments +None + +#### Output +None -R Library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. +## Errors +The library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. | Error Code | Type | Scenario| | :--- | :--- |:---| diff --git a/vignettes/pythonLibrary_usage.md b/vignettes/pythonLibrary_usage.md deleted file mode 100644 index f57ae7a..0000000 --- a/vignettes/pythonLibrary_usage.md +++ /dev/null @@ -1,118 +0,0 @@ -# DataConnect Python Usage Guide - -This guide contains runnable examples for all public `DataConnectClient` functions. - -## Prerequisites - -- Python 3.13 -- Installed package dependencies -- A valid Data Connect user token -- A valid project token - -## connect() - -```python -from dataconnect import DataConnectClient - -client = DataConnectClient.connect(token="") -try: - studies = client.get_studies() - print(f"Found {len(studies)} studies") -finally: - client.close() -``` - -## get_studies() - -```python -from dataconnect import DataConnectClient - -with DataConnectClient.connect(token="") as client: - studies = client.get_studies(search_study_name="") - for study in studies: - print(study.uuid, study.name) -``` - -## get_datasets() - -```python -from dataconnect import DataConnectClient - -with DataConnectClient.connect(token="") as client: - response = client.get_datasets( - study_environment_uuid="", - search_dataset_name="", - page=1, - page_size=25, - ) - - print(response.total_records) - for dataset in response.items: - print(dataset.dataset_uuid, dataset.dataset_name) -``` - -## get_dataset_versions() - -```python -from uuid import UUID - -from dataconnect import DataConnectClient - -with DataConnectClient.connect(token="") as client: - versions = client.get_dataset_versions(UUID("")) - for version in versions: - print(version.dataset_name, version.dataset_version) -``` - -## fetch_data() - -```python -from uuid import UUID - -from dataconnect import DataConnectClient - -with DataConnectClient.connect(token="") as client: - df = client.fetch_data(UUID(""), first_n_rows=100) - print(df.shape) - print(df.head()) -``` - -## close() - -```python -from dataconnect import DataConnectClient - -client = DataConnectClient.connect(token="") -try: - pass -finally: - client.close() -``` - -## Error Handling Example - -```python -from dataconnect import ( - AuthenticationError, - AuthorizationError, - DataConnectClient, - DataConnectError, - NotFoundError, - ValidationError, -) - -try: - with DataConnectClient.connect(token="") as client: - studies = client.get_studies() - print(studies) -except AuthenticationError as exc: - print("Authentication failed:", exc) -except AuthorizationError as exc: - print("Not authorized:", exc) -except NotFoundError as exc: - print("Resource not found:", exc) -except ValidationError as exc: - print("Invalid request:", exc) -except DataConnectError as exc: - print("DataConnect request failed:", exc) -``` From 3f97b564ade9fd774b27745521e50e6ad98cf3d2 Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Thu, 21 May 2026 16:09:54 -0400 Subject: [PATCH 8/9] fix: Remove trailing whitespace --- readme/README-v1.0.0.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/readme/README-v1.0.0.md b/readme/README-v1.0.0.md index b02cbca..0582ef6 100644 --- a/readme/README-v1.0.0.md +++ b/readme/README-v1.0.0.md @@ -155,30 +155,30 @@ None None ## Errors -The library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. +The library raises exceptions for many reasons, such as invalid parameters, authentication errors, and validation failures. We have introduced error codes for each category of errors to be handled programmatically. | Error Code | Type | Scenario| | :--- | :--- |:---| -| AUTHZ_001 | Authorization | Authorization service check failed | -| VAL_002 | Validation - Page Number | Page number is not a positive integer -| VAL_003 | Validation - Page Size | Page size is out of range [1, 100] +| AUTHZ_001 | Authorization | Authorization service check failed | +| VAL_002 | Validation - Page Number | Page number is not a positive integer +| VAL_003 | Validation - Page Size | Page size is out of range [1, 100] | VAL_004 | Validation - Study Parameter | Invalid study uuid -| VAL_005 | Validation - Study Environment Parameter | Missing or invalid study environment uuid +| VAL_005 | Validation - Study Environment Parameter | Missing or invalid study environment uuid | VAL_006 | Validation - Dataset Parameter | Invalid dataset uuid -| VAL_007 | Validation - Configuration Error | Required input parameters are missing or invalid in configuration -| VAL_008 | Validation - Project Token | Invalid project token +| VAL_007 | Validation - Configuration Error | Required input parameters are missing or invalid in configuration +| VAL_008 | Validation - Project Token | Invalid project token | VAL_009 | Validation - Unsupported Data Type | Unsupported data types. | VAL_010 | Validation - Unsupported Data Type | Unsupported datetime formats. | VAL_011 | Validation - Pagination | Pagination is out of range | VAL_012 | Validation - Concurrency | Project actively being published | VAL_013 | Validation - Formatting Error | Data validation failed. One or more records contain formatting errors. | RES_002 | Resource Exceptions - Study Environment | No authorized Study Environments found for the authenticated user -| RES_003 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. +| RES_003 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. | RES_004 | Resource Exceptions - Invalid parameter | Incorrect UUID combination. -| RES_005 | Resource Exceptions - Study Group | Study Group not found for the Dataset's Study Environment. +| RES_005 | Resource Exceptions - Study Group | Study Group not found for the Dataset's Study Environment. | RES_006 | Resource Exceptions - Study | Study Group not found for the Dataset's Study Environment. | RES_007 | Resource Exceptions - Client Division | Client Division not found for the Dataset's Study Environment. -| RES_008 | Resource Exceptions - Custom Code Project | Transformation Project is not found. +| RES_008 | Resource Exceptions - Custom Code Project | Transformation Project is not found. | INT_001 | Internal Application Exception | Something went wrong on our end. # Reporting known issues @@ -191,17 +191,17 @@ Additionally, all known issues are available [here](https://learn.medidata.com/e This library uses the Arrow open source library and the Iceberg open table format to enable data interoperability across platforms. -* [Apache arrow](https://arrow.apache.org/docs/r/): This library uses Arrow’s highly efficient format [pyarrow](https://arrow.apache.org/cookbook/py/flight.html) to transfer massive datasets over the network, allowing users to access & interact with remote datasets. - +* [Apache arrow](https://arrow.apache.org/docs/r/): This library uses Arrow’s highly efficient format [pyarrow](https://arrow.apache.org/cookbook/py/flight.html) to transfer massive datasets over the network, allowing users to access & interact with remote datasets. + * [Apache Iceberg](https://iceberg.apache.org/): This is the open table format underlying Medidata Data Connect's structured data management to support high-performance and reliable data analytics and storage. # Licensing -BY DOWNLOADING THIS FILE (“DOWNLOAD”) YOU AGREE TO THE FOLLOWING TERMS: -MEDIDATA SOLUTIONS, INC. AND ITS AFFILIATES (COLLECTIVELY “MEDIDATA”) GRANT A FREE OF CHARGE, NON-EXCLUSIVE AND NON-TRANSFERABLE RIGHT TO USE THE DOWNLOAD. USE OF THIS DOWNLOAD IS PERMITTED FOR INTERNAL BUSINESS PURPOSES ONLY. - -THIS DOWNLOAD IS MADE AVAILABLE ON AN "AS IS" BASIS WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, ORAL OR WRITTEN, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. - -MEDIDATA SHALL HAVE NO LIABILITY FOR DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES, INCLUDING, WITHOUT LIMITATION, CLAIMS FOR LOST PROFITS, BUSINESS INTERRUPTION AND LOSS OF DATA THAT IN ANY WAY RELATE TO THIS DOWNLOAD, WHETHER OR NOT MEDIDATA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND NOTWITHSTANDING THE FAILURE OF THE ESSENTIAL PURPOSE OF ANY REMEDY. - +BY DOWNLOADING THIS FILE (“DOWNLOAD”) YOU AGREE TO THE FOLLOWING TERMS: +MEDIDATA SOLUTIONS, INC. AND ITS AFFILIATES (COLLECTIVELY “MEDIDATA”) GRANT A FREE OF CHARGE, NON-EXCLUSIVE AND NON-TRANSFERABLE RIGHT TO USE THE DOWNLOAD. USE OF THIS DOWNLOAD IS PERMITTED FOR INTERNAL BUSINESS PURPOSES ONLY. + +THIS DOWNLOAD IS MADE AVAILABLE ON AN "AS IS" BASIS WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, ORAL OR WRITTEN, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. + +MEDIDATA SHALL HAVE NO LIABILITY FOR DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES, INCLUDING, WITHOUT LIMITATION, CLAIMS FOR LOST PROFITS, BUSINESS INTERRUPTION AND LOSS OF DATA THAT IN ANY WAY RELATE TO THIS DOWNLOAD, WHETHER OR NOT MEDIDATA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND NOTWITHSTANDING THE FAILURE OF THE ESSENTIAL PURPOSE OF ANY REMEDY. + YOUR USE OF THIS DOWNLOAD SHALL BE AT YOUR SOLE RISK. NO SUPPORT OF ANY KIND OF THE DOWNLOAD IS PROVIDED BY MEDIDATA. From dacf28442efa7b714d6d52538b7a9de7452a6c7e Mon Sep 17 00:00:00 2001 From: Nastassia Makarava Date: Tue, 26 May 2026 09:48:40 -0400 Subject: [PATCH 9/9] Fixes per feedbacks --- README.md | 2 +- readme/README-v1.0.0.md | 9 ++------- vignettes/pythonLibrary_setup.md | 4 ++-- 3 files changed, 5 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 5689688..f3da6ef 100644 --- a/README.md +++ b/README.md @@ -14,4 +14,4 @@ This directory contains documentation snapshots for each release of the `datacon | Version | Documentation | Current version | |---------|---------------------------------------------|-----------------| -|1.1.0 | [README-v1.0.0.md](readme/README-v1.0.0.md) | Yes | +|1.0.0 | [README-v1.0.0.md](readme/README-v1.0.0.md) | Yes | diff --git a/readme/README-v1.0.0.md b/readme/README-v1.0.0.md index 0582ef6..f9c3431 100644 --- a/readme/README-v1.0.0.md +++ b/readme/README-v1.0.0.md @@ -5,13 +5,9 @@ To use this library, you must have a valid iMedidata account and access to requi ## Table of Contents -- [Environment Setup and Requirements](#environment-setup-and-requirements) - - [System Requirements](#system-requirements) - - [Authentication and Connectivity](#authentication-and-connectivity) - [Installation](#installation) - - [Option 1: Install from source (recommended for this repository)](#option-1-install-from-source-recommended-for-this-repository) - - [Option 2: Install using Poetry](#option-2-install-using-poetry) - [Quick Start](#quick-start) +- [Authentication and Connectivity](#authentication-and-connectivity) - [Functions](#functions) - [connect()](#connect) - [get_studies()](#get_studies) @@ -20,9 +16,8 @@ To use this library, you must have a valid iMedidata account and access to requi - [fetch_data()](#fetch_data) - [close()](#close) - [Errors](#errors) -- [Reporting known issues ](#reporting-known-issues) +- [Reporting known issues](#reporting-known-issues) - [Backend](#backend) -- [Versions](#versions) - [Licensing](#licensing) ## Installation diff --git a/vignettes/pythonLibrary_setup.md b/vignettes/pythonLibrary_setup.md index 1096479..2a8fd29 100644 --- a/vignettes/pythonLibrary_setup.md +++ b/vignettes/pythonLibrary_setup.md @@ -6,7 +6,7 @@ Choose one of the following approaches. ```bash git clone -cd Dataconnect-library-python +cd dataconnect-library-python python3.13 -m venv .venv source .venv/bin/activate pip install -e . @@ -16,7 +16,7 @@ pip install -e . ```bash git clone -cd Dataconnect-library-python +cd dataconnect-library-python poetry install poetry shell ```