diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2af914f..dff9d28 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,10 +15,10 @@ jobs: runs-on: [ self-hosted, ubuntu-latest ] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Set up Python 3.11 - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.11" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index faef645..9c1993b 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -23,14 +23,14 @@ jobs: steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: fetch-depth: 0 fetch-tags: true token: ${{ secrets.GITHUB_TOKEN }} - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: '3.13' diff --git a/guides/dataconnect_quickstart.md b/guides/dataconnect_quickstart.md deleted file mode 100644 index 0d9f095..0000000 --- a/guides/dataconnect_quickstart.md +++ /dev/null @@ -1,29 +0,0 @@ -# DataConnect Python Library - Quick Start - -Instructions in this document apply only once all the steps of the [Setup document](dataconnect_setup.md) are followed, and you have the [Jupyter notebook](dataconnect_usage.ipynb) opened in your IDE where the Data Connect Python Library package was installed. - -*Note:* The `User Authentication Token` used to connect with DataConnect and make function calls can be generated from `iMedidata` > `Data Connect` > `Developer Center`. - -## Jupyter -* Make sure the Jupyter notebook being used points to the correct Python Virtual Environment. You can configure that by clicking on `Select Kernel` on the top right, and pick the `venv` that has `Python3.13` configured. -* In the Jupyter notebook, under *Preparation*, enter the user token from `Data Connect Developer Center`. -* Run all the code-cells until **Get all available studies** - * Feel free to enter a `search_study_name` wildcard value - * Confirm that the `get_studies()` call works. -* Continue running other code-cells in the notebook as desired. - -## Stand-alone code -* You may write your own Python files to access the Data Connect Python Library, but they must be in the same directory. -* Sample code: - - ```python - from uuid import UUID - from dataconnect import DataConnectClient - - with DataConnectClient.connect( - token="user-token-from-dataconnect", - ) as client: - result = client.get_studies(search_study_name="clin") - print(result.total_records) # total number of studies accessible to the user - print(result.studies) # list of Study objects - ``` diff --git a/guides/dataconnect_setup.md b/guides/dataconnect_setup.md deleted file mode 100644 index 903b524..0000000 --- a/guides/dataconnect_setup.md +++ /dev/null @@ -1,46 +0,0 @@ -# DataConnect Python Library Setup - -This document is intended for first-time end-users. - -## Prerequisites -### Environment -* Python 3.13 - * Should automatically include `pip` and `venv` (Python Virtual Environment) -* IDE of choice - `Visual Studio Code`, `PyCharm` etc. with `Jupyter` plugin - -### Credentials -* An iMedidata Account -* Access to **DataConnect**'s **Developer Center** and **Transformations** in iMedidata - -*Note:* You will need a generated `User Token` from **Developer Center** to make any function calls in this library. - -## Setup -* On your Terminal window of choice (`bash`, `zsh`, `iTerm`, `WSL`, `gitBash` etc), create a new directory and go to it. - - ```bash - mkdir dataconnect && cd $_ - ``` - -* Create a Python Virtual Environment and Activate it. Depending on your setup, you may use the `python` command instead of `python3`. - - ```bash - python3 -m venv ./.venv - source ./.venv/bin/activate - ``` - - * To confirm that the virtual environment has been created and activated, simply enter `which python3` (or `which python`) and it should point to the `dataconnect/.venv/bin` path. If not, run the above commands again. - - -* Run the following command to fetch the latest-released `dataconnect-library-python` package. In this example, version `1.0.0` is assumed to be the latest release: - - ```bash - pip install git+https://github.com/mdsol/dataconnect-library-python.git@1.0.0 - ``` - -* If there are no errors in fetching the package, open the directory in your IDE. For example, run this for VS Code: - - ```bash - code . - ``` - -* Download the usage jupyter file and open it in the same IDE window. diff --git a/guides/dataconnect_usage.ipynb b/guides/dataconnect_usage.ipynb index c32c975..fd6cda6 100644 --- a/guides/dataconnect_usage.ipynb +++ b/guides/dataconnect_usage.ipynb @@ -8,11 +8,10 @@ "# Data Connect Python Library - Usage\n", "\n", "***Version Note:***\n", - "This notebook reflects the *latest version* of the `dataconnect` Python library.\n", - "If you are using an older version, the functions or parameters described here may differ from your installed version.\n", + "This notebook corresponds to the **dataconnect-library-python** `v1.0.0`. Ensure that you are working with this specific version. \n", "\n", "***View version-matched documentation:***\n", - "On GitHub, use the branch/tag selector to switch to your version's tag (e.g., `v0.1.0`)." + "On GitHub, use the branch/tag selector to switch to your version's tag (e.g., `v1.0.0`)." ] }, { @@ -20,64 +19,42 @@ "id": "491849ae", "metadata": {}, "source": [ - "### Setup\n", - "Refer to the [Setup](dataconnect_setup.md) and [Quick Start](dataconnect_quickstart.md) documents first." - ] - }, - { - "cell_type": "markdown", - "id": "64e8d2c2", - "metadata": {}, - "source": [ - "### Preparation" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "fe8bd656", - "metadata": {}, - "outputs": [], - "source": [ - "# We recommend storing user_token in a separate file or environment variable.\n", - "# This is the user authentication token generated from the Developer Center in iMedidata Platform.\n", + "## Setup\n", "\n", - "user_token = \"usertoken\"" + "**Prerequisites**\n", + "* Python 3.13\n", + "* pip3 (pip for Windows)\n", + "\n", + "**Credentials**\n", + "* An iMedidata Account\n", + "* Access to **DataConnect**'s **Developer Center** and **Transformations** in iMedidata\n", + "\n", + "*Note:* You will need a generated `User Authentication Token` from **Developer Center** to use this library." ] }, { "cell_type": "code", - "execution_count": 2, - "id": "40152cc2", + "execution_count": null, + "id": "5dabdeac", "metadata": {}, "outputs": [], "source": [ - "# Load prerequisite libraries\n", - "from uuid import UUID\n", - "\n", - "import pandas as pd" + "%pip install git+https://github.com/mdsol/dataconnect-library-python.git@v1.0.0" ] }, { "cell_type": "markdown", - "id": "62e5659ef27811b7", + "id": "3155af56", "metadata": {}, "source": [ - "### Initialize the connection to Data Connect\n", + "## Usage\n", + "It is recommended that the `User Authentication Token` generated from `Developer Center` is stored in a separate secrets file. When using functions without context, the connection has to be explicitly closed. Alternatively, a context manager (`with` statement, for example) can be used to ensure proper resource management.\n", "\n", - "_Note_: Do remember to close this connection after your operations, or use a context manager (`with` statement) to ensure proper resource management." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "c60b75ab2e67426b", - "metadata": {}, - "outputs": [], - "source": [ - "from dataconnect import DataConnectClient\n", + "*Note:* By default, the `connect()` function assumes we're working with iMedidata production. If you need to connect to the `Innovate` (pre-production) stage for any reason, add an additional `host` parameter to the `DataConnectClient.connect(...)` call as:\n", "\n", - "dataconnect_client = DataConnectClient.connect(token=user_token)" + "```python\n", + "DataConnectClient.connect(host=\"enodia-gateway-innovate.platform.imedidata.com\", token=user_token)\n", + "```" ] }, { @@ -87,7 +64,7 @@ "source": [ "### Get all available studies\n", "\n", - "Get all available studies or search for a study by name." + "Get all available studies or search for a study by name (without using context)." ] }, { @@ -97,6 +74,12 @@ "metadata": {}, "outputs": [], "source": [ + "from dataconnect import DataConnectClient\n", + "\n", + "user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", + "\n", + "dataconnect_client = DataConnectClient.connect(token=user_token)\n", + "\n", "try:\n", " study_result = dataconnect_client.get_studies(search_study_name=\"\")\n", "\n", @@ -104,132 +87,112 @@ " print(f\"\\n--- Total Studies: {study_result.total_records} ---\\n\")\n", "\n", " for study in study_result.studies:\n", - " envs_list = \", \".join(env.name for env in study.environments if env.name)\n", " print(f\"• Study: {study.name}\")\n", - " if envs_list:\n", - " print(f\" Envs: {envs_list}\\n\")\n", - " else:\n", - " print(\" Envs: (none)\\n\")\n", + " print(\" Envs: \")\n", + "\n", + " if study.environments:\n", + " for env in study.environments:\n", + " print(f\" - {env.uuid} : {env.name}\")\n", + " print(\"\\n\")\n", " else:\n", " print(\"No Studies found.\")\n", "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "07ac39b7", - "metadata": {}, - "source": [ - "### Work with datasets with `dataconnect` library\n", - "\n", - "#### Search datasets in a study environment\n", - "\n", - "Retrieve the study and study environment information from \"Developer info\" in iMedidata Platform." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9fd535ac", - "metadata": {}, - "outputs": [], - "source": [ - "pwb_study = {\n", - " \"study_id\": \"studyid\",\n", - " \"prod_env_id\": \"studyenvironmentid\",\n", - "}\n", - "\n", - "exciter_study = {\n", - " \"study_id\": \"studyid\",\n", - " \"prod_env_id\": \"studyenvironmentid\",\n", - "}" - ] - }, - { - "cell_type": "markdown", - "id": "b357b826", - "metadata": {}, - "source": [ - "Set search keyword." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f1de9ddb", - "metadata": {}, - "outputs": [], - "source": [ - "search_name = \"datasetname\"" + " print(e)\n", + "finally:\n", + " dataconnect_client.close()" ] }, { "cell_type": "markdown", - "id": "f900625f", + "id": "6a49e034", "metadata": {}, "source": [ - "Search for a dataset in a specific study environment." + "Get all available studies or search for a study by name (using `with` for context management)." ] }, { "cell_type": "code", "execution_count": null, - "id": "25c51a7d", + "id": "63c6a71d", "metadata": {}, "outputs": [], "source": [ + "from dataconnect import DataConnectClient\n", + "\n", + "user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", + "\n", "try:\n", - " datasets = dataconnect_client.get_datasets(\n", - " study_environment_uuid=UUID(exciter_study[\"prod_env_id\"]),\n", - " search_dataset_name=search_name,\n", - " page=1,\n", - " page_size=5,\n", - " )\n", - "\n", - " print(f\"Total datasets available across all pages: {datasets.total_records}\")\n", - " print(\n", - " f\"Page: {datasets.pagination.page}, \"\n", - " f\"Page size: {datasets.pagination.page_size}, \"\n", - " f\"Total pages: {datasets.pagination.total_pages}\"\n", - " )\n", - "\n", - " for dataset in datasets.items:\n", - " print(f\"\\n Dataset: {dataset.dataset_name}\")\n", - " print(f\" UUID: {dataset.dataset_uuid}\")\n", + " with DataConnectClient.connect(token=user_token) as dataconnect_client:\n", + " study_result = dataconnect_client.get_studies(search_study_name=\"\")\n", + "\n", + " if study_result:\n", + " print(f\"\\n--- Total Studies: {study_result.total_records} ---\\n\")\n", + "\n", + " for study in study_result.studies:\n", + " print(f\"• Study: {study.name}\")\n", + " print(\" Envs: \")\n", "\n", + " if study.environments:\n", + " for env in study.environments:\n", + " print(f\" - {env.uuid} : {env.name}\")\n", + " print(\"\\n\")\n", + "\n", + " else:\n", + " print(\"No Studies found.\")\n", "except Exception as e:\n", - " print(f\"Error: {e}\")" + " print(e)" ] }, { "cell_type": "markdown", - "id": "7a1b4890", + "id": "07ac39b7", "metadata": {}, "source": [ - "#### Identify different versions of a specific dataset\n", + "## Work with datasets\n", + "\n", + "### Search datasets in a study environment\n", "\n", - "Prepare a list of datasets that are needed." + "Retrieve the study environment uuid from \"Developer Info\" in iMedidata Platform." ] }, { "cell_type": "code", "execution_count": null, - "id": "4727acb6", + "id": "25c51a7d", "metadata": {}, "outputs": [], "source": [ - "# dataset_uuid values can be retrieved from get_datasets()\n", - "dataset_ids = {\n", - " \"pwb_der_id\": \"datasetid\",\n", - " \"pwb_import_id\": \"datasetid\",\n", - " \"pwb_rave_id\": \"datasetid\",\n", - " \"pwb_custom_id\": \"datasetid\",\n", - " \"exciter_der_id\": \"datasetid\",\n", - " \"exciter_import_id\": \"datasetid\",\n", - " \"exciter_rave_id\": \"datasetid\",\n", - " \"exciter_custom_id\": \"datasetid\",\n", - "}" + "from uuid import UUID\n", + "\n", + "from dataconnect import DataConnectClient\n", + "\n", + "try:\n", + " user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", + " search_name = \"datasetname\" # complete dataset name or wildcard\n", + "\n", + " with DataConnectClient.connect(token=user_token) as dataconnect_client:\n", + " study_env_uuid = UUID(\"\")\n", + "\n", + " datasets = dataconnect_client.get_datasets(\n", + " study_environment_uuid=study_env_uuid,\n", + " search_dataset_name=search_name,\n", + " page=1,\n", + " page_size=5,\n", + " )\n", + "\n", + " print(f\"Total datasets available across all pages: {datasets.total_records}\")\n", + " print(\n", + " f\"Page: {datasets.pagination.page}, \"\n", + " f\"Page size: {datasets.pagination.page_size}, \"\n", + " f\"Total pages: {datasets.pagination.total_pages}\"\n", + " )\n", + "\n", + " for dataset in datasets.items:\n", + " print(f\"\\n Dataset: {dataset.dataset_name}\")\n", + " print(f\" UUID: {dataset.dataset_uuid}\")\n", + "\n", + "except Exception as e:\n", + " print(e)" ] }, { @@ -237,7 +200,7 @@ "id": "7985f2b9", "metadata": {}, "source": [ - "Retrieve other dataset versions if they are available." + "### Retrieve all dataset versions." ] }, { @@ -247,378 +210,213 @@ "metadata": {}, "outputs": [], "source": [ - "try:\n", - " dataset_versions = dataconnect_client.get_dataset_versions(dataset_uuid=UUID(dataset_ids[\"exciter_custom_id\"]))\n", + "from uuid import UUID\n", "\n", - " for version in dataset_versions:\n", - " print(f\" Dataset: {version.dataset_name}\")\n", - " print(f\" Version: {version.dataset_version}\")\n", - " print(f\" UUID: {version.dataset_uuid}\\n\")\n", + "from dataconnect import DataConnectClient\n", "\n", - "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "fb0f253e", - "metadata": {}, - "source": [ - "#### Fetch records for specific datasets" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "dda6f0cc", - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " rave_data = dataconnect_client.fetch_data(dataset_uuid=UUID(dataset_ids[\"exciter_rave_id\"]))\n", - " # Fetching first 3 rows of data\n", - " rave_data.head(3)\n", + "user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", "\n", - "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c73ccece", - "metadata": {}, - "outputs": [], - "source": [ "try:\n", - " der_data = dataconnect_client.fetch_data(dataset_uuid=UUID(dataset_ids[\"pwb_der_id\"]))\n", - " # Fetching the first 6 rows of data\n", - " der_data.head(6)\n", + " with DataConnectClient.connect(token=user_token) as dataconnect_client:\n", + " dataset_uuid = UUID(\"\")\n", "\n", - "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7413a5f5", - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " import_data = dataconnect_client.fetch_data(dataset_uuid=UUID(dataset_ids[\"pwb_import_id\"]))\n", - " # Fetching the first 3 rows of data\n", - " import_data.head(3)\n", + " dataset_versions = dataconnect_client.get_dataset_versions(dataset_uuid=dataset_uuid)\n", "\n", - "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "09b7aca3", - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " custom_data = dataconnect_client.fetch_data(dataset_uuid=UUID(dataset_ids[\"exciter_custom_id\"]))\n", - " # Fetching the first 3 rows of data\n", - " custom_data.head(3)\n", + " for version in dataset_versions:\n", + " print(f\" Dataset: {version.dataset_name}\")\n", + " print(f\" Version: {version.dataset_version}\")\n", + " print(f\" Dataset Version UUID: {version.dataset_uuid}\\n\")\n", "\n", "except Exception as e:\n", - " print(f\"Error: {e}\")" + " print(e)" ] }, { "cell_type": "markdown", - "id": "d2322e61", + "id": "fb0f253e", "metadata": {}, "source": [ - "You can also use `first_n_rows` to limit the number of rows fetched, which is useful for large datasets." + "### Fetch records of a specific dataset\n", + "Data fetched based on a `dataset_uuid` are in pandas `DataFrame` format. You can use the optional parameter `first_n_rows` to limit the number of rows fetched, which is useful for large datasets." ] }, { "cell_type": "code", "execution_count": null, - "id": "b797d800ac0de9ef", + "id": "331ca00b", "metadata": {}, "outputs": [], "source": [ - "try:\n", - " # Fetch only the first 10 rows from the server\n", - " rave_sample = dataconnect_client.fetch_data(\n", - " dataset_uuid=UUID(dataset_ids[\"exciter_rave_id\"]),\n", - " first_n_rows=10,\n", - " )\n", - " rave_sample.head()\n", + "from uuid import UUID\n", "\n", - "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "7319083fd0e6ca5", - "metadata": {}, - "source": [ - "### Data Transformation Example Using pandas\n", + "from dataconnect import DataConnectClient\n", "\n", - "#### Data transformation using native pandas capability in Python\n", + "user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", "\n", - "The fetched data is returned as a pandas `DataFrame`. You can use any pandas operations\n", - "for data manipulation.\n", + "try:\n", + " with DataConnectClient.connect(token=user_token) as dataconnect_client:\n", + " dataset_uuid = UUID(\"\")\n", "\n", - "If you have limited memory allocation, we recommend working on a limited set of\n", - "records to reduce development time, and then remove the record limit during\n", - "recurring execution.\n", + " # Fetch only the first 10 rows from the server\n", + " data = dataconnect_client.fetch_data(dataset_uuid=dataset_uuid, first_n_rows=10)\n", "\n", - "These functions are not provided by the `dataconnect` library. Below are just\n", - "examples of how common Python libraries can be used with the `dataconnect` library.\n", - "For details on how these libraries can be used, please consult the respective\n", - "library documentation.\n", + " print(data)\n", "\n", - "#### Set up: Using pandas to perform data transformation on fetched DataFrames" + "except Exception as e:\n", + " print(e)" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "9bf1896ca482bfd9", + "cell_type": "markdown", + "id": "58dcccda", "metadata": {}, - "outputs": [], "source": [ - "# Pivot der_data\n", - "pivot_df = der_data.rename(\n", - " columns={\n", - " \"HCT\": \"HCT_result\",\n", - " \"PLAT\": \"plat_result\",\n", - " \"WBC\": \"wbc_result\",\n", - " \"HCT_UNIT\": \"HCT_unit\",\n", - " \"PLAT_UNIT\": \"plat_unit\",\n", - " \"WBC_UNIT\": \"wbc_unit\",\n", - " }\n", - ").melt(\n", - " id_vars=[\"patient_id\", \"site_id\", \"LBTIM\"],\n", - " var_name=\"lab_test_value\",\n", - " value_name=\"value\",\n", - ")\n", + "## Dry Publish and Publish\n", + "Along with the `User Authentication Token` as mentioned above, a `Project Token` is required to publish datasets to a `Data Connect` Transformation project. \n", "\n", - "# Split the lab_test_value column into lab_test and measurement type\n", - "pivot_df[[\"lab_test\", \"measure\"]] = pivot_df[\"lab_test_value\"].str.rsplit(\"_\", n=1, expand=True)\n", - "pivot_df = pivot_df.pivot_table(\n", - " index=[\"patient_id\", \"site_id\", \"LBTIM\", \"lab_test\"],\n", - " columns=\"measure\",\n", - " values=\"value\",\n", - " aggfunc=\"first\",\n", - ").reset_index()\n", - "pivot_df = pivot_df.rename(columns={\"result\": \"lab_result\", \"unit\": \"lab_unit\"})\n", - "\n", - "# Union the datasets together (pivot_df, import_data)\n", - "pivot_df_renamed = pivot_df.rename(\n", - " columns={\n", - " \"patient_id\": \"subjid\",\n", - " \"site_id\": \"siteid\",\n", - " \"lab_test\": \"lbtest\",\n", - " \"lab_result\": \"lbresn\",\n", - " \"lab_unit\": \"lbresu\",\n", - " }\n", - ")\n", - "union_df = pd.concat([import_data, pivot_df_renamed], ignore_index=True)\n", - "\n", - "# Create publish_df by outer-joining import_data to the union_df key set\n", - "# (to include keys only present in derived data)\n", - "publish_df = import_data.merge(union_df[[\"subjid\", \"siteid\", \"visitnum\"]].drop_duplicates(), ...)\n", - "\n", - "publish_df.head()" + "You can retrieve this by navigating to `iMedidata` > `Data Connect` > `Transformations` and by clicking on the kebab menu on a project row and selecting `View/Modify`." ] }, { "cell_type": "markdown", - "id": "5132a8c3681180aa", + "id": "69e733cb", "metadata": {}, "source": [ - "### Publish datasets with `dataconnect` library\n", - "\n", - "#### Set up the project token and publish parameters\n", + "### Dry Publish (validate without persisting)\n", "\n", - "The `project_token` is a Base64-encoded token that identifies the target study, study environment, and project.\n", - "You can retrieve this from the \"Developer info\" in the iMedidata Platform." + "Before publishing, use `dry_publish` to validate your dataset against the server without committing any changes. This lets you catch errors early." ] }, { "cell_type": "code", "execution_count": null, - "id": "fe7af53732e8cf31", + "id": "27e85026", "metadata": {}, "outputs": [], "source": [ - "project_token = \"projecttoken\"\n", + "from uuid import UUID\n", "\n", - "# Name for the dataset you want to publish\n", - "publish_dataset_name = \"my_published_dataset\"\n", + "import pandas as pd\n", "\n", - "# Key columns define the unique key for deduplication\n", - "key_columns = [\"subjid\", \"siteid\", \"visitnum\"]\n", + "from dataconnect import DataConnectClient\n", "\n", - "# Source dataset UUIDs that the published dataset is derived from\n", - "source_dataset_uuids = [\n", - " UUID(dataset_ids[\"pwb_import_id\"]),\n", - " UUID(dataset_ids[\"pwb_der_id\"]),\n", - "]" - ] - }, - { - "cell_type": "markdown", - "id": "665d73cc31b99b17", - "metadata": {}, - "source": [ - "#### Dry publish (validate without persisting)\n", + "user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", + "project_token = \"project_token\" # From iMedidata > Data Connect > Transformations\n", + "\n", + "# Configuration\n", + "dataset_name = \"my_new_dataset\"\n", + "key_columns: list[str] = [\"patient_id\", \"site_id\"]\n", + "source_datasets = [\n", + " UUID(\"f30a8619-8348-330b-8a66-34d837303a8c\")\n", + "] # Any number of source datasets, in UUID format and comma-separated as a list\n", + "datetime_formats = {\"visit_date\": \"yyyy-MM-dd\"}\n", + "\n", + "# Data to Publish - pandas DataFrame\n", + "data = pd.DataFrame(\n", + " {\n", + " \"VISIT_NAME\": [\"Adverse Events\", \"Adverse Events\", \"Adverse Events\"],\n", + " \"SITE_ID\": [\"1010\", \"1042\", \"1001\"],\n", + " \"PATIENT_ID\": [\"10101097\", \"10421089\", \"10011029\"],\n", + " \"VISIT_DATE\": pd.to_datetime([\"2020-02-01\", \"2021-02-01\", \"2022-02-01\"]),\n", + " }\n", + ")\n", "\n", - "Before publishing, use `dry_publish` to validate your dataset against the server\n", - "without committing any changes. This lets you catch errors early." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "23e2cabf3fead3bb", - "metadata": {}, - "outputs": [], - "source": [ "try:\n", - " dry_result = dataconnect_client.dry_publish(\n", - " project_token=project_token,\n", - " dataset_name=publish_dataset_name,\n", - " key_columns=key_columns,\n", - " source_datasets=source_dataset_uuids,\n", - " data=publish_df,\n", - " )\n", - "\n", - " print(f\"Dry publish status: {dry_result.status}\")\n", - " print(f\"Schema valid: {dry_result.is_schema_valid}\")\n", - " print(f\"Config valid: {dry_result.is_config_valid}\")\n", - " print(f\"Dataset valid: {dry_result.is_dataset_valid}\")\n", - " print(f\"Dataset name: {dry_result.dataset_name}\")\n", - " print(f\"Dataset version: {dry_result.dataset_version}\")\n", - " print(f\"Number of columns: {dry_result.no_of_columns}\")\n", - " print(f\"Valid record count: {dry_result.valid_record_count}\")\n", - " print(f\"Duplicate record count: {dry_result.duplicate_record_count}\")\n", - " print(f\"Invalid record count: {dry_result.invalid_record_count}\")\n", - "\n", - " if dry_result.errors:\n", - " print(f\"\\nErrors: {dry_result.errors}\")\n", - "\n", - " if dry_result.invalid_datetime_formats:\n", - " print(f\"\\nInvalid datetime formats: {dry_result.invalid_datetime_formats}\")\n", - "\n", - " if dry_result.invalid_records is not None and not dry_result.invalid_records.empty:\n", - " print(\"\\nInvalid records:\")\n", - " print(dry_result.invalid_records.head())\n", + " with DataConnectClient.connect(token=user_token) as client:\n", + " dry_publish_result = client.dry_publish(\n", + " project_token=project_token,\n", + " dataset_name=dataset_name,\n", + " key_columns=key_columns,\n", + " source_datasets=source_datasets,\n", + " datetime_formats=datetime_formats,\n", + " data=data,\n", + " ) # pandas DataFrame to publish\n", + "\n", + " print(f\"Dry publish result: \t {dry_publish_result.status}\")\n", + " print(f\"Schema valid: {dry_publish_result.is_schema_valid}\")\n", + " print(f\"Config valid: {dry_publish_result.is_config_valid}\")\n", + " print(f\"Dataset valid: {dry_publish_result.is_dataset_valid}\")\n", + " print(f\"Datetime Formats invalid: {dry_publish_result.invalid_datetime_formats}\")\n", + " print(f\"Dataset name: {dry_publish_result.dataset_name}\")\n", + " print(f\"Dataset version: {dry_publish_result.dataset_version}\")\n", + " print(f\"Number of columns: {dry_publish_result.no_of_columns}\")\n", + " print(f\"Valid record count: {dry_publish_result.valid_record_count}\")\n", + " print(f\"Duplicate record count: {dry_publish_result.duplicate_record_count}\")\n", + " print(f\"Invalid record count: {dry_publish_result.invalid_record_count}\")\n", + " print(f\"Errors:\t\t\t\t\t {dry_publish_result.errors}\")\n", "\n", "except Exception as e:\n", - " print(f\"Error: {e}\")" + " print(e)" ] }, { "cell_type": "markdown", - "id": "599eddc1f46e945b", + "id": "829bc003", "metadata": {}, "source": [ - "#### Publish (persist the dataset)\n", + "### Publish (persist the dataset)\n", "\n", - "Once the dry publish validates successfully, publish the dataset to the server." + "Once the dry publish validates successfully, publish the dataset to the server. The now published dataset should be visible in `iMedidata` > `Data Connect` > `Datasets` within a few minutes. " ] }, { "cell_type": "code", "execution_count": null, - "id": "8429cd0eb7e74e1c", + "id": "960ea8e3", "metadata": {}, "outputs": [], "source": [ - "try:\n", - " publish_result = dataconnect_client.publish(\n", - " project_token=project_token,\n", - " dataset_name=publish_dataset_name,\n", - " key_columns=key_columns,\n", - " source_datasets=source_dataset_uuids,\n", - " data=publish_df,\n", - " )\n", - "\n", - " print(f\"Publish status: {publish_result.status}\")\n", - " print(f\"Dataset name: {publish_result.dataset_name}\")\n", - " print(f\"Dataset UUID: {publish_result.dataset_uuid}\")\n", - " print(f\"Dataset version: {publish_result.dataset_version}\")\n", - " print(f\"Dataset batch number: {publish_result.dataset_batch_number}\")\n", - " print(f\"Valid record count: {publish_result.valid_record_count}\")\n", - " print(f\"Duplicate record count: {publish_result.duplicate_record_count}\")\n", - " print(f\"Invalid record count: {publish_result.invalid_record_count}\")\n", - "\n", - " if publish_result.invalid_records is not None and not publish_result.invalid_records.empty:\n", - " print(\"\\nInvalid records:\")\n", - " print(publish_result.invalid_records.head())\n", + "from dataconnect import DataConnectClient\n", "\n", - "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "759eea057d0957b9", - "metadata": {}, - "source": [ - "You can also specify `datetime_formats` if your dataset contains datetime columns\n", - "that need a specific format for validation and publishing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cb6d988fcf8854e5", - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " publish_result_with_formats = dataconnect_client.publish(\n", - " project_token=project_token,\n", - " dataset_name=publish_dataset_name,\n", - " key_columns=key_columns,\n", - " source_datasets=source_dataset_uuids,\n", - " data=publish_df,\n", - " datetime_formats={\"visit_date\": \"yyyy-MM-dd\"},\n", - " )\n", + "user_token = \"usertoken\" # From iMedidata > Data Connect > Developer Center\n", + "project_token = \"project_token\" # From iMedidata > Data Connect > Transformations\n", + "\n", + "# Configuration\n", + "dataset_name = \"my_new_dataset\"\n", + "key_columns: list[str] = [\"patient_id\", \"site_id\"]\n", + "source_datasets = [\n", + " UUID(\"f30a8619-8348-330b-8a66-34d837303a8c\")\n", + "] # Any number of source datasets, in UUID format and comma-separated as a list\n", + "datetime_formats = {\"visit_date\": \"yyyy-MM-dd\"}\n", + "\n", + "# Data to Publish - pandas DataFrame\n", + "data = pd.DataFrame(\n", + " {\n", + " \"VISIT_NAME\": [\"Adverse Events\", \"Adverse Events\", \"Adverse Events\"],\n", + " \"SITE_ID\": [\"1010\", \"1042\", \"1001\"],\n", + " \"PATIENT_ID\": [\"10101097\", \"10421089\", \"10011029\"],\n", + " \"VISIT_DATE\": pd.to_datetime([\"2020-02-01\", \"2021-02-01\", \"2022-02-01\"]),\n", + " }\n", + ")\n", "\n", - " print(f\"Publish status: {publish_result_with_formats.status}\")\n", + "try:\n", + " with DataConnectClient.connect(token=user_token) as client:\n", + " publish_result = client.publish(\n", + " project_token=project_token,\n", + " dataset_name=dataset_name,\n", + " key_columns=key_columns,\n", + " source_datasets=source_datasets,\n", + " datetime_formats=datetime_formats,\n", + " data=data,\n", + " ) # pandas DataFrame to publish\n", + "\n", + " print(f\"Publish status: {publish_result.status}\")\n", + " print(f\"Dataset name: {publish_result.dataset_name}\")\n", + " print(f\"Dataset UUID: {publish_result.dataset_uuid}\")\n", + " print(f\"Dataset version: {publish_result.dataset_version}\")\n", + " print(f\"Dataset batch number: {publish_result.dataset_batch_number}\")\n", + " print(f\"Valid record count: {publish_result.valid_record_count}\")\n", + " print(f\"Duplicate record count: {publish_result.duplicate_record_count}\")\n", + " print(f\"Invalid record count: {publish_result.invalid_record_count}\")\n", "\n", "except Exception as e:\n", - " print(f\"Error: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "608fe2d6aa684bd2", - "metadata": {}, - "source": [ - "### Close the Client Connection" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3fce61d2f447c8e0", - "metadata": {}, - "outputs": [], - "source": [ - "if dataconnect_client:\n", - " dataconnect_client.close()" + " print(e)" ] } ], "metadata": { "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3", "language": "python", "name": "python3" }, diff --git a/guides/transform_example.md b/guides/transform_example.md new file mode 100644 index 0000000..d326140 --- /dev/null +++ b/guides/transform_example.md @@ -0,0 +1,75 @@ + +# DataConnect Python Library v1.0.0 + +## Data Transformation Example Using pandas + +The fetched *data* is returned as a pandas `DataFrame`. You can use any pandas operations +for data manipulation. + +If you have limited memory allocation, we recommend working on a limited set of +records to reduce development time, and then remove the record limit during +recurring execution. + +These functions are not provided by the `dataconnect` library. Below are just +examples of how common Python libraries can be used with `dataconnect`. +For details on how these libraries can be used, please consult the respective +library documentation. + + +```python +from dataconnect import DataConnectClient +import pandas as pd +from uuid import UUID + +user_token = "usertoken_from_dataconnect_developer_center" +data = + +with DataConnectClient.connect(token=user_token) as dataconnect_client: + import_data = dataconnect_client.fetch_data(dataset_uuid=UUID("import_dataset_uuid")) + + data = pd.DataFrame() # should be populated + + # Pivot data + pivot_df = data.rename( + columns={ + "HCT": "HCT_result", + "PLAT": "plat_result", + "WBC": "wbc_result", + "HCT_UNIT": "HCT_unit", + "PLAT_UNIT": "plat_unit", + "WBC_UNIT": "wbc_unit", + } + ).melt( + id_vars=["patient_id", "site_id", "LBTIM"], + var_name="lab_test_value", + value_name="value", + ) + + # Split the lab_test_value column into lab_test and measurement type + pivot_df[["lab_test", "measure"]] = pivot_df["lab_test_value"].str.rsplit("_", n=1, expand=True) + pivot_df = pivot_df.pivot_table( + index=["patient_id", "site_id", "LBTIM", "lab_test"], + columns="measure", + values="value", + aggfunc="first", + ).reset_index() + pivot_df = pivot_df.rename(columns={"result": "lab_result", "unit": "lab_unit"}) + + # Union the datasets together (pivot_df, import_data) + pivot_df_renamed = pivot_df.rename( + columns={ + "patient_id": "subjid", + "site_id": "siteid", + "lab_test": "lbtest", + "lab_result": "lbresn", + "lab_unit": "lbresu", + } + ) + union_df = pd.concat([import_data, pivot_df_renamed], ignore_index=True) + + # Create publish_df by outer-joining import_data to the union_df key set + # (to include keys only present in derived data) + publish_df = import_data.merge(union_df[["subjid", "siteid", "visitnum"]].drop_duplicates(), ...) + + print(publish_df.head()) +``` diff --git a/pyproject.toml b/pyproject.toml index 768f12e..59eb53b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "dataconnect-library-python" -version = "0.1.0" +version = "1.0.0" description = "Python library built by Medidata for connecting to Data Connect and programmatically querying and retrieving data." authors = ["Team 348 (SDK) "] packages = [{include = "dataconnect"}] diff --git a/readme/README-v1.0.0.md b/readme/README-v1.0.0.md index 16dfa09..7ec8290 100644 --- a/readme/README-v1.0.0.md +++ b/readme/README-v1.0.0.md @@ -5,28 +5,50 @@ To use this library, you must have a valid iMedidata account and access to requi ## Table of Contents -- [Installation](#installation) -- [Quick Start](#quick-start) -- [Authentication and Connectivity](#authentication-and-connectivity) -- [Functions](#functions) - - [connect()](#connect) - - [get_studies()](#get_studies) - - [get_datasets()](#get_datasets) - - [get_dataset_versions()](#get_dataset_versions) - - [fetch_data()](#fetch_data) - - [close()](#close) -- [Errors](#errors) +- [DataConnect Python Library v1.0.0](#dataconnect-python-library-v100) + - [Table of Contents](#table-of-contents) + - [Setup and Usage](#setup-and-usage) + - [Authentication and Connectivity](#authentication-and-connectivity) + - [Functions](#functions) + - [connect()](#connect) + - [Description](#description) + - [Usage](#usage) + - [Arguments](#arguments) + - [Output](#output) + - [get\_studies()](#get_studies) + - [Description](#description-1) + - [Usage](#usage-1) + - [Arguments](#arguments-1) + - [Output](#output-1) + - [get\_datasets()](#get_datasets) + - [Description](#description-2) + - [Usage](#usage-2) + - [Arguments](#arguments-2) + - [Output](#output-2) + - [get\_dataset\_versions()](#get_dataset_versions) + - [Description](#description-3) + - [Usage](#usage-3) + - [Arguments](#arguments-3) + - [Output](#output-3) + - [fetch\_data()](#fetch_data) + - [Description](#description-4) + - [Usage](#usage-4) + - [Arguments](#arguments-4) + - [Output](#output-4) + - [close()](#close) + - [Description](#description-5) + - [Usage](#usage-5) + - [Arguments](#arguments-5) + - [Output](#output-5) + - [Errors](#errors) - [Reporting known issues](#reporting-known-issues) - [Backend](#backend) - [Licensing](#licensing) -## Installation +## Setup and Usage -To install, follow the [Installation Guide](https://github.com/mdsol/dataconnect-library-python/blob/main/guides/dataconnect_setup.md). - -## Quick Start - -For end-to-end examples, see the [Usage Guide](https://github.com/mdsol/dataconnect-library-python/blob/main/guides/dataconnect_quickstart.md). +* For instructions on how to install and use this library, follow the [Usage Notebook](../guides/dataconnect_usage.ipynb). +* For an example of how to transform data, go [here](../guides/transform_example.md). ### Authentication and Connectivity