Release v 1.3#210
Closed
pendingintent wants to merge 18 commits into
Closed
Conversation
… are assigned to the selected timeline
| return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/edit"}) | ||
| return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/edit';</script>") | ||
| return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/freezes"}) | ||
| return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/freezes';</script>") |
| redirect_url = f"/ui/soa/{soa_id}/edit" | ||
| if request.headers.get("HX-Request") == "true": | ||
| return HTMLResponse("", headers={"HX-Redirect": redirect_url}) | ||
| return RedirectResponse(redirect_url, status_code=303) |
| redirect_url = f"/ui/soa/{soa_id}/edit" | ||
| if request.headers.get("HX-Request") == "true": | ||
| return HTMLResponse("", headers={"HX-Redirect": redirect_url}) | ||
| return RedirectResponse(redirect_url, status_code=303) |
| redirect_url = f"/ui/soa/{soa_id}/edit" | ||
| if request.headers.get("HX-Request") == "true": | ||
| return HTMLResponse("", headers={"HX-Redirect": redirect_url}) | ||
| return RedirectResponse(redirect_url, status_code=303) |
Contributor
There was a problem hiding this comment.
Pull request overview
This release bundles a broad set of changes across the SoA Workbench: new objective/endpoint authoring in the FastAPI UI/API, richer USDM biomedical concept export data, freeze-page refactoring, and first-pass Azure deployment assets.
Changes:
- Added objective and endpoint schemas, routers, edit-page UI, migrations, audits, and tests.
- Added USDM biomedical concept property / extension-attribute generation plus related UID helpers and tests.
- Moved freeze management to a dedicated page and added Azure deployment scripts/workflow/docs.
Reviewed changes
Copilot reviewed 29 out of 31 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_routers_objectives.py |
New API/UI coverage for objectives CRUD and orphaning behavior. |
tests/test_routers_freezes.py |
Added coverage for the dedicated freezes page and delete flow. |
tests/test_routers_endpoints.py |
New API/UI coverage for endpoints CRUD and objective linkage. |
tests/test_generate_extension_attributes.py |
Tests for extension-attribute UID population and export shape. |
tests/test_generate_biomedical_concept_properties.py |
Tests for biomedical concept property persistence and export shape. |
startup.sh |
New App Service startup command using Gunicorn/Uvicorn. |
src/usdm/generate_extension_attributes.py |
New generator for BC extension attributes from DSS mappings. |
src/usdm/generate_biomedical_concepts.py |
Wires properties/extension attributes into BC export. |
src/usdm/generate_biomedical_concept_properties.py |
New persisted BC property generator from CDISC BC metadata. |
src/soa_builder/web/utils.py |
Added UID helpers for BC properties and extension attributes. |
src/soa_builder/web/templates/freezes.html |
New dedicated frozen-versions page and actions. |
src/soa_builder/web/templates/edit.html |
Removed inline freeze bar, added objectives section, filtered matrix rows by timeline. |
src/soa_builder/web/templates/base.html |
Added navigation entry for Frozen Versions. |
src/soa_builder/web/templates/_objectives_section.html |
New edit-page UI for objectives/endpoints CRUD and orphan handling. |
src/soa_builder/web/schemas.py |
Added request models for objectives and endpoints. |
src/soa_builder/web/routers/objectives.py |
New objective API/UI router with audit and level-code handling. |
src/soa_builder/web/routers/freezes.py |
Refactored freeze routes to shared helpers and dedicated listing/delete UI. |
src/soa_builder/web/routers/endpoints.py |
New endpoint API/UI router with parent-objective validation. |
src/soa_builder/web/routers/_freeze_helpers.py |
Extracted freeze/rollback implementation from app.py. |
src/soa_builder/web/migrate_database.py |
Added migrations for objectives, endpoints, audits, and extension-attribute UID column. |
src/soa_builder/web/audit.py |
Added objective and endpoint audit writers. |
src/soa_builder/web/app.py |
Registered new routers/migrations and expanded edit-page data loading. |
src/soa_builder.egg-info/top_level.txt |
Refreshed package metadata for additional top-level modules. |
src/soa_builder.egg-info/SOURCES.txt |
Refreshed package source manifest. |
src/soa_builder.egg-info/requires.txt |
Refreshed dependency metadata. |
src/soa_builder.egg-info/PKG-INFO |
Refreshed package metadata/README snapshot. |
scripts/upload_database_to_azure.sh |
Helper script for uploading SQLite data to Azure Files. |
requirements.txt |
Added Gunicorn runtime dependency. |
docs/DEPLOYMENT_AZURE.md |
Large Azure deployment runbook and rollout guidance. |
.gitignore |
Added Azure/deployment and output ignore patterns. |
.github/workflows/azure-deploy.yml |
New GitHub Actions workflow for Azure deployment. |
Comment on lines
+166
to
+170
| snapshot = { | ||
| "soa_id": soa_id, | ||
| "soa_name": soa_name, | ||
| "study_id": study_id_val, | ||
| "study_label": study_label_val, |
Comment on lines
+416
to
+420
| cur.execute("DELETE FROM matrix_cells WHERE soa_id=?", (soa_id,)) | ||
| cur.execute( | ||
| "DELETE FROM activity_concept WHERE activity_id IN" | ||
| " (SELECT id FROM activity WHERE soa_id=? )", | ||
| (soa_id,), |
Comment on lines
+3997
to
+4001
| activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()} | ||
| for c in cells: | ||
| tl = instance_timeline.get(c["instance_id"]) | ||
| if tl is None or tl not in activity_ids_by_timeline: | ||
| continue |
| raise HTTPException(404, "Objective not found") | ||
| before = _row_to_dict(row) | ||
|
|
||
| new_name = body.name if body.name is not None else before["name"] |
Comment on lines
+231
to
+237
| new_name = body.name if body.name is not None else before["name"] | ||
| new_label = body.label if body.label is not None else before["label"] | ||
| new_desc = ( | ||
| body.description if body.description is not None else before["description"] | ||
| ) | ||
| new_text = body.text if body.text is not None else before["text"] | ||
| new_purpose = body.purpose if body.purpose is not None else before["purpose"] |
|
|
||
| </div> | ||
|
|
||
| {% include '_objectives_section.html' %} |
Comment on lines
+35
to
+36
| populate_biomedical_concept_properties(soa_id) | ||
| populate_extension_attributes(soa_id) |
…e cross-site scripting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…e cross-site scripting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
| return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/edit"}) | ||
| return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/edit';</script>") | ||
| return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/freezes"}) | ||
| return RedirectResponse(url=f"/ui/soa/{soa_id}/freezes", status_code=303) |
…e cross-site scripting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
| return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/edit"}) | ||
| return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/edit';</script>") | ||
| return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/freezes"}) | ||
| return RedirectResponse(url=f"/ui/soa/{soa_id}/freezes", status_code=303) |
Comment on lines
+3989
to
+4005
| # Activities per timeline: an activity is shown in timeline T's matrix | ||
| # if any matrix_cells row connects it to an instance whose | ||
| # member_of_timeline == T. The instance->timeline link (set on the | ||
| # study_timing page) is the authoritative criterion. | ||
| instance_timeline = { | ||
| inst["id"]: (inst.get("member_of_timeline") or "unassigned") | ||
| for inst in instances | ||
| } | ||
| activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()} | ||
| for c in cells: | ||
| tl = instance_timeline.get(c["instance_id"]) | ||
| if tl is None or tl not in activity_ids_by_timeline: | ||
| continue | ||
| activity_ids_by_timeline[tl].add(c["activity_id"]) | ||
| activities_by_timeline: dict = { | ||
| tl: [a for a in activities_page if a["id"] in ids] | ||
| for tl, ids in activity_ids_by_timeline.items() |
Comment on lines
+166
to
+180
| snapshot = { | ||
| "soa_id": soa_id, | ||
| "soa_name": soa_name, | ||
| "study_id": study_id_val, | ||
| "study_label": study_label_val, | ||
| "study_description": study_description_val, | ||
| "version_label": version_label, | ||
| "frozen_at": datetime.now(timezone.utc).isoformat(), | ||
| "epochs": epochs, | ||
| "elements": elements, | ||
| "visits": visits, | ||
| "activities": activities, | ||
| "cells": cells, | ||
| "activity_concepts": concepts_map, | ||
| } |
Comment on lines
+422
to
+425
| cur.execute("DELETE FROM biomedical_concept WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM alias_code WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM code WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM code_association WHERE soa_id=?", (soa_id,)) |
Comment on lines
+262
to
+266
| _migrate_activity_concept_dss_add_extension_attribute_uid() | ||
| _migrate_add_objective_table() | ||
| _migrate_add_objective_audit_table() | ||
| _migrate_add_endpoint_table() | ||
| _migrate_add_endpoint_audit_table() |
| <h3 style="margin:0 0 8px;font-size:1em;color:#455a64;">Audit Logs</h3> | ||
| <div style="display:flex;gap:8px;flex-wrap:wrap;font-size:0.85em;"> | ||
| <button hx-get="/ui/soa/{{ soa_id }}/rollback_audit" hx-target="#modal-host" hx-swap="innerHTML" style="background:#37474f;color:#fff;border:none;padding:4px 10px;border-radius:4px;cursor:pointer;">Rollback Audit Log</button> | ||
| <a href="/soa/{{ soa_id }}/rollback_audit/export/xlsx" style="background:#1976d2;color:#fff;text-decoration:none;padding:4px 10px;border-radius:4px;">Rollback Audit XLSX</a> |
Comment on lines
+193
to
+196
| "isRequired": bool(is_required) if is_required is not None else True, | ||
| "isEnabled": bool(is_enabled) if is_enabled is not None else True, | ||
| "datatype": datatype or "", | ||
| "responseCodes": [], |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
… remote source' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Comment on lines
+3989
to
+4005
| # Activities per timeline: an activity is shown in timeline T's matrix | ||
| # if any matrix_cells row connects it to an instance whose | ||
| # member_of_timeline == T. The instance->timeline link (set on the | ||
| # study_timing page) is the authoritative criterion. | ||
| instance_timeline = { | ||
| inst["id"]: (inst.get("member_of_timeline") or "unassigned") | ||
| for inst in instances | ||
| } | ||
| activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()} | ||
| for c in cells: | ||
| tl = instance_timeline.get(c["instance_id"]) | ||
| if tl is None or tl not in activity_ids_by_timeline: | ||
| continue | ||
| activity_ids_by_timeline[tl].add(c["activity_id"]) | ||
| activities_by_timeline: dict = { | ||
| tl: [a for a in activities_page if a["id"] in ids] | ||
| for tl, ids in activity_ids_by_timeline.items() |
Comment on lines
+174
to
+179
| "epochs": epochs, | ||
| "elements": elements, | ||
| "visits": visits, | ||
| "activities": activities, | ||
| "cells": cells, | ||
| "activity_concepts": concepts_map, |
Comment on lines
+422
to
+425
| cur.execute("DELETE FROM biomedical_concept WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM alias_code WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM code WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM code_association WHERE soa_id=?", (soa_id,)) |
| <h3 style="margin:0 0 8px;font-size:1em;color:#455a64;">Audit Logs</h3> | ||
| <div style="display:flex;gap:8px;flex-wrap:wrap;font-size:0.85em;"> | ||
| <button hx-get="/ui/soa/{{ soa_id }}/rollback_audit" hx-target="#modal-host" hx-swap="innerHTML" style="background:#37474f;color:#fff;border:none;padding:4px 10px;border-radius:4px;cursor:pointer;">Rollback Audit Log</button> | ||
| <a href="/soa/{{ soa_id }}/rollback_audit/export/xlsx" style="background:#1976d2;color:#fff;text-decoration:none;padding:4px 10px;border-radius:4px;">Rollback Audit XLSX</a> |
| raise HTTPException(404, "Objective not found") | ||
| before = _row_to_dict(row) | ||
|
|
||
| new_name = body.name if body.name is not None else before["name"] |
Comment on lines
+444
to
+462
| def get_next_extension_attribute_uid(cur: Any, soa_id: int) -> str: | ||
| """Compute next unique ExtensionAttribute_N for the given SOA. | ||
|
|
||
| Assumes `cur` is a sqlite cursor within an open transaction. | ||
| """ | ||
| cur.execute( | ||
| "SELECT extension_attribute_uid FROM activity_concept_dss" | ||
| " WHERE soa_id=?" | ||
| " AND extension_attribute_uid LIKE 'ExtensionAttribute_%'", | ||
| (soa_id,), | ||
| ) | ||
| existing = [x[0] for x in cur.fetchall() if x[0]] | ||
| n = 1 | ||
| if existing: | ||
| try: | ||
| n = max(int(x.split("_")[1]) for x in existing) + 1 | ||
| except Exception: | ||
| n = len(existing) + 1 | ||
| return f"ExtensionAttribute_{n}" |
Comment on lines
+4052
to
+4063
| objective_level_options = sorted({v for v in c188725_map.values() if v}) | ||
| endpoint_level_options = sorted({v for v in c188726_map.values() if v}) | ||
| conn_obj = _connect() | ||
| cur_obj = conn_obj.cursor() | ||
| cur_obj.execute( | ||
| "SELECT code_uid, code FROM code_association " | ||
| "WHERE soa_id=? AND codelist_code IN ('C188725','C188726')", | ||
| (soa_id,), | ||
| ) | ||
| level_code_to_sv: dict = {} | ||
| for code_uid, code_val in cur_obj.fetchall(): | ||
| level_code_to_sv[code_uid] = code_val or "" |
Comment on lines
+82
to
+92
| # Upload database | ||
| echo -e "${YELLOW}Uploading database to Azure Files...${NC}" | ||
| echo "This may take a few moments depending on file size..." | ||
|
|
||
| az storage file upload \ | ||
| --account-name "$STORAGE_ACCOUNT" \ | ||
| --account-key "$STORAGE_KEY" \ | ||
| --share-name "$FILE_SHARE" \ | ||
| --source "$LOCAL_DB" \ | ||
| --path soa_builder_web.db \ | ||
| --no-progress |
Comment on lines
+462
to
+465
| **⚠️ Important Notes:** | ||
| - **WAL mode files**: SQLite creates `-wal` and `-shm` files in WAL mode. These will be recreated automatically by Azure; you only need to upload the main `.db` file. | ||
| - **Database locking**: Ensure your local app is stopped before uploading the database to avoid corruption. | ||
| - **Schema migrations**: The app will automatically run any pending migrations on startup, so your local database will be updated to match the latest schema. |
| "DELETE FROM activity_concept WHERE activity_id IN" | ||
| " (SELECT id FROM activity WHERE soa_id=? )", | ||
| (soa_id,), | ||
| ) |
Comment on lines
4149
to
+4154
| "superscript_map": superscript_map, | ||
| "objectives": objectives, | ||
| "endpoints_by_objective": endpoints_by_objective, | ||
| "orphan_endpoints": orphan_endpoints, | ||
| "objective_level_options": objective_level_options, | ||
| "endpoint_level_options": endpoint_level_options, |
| <a href="/soa/{{ soa_id }}/freeze/{{ f.id }}" target="_blank" style="background:#37474f;color:#fff;text-decoration:none;padding:2px 8px;border-radius:3px;font-size:0.85em;">JSON</a> | ||
| </td> | ||
| <td style="padding:4px;"> | ||
| <form method="post" action="/ui/soa/{{ soa_id }}/freeze/{{ f.id }}/delete" onsubmit="return confirm('Delete frozen version {{ f.version_label }}? This cannot be undone.');" style="display:inline;"> |
Comment on lines
+3989
to
+4005
| # Activities per timeline: an activity is shown in timeline T's matrix | ||
| # if any matrix_cells row connects it to an instance whose | ||
| # member_of_timeline == T. The instance->timeline link (set on the | ||
| # study_timing page) is the authoritative criterion. | ||
| instance_timeline = { | ||
| inst["id"]: (inst.get("member_of_timeline") or "unassigned") | ||
| for inst in instances | ||
| } | ||
| activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()} | ||
| for c in cells: | ||
| tl = instance_timeline.get(c["instance_id"]) | ||
| if tl is None or tl not in activity_ids_by_timeline: | ||
| continue | ||
| activity_ids_by_timeline[tl].add(c["activity_id"]) | ||
| activities_by_timeline: dict = { | ||
| tl: [a for a in activities_page if a["id"] in ids] | ||
| for tl, ids in activity_ids_by_timeline.items() |
Comment on lines
+422
to
+425
| cur.execute("DELETE FROM biomedical_concept WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM alias_code WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM code WHERE soa_id=?", (soa_id,)) | ||
| cur.execute("DELETE FROM code_association WHERE soa_id=?", (soa_id,)) |
Comment on lines
4149
to
+4154
| "superscript_map": superscript_map, | ||
| "objectives": objectives, | ||
| "endpoints_by_objective": endpoints_by_objective, | ||
| "orphan_endpoints": orphan_endpoints, | ||
| "objective_level_options": objective_level_options, | ||
| "endpoint_level_options": endpoint_level_options, |
Comment on lines
+61
to
+71
| - name: Azure login | ||
| uses: azure/login@v1 | ||
| with: | ||
| creds: ${{ secrets.AZURE_CREDENTIALS }} | ||
| - name: Stop Azure Web App | ||
| uses: azure/cli@v1 | ||
| with: | ||
| azcliversion: 2.36.0 | ||
| inlineScript: | | ||
| az webapp stop --name ${{ secrets.AZURE_WEBAPP_NAME }} --resource-group cdisc-soa-workbench-rg | ||
|
|
Comment on lines
+202
to
+225
| new_name = body.name if body.name is not None else before["name"] | ||
| new_label = body.label if body.label is not None else before["label"] | ||
| new_desc = ( | ||
| body.description if body.description is not None else before["description"] | ||
| ) | ||
| new_text = body.text if body.text is not None else before["text"] | ||
|
|
||
| new_level_code_uid = before["level_code_uid"] | ||
| if body.level is not None: | ||
| new_level = body.level.strip() | ||
| if not new_level: | ||
| conn.close() | ||
| raise HTTPException(400, "Objective level cannot be empty") | ||
| if before["level_code_uid"]: | ||
| # Update the submission value in the existing Code_N row. | ||
| cur.execute( | ||
| "UPDATE code_association SET code=? WHERE soa_id=? AND code_uid=?", | ||
| (new_level, soa_id, before["level_code_uid"]), | ||
| ) | ||
| else: | ||
| new_level_code_uid = _insert_level_code(cur, soa_id, new_level) | ||
|
|
||
| cur.execute( | ||
| "UPDATE objective SET name=?, label=?, description=?, text=?, " |
Comment on lines
+58
to
+71
| echo "Retrieving storage account key..." | ||
| STORAGE_KEY=$(az storage account keys list \ | ||
| --resource-group "$RESOURCE_GROUP" \ | ||
| --account-name "$STORAGE_ACCOUNT" \ | ||
| --query '[0].value' \ | ||
| --output tsv 2>&1) | ||
|
|
||
| if [ $? -ne 0 ]; then | ||
| echo -e "${RED}Error: Failed to retrieve storage account key${NC}" | ||
| echo "Please verify:" | ||
| echo " 1. Resource group '$RESOURCE_GROUP' exists" | ||
| echo " 2. Storage account '$STORAGE_ACCOUNT' exists" | ||
| echo " 3. You have permission to access the storage account" | ||
| exit 1 |
Comment on lines
+451
to
+463
| for c in cells: | ||
| old_vid = c.get("visit_id") | ||
| old_aid = c.get("activity_id") | ||
| status = c.get("status", "").strip() | ||
| if status == "": | ||
| continue | ||
| vid = visit_id_map.get(old_vid) | ||
| aid = activity_id_map.get(old_aid) | ||
| if vid and aid: | ||
| cur.execute( | ||
| "INSERT INTO matrix_cells (soa_id, visit_id, activity_id, status)" | ||
| " VALUES (?,?,?,?)", | ||
| (soa_id, vid, aid, status), |
| study_id_val = row[2] if row else None | ||
| study_label_val = row[3] if row else None | ||
| study_description_val = row[4] if row else None | ||
| visits, activities, cells = _fetch_matrix(soa_id) |
Comment on lines
+114
to
+122
| cur.execute( | ||
| "SELECT id,soa_id,objective_uid,name,label,description,text," | ||
| "level_code_uid,order_index " | ||
| "FROM objective WHERE soa_id=? ORDER BY order_index, id", | ||
| (soa_id,), | ||
| ) | ||
| rows = [_row_to_dict(r) for r in cur.fetchall()] | ||
| conn.close() | ||
| return JSONResponse(rows) |
Comment on lines
+122
to
+130
| cur.execute( | ||
| "SELECT id,soa_id,endpoint_uid,objective_uid,name,label," | ||
| "description,text,purpose,level_code_uid,order_index " | ||
| "FROM endpoint WHERE soa_id=? ORDER BY order_index, id", | ||
| (soa_id,), | ||
| ) | ||
| rows = [_row_to_dict(r) for r in cur.fetchall()] | ||
| conn.close() | ||
| return JSONResponse(rows) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.