Skip to content

Release v 1.3#210

Closed
pendingintent wants to merge 18 commits into
masterfrom
release-v-1.3
Closed

Release v 1.3#210
pendingintent wants to merge 18 commits into
masterfrom
release-v-1.3

Conversation

@pendingintent

Copy link
Copy Markdown
Owner

No description provided.

@pendingintent pendingintent added this to the v1.3 milestone May 4, 2026
@pendingintent pendingintent self-assigned this May 4, 2026
Copilot AI review requested due to automatic review settings May 4, 2026 13:22
@pendingintent pendingintent added the bug Something isn't working label May 4, 2026
@pendingintent pendingintent added documentation Improvements or additions to documentation enhancement New feature or request labels May 4, 2026
Comment thread src/soa_builder/web/routers/freezes.py Fixed
Comment thread src/soa_builder/web/routers/freezes.py Outdated
return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/edit"})
return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/edit';</script>")
return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/freezes"})
return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/freezes';</script>")
Comment thread src/soa_builder/web/routers/freezes.py Fixed
Comment thread src/soa_builder/web/routers/freezes.py Fixed
Comment thread src/soa_builder/web/routers/endpoints.py Fixed
Comment thread src/soa_builder/web/routers/endpoints.py Fixed
Comment thread src/soa_builder/web/routers/endpoints.py Fixed
redirect_url = f"/ui/soa/{soa_id}/edit"
if request.headers.get("HX-Request") == "true":
return HTMLResponse("", headers={"HX-Redirect": redirect_url})
return RedirectResponse(redirect_url, status_code=303)
redirect_url = f"/ui/soa/{soa_id}/edit"
if request.headers.get("HX-Request") == "true":
return HTMLResponse("", headers={"HX-Redirect": redirect_url})
return RedirectResponse(redirect_url, status_code=303)
redirect_url = f"/ui/soa/{soa_id}/edit"
if request.headers.get("HX-Request") == "true":
return HTMLResponse("", headers={"HX-Redirect": redirect_url})
return RedirectResponse(redirect_url, status_code=303)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This release bundles a broad set of changes across the SoA Workbench: new objective/endpoint authoring in the FastAPI UI/API, richer USDM biomedical concept export data, freeze-page refactoring, and first-pass Azure deployment assets.

Changes:

  • Added objective and endpoint schemas, routers, edit-page UI, migrations, audits, and tests.
  • Added USDM biomedical concept property / extension-attribute generation plus related UID helpers and tests.
  • Moved freeze management to a dedicated page and added Azure deployment scripts/workflow/docs.

Reviewed changes

Copilot reviewed 29 out of 31 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/test_routers_objectives.py New API/UI coverage for objectives CRUD and orphaning behavior.
tests/test_routers_freezes.py Added coverage for the dedicated freezes page and delete flow.
tests/test_routers_endpoints.py New API/UI coverage for endpoints CRUD and objective linkage.
tests/test_generate_extension_attributes.py Tests for extension-attribute UID population and export shape.
tests/test_generate_biomedical_concept_properties.py Tests for biomedical concept property persistence and export shape.
startup.sh New App Service startup command using Gunicorn/Uvicorn.
src/usdm/generate_extension_attributes.py New generator for BC extension attributes from DSS mappings.
src/usdm/generate_biomedical_concepts.py Wires properties/extension attributes into BC export.
src/usdm/generate_biomedical_concept_properties.py New persisted BC property generator from CDISC BC metadata.
src/soa_builder/web/utils.py Added UID helpers for BC properties and extension attributes.
src/soa_builder/web/templates/freezes.html New dedicated frozen-versions page and actions.
src/soa_builder/web/templates/edit.html Removed inline freeze bar, added objectives section, filtered matrix rows by timeline.
src/soa_builder/web/templates/base.html Added navigation entry for Frozen Versions.
src/soa_builder/web/templates/_objectives_section.html New edit-page UI for objectives/endpoints CRUD and orphan handling.
src/soa_builder/web/schemas.py Added request models for objectives and endpoints.
src/soa_builder/web/routers/objectives.py New objective API/UI router with audit and level-code handling.
src/soa_builder/web/routers/freezes.py Refactored freeze routes to shared helpers and dedicated listing/delete UI.
src/soa_builder/web/routers/endpoints.py New endpoint API/UI router with parent-objective validation.
src/soa_builder/web/routers/_freeze_helpers.py Extracted freeze/rollback implementation from app.py.
src/soa_builder/web/migrate_database.py Added migrations for objectives, endpoints, audits, and extension-attribute UID column.
src/soa_builder/web/audit.py Added objective and endpoint audit writers.
src/soa_builder/web/app.py Registered new routers/migrations and expanded edit-page data loading.
src/soa_builder.egg-info/top_level.txt Refreshed package metadata for additional top-level modules.
src/soa_builder.egg-info/SOURCES.txt Refreshed package source manifest.
src/soa_builder.egg-info/requires.txt Refreshed dependency metadata.
src/soa_builder.egg-info/PKG-INFO Refreshed package metadata/README snapshot.
scripts/upload_database_to_azure.sh Helper script for uploading SQLite data to Azure Files.
requirements.txt Added Gunicorn runtime dependency.
docs/DEPLOYMENT_AZURE.md Large Azure deployment runbook and rollout guidance.
.gitignore Added Azure/deployment and output ignore patterns.
.github/workflows/azure-deploy.yml New GitHub Actions workflow for Azure deployment.

Comment on lines +166 to +170
snapshot = {
"soa_id": soa_id,
"soa_name": soa_name,
"study_id": study_id_val,
"study_label": study_label_val,
Comment on lines +416 to +420
cur.execute("DELETE FROM matrix_cells WHERE soa_id=?", (soa_id,))
cur.execute(
"DELETE FROM activity_concept WHERE activity_id IN"
" (SELECT id FROM activity WHERE soa_id=? )",
(soa_id,),
Comment on lines +3997 to +4001
activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()}
for c in cells:
tl = instance_timeline.get(c["instance_id"])
if tl is None or tl not in activity_ids_by_timeline:
continue
raise HTTPException(404, "Objective not found")
before = _row_to_dict(row)

new_name = body.name if body.name is not None else before["name"]
Comment on lines +231 to +237
new_name = body.name if body.name is not None else before["name"]
new_label = body.label if body.label is not None else before["label"]
new_desc = (
body.description if body.description is not None else before["description"]
)
new_text = body.text if body.text is not None else before["text"]
new_purpose = body.purpose if body.purpose is not None else before["purpose"]

</div>

{% include '_objectives_section.html' %}
Comment on lines +35 to +36
populate_biomedical_concept_properties(soa_id)
populate_extension_attributes(soa_id)
…e cross-site scripting'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 4, 2026 13:55
…e cross-site scripting'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/edit"})
return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/edit';</script>")
return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/freezes"})
return RedirectResponse(url=f"/ui/soa/{soa_id}/freezes", status_code=303)
…e cross-site scripting'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/edit"})
return HTMLResponse(f"<script>window.location='/ui/soa/{soa_id}/edit';</script>")
return HTMLResponse("", headers={"HX-Redirect": f"/ui/soa/{soa_id}/freezes"})
return RedirectResponse(url=f"/ui/soa/{soa_id}/freezes", status_code=303)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 31 changed files in this pull request and generated 7 comments.

Comment on lines +3989 to +4005
# Activities per timeline: an activity is shown in timeline T's matrix
# if any matrix_cells row connects it to an instance whose
# member_of_timeline == T. The instance->timeline link (set on the
# study_timing page) is the authoritative criterion.
instance_timeline = {
inst["id"]: (inst.get("member_of_timeline") or "unassigned")
for inst in instances
}
activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()}
for c in cells:
tl = instance_timeline.get(c["instance_id"])
if tl is None or tl not in activity_ids_by_timeline:
continue
activity_ids_by_timeline[tl].add(c["activity_id"])
activities_by_timeline: dict = {
tl: [a for a in activities_page if a["id"] in ids]
for tl, ids in activity_ids_by_timeline.items()
Comment on lines +166 to +180
snapshot = {
"soa_id": soa_id,
"soa_name": soa_name,
"study_id": study_id_val,
"study_label": study_label_val,
"study_description": study_description_val,
"version_label": version_label,
"frozen_at": datetime.now(timezone.utc).isoformat(),
"epochs": epochs,
"elements": elements,
"visits": visits,
"activities": activities,
"cells": cells,
"activity_concepts": concepts_map,
}
Comment on lines +422 to +425
cur.execute("DELETE FROM biomedical_concept WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM alias_code WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM code WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM code_association WHERE soa_id=?", (soa_id,))
Comment on lines +262 to +266
_migrate_activity_concept_dss_add_extension_attribute_uid()
_migrate_add_objective_table()
_migrate_add_objective_audit_table()
_migrate_add_endpoint_table()
_migrate_add_endpoint_audit_table()
<h3 style="margin:0 0 8px;font-size:1em;color:#455a64;">Audit Logs</h3>
<div style="display:flex;gap:8px;flex-wrap:wrap;font-size:0.85em;">
<button hx-get="/ui/soa/{{ soa_id }}/rollback_audit" hx-target="#modal-host" hx-swap="innerHTML" style="background:#37474f;color:#fff;border:none;padding:4px 10px;border-radius:4px;cursor:pointer;">Rollback Audit Log</button>
<a href="/soa/{{ soa_id }}/rollback_audit/export/xlsx" style="background:#1976d2;color:#fff;text-decoration:none;padding:4px 10px;border-radius:4px;">Rollback Audit XLSX</a>
Comment thread startup.sh Outdated
Comment on lines +193 to +196
"isRequired": bool(is_required) if is_required is not None else True,
"isEnabled": bool(is_enabled) if is_enabled is not None else True,
"datatype": datatype or "",
"responseCodes": [],
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 4, 2026 14:33
… remote source'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 31 changed files in this pull request and generated 11 comments.

Comment on lines +3989 to +4005
# Activities per timeline: an activity is shown in timeline T's matrix
# if any matrix_cells row connects it to an instance whose
# member_of_timeline == T. The instance->timeline link (set on the
# study_timing page) is the authoritative criterion.
instance_timeline = {
inst["id"]: (inst.get("member_of_timeline") or "unassigned")
for inst in instances
}
activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()}
for c in cells:
tl = instance_timeline.get(c["instance_id"])
if tl is None or tl not in activity_ids_by_timeline:
continue
activity_ids_by_timeline[tl].add(c["activity_id"])
activities_by_timeline: dict = {
tl: [a for a in activities_page if a["id"] in ids]
for tl, ids in activity_ids_by_timeline.items()
Comment on lines +174 to +179
"epochs": epochs,
"elements": elements,
"visits": visits,
"activities": activities,
"cells": cells,
"activity_concepts": concepts_map,
Comment on lines +422 to +425
cur.execute("DELETE FROM biomedical_concept WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM alias_code WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM code WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM code_association WHERE soa_id=?", (soa_id,))
<h3 style="margin:0 0 8px;font-size:1em;color:#455a64;">Audit Logs</h3>
<div style="display:flex;gap:8px;flex-wrap:wrap;font-size:0.85em;">
<button hx-get="/ui/soa/{{ soa_id }}/rollback_audit" hx-target="#modal-host" hx-swap="innerHTML" style="background:#37474f;color:#fff;border:none;padding:4px 10px;border-radius:4px;cursor:pointer;">Rollback Audit Log</button>
<a href="/soa/{{ soa_id }}/rollback_audit/export/xlsx" style="background:#1976d2;color:#fff;text-decoration:none;padding:4px 10px;border-radius:4px;">Rollback Audit XLSX</a>
raise HTTPException(404, "Objective not found")
before = _row_to_dict(row)

new_name = body.name if body.name is not None else before["name"]
Comment on lines +444 to +462
def get_next_extension_attribute_uid(cur: Any, soa_id: int) -> str:
"""Compute next unique ExtensionAttribute_N for the given SOA.

Assumes `cur` is a sqlite cursor within an open transaction.
"""
cur.execute(
"SELECT extension_attribute_uid FROM activity_concept_dss"
" WHERE soa_id=?"
" AND extension_attribute_uid LIKE 'ExtensionAttribute_%'",
(soa_id,),
)
existing = [x[0] for x in cur.fetchall() if x[0]]
n = 1
if existing:
try:
n = max(int(x.split("_")[1]) for x in existing) + 1
except Exception:
n = len(existing) + 1
return f"ExtensionAttribute_{n}"
Comment on lines +4052 to +4063
objective_level_options = sorted({v for v in c188725_map.values() if v})
endpoint_level_options = sorted({v for v in c188726_map.values() if v})
conn_obj = _connect()
cur_obj = conn_obj.cursor()
cur_obj.execute(
"SELECT code_uid, code FROM code_association "
"WHERE soa_id=? AND codelist_code IN ('C188725','C188726')",
(soa_id,),
)
level_code_to_sv: dict = {}
for code_uid, code_val in cur_obj.fetchall():
level_code_to_sv[code_uid] = code_val or ""
Comment on lines +82 to +92
# Upload database
echo -e "${YELLOW}Uploading database to Azure Files...${NC}"
echo "This may take a few moments depending on file size..."

az storage file upload \
--account-name "$STORAGE_ACCOUNT" \
--account-key "$STORAGE_KEY" \
--share-name "$FILE_SHARE" \
--source "$LOCAL_DB" \
--path soa_builder_web.db \
--no-progress
Comment thread docs/DEPLOYMENT_AZURE.md
Comment on lines +462 to +465
**⚠️ Important Notes:**
- **WAL mode files**: SQLite creates `-wal` and `-shm` files in WAL mode. These will be recreated automatically by Azure; you only need to upload the main `.db` file.
- **Database locking**: Ensure your local app is stopped before uploading the database to avoid corruption.
- **Schema migrations**: The app will automatically run any pending migrations on startup, so your local database will be updated to match the latest schema.
"DELETE FROM activity_concept WHERE activity_id IN"
" (SELECT id FROM activity WHERE soa_id=? )",
(soa_id,),
)
Copilot AI review requested due to automatic review settings May 4, 2026 15:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 31 changed files in this pull request and generated 7 comments.

Comment thread src/soa_builder/web/app.py
Comment on lines 4149 to +4154
"superscript_map": superscript_map,
"objectives": objectives,
"endpoints_by_objective": endpoints_by_objective,
"orphan_endpoints": orphan_endpoints,
"objective_level_options": objective_level_options,
"endpoint_level_options": endpoint_level_options,
Comment thread src/soa_builder/web/routers/_freeze_helpers.py
Comment thread src/soa_builder/web/routers/_freeze_helpers.py
Comment thread .github/workflows/azure-deploy.yml
Comment thread src/usdm/generate_biomedical_concept_properties.py
<a href="/soa/{{ soa_id }}/freeze/{{ f.id }}" target="_blank" style="background:#37474f;color:#fff;text-decoration:none;padding:2px 8px;border-radius:3px;font-size:0.85em;">JSON</a>
</td>
<td style="padding:4px;">
<form method="post" action="/ui/soa/{{ soa_id }}/freeze/{{ f.id }}/delete" onsubmit="return confirm('Delete frozen version {{ f.version_label }}? This cannot be undone.');" style="display:inline;">
Copilot AI review requested due to automatic review settings May 4, 2026 16:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 31 changed files in this pull request and generated 13 comments.

Comment on lines +3989 to +4005
# Activities per timeline: an activity is shown in timeline T's matrix
# if any matrix_cells row connects it to an instance whose
# member_of_timeline == T. The instance->timeline link (set on the
# study_timing page) is the authoritative criterion.
instance_timeline = {
inst["id"]: (inst.get("member_of_timeline") or "unassigned")
for inst in instances
}
activity_ids_by_timeline: dict = {tl: set() for tl in instances_by_timeline.keys()}
for c in cells:
tl = instance_timeline.get(c["instance_id"])
if tl is None or tl not in activity_ids_by_timeline:
continue
activity_ids_by_timeline[tl].add(c["activity_id"])
activities_by_timeline: dict = {
tl: [a for a in activities_page if a["id"] in ids]
for tl, ids in activity_ids_by_timeline.items()
Comment on lines +422 to +425
cur.execute("DELETE FROM biomedical_concept WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM alias_code WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM code WHERE soa_id=?", (soa_id,))
cur.execute("DELETE FROM code_association WHERE soa_id=?", (soa_id,))
Comment on lines 4149 to +4154
"superscript_map": superscript_map,
"objectives": objectives,
"endpoints_by_objective": endpoints_by_objective,
"orphan_endpoints": orphan_endpoints,
"objective_level_options": objective_level_options,
"endpoint_level_options": endpoint_level_options,
Comment on lines +61 to +71
- name: Azure login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Stop Azure Web App
uses: azure/cli@v1
with:
azcliversion: 2.36.0
inlineScript: |
az webapp stop --name ${{ secrets.AZURE_WEBAPP_NAME }} --resource-group cdisc-soa-workbench-rg

Comment on lines +202 to +225
new_name = body.name if body.name is not None else before["name"]
new_label = body.label if body.label is not None else before["label"]
new_desc = (
body.description if body.description is not None else before["description"]
)
new_text = body.text if body.text is not None else before["text"]

new_level_code_uid = before["level_code_uid"]
if body.level is not None:
new_level = body.level.strip()
if not new_level:
conn.close()
raise HTTPException(400, "Objective level cannot be empty")
if before["level_code_uid"]:
# Update the submission value in the existing Code_N row.
cur.execute(
"UPDATE code_association SET code=? WHERE soa_id=? AND code_uid=?",
(new_level, soa_id, before["level_code_uid"]),
)
else:
new_level_code_uid = _insert_level_code(cur, soa_id, new_level)

cur.execute(
"UPDATE objective SET name=?, label=?, description=?, text=?, "
Comment on lines +58 to +71
echo "Retrieving storage account key..."
STORAGE_KEY=$(az storage account keys list \
--resource-group "$RESOURCE_GROUP" \
--account-name "$STORAGE_ACCOUNT" \
--query '[0].value' \
--output tsv 2>&1)

if [ $? -ne 0 ]; then
echo -e "${RED}Error: Failed to retrieve storage account key${NC}"
echo "Please verify:"
echo " 1. Resource group '$RESOURCE_GROUP' exists"
echo " 2. Storage account '$STORAGE_ACCOUNT' exists"
echo " 3. You have permission to access the storage account"
exit 1
Comment on lines +451 to +463
for c in cells:
old_vid = c.get("visit_id")
old_aid = c.get("activity_id")
status = c.get("status", "").strip()
if status == "":
continue
vid = visit_id_map.get(old_vid)
aid = activity_id_map.get(old_aid)
if vid and aid:
cur.execute(
"INSERT INTO matrix_cells (soa_id, visit_id, activity_id, status)"
" VALUES (?,?,?,?)",
(soa_id, vid, aid, status),
study_id_val = row[2] if row else None
study_label_val = row[3] if row else None
study_description_val = row[4] if row else None
visits, activities, cells = _fetch_matrix(soa_id)
Comment on lines +114 to +122
cur.execute(
"SELECT id,soa_id,objective_uid,name,label,description,text,"
"level_code_uid,order_index "
"FROM objective WHERE soa_id=? ORDER BY order_index, id",
(soa_id,),
)
rows = [_row_to_dict(r) for r in cur.fetchall()]
conn.close()
return JSONResponse(rows)
Comment on lines +122 to +130
cur.execute(
"SELECT id,soa_id,endpoint_uid,objective_uid,name,label,"
"description,text,purpose,level_code_uid,order_index "
"FROM endpoint WHERE soa_id=? ORDER BY order_index, id",
(soa_id,),
)
rows = [_row_to_dict(r) for r in cur.fetchall()]
conn.close()
return JSONResponse(rows)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants