Release v 1.1 by pendingintent · Pull Request #187 · pendingintent/soa-workbench

pendingintent · 2026-03-30T19:54:05Z

Added submodule to support validation against the USDM v4.0 schema .
Added a markdown file to explain the biomedical concept workflow.
Various fixes.

…tnotes

+    _record_bc_surrogate_audit(
+        soa_id, "update", surrogate_id, before=before, after=after
+    )
+    return RedirectResponse(f"/ui/soa/{soa_id}/activities", status_code=303)


+    _record_bc_surrogate_audit(
+        soa_id, "delete", surrogate_id, before=before, after=None
+    )
+    return RedirectResponse(f"/ui/soa/{soa_id}/activities", status_code=303)


Copilot

Pull request overview

Release-oriented update for the SoA Workbench adding USDM v4.0 support pieces (schema validation submodule + USDM output extensions), plus new UI/API capabilities for biomedical concept surrogates and matrix footnotes/superscripts.

Changes:

Add Biomedical Concept Surrogates (DB tables, API/UI router, USDM export, tests, and UI integration in concepts cells / XLSX export).
Add footnotes + matrix cell superscripts (DB migrations, UI endpoints, edit-page rendering).
USDM generator tweaks and performance improvements (prefetching mappings, safer timing code objects, extra indexes).

Reviewed changes

Copilot reviewed 30 out of 46 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
tests/test_routers_bc_surrogates.py	Adds API + USDM integration tests for BC surrogates.
tests/test_routers_activities.py	Updates reorder tests to new request shape.
src/usdm/usdm_utils.py	Adjusts BC property output fields and AliasCode typing.
src/usdm/generate_usdm.py	Adds `bcSurrogates` to the full USDM StudyVersion output; improves logging.
src/usdm/generate_study_timings.py	Makes timing `type` / `relativeToFrom` optional code objects.
src/usdm/generate_encounters.py	Prefetches code/rule/timing lookups to reduce per-row queries; tweaks transition rule nullability.
src/usdm/generate_biomedical_concepts.py	Provides a default `reference` URL when missing.
src/usdm/generate_bc_surrogates.py	New USDM generator for BC surrogates.
src/usdm/generate_activities.py	Prefetches concept/surrogate mappings; emits `bcSurrogateIds`.
src/soa_builder/web/templates/edit.html	Renders superscripts + inline edit controls; adds footnote display and management UI.
src/soa_builder/web/templates/concepts_cell.html	Shows surrogate chips alongside concept chips and updates counts/actions.
src/soa_builder/web/templates/activities.html	Adds UI for defining/editing BC surrogates and linking them to activities.
src/soa_builder/web/schemas.py	Adds Pydantic schemas for BC surrogate create/update payloads.
src/soa_builder/web/routers/visits.py	Adjusts transaction commit placement; fixes connection close on 404.
src/soa_builder/web/routers/rollback.py	Removes unused DB_PATH constant.
src/soa_builder/web/routers/freezes.py	Uses shared `_connect()`; removes local SQLite wiring.
src/soa_builder/web/routers/footnotes.py	New footnotes API/UI router with audit logging.
src/soa_builder/web/routers/bc_surrogates.py	New BC surrogate API/UI router + activity linking endpoints + HTMX partial rendering.
src/soa_builder/web/routers/activities.py	Makes activity_uid immutable; changes reorder body shape; adds surrogate data for templates.
src/soa_builder/web/migrate_database.py	Adds indexes + new tables/columns for footnotes, superscripts, and surrogates.
src/soa_builder/web/initialize_database.py	Creates footnote table during fresh DB init.
src/soa_builder/web/db.py	Improves PRAGMA failure logging.
src/soa_builder/web/audit.py	Adds audit recorders for footnotes and BC surrogates; removes defensive CREATE TABLE blocks.
src/soa_builder/web/app.py	Integrates new migrations/routers; adds superscript helpers and surrogate export/XLSX support.
requirements.txt	Loosens requests dependency constraint.
README.md	Adds submodule clone instructions; updates USDM export section.
output/report/NCT01797120_validation_report.xlsx	Adds a generated validation report artifact.
images/image-7.png	Documentation screenshot (surrogate selection UI).
images/image-5.png	Documentation screenshot.
images/image-4.png	Documentation screenshot.
files/NCT01797120_Footnote_2.html	Example footnote HTML content.
files/NCT01797120_Footnote_1.html	Example footnote HTML content (currently malformed).
docs/BC_WORKFLOW.md	New/expanded documentation for BC/DSS workflow and surrogates.
.gitmodules	Adds `cdisc-json-validation` submodule reference.
.gitignore	Adjusts ignored output patterns and adds additional local artifact ignores.

Copilot · 2026-03-30T20:03:50Z

+    resp = client.post(
+        f"/soa/{soa_id}/activities/reorder", json={"order": [a3, a1, a2]}
+    )


The reorder tests now POST a JSON object {order: [...]} to /soa/{soa_id}/activities/reorder, but the current handler (src/soa_builder/web/app.py at the same route) expects the request body to be a bare JSON array (List[int]). This will result in a 422 and make the test fail.

Align the test payload shape with the actual endpoint contract (or update the endpoint to Body(..., embed=True) consistently).

…issues

+    if request.headers.get("HX-Request") == "true":
+        return HTMLResponse("", headers={"HX-Redirect": redirect_url})
+
+    return RedirectResponse(redirect_url, status_code=303)


+    if request.headers.get("HX-Request") == "true":
+        return HTMLResponse("", headers={"HX-Redirect": redirect_url})
+
+    return RedirectResponse(redirect_url, status_code=303)


+    if request.headers.get("HX-Request") == "true":
+        return HTMLResponse("", headers={"HX-Redirect": redirect_url})
+
+    return RedirectResponse(redirect_url, status_code=303)


Copilot

Pull request overview

Copilot reviewed 30 out of 46 changed files in this pull request and generated 3 comments.

… so deleted UIDs are never recycled

Copilot

Pull request overview

Copilot reviewed 30 out of 46 changed files in this pull request and generated 6 comments.

Copilot · 2026-03-31T15:57:28Z

+    hx_vals_attr = _html.escape(hx_vals_json, quote=True)
+
+    return (
+        f'<td hx-post="/ui/soa/{soa_id_safe}/toggle_cell"'


_render_cell_td() returns a <td> without data-visit-id / data-instance-id. The matrix reorder JS in edit.html relies on td[data-visit-id] to reorder columns; after an HTMX swap from toggle/superscript-save, those cells will no longer participate in reordering. Include the same data-* attributes in the returned <td> so swapped cells keep working with the existing client-side reorder logic.

Suggested change

f'<td hx-post="/ui/soa/{soa_id_safe}/toggle_cell"'

f'<td data-instance-id="{instance_id_safe}" data-activity-id="{activity_id_safe}"'

f' hx-post="/ui/soa/{soa_id_safe}/toggle_cell"'

Copilot · 2026-03-31T15:57:29Z

+    conn.close()
+    _record_bc_surrogate_audit(
+        soa_id, "delete", surrogate_id, before=before, after=None
+    )
+    return RedirectResponse(f"/ui/soa/{soa_id}/activities", status_code=303)


_render_concepts_cell() queries activity_concept with ... WHERE activity_id=? AND soa_id=?. Elsewhere the codebase defensively checks whether activity_concept has an soa_id column for backward-compatible databases. To avoid breaking upgraded deployments with legacy schemas, add the same column-existence guard here (or join via activity when soa_id is absent).

Copilot · 2026-03-31T15:57:29Z

+    """Add standalone soa_id indexes on high-traffic tables.
+
+    The existing UNIQUE constraints cover (soa_id, uid) lookups, but bare
+    WHERE soa_id=? list queries do full table scans without a leading index.
+    These indexes cover the ~259 soa_id filter sites in the codebase.
+    """
+    try:
+        conn = _connect()
+        cur = conn.cursor()
+        indexes = [
+            ("idx_activity_soa", "activity", "soa_id"),
+            ("idx_visit_soa", "visit", "soa_id"),
+            ("idx_matrix_cells_soa", "matrix_cells", "soa_id"),
+            ("idx_activity_concept_soa", "activity_concept", "soa_id"),
+            ("idx_instances_soa", "instances", "soa_id"),
+            ("idx_timing_soa", "timing", "soa_id"),


Most of these new indexes appear redundant because the tables already have UNIQUE constraints whose backing indexes start with soa_id (e.g., UNIQUE(soa_id, activity_uid) on activity, UNIQUE(soa_id, encounter_uid) on visit, etc.), which SQLite can use for WHERE soa_id=? queries. Adding additional single-column soa_id indexes increases write cost and DB size without improving those queries. Consider limiting this migration to tables that don’t already have a leading (soa_id, ...) index (e.g., matrix_cells, activity_concept).

Suggested change

"""Add standalone soa_id indexes on high-traffic tables.

The existing UNIQUE constraints cover (soa_id, uid) lookups, but bare

WHERE soa_id=? list queries do full table scans without a leading index.

These indexes cover the ~259 soa_id filter sites in the codebase.

"""

try:

conn = _connect()

cur = conn.cursor()

indexes = [

("idx_activity_soa", "activity", "soa_id"),

("idx_visit_soa", "visit", "soa_id"),

("idx_matrix_cells_soa", "matrix_cells", "soa_id"),

("idx_activity_concept_soa", "activity_concept", "soa_id"),

("idx_instances_soa", "instances", "soa_id"),

("idx_timing_soa", "timing", "soa_id"),

"""Add standalone soa_id indexes on selected high-traffic tables.

The existing UNIQUE constraints cover (soa_id, uid) lookups, but bare

WHERE soa_id=? list queries do full table scans without a leading index.

These indexes cover the ~259 soa_id filter sites in the codebase, limited

to tables that do not already have a leading (soa_id, ...) index.

"""

try:

conn = _connect()

cur = conn.cursor()

# Only add standalone soa_id indexes for tables that lack an existing

# leading (soa_id, ...) index from a UNIQUE constraint.

indexes = [

("idx_matrix_cells_soa", "matrix_cells", "soa_id"),

("idx_activity_concept_soa", "activity_concept", "soa_id"),

Copilot · 2026-03-31T15:57:29Z

+        _type_entry = type_code_map.get(type, ([], [], [], []))
+        t_code, t_decode, t_codeSystem, t_codeSystemVersion = _type_entry



type_code_map.get(type, ([],[],[],[])) can yield empty lists when visit.type is NULL/blank or when the code association is missing. The code later indexes into t_code[0] / t_decode[0], which will raise IndexError and fail USDM encounter generation. Treat type as optional (set it to None when missing/unmapped), similar to how timings now handle missing codes.

Copilot · 2026-03-31T15:57:30Z

@@ -0,0 +1,414 @@
+# Workflow for linking Biomedical Concept with Scheduled Activity Instance
+
+Assumes that the Activties and Scheduled Activity Instances have been created to form the SOA Matrix.


Typo: “Activties” → “Activities”. This affects readability and searchability in the documentation.

Suggested change

Assumes that the Activties and Scheduled Activity Instances have been created to form the SOA Matrix.

Assumes that the Activities and Scheduled Activity Instances have been created to form the SOA Matrix.

Copilot · 2026-03-31T15:57:30Z

+    cur.execute(
+        "INSERT INTO footnote (soa_id, footnote_uid, name, label, description, text, dictionary_uid) VALUES (?,?,?,?,?,?,?)",
+        (
+            soa_id,
+            uid,
+            body.name,
+            body.label or None,
+            body.description or None,
+            body.text or None,
+            body.dictionary_uid or None,
+        ),
+    )


Footnote text is stored verbatim (intended XHTML) and later rendered with | safe in the edit template. Without server-side sanitization/validation, this creates a stored-XSS risk. Consider sanitizing text on create/update (e.g., allowlist tags/attributes) or storing a safe structured representation.

pendingintent added 17 commits March 10, 2026 09:25

Issue #153: Reviewed database operations and addressed identified issues

316ea49

Added lines for .scripts directory and interim USDM

5ea1556

Fixed 404 on activity save

ece559e

Issue #164: Added new footnotes feature

95ee796

Added cell superscript functionality to the SOA MAtrix to support foo…

eea7f0b

…tnotes

Ignore more local files

8b79937

Issue #168: Added support for Biomedical Concept Surrogates

c6261ab

Fixed USDM issues #176, #177, #178, #179

1d0c2a9

Added cdisc-json-validation repository as a submodule

9c38f19

Added to support JSON validation against schema

8a03176

USDM JSON generated with version 2 beta

458a5c5

Schema validation of version 2.0 beta USDM JSON

27e4aaa

More explicit rules

922f0c9

Issue #180: fixed schedule timelines USDM

a95699b

Updated with latest instructions

f3c7115

Added images and markdown to explain bc workflow

ff780c0

Updated to latest version of requests module

60317d1

pendingintent added this to the v1.1 milestone Mar 30, 2026

pendingintent self-assigned this Mar 30, 2026

pendingintent added the documentation Improvements or additions to documentation label Mar 30, 2026

Copilot AI review requested due to automatic review settings March 30, 2026 19:54

pendingintent added the enhancement New feature or request label Mar 30, 2026

github-advanced-security AI found potential problems Mar 30, 2026

View reviewed changes

Copilot started reviewing on behalf of pendingintent March 30, 2026 19:54 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

pendingintent added 5 commits March 30, 2026 16:10

Fixed Uncontrolled data used in path expression

51aa11c

Fixed documentation syntax issues

b260fc3

Added pydantic request modesls for the JSON API routes

40e5826

Merge branch 'fix-footnotes' into fix-documentation-issues

1cd8a9d

Merge branch 'fix-uncontrolled-data-security' into fix-documentation-…

09c64ad

…issues

pendingintent added 7 commits March 31, 2026 08:52

Fixed syntax issues

46179a1

Fixed issues reported in the app.py;removed deprecated code

cc2ffb5

Removed activities reoreder from app.py

ead3b4a

Fixed potential XSS issue

62af511

Merge branch 'fix-footnotes' into release-v-1.1

3a0545c

Fixed potential XSS issue

73f087c

Merge branch 'fix-app-scan-issues' into release-v-1.1

1582b6e

github-advanced-security AI found potential problems Mar 31, 2026

View reviewed changes

pendingintent requested a review from Copilot March 31, 2026 14:43

Copilot started reviewing on behalf of pendingintent March 31, 2026 14:44 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Comment thread src/soa_builder/web/templates/edit.html Outdated

Comment thread src/soa_builder/web/routers/bc_surrogates.py Outdated

Comment thread src/soa_builder/web/routers/activities.py

pendingintent added 3 commits March 31, 2026 11:01

Wrapped form tag in <td>

ccf1b1f

Merge branch 'fix-edit-syntax' into release-v-1.1

9630bf4

_next_surrogate_uid now Scans both the live table and the audit trail…

c83e8ad

… so deleted UIDs are never recycled

pendingintent requested a review from Copilot March 31, 2026 15:48

Copilot started reviewing on behalf of pendingintent March 31, 2026 15:49 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

pendingintent merged commit 60ce0d1 into master Mar 31, 2026
9 checks passed

	f'<td hx-post="/ui/soa/{soa_id_safe}/toggle_cell"'
	f'<td data-instance-id="{instance_id_safe}" data-activity-id="{activity_id_safe}"'
	f' hx-post="/ui/soa/{soa_id_safe}/toggle_cell"'

		_type_entry = type_code_map.get(type, ([], [], [], []))
		t_code, t_decode, t_codeSystem, t_codeSystemVersion = _type_entry

		@@ -0,0 +1,414 @@
		# Workflow for linking Biomedical Concept with Scheduled Activity Instance

		Assumes that the Activties and Scheduled Activity Instances have been created to form the SOA Matrix.

	Assumes that the Activties and Scheduled Activity Instances have been created to form the SOA Matrix.
	Assumes that the Activities and Scheduled Activity Instances have been created to form the SOA Matrix.

Conversation

pendingintent commented Mar 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Copilot Autofix

Check warning

Uh oh!

Copilot Autofix

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Copilot Autofix

Check warning

Uh oh!

Copilot Autofix

Check warning

Uh oh!

Copilot Autofix

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!