From dcd83cf4d3a9b583fbc7e20d01edaaccb23f9e27 Mon Sep 17 00:00:00 2001 From: Otto Date: Tue, 6 Oct 2026 15:44:11 -0400 Subject: [PATCH 1/4] Add Ward linkage clustering protocol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../protocol.md | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 protocols/ward-linkage-signature-matrix-clustering/protocol.md diff --git a/protocols/ward-linkage-signature-matrix-clustering/protocol.md b/protocols/ward-linkage-signature-matrix-clustering/protocol.md new file mode 100644 index 0000000..a5f5eb6 --- /dev/null +++ b/protocols/ward-linkage-signature-matrix-clustering/protocol.md @@ -0,0 +1,141 @@ +--- +name: "ward-linkage-signature-matrix-clustering" +description: "Cluster and display a signature similarity matrix using Ward minimum-variance hierarchical linkage." +version: "1.0.0" +authors: + - name: "Otto Infield-Harm" + orcid: "0009-0009-8629-6124" + - name: "Levi Waldron" + orcid: "0000-0003-2725-0694" +date: "2026-10-06" +status: draft +type: "atomic" + +artifact_doi: ~ +collection_doi: "10.5281/zenodo.22731694" +protocol_citation: "10.1080/01621459.1963.10500845" +method_origin_citation: "10.1080/01621459.1963.10500845" +license: "CC-BY-4.0" +protocols_used: [] +category: "Statistical Analysis" +tags: [clustering, hierarchical-clustering, heatmap, dendrogram, similarity-matrix] +--- + +# Ward-Linkage Clustering of a Signature Similarity Matrix + +This protocol orders and displays a signature-by-signature similarity matrix +using agglomerative hierarchical clustering with Ward minimum-variance +linkage. It is independent of how the similarities were calculated, so the +input may come from any similarity protocol that produces the required matrix +layout. + +## Materials + +- **Similarity matrix** + - A square numeric matrix with one row and one column for every signature. + - The row and column identifiers must be unique and must occur in the same + order. + - The matrix must be symmetric, with a documented diagonal and a documented + interpretation of larger values. +- **Ward-compatible distances** + - A distance representation for exactly the same signatures and ordering. + - Distances must be non-negative, symmetric, have zero diagonal, and be + appropriate for Ward's minimum-variance criterion. In particular, they + must be Euclidean-compatible. + - Any similarity-to-distance conversion or other repair of the input is + completed and documented before this protocol begins; this protocol does + not prescribe that preprocessing. +- **Signature annotations** + - Study, condition, body site, and signature length for every signature when + those metadata are available. + - Annotation values must be joined by signature identifier, not by matrix + position inferred from a separate ordering. + +## Steps + +### Step 1: Verify and record the inputs + +Confirm that the similarity matrix and distance representation contain the +same signatures in the same order. Verify the matrix properties and the +distance prerequisites before clustering. Reject the input and report the +failed property if an identifier is duplicated or missing, the two +representations do not align, or the distance representation is not +Ward-compatible. Do not silently reorder, impute, symmetrize, or transform +the inputs. + +Record the source and version of the similarity matrix, the distance +representation and its construction, the signature ordering, the number of +signatures, and the checks performed. + +### Step 2: Perform Ward hierarchical clustering + +Start with each signature as its own cluster and repeatedly merge the pair of +clusters that gives the smallest increase in within-cluster sum of squares +under Ward linkage. Continue until all signatures form one hierarchy. + +Use a deterministic tie rule and record it when two candidate merges have the +same objective increase. Record the linkage method, distance representation, +tie rule, merge sequence or dendrogram, and any software and version used. + +The dendrogram represents the sequence and relative objective cost of the +merges. A branch groups signatures according to the supplied distances; it +does not establish a biological taxon, causal relationship, statistical +significance, or condition enrichment. + +### Step 3: Choose an ordering and display the matrix + +Use the leaf order of the Ward dendrogram to order both rows and columns of +the original similarity matrix. Apply the identical order to every annotation +track. Do not replace the similarity values with distances or reorder rows +and columns independently. + +Display the ordered similarity matrix with the dendrogram and annotation +tracks for the available study, condition, body-site, and signature-length +metadata. Include a legend or scale identifying the meaning and range of the +similarity values. + +If discrete cluster labels are required, choose the number of clusters or +cutting rule before interpreting the annotations, apply it to the recorded +dendrogram, and report the rule and resulting membership. A dendrogram and +heatmap do not require a fixed number of clusters. + +## Output + +Produce: + +- the original similarity matrix with rows and columns in the recorded Ward + leaf order; +- the Ward dendrogram and its merge heights or objective increases; +- annotation tracks joined to the same ordered signature identifiers; and +- the recorded input checks, clustering parameters, tie rule, and any + discrete cluster-cut rule. + +The similarity values must remain unchanged apart from their row and column +ordering. + + +## Notes + +Ward's paper introduced hierarchical grouping by minimizing an objective +function; this protocol specifies its minimum-variance linkage form. Ward +linkage is therefore a deliberate choice here, not an implementation default. +The Euclidean-compatibility requirement belongs to the clustering method, not +to any particular similarity measure. + +## Out of scope + +This protocol does not calculate similarity, harmonize taxa, convert +similarities to distances, repair invalid matrices, test clusters for +condition enrichment, or interpret biological associations. Those operations +must be specified separately. + +## History & Reviews + + +### Version 1.0.0 (2026-10-06) + +#### Changes +- Initial protocol creation. + +#### Reviews +*No reviews yet.* From 615953b1194b89aefa85d89b074474b0a52538ec Mon Sep 17 00:00:00 2001 From: Levi Waldron Date: Sat, 17 Oct 2026 22:42:07 -0400 Subject: [PATCH 2/4] Define distance conversion, squared-distance Ward criterion, and tie rule Take the similarity-to-distance conversion (1 - s) into this protocol as issue #9 specifies, state that the Ward criterion is applied to squared distances (ward.D2, not ward.D), give the Lance-Williams update, define the default tie rule by input index, and require the leaf order to be recorded. Drafted with Claude Code. --- .../protocol.md | 107 +++++++++++------- 1 file changed, 69 insertions(+), 38 deletions(-) diff --git a/protocols/ward-linkage-signature-matrix-clustering/protocol.md b/protocols/ward-linkage-signature-matrix-clustering/protocol.md index a5f5eb6..5969f13 100644 --- a/protocols/ward-linkage-signature-matrix-clustering/protocol.md +++ b/protocols/ward-linkage-signature-matrix-clustering/protocol.md @@ -23,11 +23,12 @@ tags: [clustering, hierarchical-clustering, heatmap, dendrogram, similarity-matr # Ward-Linkage Clustering of a Signature Similarity Matrix -This protocol orders and displays a signature-by-signature similarity matrix -using agglomerative hierarchical clustering with Ward minimum-variance -linkage. It is independent of how the similarities were calculated, so the -input may come from any similarity protocol that produces the required matrix -layout. +This protocol converts a signature-by-signature similarity matrix to +distances, clusters the signatures by agglomerative hierarchical clustering +with Ward minimum-variance linkage, and displays the similarity matrix in the +resulting order. It is independent of how the similarities were calculated, +so the input may come from any similarity protocol that produces the required +matrix layout, such as `jaccard-signature-similarity`. ## Materials @@ -35,16 +36,16 @@ layout. - A square numeric matrix with one row and one column for every signature. - The row and column identifiers must be unique and must occur in the same order. - - The matrix must be symmetric, with a documented diagonal and a documented - interpretation of larger values. -- **Ward-compatible distances** - - A distance representation for exactly the same signatures and ordering. - - Distances must be non-negative, symmetric, have zero diagonal, and be - appropriate for Ward's minimum-variance criterion. In particular, they - must be Euclidean-compatible. - - Any similarity-to-distance conversion or other repair of the input is - completed and documented before this protocol begins; this protocol does - not prescribe that preprocessing. + - The matrix must be symmetric, with every value between 0 and 1 + inclusive, a diagonal of 1, and larger values meaning more similar. + Jaccard similarity satisfies these conditions. + - Every entry must be present. A matrix with missing entries, for example + one produced with pair restrictions, cannot be clustered by this protocol. +- **Parameters** + - `tie_rule`: when two or more candidate merges have exactly the same + objective increase, merge the pair whose members have the smallest + original input indices, comparing the smaller index of each pair first + and then the larger. This is the default and the only rule defined here. - **Signature annotations** - Study, condition, body site, and signature length for every signature when those metadata are available. @@ -53,29 +54,51 @@ layout. ## Steps -### Step 1: Verify and record the inputs +### Step 1: Verify the input and convert similarity to distance -Confirm that the similarity matrix and distance representation contain the -same signatures in the same order. Verify the matrix properties and the -distance prerequisites before clustering. Reject the input and report the -failed property if an identifier is duplicated or missing, the two -representations do not align, or the distance representation is not -Ward-compatible. Do not silently reorder, impute, symmetrize, or transform -the inputs. +Verify the matrix properties before clustering. Reject the input and report +the failed property if an identifier is duplicated or missing, the matrix is +not square or symmetric, any entry is missing or outside 0 to 1, or the +diagonal is not 1. Do not silently reorder, impute, symmetrize, or transform +the input. -Record the source and version of the similarity matrix, the distance -representation and its construction, the signature ordering, the number of -signatures, and the checks performed. +Convert the similarity matrix to a distance matrix by subtracting every +entry from 1: + +```text +d(i, j) = 1 - s(i, j) +``` + +The result is non-negative, symmetric, and zero on the diagonal. Use it only +for clustering; the similarity values are what is displayed in Step 3. + +Record the source and version of the similarity matrix, the signature +ordering, the number of signatures, and the checks performed. ### Step 2: Perform Ward hierarchical clustering Start with each signature as its own cluster and repeatedly merge the pair of -clusters that gives the smallest increase in within-cluster sum of squares -under Ward linkage. Continue until all signatures form one hierarchy. +clusters whose merge gives the smallest increase in the within-cluster sum of +squared distances. Continue until all signatures form one hierarchy. + +Apply Ward's criterion to the squared distances. In the Lance-Williams +update, after merging clusters `A` and `B`, the squared distance from the +merged cluster to any other cluster `C` is: + +```text +d²(A∪B, C) = ((|A| + |C|) d²(A, C) + (|B| + |C|) d²(B, C) - |C| d²(A, B)) + / (|A| + |B| + |C|) +``` + +where `|X|` is the number of signatures in cluster `X`. The merge height +reported for each step is the distance, not the squared distance. In R this +is `hclust(as.dist(d), method = "ward.D2")`. Do not use `method = "ward.D"`, +which applies the same update to unsquared distances and produces a +different hierarchy from the same input. -Use a deterministic tie rule and record it when two candidate merges have the -same objective increase. Record the linkage method, distance representation, -tie rule, merge sequence or dendrogram, and any software and version used. +Break ties with `tie_rule`. Record the linkage method, the tie rule, whether +any tie occurred, the merge sequence with its heights, the leaf order of the +resulting dendrogram, and the software and version used. The dendrogram represents the sequence and relative objective cost of the merges. A branch groups signatures according to the supplied distances; it @@ -117,16 +140,24 @@ ordering. ## Notes Ward's paper introduced hierarchical grouping by minimizing an objective -function; this protocol specifies its minimum-variance linkage form. Ward -linkage is therefore a deliberate choice here, not an implementation default. -The Euclidean-compatibility requirement belongs to the clustering method, not -to any particular similarity measure. +function; this protocol specifies its minimum-variance linkage form applied +to squared distances, which is the criterion Ward described. Ward linkage is +therefore a deliberate choice here, not an implementation default. The +source analyses applied it to one minus Jaccard similarity, which is a +metric but not in general a Euclidean distance; the criterion is applied to +it regardless, as a documented choice. + +The leaf order of a dendrogram is not unique: either child of a merge can be +drawn first without changing the hierarchy, and implementations differ in +the convention they use. This protocol does not prescribe one. It requires +the leaf order to be recorded so that the displayed matrix can be +reproduced. ## Out of scope -This protocol does not calculate similarity, harmonize taxa, convert -similarities to distances, repair invalid matrices, test clusters for -condition enrichment, or interpret biological associations. Those operations +This protocol does not calculate similarity, harmonize taxa, repair invalid +matrices, test clusters for condition enrichment, or interpret biological +associations. Those operations must be specified separately. ## History & Reviews From ed30afa2b7e4a3dcb7fb3e1cc8d8ff6c9343bc8c Mon Sep 17 00:00:00 2001 From: Levi Waldron Date: Sat, 17 Oct 2026 22:44:50 -0400 Subject: [PATCH 3/4] Replace the Lance-Williams formula with named R and Python calls Drafted with Claude Code. --- .../protocol.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/protocols/ward-linkage-signature-matrix-clustering/protocol.md b/protocols/ward-linkage-signature-matrix-clustering/protocol.md index 5969f13..45a83b4 100644 --- a/protocols/ward-linkage-signature-matrix-clustering/protocol.md +++ b/protocols/ward-linkage-signature-matrix-clustering/protocol.md @@ -81,20 +81,21 @@ Start with each signature as its own cluster and repeatedly merge the pair of clusters whose merge gives the smallest increase in the within-cluster sum of squared distances. Continue until all signatures form one hierarchy. -Apply Ward's criterion to the squared distances. In the Lance-Williams -update, after merging clusters `A` and `B`, the squared distance from the -merged cluster to any other cluster `C` is: +Apply Ward's criterion to the squared distances, and report each merge +height as a distance rather than a squared distance. This is the criterion +Ward described, and it is what the following calls compute from the +distance matrix `d` of Step 1: -```text -d²(A∪B, C) = ((|A| + |C|) d²(A, C) + (|B| + |C|) d²(B, C) - |C| d²(A, B)) - / (|A| + |B| + |C|) +```r +hclust(as.dist(d), method = "ward.D2") +``` + +```python +scipy.cluster.hierarchy.linkage(scipy.spatial.distance.squareform(d), method="ward") ``` -where `|X|` is the number of signatures in cluster `X`. The merge height -reported for each step is the distance, not the squared distance. In R this -is `hclust(as.dist(d), method = "ward.D2")`. Do not use `method = "ward.D"`, -which applies the same update to unsquared distances and produces a -different hierarchy from the same input. +Do not use R's `method = "ward.D"`, which applies the criterion to unsquared +distances and produces a different hierarchy from the same input. Break ties with `tie_rule`. Record the linkage method, the tie rule, whether any tie occurred, the merge sequence with its heights, the leaf order of the From 84159809cf28a0ac14b1a15783694b51f20db3ab Mon Sep 17 00:00:00 2001 From: Levi Waldron Date: Sat, 17 Oct 2026 23:09:05 -0400 Subject: [PATCH 4/4] Record review by Levi Waldron Drafted with Claude Code. --- .../protocol.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/protocols/ward-linkage-signature-matrix-clustering/protocol.md b/protocols/ward-linkage-signature-matrix-clustering/protocol.md index 45a83b4..fdc36a7 100644 --- a/protocols/ward-linkage-signature-matrix-clustering/protocol.md +++ b/protocols/ward-linkage-signature-matrix-clustering/protocol.md @@ -7,6 +7,12 @@ authors: orcid: "0009-0009-8629-6124" - name: "Levi Waldron" orcid: "0000-0003-2725-0694" +reviews: + - name: "Levi Waldron" + orcid: "0000-0003-2725-0694" + date: "2026-10-17" + protocol_version: "1.0.0" + status: "approved" date: "2026-10-06" status: draft type: "atomic" @@ -170,4 +176,8 @@ must be specified separately. - Initial protocol creation. #### Reviews -*No reviews yet.* + +**Review by Levi Waldron ([0000-0003-2725-0694](https://orcid.org/0000-0003-2725-0694))** +- **Date:** 2026-10-17 +- **Status:** `approved` +- **Notes:** Brought the similarity-to-distance conversion into the protocol as issue #9 specifies, stated that Ward's criterion is applied to squared distances and named the R (`ward.D2`) and scipy calls that compute it, defined the default tie rule by input index, and required the leaf order to be recorded.