diff --git a/.github/workflows/compile_go.yml b/.github/workflows/compile_go.yml
new file mode 100644
index 0000000..9b8c9c8
--- /dev/null
+++ b/.github/workflows/compile_go.yml
@@ -0,0 +1,28 @@
+# This workflow will build a golang project
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
+
+name: Go
+
+on:
+ push:
+ branches: [ "master" ]
+ pull_request:
+
+jobs:
+
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Set up Go
+ uses: actions/setup-go@v4
+ with:
+ go-version: '1.23'
+
+ - name: Build
+ run: go build -v -o ptra cmd/main.go
+
+ - name: Test
+ run: cd ptra_test && go test
+
diff --git a/LICENSE.txt b/LICENSE.txt
index 917c2f2..854920e 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -1,5 +1,5 @@
PTRA: Patient Trajectory Analysis Library.
-Copyright (c) 2022 imec vzw.
+Copyright (c) 2022-2026 imec vzw.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
diff --git a/README.md b/README.md
index b0dda95..2c5c099 100644
--- a/README.md
+++ b/README.md
@@ -89,9 +89,10 @@ The clustering is by default done via the [MCL](https://micans.org/mcl/) tool.
# 6. Building
-The following information is relevant if you want to build `ptra` to obtain an executable. `ptra` out of the box implements
-one use case (TriNetX), and building an executable hence by default generates a binary for this use case. For
-instructions how to organise the code to build a binary for a new use case, please see the Implementation Manual.
+The following information is relevant if you want to build `ptra` to obtain an executable. The `ptra` repository also
+contains code for one use case (TriNetX). You can either build a generic `ptra` binary or a binary for the TriNetX use
+case. For instructions how to organise the code to build a binary for a new use case, please see the Implementation
+Manual.
`ptra` is implemented in Go. Please make sure you have a working installation of Go. You can install Go from the [Go
website](https://golang.org/). Alternatively, most package managers provide options to install Go as a development tool.
@@ -99,17 +100,217 @@ Check the documentation of the package manager of your Linux distribution for de
First checkout the `ptra` sources using the following command:
- go get -u github.com/exascience/ptra
+ git clone https://github.com/ExaScience/ptra.git
-This downloads the `ptra` Go source code, and creates the `ptra` binary in your configured Go home folder, for example
-`~/go/bin/ptra`. See the [GOPATH](https://golang.org/cmd/go/#hdr-GOPATH_environment_variable) variable for your Go home
-folder.
+This downloads the `ptra` Go source code.
+
+To build the generic binary:
+
+ go build -o ptra cmd/main.go
+
+To build the binary for the TrinetX use case:
+
+ go build -o ptra_trinetx cmd/main_trinetx.go
Add the binary to your path, for example:
- export PATH=$PATH:~/go/bin
+ export PATH=$PATH:~/pra/ptra
# 7. Command Line Interface Reference (CLI)
+## Generic interface
+### Name
+`ptra` - a commandline tool for extracting statistically relevant trajectories from ICD10 diagnosis histories in CDC or
+WHO format
+
+### Synopsis
+
+```
+ ptra patientInfoFile diagnosisInfoFile diagnosesFile outputPath
+ --nofAgeGroups nr --lvl nr --minPatients nr --maxYears nr --minYears nr --maxTrajectoryLength nr
+ --minTrajectoryLength nr --name string --cluster --mclPath string --iter nr --saveRR file --loadRR file
+ --pfilters age+:nr,age-:nr,sex:[male | female]
+ --tfilters cat:[neoplasms|bc],code:icd10;icd10,...
+ --actFile file
+ --loglvl error | warning | info | debug | all
+```
+### Description
+The `ptra` command requires four arguments:
+
+1. `patientInfoFile`: this is a csv file containing patient information. The expected csv header is:
+ `patient_id, sex, year_of_birth, patient_regional_location, month_year_death`
+
+2. `diagnosisInfoFile`: this is a file mapping diagnosis IDs (ICD10) onto medical descriptions. This can
+ either be:
+ * an XML file containing the CDC ICD10 hierarchy with medical descriptors ([icd10cm_tabular_2022.xml](https://www.cms.gov/medicare/icd-10/2022-icd-10-cm))
+ * an XML file containing the WHO ICD10 hierarchy
+ * a CCSR with CCSR categorization of the ICD10 hierarchy ([DXCCSR_v2022-1.CSV](https://www.hcup-us.ahrq.gov/toolssoftware/ccsr/dxccsr.jsp))
+
+3. `diagnosesFile`: this is a csv file containing dated diagnoses for the patients in patientInfoFile. The expected
+ csv header is: `patient_id,encounter_id,code_system, code, principal_diagnosis_indicator, admiting_diagnosis, reason_for_visit, date,
+ derived_by_trinetx, source_id`
+
+4. `outputPath`: a path where the outputs of the `ptra` run can be written.
+
+`ptra` creates multiple output files. This output is structured as follows:
+
+1. A folder `unclustered-trajectories` with all trajectories found by `ptra` (before clustering). This folder contains the
+ following files:
+
+ 1. a tab file with the found trajectories. The tab file contains two lines per trajectory. The first line lists the diagnoses
+ in the trajectory, separated by tabs. The second line lists the number of patients between each transition in the trajectory.
+
+ Example:
+
+ ```
+ Cough \tab Dyspnea \tab COPD
+ 150 \tab 50
+ ```
+ 2. a tab file with the found diagnosis pairs and their relative risk scores. There is a single line that list the diagnoses and the RR.
+
+ Example:
+
+ ```Cough \tab Dyspnea \tab 1.95```
+
+ 3. a gml file with all found trajectories as individual graphs. Useful for verifying outputs with few trajectories.
+
+ 4. a gml file with all found trajectories plotted as one (large) graph. Useful for verifying output, but may be too
+ dense to view for large numbers of trajectories. In this case, clustering is recommended.
+
+2. a folder `clustered-trajectories` with clustered trajectories --if `ptra` was requested to cluster its output
+ (`--cluster` flag). This folder contains multiple subfolders:
+
+ 1. A folder `mclOutput` with intermediate clustering output generated by the MCL tool. This is normally not relevant
+ to view for `ptra` users. It is useful for the `ptra` developers for debugging.
+
+ 2. A number of `clusters` folders containing clustered trajectories. A folder is generated for each chosen cluster
+ granularity when calling `ptra`, cf. `--clusterGranularities` option. By default, 4 granuarities are tried, so 4
+ `clusters` folders are created. Each `clusters` folder contains the following files:
+
+ 1. a csv file with cluster information. The header is: `PID,CID,TID,Age`. These represent the patient identifier, cluster
+ identifier, trajectory identifier, and age of the patient at the time they completed the trajectory.
+
+ 2. a csv file with information to link the patient analysis identifier used in `ptra` back to the TriNetX identifier. The
+ header of the csv file is: `PID,AgeEOI,Sex,PIDString`. This represents the patient id used in `ptra`, the age of the
+ patient at the event of interest, the sex of the patient, and the TriNetX identifier of the patient.
+
+ 3. two graph modeling language (.gml) files with the clustered trajectories organised as a subgraph per cluster. gml files
+ can be visualised with other tools such as [yEd](https://www.yworks.com/products/yed). There is one .gml file where
+ the trajectory transitions are annotated with the number of patients in the trajectory so far, and second .gml file
+ where the trajectory transitions are annotated with the relative risk score (RR) for the diagnosis pairs.
+
+ 4. a DOT (.dot) file as alternative to the .gml files. DOT files can be visualised using [graphviz](https://graphviz.org/).
+ E.g. using the command:
+ ```dot -T png exp1.mci.I40.trajectories.RR.dot -o clusters.png```
+ The edges in the graph are annotated respectively with the RR scores or the number of patients. In case of the RR scores, these are itself annotated with an empirical probability. E.g. for the RR for A -> B, this probability counts how often #(not(A) -> B) > #(A ->B) is sampled. Ideally this value is very small. The ptra algorithm filters out all disease pairs (A, B) where this value is > 0.001.
+
+3. a file ending on `.scores`. This file contains the extended modularity scores calculated for each clustering. The extended modularity score is a quality metric for clustering. A modularity score > 0.03 is considered good. The file lists three tables containing 1) the granularity of the clustering 2) the modularity score and 3) a `x` if it is the best score.
+
+ Example:
+
+ 
+
+### Optional flags
+
+The `ptra` command accepts the following optional flags:
+
+* `--nofAgeGroups nr`
+
+The number of age groups to consider when dividing the population into cohorts based on age. The tool automatically
+detects the oldest and youngest patients from input. The number of age groups is used to divide the minimum birth
+year and maximum birth year into age ranges. E.g. min birth year: 1900, max birth year: 2020, and nr of age groups:
+10, will create 10 cohorts where age ranges from [0,12],[12,22],...[108,120].
+
+* `--lvl nr`
+
+Sets the ICD10 level for the diagnosis input [0-6]. The tool maps all ICD10 codes to a medical meaningful term based on
+this chosen level. ICD10 codes of lower levels may be combined into the same code of a higher level. E.g. A00.0
+Cholera due to Vibrio cholerae 01, biovar cholerae and A00.1 Cholera due to Vibrio cholerae 01, biovar eltor are lvl
+3 codes and may be collapsed to A00 Cholera in lvl 2, or A00-A09 Intestinal infectious diseases in lvl 1, or A00-B99
+Certain infectious and parasitic diseases in lvl 0.
+
+* `--minPatients nr`
+
+Sets the minimum required number of patients in a trajectory.
+
+* `--maxYears nr`
+
+Sets the maximum number of years between subsequent diagnoses to be considered for inclusion in a trajectory. E.g.
+0.5 for half a year.
+
+* `--minYears nr`
+*
+Sets the minimum number of years between subsequent diagnoses to be considered for inclusion in a trajectory. E.g. 0.5
+for half a year.
+
+* `--maxTrajectoryLength nr`
+
+Sets the maximum length of trajectories to be included in the output. E.g. 5 for trajectories with maximum 5 diagnoses.
+
+* `--minTrajectoryLength nr`
+
+Sets the minimum length of trajectories to be included in the output. E.g. 3 for trajectories with minimum 3 diagnoses.
+
+* `--name string`
+
+Sets the name of the experiment. This name is used to generate names for output files.
+
+* `--cluster`
+
+If this flag is passed, the computed trajectories are clustered and the clusters are outputted to file.
+
+* `--mclPath`
+
+Sets the path where the mcl binaries can be found.
+
+* `--iter nr`
+
+Sets the number of iterations to be used in the sampling experiments for calculating relative risk ratios. If iter
+is 400, the calculated p-values are within 0.05 of the true p-values. For iter = 10000, the true p-values are within
+0.01 of the true p-values. The higher the number of iterations, the higher the runtime.
+
+* `--saveRR file`
+
+Save the RR matrix, a matrix that represents the RR calculated from the population for each possible combination of
+ICD10 diagnosis pairs. This matrix can be loaded in other `ptra` runs to avoid recalculating the RR scores. This can
+be useful if parameters want to be explored that do not impact the RR calculation itself. Only `iter`, `maxYears` and
+`minYears`, and `filters` influence RR calculation. Variations of other parameters for constructing trajectories from RR
+scores, such as `maxTrajectoryLenght`, `minTrajectoryLength`, `minPatients`, `RR` etc might be explored in other runs.
+
+* `--loadRR file`
+
+Load the RR matrix from file. Such a file must be created by a previous run of `ptra` with the `--saveRR` flag.
+
+* `--pfilters age+:nr,age-:nr, [eoi+ | eoi-], sex:[male | female]`
+
+A list of filters for selecting/filtering patients from which to derive trajectories.
+
+* `--tfilters cat:[neoplasm | bc],code:icd10,icd10,...`
+
+A list of filters for reducing the output of trajectories. E.g. `cat:neoplasm` only outputs trajectories where there is
+at least one diagnosis related to cancer. `cat:bc` only outputs trajectories where one diagnosis is assuming to be
+related to bladder cancer. `code:icd10;icd10;...` filters trajectories where at least one of the listed icd10 codes is
+present.
+
+* `--eoid icd10,icd10,...`
+
+A list of icd10 codes that represent the event of interest to track (e.g. cancer diagnosis). The event of interest can
+for example be used in the patient filter (`--pfilters`) option (`eoi+ | eoi-`) to remove all diagnoses before/after the
+event of interest. Is also used in the metrics that are generated for the clusters, e.g. to annotate the average age of
+patients at the time of the event of interest.
+
+* `--actFile file`
+
+A file containing WHO ACT codes, mapping a unique ACT ID to description. ACT codes may be registered in a similar way as
+diagnoses and are included as events in trajectory building.
+
+* `--loglvl error | warning | info | debug | all`
+
+An option for toggling structured logging in ptra. You can choose the following level of detail for logging:
+- info: logs general information about the ptra execution. Includes all messages from the Warn, Error, and Info levels.
+- warn: logs warning messages that indicate potential issues with the ptra execution. Includes all messages from Warn
+and Error levels.
+- error: logs only error messages.
+- debug: most detailed logging level. Logs messages in the Debug, Info, Warn, and Error level.
## TriNetX Use Case
### Name
`ptra` - a commandline tool for extracting statistically relevant trajectories from ICD9/ICD10 diagnosis histories in
@@ -141,38 +342,60 @@ The `ptra` command requires four arguments:
derived_by_trinetx, source_id`
4. `outputPath`: a path where the outputs of the `ptra` run can be written.
-`ptra` creates multiple output files:
+`ptra` creates multiple output files. This output is structured as follows:
+
+1. A folder `unclustered-trajectories` with all trajectories found by `ptra` (before clustering). This folder contains the
+following files:
-1. a tab file with the found trajectories. The tab file contains two lines per trajectory. The first line lists the diagnoses
- in the trajectory, separated by tabs. The second line lists the number of patients between each transition in the trajectory.
+ 1. a tab file with the found trajectories. The tab file contains two lines per trajectory. The first line lists the diagnoses
+ in the trajectory, separated by tabs. The second line lists the number of patients between each transition in the trajectory.
- Example:
+ Example:
- ```
- Cough \tab Dyspnea \tab COPD
- 150 \tab 50
- ```
-2. a tab file with the found diagnosis pairs and their relative risk scores. There is a single line that list the diagnoses and the RR.
+ ```
+ Cough \tab Dyspnea \tab COPD
+ 150 \tab 50
+ ```
+ 2. a tab file with the found diagnosis pairs and their relative risk scores. There is a single line that list the diagnoses and the RR.
- Example:
-
- ```Cough \tab Dyspnea \tab 1.95```
-
-3. a folder with clustered trajectory output --if `ptra` was requested to cluster its output (`--cluster` flag). This folder
- contains per requested cluster granularity (`--cluster-granularities`) up to 4 files:
- 1. a csv file with cluster information. The header is: `PID,CID,TID,Age`. These represent the patient identifier, cluster
- identifier, trajectory identifier, and age of the patient at the time they completed the trajectory.
- 2. a csv file with information to link the patient analysis identifier used in `ptra` back to the TriNetX identifier. The
- header of the csv file is: `PID,AgeEOI,Sex,PIDString`. This represents the patient id used in `ptra`, the age of the
- patient at the event of interest, the sex of the patient, and the TriNetX identifier of the patient.
- 3. two graph modeling language (.gml) files with the clustered trajectories organised as a subgraph per cluster. gml files
- can be visualised with other tools such as [yEd](https://www.yworks.com/products/yed). There is one .gml file where
- the trajectory transitions are annotated with the number of patients in the trajectory so far, and second .gml file
- where the trajectory transitions are annotated with the relative risk score (RR) for the diagnosis pairs.
+ Example:
+
+ ```Cough \tab Dyspnea \tab 1.95```
+
+ 3. a gml file with all found trajectories as individual graphs. Useful for verifying outputs with few trajectories.
+
+ 4. a gml file with all found trajectories plotted as one (large) graph. Useful for verifying output, but may be too
+ dense to view for large numbers of trajectories. In this case, clustering is recommended.
+
+2. a folder `clustered-trajectories` with clustered trajectories --if `ptra` was requested to cluster its output
+(`--cluster` flag). This folder contains multiple subfolders:
+
+ 1. A folder `mclOutput` with intermediate clustering output generated by the MCL tool. This is normally not relevant
+ to view for `ptra` users. It is useful for the `ptra` developers for debugging.
+
+ 2. A number of `clusters` folders containing clustered trajectories. A folder is generated for each chosen cluster
+ granularity when calling `ptra`, cf. `--clusterGranularities` option. By default, 4 granuarities are tried, so 4
+ `clusters` folders are created. Each `clusters` folder contains the following files:
+
+ 1. a csv file with cluster information. The header is: `PID,CID,TID,Age`. These represent the patient identifier, cluster
+ identifier, trajectory identifier, and age of the patient at the time they completed the trajectory.
+
+ 2. a csv file with information to link the patient analysis identifier used in `ptra` back to the TriNetX identifier. The
+ header of the csv file is: `PID,AgeEOI,Sex,PIDString`. This represents the patient id used in `ptra`, the age of the
+ patient at the event of interest, the sex of the patient, and the TriNetX identifier of the patient.
+
+ 3. two graph modeling language (.gml) files with the clustered trajectories organised as a subgraph per cluster. gml files
+ can be visualised with other tools such as [yEd](https://www.yworks.com/products/yed). There is one .gml file where
+ the trajectory transitions are annotated with the number of patients in the trajectory so far, and second .gml file
+ where the trajectory transitions are annotated with the relative risk score (RR) for the diagnosis pairs.
- Example:
+ Example:
+
+ 
- 
+ 4. a DOT (.dot) file as alternative to the .gml files. DOT files can be visualised using [graphviz](https://graphviz.org/).
+ E.g. using the command:
+ ```dot -T png exp1.mci.I40.trajectories.RR.dot -o clusters.png```
### Optional flags
diff --git a/app/filters.go b/app/filters.go
index da905f7..07aa3b4 100644
--- a/app/filters.go
+++ b/app/filters.go
@@ -269,3 +269,36 @@ func BladderCancerTrajectoryFilter(exp *trajectory.Experiment) trajectory.Trajec
return false
}
}
+
+func codeMember(code string, categories []string) bool {
+ for _, cat := range categories {
+ if code == cat {
+ return true
+ }
+ }
+ return false
+}
+
+func Icd10TrajectoryFilter(exp *trajectory.Experiment, codes []string) trajectory.TrajectoryFilter {
+ //Determine all diagnoses that match the given codes, either exactly or either the top-level category
+ codeRelatedMap := map[int]bool{}
+ for did, _ := range exp.NameMap {
+ icdCode := exp.IdMap[did]
+ if len(icdCode) >= 3 {
+ subCode := icdCode[0:3]
+ if codeMember(subCode, codes) {
+ codeRelatedMap[did] = true
+ } else {
+ codeRelatedMap[did] = false
+ }
+ }
+ }
+ return func(t *trajectory.Trajectory) bool {
+ for _, did := range t.Diagnoses {
+ if codeRelatedMap[did] {
+ return true
+ }
+ }
+ return false
+ }
+}
diff --git a/app/parseData.go b/app/parseData.go
index 13cca05..1281b5e 100644
--- a/app/parseData.go
+++ b/app/parseData.go
@@ -19,228 +19,20 @@
package app
import (
+ "bytes"
"encoding/csv"
- "encoding/json"
- "encoding/xml"
"fmt"
"io"
- "io/ioutil"
+ "log/slog"
"math"
"os"
"path/filepath"
"ptra/trajectory"
"ptra/utils"
- "sort"
"strconv"
- "strings"
)
-//Package ptra implements a patient trajectory analysis tool.
-//The ptra program has 3 data inputs:
-//A file mapping diagnosis ID (DID) -> medical name.
-//A file with patient info, associating a patient ID (PID) with YOB, sex, etc.
-//A file with diagnosis info, mapping PID -> DID, date.
-
-//TriNetX Database stores diagnoses data using a mix of ICD10 and ICD9 codes.
-//We have an additional file that maps ICD9 IDs -> ICD10 IDs.
-//We can download the ICD10 ID -> medical name from https://www.cms.gov/medicare/icd-10/2022-icd-10-cm as an xml file.
-//TriNetX stores patient info as a csv file, as well as the diagnoses info.
-//For the medical name mapping, we can also use the ICD10 -> CCSR Mapping which maps ICD10 onto 530 categories with medical meaning. This
-//mapping can be downloaded from https://www.hcup-us.ahrq.gov/toolssoftware/ccsr/dxccsr.jsp#download as a CSV file. This
-//mapping performs a mapping ICD10 -> CCSR categpries -> medical name.
-
-//Parsing ICD10 names hierarchy from xml
-//Structs for unmarshalling ICD10 xml data
-//Structure of an ICD10 code: ABC.XYZ(D): up to 7 characters
-//The xml file structures these codes using:
-//
-//with chapter the first level of the diagnosis code (A), section the second level (D) and desc the rest (C.XYZ(D))
-
-// diag captures the lowest levels of the ICD10 code
-type diag struct {
- Name string `xml:"name"` //Unique diagnosis ID (DID) in ICD10 encoding
- Desc string `xml:"desc"` //A medical name/description for a DID
- Diagnoses []diag `xml:"diag"` //A diagnosis can be split into more detailed diagnoses descriptors.
-}
-
-// section captures the second level of the ICD10 code
-type section struct {
- Desc string `xml:"desc"` //A medical name/description for a DID.
- Id string `xml:"id,attr"` //Unique diagnosis ID (DID) in ICD10 encoding
- Diagnoses []diag `xml:"diag"` //A diagnosis can be split into more detailed diagnoses descriptors.
-}
-
-// chapter captures the first (highest) level of the ICD10 code
-type chapter struct {
- XmlName xml.Name `xml:"chapter"`
- Desc string `xml:"desc"`
- Sections []section `xml:"section"`
-}
-
-// icd10Hierarchy contains the full xml table with the ICD10 code hierarchy.
-type icd10Hierarchy struct {
- XmlName xml.Name `xml:"ICD10CM.tabular"`
- Chapters []chapter `xml:"chapter"`
-}
-
-// parseIcd10HierarchyFromXML parses the xml file with the ICD10 hierarchy into an icd10Hierarchy object.
-func parseIcd10HierarchyFromXml(file string) icd10Hierarchy {
- fmt.Println("Parsing ICD10 code hierarchy from XML file: ", file)
- //open file
- xmlFile, err := os.Open(file)
- if err != nil {
- panic(err)
- }
- defer xmlFile.Close()
- xmlFileBytes, _ := ioutil.ReadAll(xmlFile)
- //unmarshall
- icd10Hierarchy := icd10Hierarchy{}
- xml.Unmarshal(xmlFileBytes, &icd10Hierarchy)
- return icd10Hierarchy
-}
-
-// printIcd10Hierarchy prints an ICD10 hierarchy parsed from an XML file.
-func printIcd10Hierarchy(hierarchy icd10Hierarchy) {
- fmt.Println("Printing ICD10 code hierarchy.")
- // count # DID per level
- ctr1, ctr2, ctr3, ctr4, ctr5, ctr6, ctr7 := 0, 0, 0, 0, 0, 0, 0
- for _, chap := range hierarchy.Chapters {
- // level 1
- ctr1++
- fmt.Println("Chapter: ", chap.Desc)
- for _, section := range chap.Sections {
- // level 2
- ctr2++
- fmt.Println("Section: ", section.Desc)
- for _, diag := range section.Diagnoses {
- // level 3
- ctr3++
- fmt.Println(diag.Name, " : ", diag.Desc)
- if len(diag.Diagnoses) == 0 {
- continue
- }
- for _, diag := range diag.Diagnoses {
- // level 4
- ctr4++
- fmt.Println(diag.Name, " : ", diag.Desc)
- if len(diag.Diagnoses) == 0 {
- continue
- }
- for _, diag := range diag.Diagnoses {
- // level 5
- ctr5++
- fmt.Println(diag.Name, " : ", diag.Desc)
- if len(diag.Diagnoses) == 0 {
- continue
- }
- for _, diag := range diag.Diagnoses {
- // level 6
- ctr6++
- fmt.Println(diag.Name, " : ", diag.Desc)
- if len(diag.Diagnoses) == 0 {
- continue
- }
- // level 7
- ctr7++
- fmt.Println(diag.Name, " : ", diag.Desc)
- }
- }
- }
- }
- }
- }
- fmt.Println("#ICD10 codes/descriptors per level: ")
- fmt.Println("Lvl 0: ", ctr1, " Lvl 1: ", ctr2, " Lvl 2: ",
- ctr3, " Lvl 3: ", ctr4, " Lvl 4: ", ctr5, " Lvl 5: ", ctr6, " Lvl 6: ", ctr7)
-}
-
-//The ptra program needs a names map that maps DID -> medical name. The following code extracts a name map from an ICD10
-//hierarchy and a given level.
-
-// icd10Name is a struct for containing a medical name + level + the categories of a DID in ICD10 encoding.
-type icd10Name struct {
- name string //medical name for a DID in ICD10 encoding
- categories [6]string //the names of the ICD10 encoding higher and lower in the hierarchy.
- level int //the ICD10 hierarchy level of this name.
-}
-
-type icd10Table struct {
- namesMap map[string]icd10Name //maps ICD10 DID to a medical name, level, and categories to which it belongs.
-}
-
-// selectParentCategory returns the ICD10 name of the parent category to which a DID belongs.
-func selectParentCategory(name icd10Name) string {
- if name.level > 0 {
- return name.categories[name.level-1]
- }
- return "None"
-}
-
-func printIcd10NameMap(table map[string]icd10Name) {
- fmt.Println("ICD10 Name map: ")
- for id, name := range table {
- fmt.Println(id, " : ", name)
- }
-}
-
-// initializeIcd10NameMap initializes a name map for ICD10 DID -> medical name, level, and categories it belongs to.
-func initializeIcd10NameMap(file string) map[string]icd10Name {
- icd10NameMap := map[string]icd10Name{} //maps ICD10 DID to a medical name, level, and categories to which it belongs.
- icd10Hierarchy := parseIcd10HierarchyFromXml(file)
- for _, chap := range icd10Hierarchy.Chapters {
- category0 := chap.Desc
- for _, section := range chap.Sections {
- category1 := section.Desc
- // manually unrolled loop since we know hierarchy is max 7 levels, otherwise recursive code
- for _, diag := range section.Diagnoses {
- if len(diag.Diagnoses) == 0 {
- icd10Name := icd10Name{name: diag.Desc,
- categories: [6]string{category0, category1, "NONE", "NONE", "NONE", "NONE"}, level: 2}
- icd10NameMap[diag.Name] = icd10Name
- continue
- }
- category2 := diag.Desc
- for _, diag := range diag.Diagnoses {
- if len(diag.Diagnoses) == 0 {
- icd10Name := icd10Name{name: diag.Desc,
- categories: [6]string{category0, category1, category2, "NONE", "NONE", "NONE"},
- level: 3}
- icd10NameMap[diag.Name] = icd10Name
- continue
- }
- category3 := diag.Desc
- for _, diag := range diag.Diagnoses {
- if len(diag.Diagnoses) == 0 {
- ICD10Name := icd10Name{name: diag.Desc,
- categories: [6]string{category0, category1, category2, category3, "NONE", "NONE"},
- level: 4}
- icd10NameMap[diag.Name] = ICD10Name
- continue
- }
- category4 := diag.Desc
- for _, diag := range diag.Diagnoses {
- if len(diag.Diagnoses) == 0 {
- ICD10Name := icd10Name{name: diag.Desc,
- categories: [6]string{category0, category1, category2, category3, category4, "NONE"},
- level: 5}
- icd10NameMap[diag.Name] = ICD10Name
- continue
- }
- category5 := diag.Desc
- for _, diag := range diag.Diagnoses {
- ICD10Name := icd10Name{name: diag.Desc,
- categories: [6]string{category0, category1, category2, category3, category4, category5},
- level: 6}
- icd10NameMap[diag.Name] = ICD10Name
- }
- }
- }
- }
- }
- }
- }
- return icd10NameMap
-}
+//Generic ptra parsing
// getIcd10DescToExcludeFromAnalysis returns a map that lists ICD10 categories to be excluded from analysis by mapping
// the ICD10 category description (string) onto a boolean.
@@ -270,81 +62,11 @@ func getIcd10CodesToExcludeFromAnalysis() map[string]bool {
return exclude
}
-// getNonICD10CodesToAddToAnalysis returns a set of mockup ICD10 codes to be able to introduce non ICD codes to be
-// included for analysis. It returns a map from mockup ICD10 code (string) to description string. It introduces "C98" for
-// "Radical custectomy (bladder cancer)", "C99" for "MVAC Chemotherapy (bladder cancer)", and "C100" for "Intravesical
-// therapy (bladder cancer)".
-func getNonICD10CodesToAddToAnalysis() map[string]string {
- return map[string]string{
- "C98": "Radical cystectomy (bladder cancer)",
- "C99": "MVAC Chemotherapy (bladder cancer)",
- "C100": "Intravesical therapy (bladder cancer)",
- }
-}
-
-// initializeIcd10AnalysisIDMap creates a map ICD10 DID -> analysis DID and a map analysis ID -> medical name. This is
-// useful to remap diagnosis codes used in the input to a higher level in the ICD10 hierarchy. E.g "typhoid fever" and
-// "cholera" are both "infectuous intestinal diseases", so they could both be identified as such during the analysis.
-// This can be interesting to obtain more global patient trajectories/clusters.
-func intializeIcd10AnalysisMaps(icd10NameMap map[string]icd10Name, level int) (map[string]int, map[int]string, int) {
- analysisIdMap := map[string]int{} // maps icd 10 code to analysis ID
- analysisNameMap := map[int]string{} // maps analysis ID to a medical name
- nameToAnalysisIdMap := map[string]int{} // maps medical name to analysis ID
- ctr := 0 //serves as analysis ID generator
- icd10ToExclude := getIcd10DescToExcludeFromAnalysis() // a list of level 0 categories to exclude from analysis
- for icd10Code, icd10Name := range icd10NameMap {
- if _, ok := icd10ToExclude[icd10Name.categories[0]]; ok {
- // code to exclude from analysis
- continue
- }
- var name string
- if level == icd10Name.level || level > icd10Name.level {
- name = icd10Name.name
- } else {
- name = icd10Name.categories[level]
- }
- // may already have seen name, because of level
- newID, ok := nameToAnalysisIdMap[name]
- if !ok {
- newID = ctr
- ctr++
- analysisNameMap[newID] = name
- nameToAnalysisIdMap[name] = newID
- }
- analysisIdMap[icd10Code] = newID
- }
- extra := getNonICD10CodesToAddToAnalysis()
- for code, name := range extra {
- analysisNameMap[ctr] = name
- nameToAnalysisIdMap[name] = ctr
- analysisIdMap[code] = ctr
- ctr++
- }
- fmt.Println("Mapped ", len(icd10NameMap), " ICD10 codes to ", ctr, " analysis IDs of level ", level)
- return analysisIdMap, analysisNameMap, ctr
-}
-
-// ccsrCategory is a struct for containing CCSR categories, encoding medically meaningful names for a DID in ICD10
-// encoding.
-type ccsrCategory struct {
- name string //default CCSR category/medical name
- id string //CCSR ID for default category
- categories map[string]string //Up to 6 different CCSR categories an ICD10 code is mapped to
-}
-
-type icd10ToCCSRTable map[string]ccsrCategory //maps ICD10 DID to its CCSR categories
-
-// ccsrIcd10ToProperIcd10 transforms the ICD10 code from a ccsr file into a proper ICD10 code. The ICD10 codes in the
-// ccsr file are stored without the ".", so this needs to be added to be able to compare to any other data that uses
-// ICD10 codes. Also removes superfluous quotes stored in the ccsr file.
-func ccsrIcd10ToProperIcd10(code string) string {
- return code[1:4] + "." + code[4:len(code)-1]
-}
-
-// initializeIcd10NameMapFromCCSR initializes a name map for ICD10 DID -> CCSR categories (medical names)
-func initializeIcd10ToCCSRMap(file string) map[string]ccsrCategory {
- //map to collect data
- icd10ToCCSRTable := map[string]ccsrCategory{}
+// Parsing ATC codes
+// parseATCCodes parses a files with ATC codes + descriptions. Input: a csv file with the header:
+// atc_code,atc_name,ddd,uom,adm_r,note. Returns a map: code -> description.
+func parseATCCodes(file string) map[string]string {
+ dct := map[string]string{}
//open file
csvFile, err := os.Open(file)
if err != nil {
@@ -355,15 +77,8 @@ func initializeIcd10ToCCSRMap(file string) map[string]ccsrCategory {
panic(err)
}
}()
- //parse file
reader := csv.NewReader(csvFile)
- //the header is 'ICD-10-CM CODE','ICD-10-CM CODE DESCRIPTION','Default CCSR CATEGORY IP','
- //Default CCSR CATEGORY DESCRIPTION IP','Default CCSR CATEGORY OP','Default CCSR CATEGORY DESCRIPTION OP','
- //CCSR CATEGORY 1','CCSR CATEGORY 1 DESCRIPTION','CCSR CATEGORY 2','CCSR CATEGORY 2 DESCRIPTION',
- //'CCSR CATEGORY 3','CCSR CATEGORY 3 DESCRIPTION','CCSR CATEGORY 4','CCSR CATEGORY 4 DESCRIPTION',
- //'CCSR CATEGORY 5','CCSR CATEGORY 5 DESCRIPTION','CCSR CATEGORY 6','CCSR CATEGORY 6 DESCRIPTION'
- // skip header
- reader.Read()
+ reader.Read() //skip header
for {
record, err := reader.Read()
if err == io.EOF {
@@ -372,250 +87,16 @@ func initializeIcd10ToCCSRMap(file string) map[string]ccsrCategory {
if err != nil {
panic(err)
}
- //create CSSR category, set default category
- category := ccsrCategory{name: record[2], id: record[3], categories: map[string]string{}}
- //fill in unique CSSR alternative categories, up to 6 possible
- for i := 6; i <= 17; i = i + 2 {
- catID := record[i]
- catName := record[i+1]
- if catName == "" || catID == "' '" {
- continue
- }
- if _, ok := category.categories[catID]; !ok {
- category.categories[catID] = catName
- }
- }
- //add category to result
- icd10Code := ccsrIcd10ToProperIcd10(record[0])
- icd10ToCCSRTable[icd10Code] = category
- }
- return icd10ToCCSRTable
-}
-
-// printIcd10ToCSSRTable is a simple function to print the map from iCD10 code to ccsr category. Useful for debugging.
-func printIcd10ToCCSRTable(tab map[string]ccsrCategory) {
- fmt.Println("ICD10 to CCSR table")
- ctr := 0
- for icd10Code, ccsr := range tab {
- ctr++
- fmt.Println(icd10Code, " : ", ccsr.categories)
- if ctr >= 1000 {
- return
- }
- }
-}
-
-// initializeIcd10AnalysisMapsCCSR creates a map ICD10 DID -> [analysis DID] and a map analysis ID -> medical name,
-// starting from a CCSR mapping, which maps ICD10 codes onto medical meaningful categories.
-// Each icd10 code can be mapped to multiple ccsr categories, and therefore to multiple analysis IDs.
-// TO DO: exclude specific ICD10 codes from the analysis.
-func initializeIcd10AnalysisMapsCCSR(icd10ToCssrMap map[string]ccsrCategory) (map[string][]int, map[int]string, int) {
- analysisIdMap := map[string][]int{} // maps icd 10 code to analysis IDs
- analysisNameMap := map[int]string{} // maps analysis ID to a medical name
- ccsrIDMap := map[string]int{}
- ctr := 0 //serves as analysis ID generator
- icd10ToExclude := getIcd10CodesToExcludeFromAnalysis()
- for icd10Code, ccsr := range icd10ToCssrMap {
- if _, ok := icd10ToExclude[icd10Code[0:1]]; ok {
- continue
- }
- ids := []int{}
- for id, name := range ccsr.categories {
- var ccsrID int
- var ok bool
- if ccsrID, ok = ccsrIDMap[id]; !ok {
- ccsrID = ctr
- analysisNameMap[ctr] = name
- ccsrIDMap[id] = ccsrID
- ctr++
- }
- ids = append(ids, ccsrID)
- }
- analysisIdMap[icd10Code] = ids
- }
- extra := getNonICD10CodesToAddToAnalysis()
- for code, name := range extra {
- analysisNameMap[ctr] = name
- analysisIdMap[code] = []int{ctr}
- ctr++
- }
- fmt.Println("Mapped ", len(icd10ToCssrMap), " ICD10 codes to ", ctr, " analysis IDs")
- return analysisIdMap, analysisNameMap, ctr
-}
-
-type icd10AnalysisMapsFromCCSR struct {
- NameMap map[int]string // map analysis DID -> medical name
- NofDiagnosisCodes int // nr of different diagnosis codes
- DIDMap map[string][]int // maps ICD10 Code onto multiple DIDs
-}
-
-type icd10AnalysisMapsFromXML struct {
- NameMap map[int]string // map analysis DID -> medical name
- NofDiagnosisCodes int // nr of different diagnosis codes
- DIDMap map[string]int // map ICD10 Code -> DID
-}
-
-func (analysisMap icd10AnalysisMapsFromXML) getDID(icd10DID string) int {
- if v, ok := analysisMap.DIDMap[icd10DID]; ok {
- return v
- } else {
- return -1
- }
-}
-
-func (analysisMap icd10AnalysisMapsFromCCSR) getDID(icd10DID string) []int {
- if v, ok := analysisMap.DIDMap[icd10DID]; ok {
- return v
- }
- return nil
-}
-
-func (analysisMap icd10AnalysisMapsFromXML) GetICDCode(did int) string {
- for icd10Code, didCode := range analysisMap.DIDMap {
- if didCode == did {
- return icd10Code
- }
- }
- return ""
-}
-
-func (analysisMap icd10AnalysisMapsFromCCSR) GetICDCode(did int) string {
- for icd10Code, didCodes := range analysisMap.DIDMap {
- for _, didCode := range didCodes {
- if didCode == did {
- return icd10Code
- }
- }
- }
- return ""
-}
-
-func (analysisMap icd10AnalysisMapsFromXML) getIdMap() map[int]string {
- res := map[int]string{}
- for icd10Code, didCode := range analysisMap.DIDMap {
- res[didCode] = icd10Code
- }
- return res
-}
-
-func (analysisMap icd10AnalysisMapsFromCCSR) getIdMap() map[int]string {
- res := map[int]string{}
- for icd10Code, didCodes := range analysisMap.DIDMap {
- for _, didCode := range didCodes {
- res[didCode] = icd10Code
- }
- }
- return res
-}
-
-// AnalysisMaps represent maps extracted from the input that map analysis IDs onto medical terms and vice versa. This is
-// an interface that defines several methods. getICDCode returns for a did the original id in the input for the
-// diagnostic event. fillInPatientDiagnoses creates for a given diagnosis identifier from the input a Diagnosis object
-// and adds it to a patient's list of diagnoses.
-type AnalysisMaps interface {
- fillInPatientDiagnoses(patient *trajectory.Patient, DidString string, date trajectory.DiagnosisDate) int
- fillInNonICDPatientDiagnoses(patient *trajectory.Patient, infoMap map[string]*TreatmentInfo) int
- GetICDCode(did int) string
- getIdMap() map[int]string
-}
-
-func (analysisMap icd10AnalysisMapsFromXML) fillInPatientDiagnoses(patient *trajectory.Patient, DIDString string, date trajectory.DiagnosisDate) int {
- DID := analysisMap.getDID(DIDString)
- if DID == -1 {
- return 1 // icd10 diagnosis excluded from analysis
- }
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: DID, Date: date}
- trajectory.AddDiagnosis(patient, diagnosis)
- return 0
-}
-
-func (analysisMap icd10AnalysisMapsFromCCSR) fillInPatientDiagnoses(patient *trajectory.Patient, DIDString string, date trajectory.DiagnosisDate) int {
- DIDs := analysisMap.getDID(DIDString)
- if DIDs == nil {
- return 1 // icd10 code excluded from analysis
- }
- for _, DID := range DIDs {
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: DID, Date: date}
- trajectory.AddDiagnosis(patient, diagnosis)
- }
- return 0
-}
-
-func (analysisMap icd10AnalysisMapsFromXML) fillInNonICDPatientDiagnoses(patient *trajectory.Patient, infoMap map[string]*TreatmentInfo) int {
- nonIcd := 0
- if info, ok := infoMap[patient.PIDString]; ok {
- if info.RCDate != nil {
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: analysisMap.DIDMap["C98"], Date: *info.RCDate}
- nonIcd = 1
- trajectory.AddDiagnosis(patient, diagnosis)
- }
- if info.MVACDate != nil {
- nonIcd = 1
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: analysisMap.DIDMap["C99"], Date: *info.MVACDate}
- trajectory.AddDiagnosis(patient, diagnosis)
- }
- if info.IVTDate != nil {
- nonIcd = 1
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: analysisMap.DIDMap["C100"], Date: *info.IVTDate}
- trajectory.AddDiagnosis(patient, diagnosis)
- }
+ dct[record[0]] = record[1]
}
- return nonIcd
-}
-
-func (analysisMap icd10AnalysisMapsFromCCSR) fillInNonICDPatientDiagnoses(patient *trajectory.Patient, infoMap map[string]*TreatmentInfo) int {
- nonIcd := 0
- if info, ok := infoMap[patient.PIDString]; ok {
- if info.RCDate != nil {
- dids := analysisMap.DIDMap["C98"]
- for _, did := range dids {
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: did, Date: *info.RCDate}
- nonIcd = 1
- trajectory.AddDiagnosis(patient, diagnosis)
- }
- }
- if info.MVACDate != nil {
- dids := analysisMap.DIDMap["C99"]
- for _, did := range dids {
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: did, Date: *info.MVACDate}
- nonIcd = 1
- trajectory.AddDiagnosis(patient, diagnosis)
- }
- }
- if info.IVTDate != nil {
- dids := analysisMap.DIDMap["C100"]
- for _, did := range dids {
- nonIcd = 1
- diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: did, Date: *info.IVTDate}
- trajectory.AddDiagnosis(patient, diagnosis)
- }
- }
- }
- return nonIcd
-}
-
-// initializeIcd10AnalysisMaps returns a map ICD10 DID -> internal analysis DID and a map analysis DID ->
-// medical name for an ICD10 Hierarchy passed as xml file and a requested hierarchy level.
-func initializeIcd10AnalysisMapsFromXML(file string, level int) icd10AnalysisMapsFromXML {
- icd10NameMapFromXml := initializeIcd10NameMap(file) // map ICD10 DID -> ICD 10 Name (medical desc, categories, level)
- analysisIdMap, analysisNameMap, ctr := intializeIcd10AnalysisMaps(icd10NameMapFromXml, level)
- return icd10AnalysisMapsFromXML{DIDMap: analysisIdMap, NameMap: analysisNameMap, NofDiagnosisCodes: ctr}
-}
-
-// initializeIcd10AnalysisMapsFromCCSR returns a map ICD10 -> []{internal analysis DID} and map analysis DID -> medical
-// name for ICD10 CCSR categorization passed as a csv file.
-func initializeIcd10AnalysisMapsFromCCSR(file string) icd10AnalysisMapsFromCCSR {
- icd10ToCssrMap := initializeIcd10ToCCSRMap(file) // map ICD10 Code -> CCSR Name
- analysisIdMap, analysisNameMap, ctr := initializeIcd10AnalysisMapsCCSR(icd10ToCssrMap)
- return icd10AnalysisMapsFromCCSR{DIDMap: analysisIdMap, NameMap: analysisNameMap, NofDiagnosisCodes: ctr}
+ return dct
}
-//Parsing patient information.
-
-// parseTriNetXPatientData parses a file with patient information from the TriNetX database. Input: a patient file in csv
+// Parsing patient information.
+// parsePatientData parses a file with patient information. Input: a patient file in csv
// format, a desired number of age groups to initialize cohorts. Diagnoses of the patient need to be filled in after
// parsing the diagnoses file.
-func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.PatientMap, int) {
+func parsePatientData(file string, nofCohortAges int) (*trajectory.PatientMap, int) {
//open file
csvFile, err := os.Open(file)
if err != nil {
@@ -634,9 +115,11 @@ func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.Patien
regionIds := map[string]int{}
//parse file
reader := csv.NewReader(csvFile)
- //the header is omitted from the TriNetX file, but is should be: patient_id, sex, race, ethnicity, year_of_birth,
+ //the header should be: patient_id, sex, race, ethnicity, year_of_birth,
//age_at_death, patient_regional_location, postal_code, marital_status, reason_yob_missing, month_year_death,
//source_id
+ //skip header
+ reader.Read()
for {
record, err := reader.Read()
if err == io.EOF {
@@ -646,7 +129,7 @@ func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.Patien
panic(err)
}
var yob int
- if yob, err = strconv.Atoi(record[4]); err != nil {
+ if yob, err = strconv.Atoi(record[2]); err != nil {
continue //skip patients without year of birth
}
pidString := record[0]
@@ -661,7 +144,7 @@ func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.Patien
sex = trajectory.Female
patientMap.FemaleCtr++
}
- dateOfDeathString := record[10]
+ dateOfDeathString := record[4]
var dateOfDeath *trajectory.DiagnosisDate
if len(dateOfDeathString) == 6 {
year, err := strconv.Atoi(dateOfDeathString[0:4])
@@ -677,13 +160,17 @@ func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.Patien
}
}
}
- region := record[6]
+ region := record[3]
if _, ok := regions[region]; !ok {
regions[region] = 0
regionIds[region] = len(regionIds)
} else {
regions[region]++
}
+ control := false
+ if record[5] == "yes" {
+ control = true
+ }
patient := trajectory.Patient{
PID: pid,
PIDString: pidString,
@@ -693,6 +180,7 @@ func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.Patien
Diagnoses: []*trajectory.Diagnosis{},
DeathDate: dateOfDeath,
Region: regionIds[region],
+ Control: control,
}
patientMap.PIDMap[pid] = &patient
patientMap.PIDStringMap[pidString] = pid
@@ -707,24 +195,35 @@ func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.Patien
p.CohortAge = int(math.Floor(float64(p.YOB-minYOB) / float64(ageRange)))
}
}
- fmt.Println("Parsed patient data.")
- fmt.Print("Parsed ", patientMap.Ctr, " patients with year of birth known ")
- fmt.Print("of which ", patientMap.FemaleCtr, " females and ")
- fmt.Println(patientMap.MaleCtr, "males; and of which ", deathCr, " have a known date of death.")
- fmt.Println("Year of birth oldest patient:", minYOB)
- fmt.Println("Year of birth youngest patient:", maxYOB)
- fmt.Println("Patients are of ", len(regions), " regions: ")
+
+ var buffer bytes.Buffer
for region, nr := range regions {
- fmt.Print(region, ": ", nr, ", ")
- }
- fmt.Println("")
+ buffer.WriteString(fmt.Sprintf("[%s: %d] ", region, nr))
+ }
+ slog.Info("Parsed patient data",
+ slog.Int("#regions", len(regions)),
+ )
+ slog.Debug("Parsed patient data",
+ slog.String("Regions", buffer.String()),
+ )
+ slog.Info("Parsed patients with year of birth known",
+ slog.Int("N", patientMap.Ctr),
+ slog.Int("females", patientMap.FemaleCtr),
+ slog.Int("males", patientMap.MaleCtr),
+ slog.Int("#deaths", deathCr),
+ )
+ slog.Info("Year of birth",
+ slog.Int("oldest", minYOB),
+ slog.Int("youngest", maxYOB),
+ )
+
return patientMap, len(regions)
}
//Parsing patient diagnoses
-// parseTriNetXDiagnosisDate turns a TriNetX date string into DiagnosisDate object.
-func parseTriNetXDiagnosisDate(date string) trajectory.DiagnosisDate {
+// parseDiagnosisDate turns a date string into DiagnosisDate object.
+func parseDiagnosisDate(date string) trajectory.DiagnosisDate {
year, err := strconv.Atoi(date[0:4])
if err != nil {
panic(err)
@@ -740,69 +239,25 @@ func parseTriNetXDiagnosisDate(date string) trajectory.DiagnosisDate {
return trajectory.DiagnosisDate{Year: year, Month: month, Day: day}
}
-// TriNetXEventOfInterest checks if the ICD10 code is related to bladder cancer
-func TriNetXEventOfInterest(icd10ID string) bool {
- if icd10ID == "Z85.1" {
- return true
- }
- if len(icd10ID) >= 3 && icd10ID[0:3] == "C67" {
- return true
- }
- return false
-}
-
-// TreatmentInfo implements a structure for storing the dates of certain bladder cancer treatments.
-type TreatmentInfo struct {
- RCDate *trajectory.DiagnosisDate //Date of radical cystectomy
- MVACDate *trajectory.DiagnosisDate //Date of MVAC chemotherapy
- IVTDate *trajectory.DiagnosisDate //Date of intravesical therapy
-}
-
-// parseTriNetXTreatmentFile parses a csv file that contains information of patient's treatments at different time stamps.
-// It returns a map from PID -> TreatmentInfo.
-func parseTriNetXTreatmentFile(fileName string) map[string]*TreatmentInfo {
- result := map[string]*TreatmentInfo{}
- file, err := os.Open(fileName)
- if err != nil {
- panic(err)
- }
- defer func() {
- if err := file.Close(); err != nil {
- panic(err)
- }
- }()
- reader := csv.NewReader(file)
- for {
- record, err := reader.Read()
- if err == io.EOF {
- break
- }
- if err != nil {
- panic(err)
- }
- PIDString := record[0]
- var rcDate, mvacDate, ivtDate *trajectory.DiagnosisDate
- if len(record[10]) == 10 { // valid date
- d := parseTriNetXDiagnosisDate(record[10])
- rcDate = &d
+// EventOfInterest checks if the ICD10 code is related to bladder cancer
+func EventOfInterest(icd10ID string, options []string) bool {
+ for _, option := range options {
+ if icd10ID == option { //specific ICD10
+ return true
}
- if len(record[11]) == 10 {
- d := parseTriNetXDiagnosisDate(record[11])
- mvacDate = &d
- }
- if len(record[13]) == 10 {
- d := parseTriNetXDiagnosisDate(record[13])
- rcDate = &d
+ if len(option) == 3 { //the whole category
+ if icd10ID[0:3] == option[0:3] {
+ return true
+ }
}
- result[PIDString] = &TreatmentInfo{RCDate: rcDate, MVACDate: mvacDate, IVTDate: ivtDate}
}
- return result
+ return false
}
-// parseTrinetXPatientDiagnoses parses a csv file containing patient diagnoses. It fills in those diagnoses for the given
+// parsePatientDiagnoses parses a csv file containing patient diagnoses. It fills in those diagnoses for the given
// patients. It uses the icd10AnalysisMap to assign internal analysis DID to the diagnoses.
-// TO DO: Handle ICD09 diagnoses.
-func parseTrinetXPatientDiagnoses(diagnosesFile, treatmentInfoFile string, patients *trajectory.PatientMap, icd10AnalysisMap AnalysisMaps, icd9ToIcd10Map map[string]string) {
+func parsePatientDiagnoses(diagnosesFile string, patients *trajectory.PatientMap,
+ icd10AnalysisMap AnalysisMaps, eoid []string) {
file, err := os.Open(diagnosesFile)
if err != nil {
panic(err)
@@ -814,9 +269,9 @@ func parseTrinetXPatientDiagnoses(diagnosesFile, treatmentInfoFile string, patie
}()
reader := csv.NewReader(file)
ctr := 0 //for counting the number of parsed diagnoses
- ctrID09 := 0
ctrExcl := 0
EOICtr := 0
+ reader.Read() //skip header
for {
record, err := reader.Read()
if err == io.EOF {
@@ -831,16 +286,8 @@ func parseTrinetXPatientDiagnoses(diagnosesFile, treatmentInfoFile string, patie
if !ok {
continue //skip unknown patients
}
- DIDCodeSystem := record[2]
- DIDString := record[3]
- if DIDCodeSystem != "ICD-10-CM" {
- // try to remap ICD9 code to ICD10 codes
- if DIDString, ok = icd9ToIcd10Map[DIDString]; !ok {
- continue // skip unkown ICD9 codes
- }
- ctrID09++
- }
- date := parseTriNetXDiagnosisDate(record[7])
+ DIDString := record[2]
+ date := parseDiagnosisDate(record[3])
nr := icd10AnalysisMap.fillInPatientDiagnoses(patient, DIDString, date)
if nr > 0 {
@@ -848,65 +295,75 @@ func parseTrinetXPatientDiagnoses(diagnosesFile, treatmentInfoFile string, patie
continue
}
//Check if diagnosis is event of interest.
- if patient.EOIDate == nil && TriNetXEventOfInterest(DIDString) {
+ if patient.EOIDate == nil && EventOfInterest(DIDString, eoid) {
EOICtr++
- patient.EOIDate = &date // mark first event of interest (e.g. bladder cancers diagnosis)
- }
- }
- var nonICD10DiagnosesMap map[string]*TreatmentInfo
- nonICDCtr := 0
- if treatmentInfoFile != "" {
- nonICD10DiagnosesMap = parseTriNetXTreatmentFile(treatmentInfoFile)
- for _, patient := range patients.PIDMap {
- //fill in non ICD10 diagnoses derived from procedure info
- r := icd10AnalysisMap.fillInNonICDPatientDiagnoses(patient, nonICD10DiagnosesMap)
- nonICDCtr = nonICDCtr + r
+ patient.EOIDate = &date // mark first event of interest
}
}
for _, patient := range patients.PIDMap {
trajectory.SortDiagnoses(patient)
trajectory.CompactDiagnoses(patient)
}
- fmt.Println("Parsed diagnosis data.")
- fmt.Print("Parsed ", ctr, " diagnoses ")
- fmt.Println("of which ", ctrID09, " ICD09 diagnoses and ", ctr-ctrID09, " ICD10 diagnoses, and ", ctrExcl, " diagnoses excluded from analysis")
- fmt.Println("and of which ", EOICtr, " events of interest.")
- fmt.Println("Parsed non ICD diagnoses for: ", nonICDCtr, " patients.")
+ slog.Info("Parsed diagnosis data",
+ slog.Int("diagnoses", ctr),
+ slog.Int("excluded", ctrExcl),
+ slog.Int("events-of-interest", EOICtr),
+ )
}
-func ParseTriNetXData(name, patientFile, diagnosisFile, diagnosisInfoFile, treatmentInfoFile string, nofCohortAges,
- level int, minYears, maxYears float64, icd9ToIcd10File string, filters []trajectory.PatientFilter) (*trajectory.Experiment, *trajectory.PatientMap) {
+func (maps *icd10AnalysisMapsFromXML) extendWithATC(file string) {
+ codes := parseATCCodes(file)
+ for code, desc := range codes {
+ maps.NofDiagnosisCodes++
+ maps.DIDMap[code] = maps.NofDiagnosisCodes
+ maps.NameMap[maps.NofDiagnosisCodes] = desc
+ }
+}
+
+func (maps *icd10AnalysisMapsFromCCSR) extendWithATC(file string) {
+ codes := parseATCCodes(file)
+ for code, desc := range codes {
+ maps.NofDiagnosisCodes++
+ maps.DIDMap[code] = []int{maps.NofDiagnosisCodes}
+ maps.NameMap[maps.NofDiagnosisCodes] = desc
+ }
+}
+
+func ParseData(name, patientFile, diagnosisFile, diagnosisInfoFile, actFile string, nofCohortAges, level int,
+ filters []trajectory.PatientFilter, eoid []string) (*trajectory.Experiment, *trajectory.PatientMap) {
// parse data
// fill in patients
- patients, nofRegions := parseTriNetXPatientData(patientFile, nofCohortAges)
+ patients, nofRegions := parsePatientData(patientFile, nofCohortAges)
// fill in icd10 to analysis map
var analysisMaps AnalysisMaps
var nofDiagnosisCodes int
var nameMap map[int]string
var idMap map[int]string
if filepath.Ext(diagnosisInfoFile) == ".xml" {
- maps := initializeIcd10AnalysisMapsFromXML(diagnosisInfoFile, level)
+ maps := initializeIcd10AnalysisMapsFromXML(diagnosisInfoFile, level, getIcd10DescToExcludeFromAnalysis())
+ if actFile != "" {
+ maps.extendWithATC(actFile)
+ }
analysisMaps = maps
nofDiagnosisCodes = maps.NofDiagnosisCodes
nameMap = maps.NameMap
idMap = maps.getIdMap()
}
if filepath.Ext(diagnosisInfoFile) == ".csv" || filepath.Ext(diagnosisInfoFile) == ".CSV" {
- maps := initializeIcd10AnalysisMapsFromCCSR(diagnosisInfoFile)
+ maps := initializeIcd10AnalysisMapsFromCCSR(diagnosisInfoFile, getIcd10CodesToExcludeFromAnalysis())
+ if actFile != "" {
+ maps.extendWithATC(actFile)
+ }
analysisMaps = maps
nofDiagnosisCodes = maps.NofDiagnosisCodes
nameMap = maps.NameMap
idMap = maps.getIdMap()
}
- icd9ToIcd10Map := map[string]string{}
- if icd9ToIcd10File != "" {
- icd9ToIcd10Map = parseIcd9ToIcd10Mapping(icd9ToIcd10File)
- }
// fill in diagnoses for patients
- parseTrinetXPatientDiagnoses(diagnosisFile, treatmentInfoFile, patients, analysisMaps, icd9ToIcd10Map)
+ parsePatientDiagnoses(diagnosisFile, patients, analysisMaps, eoid)
// Apply patient filter
patients = trajectory.ApplyPatientFilters(filters, patients)
- fmt.Println("Filtered down to: ", len(patients.PIDMap), " patients.")
+ slog.Info("Filtered down to", slog.Int("patients", len(patients.PIDMap)))
// create cohorts
cohorts := trajectory.InitializeCohorts(patients, nofCohortAges, nofRegions, nofDiagnosisCodes)
mergedCohort := trajectory.MergeCohorts(cohorts)
@@ -915,6 +372,7 @@ func ParseTriNetXData(name, patientFile, diagnosisFile, diagnosisInfoFile, treat
Level: level,
NofDiagnosisCodes: nofDiagnosisCodes,
DxDRR: trajectory.MakeDxDRR(nofDiagnosisCodes),
+ DxDRREmpProb: trajectory.MakeDxDRR(nofDiagnosisCodes),
DxDPatients: trajectory.MakeDxDPatients(nofDiagnosisCodes),
DPatients: mergedCohort.DPatients,
Cohorts: cohorts,
@@ -927,156 +385,3 @@ func ParseTriNetXData(name, patientFile, diagnosisFile, diagnosisInfoFile, treat
}
return &exp, patients
}
-
-// opening json file with ICD09 -> ICD10 mapping
-
-func parseIcd9ToIcd10Mapping(file string) map[string]string {
- jsonFile, err := os.Open(file)
- if err != nil {
- panic(err)
- }
- defer jsonFile.Close()
- fmt.Println("Parsing ICD9 to ICD10 mapping from a json file.")
- jsonBytes, _ := ioutil.ReadAll(jsonFile)
- var mapping map[string]string
- json.Unmarshal(jsonBytes, &mapping)
- return mapping
-}
-
-// TumorInfo is a struct for storing bladder cancer tumor information concerning: tumor size, tumor lymph nodes, tumor
-// metastasis
-type TumorInfo struct {
- TStage, NStage, MStage, Stage string
- Date trajectory.DiagnosisDate
-}
-
-// getTumorStage converts tumor size, number of lymph nodes, and metastatis level into an overall cancer stage.
-// T stages: Ta,T1,Tis,T2,T3,T4
-// N stages: N0,N1,N2,N3
-// M stages: M0,M1
-// Stage 0a: Ta,N0,M0
-// Stage 0is:Tis,N0,M0 known as carcinoma in situ (CIS)
-// Stage I: T1,N0,M0
-// Stage II: T2,N0,M0
-// Stage IIIA: T3a,T3b, or T4a,N0,M0 --or-- T1 to T4a,N1,M0
-// Stage IIIB: T1 to T4a, N2 or N3, M0
-// Stage IVA: T4b,any N,M0 or any T, any N, M1a
-// Stage IVB: any T, any N, M1b
-func getTumorStage(tStage, nStage, mStage string) string {
- if nStage == "N0" && mStage == "M0" {
- switch tStage {
- case "Ta":
- return "0a"
- case "Tis":
- return "0is"
- case "T1":
- return "I"
- case "T2":
- return "II"
- case "T3a", "T3b", "T4a":
- return "IIIA"
- }
- }
- if nStage == "N1" && mStage == "M0" {
- switch tStage {
- case "T1", "T1a", "T1b", "T2", "T2a", "T2b", "T3", "T3a", "T3b", "T4a":
- return "IIIA"
- }
- }
- if (nStage == "N2" || nStage == "N3") && mStage == "M0" {
- switch tStage {
- case "T1", "T1a", "T1b", "T2", "T2a", "T2b", "T3", "T3a", "T3b", "T4", "T4a":
- return "IIIB"
- }
- }
- if tStage == "T4b" && mStage == "M0" {
- return "IVA"
- }
- if mStage == "M1a" {
- return "IVA"
- }
- if mStage == "M1b" {
- return "IVB"
- }
- return tStage + nStage + mStage
-}
-
-// tumorIsCISStage checks if tumor is flat or carcinoma in situ (CIS).
-func tumorIsCISStage(tumor *TumorInfo) bool {
- return tumor.Stage == "0is"
-}
-
-// parsetTriNetXTumorData parses the tumor data from a csv file and returns a map PIDString -> []*TumorInfo.
-func ParsetTriNetXTumorData(fileName string) map[string][]*TumorInfo {
- file, err := os.Open(fileName)
- if err != nil {
- panic(err)
- }
- defer func() {
- if err := file.Close(); err != nil {
- panic(err)
- }
- }()
- result := map[string][]*TumorInfo{}
- reader := csv.NewReader(file)
- for {
- record, err := reader.Read()
- if err == io.EOF {
- break
- }
- if err != nil {
- panic(err)
- }
- tumorSite := strings.Split(record[4], ".")
- if tumorSite[0] == "C67" { //only record bladder cancer information
- PIDString := record[0]
- date := parseTriNetXDiagnosisDate(record[1])
- tumorSizeInfo := strings.Split(record[10], "_")
- numberOfLymphNodesInfo := strings.Split(record[11], "_")
- metastaticInfo := strings.Split(record[12], "_")
- if len(tumorSizeInfo) == 1 || len(numberOfLymphNodesInfo) == 1 || len(metastaticInfo) == 1 {
- continue
- }
- tumor := &TumorInfo{Date: date, TStage: tumorSizeInfo[1], NStage: numberOfLymphNodesInfo[1],
- MStage: metastaticInfo[1]}
- tumor.Stage = getTumorStage(tumorSizeInfo[1], numberOfLymphNodesInfo[1], metastaticInfo[1])
- if ts, ok := result[PIDString]; ok {
- result[PIDString] = append(ts, tumor)
- } else {
- result[PIDString] = []*TumorInfo{tumor}
- }
- }
- }
- printTumorInfoSummary(result)
- return result
-}
-
-func printTumorInfoSummary(tumorInfo map[string][]*TumorInfo) {
- fmt.Println("Parsed tumor info. Found tumor info for: ", len(tumorInfo), " patients.")
- ctr := map[string]int{}
- for _, tumors := range tumorInfo {
- for _, tumor := range tumors {
- ctr[tumor.TStage]++
- ctr[tumor.NStage]++
- ctr[tumor.MStage]++
- }
- }
- stages := []string{}
- for stage, _ := range ctr {
- stages = append(stages, stage)
- }
- sort.Strings(stages)
- for _, stage := range stages {
- fmt.Println("For stage: ", stage, ": ", ctr[stage], " entries.")
- }
-}
-
-func printTumorInfo(tumorInfo map[int][]*TumorInfo) {
- fmt.Println("Nr of patients with tumor info: ", len(tumorInfo))
- for pid, infos := range tumorInfo {
- for _, info := range infos {
- fmt.Println("Patient with pid: ", pid, " has TStage: ", info.TStage, " NStage: ", info.NStage, " MStage: ",
- info.MStage, " Global stage: ", info.Stage)
- }
- }
-}
diff --git a/app/parseICD10.go b/app/parseICD10.go
new file mode 100644
index 0000000..4fae791
--- /dev/null
+++ b/app/parseICD10.go
@@ -0,0 +1,720 @@
+package app
+
+import (
+ "encoding/csv"
+ "encoding/xml"
+ "fmt"
+ "io"
+ "log/slog"
+ "os"
+ "ptra/trajectory"
+ "slices"
+)
+
+//Package ptra implements a patient trajectory analysis tool.
+//The ptra program has 3 data inputs:
+//A file mapping diagnosis ID (DID) -> medical name.
+//A file with patient info, associating a patient ID (PID) with YOB, sex, etc.
+//A file with diagnosis info, mapping PID -> DID, date.
+
+//TriNetX Database stores diagnoses data using a mix of ICD10 and ICD9 codes.
+//We have an additional file that maps ICD9 IDs -> ICD10 IDs.
+//We can download the ICD10 ID -> medical name from https://www.cms.gov/medicare/icd-10/2022-icd-10-cm as an xml file.
+//TriNetX stores patient info as a csv file, as well as the diagnoses info.
+//For the medical name mapping, we can also use the ICD10 -> CCSR Mapping which maps ICD10 onto 530 categories with medical meaning. This
+//mapping can be downloaded from https://www.hcup-us.ahrq.gov/toolssoftware/ccsr/dxccsr.jsp#download as a CSV file. This
+//mapping performs a mapping ICD10 -> CCSR categpries -> medical name.
+
+//Parsing ICD10 names hierarchy from xml
+//Structs for unmarshalling ICD10 xml data
+//Structure of an ICD10 code: ABC.XYZ(D): up to 7 characters
+//The xml file structures these codes using:
+//
+//with chapter the first level of the diagnosis code (A), section the second level (D) and desc the rest (C.XYZ(D))
+
+//The ptra program needs a names map that maps DID -> medical name. The following code extracts a name map from an ICD10
+//hierarchy and a given level.
+
+// CDC ICD10 hierarchy
+// parseIcd10HierarchyFromXML parses the xml file with the ICD10 hierarchy into an icd10Hierarchy object.
+func parseIcd10HierarchyFromXml(file string) icd10Hierarchy {
+ slog.Info("Parsing ICD10 code hierarchy from XML", slog.String("file", file))
+ //open file
+ xmlFile, err := os.Open(file)
+ if err != nil {
+ panic(err)
+ }
+ defer xmlFile.Close()
+ xmlFileBytes, _ := io.ReadAll(xmlFile)
+ //unmarshall
+ icd10Hierarchy := icd10Hierarchy{}
+ xml.Unmarshal(xmlFileBytes, &icd10Hierarchy)
+ return icd10Hierarchy
+}
+
+// printIcd10Hierarchy prints an ICD10 hierarchy parsed from an XML file.
+func printIcd10Hierarchy(hierarchy icd10Hierarchy) {
+ slog.Info("Printing ICD10 code hierarchy.")
+ // count # DID per level
+ ctr1, ctr2, ctr3, ctr4, ctr5, ctr6, ctr7 := 0, 0, 0, 0, 0, 0, 0
+ for _, chap := range hierarchy.Chapters {
+ // level 1
+ ctr1++
+ slog.Info("\tChapter: " + chap.Desc)
+ for _, section := range chap.Sections {
+ // level 2
+ ctr2++
+ slog.Debug("\t\tSection: " + section.Desc)
+ for _, diag := range section.Diagnoses {
+ // level 3
+ ctr3++
+ slog.Debug("\t\t\t" + diag.Name + " : " + diag.Desc)
+ if len(diag.Diagnoses) == 0 {
+ continue
+ }
+ for _, diag := range diag.Diagnoses {
+ // level 4
+ ctr4++
+ slog.Debug("\t\t\t" + diag.Name + " : " + diag.Desc)
+ if len(diag.Diagnoses) == 0 {
+ continue
+ }
+ for _, diag := range diag.Diagnoses {
+ // level 5
+ ctr5++
+ slog.Debug("\t\t\t" + diag.Name + " : " + diag.Desc)
+ if len(diag.Diagnoses) == 0 {
+ continue
+ }
+ for _, diag := range diag.Diagnoses {
+ // level 6
+ ctr6++
+ slog.Debug("\t\t\t" + diag.Name + " : " + diag.Desc)
+ if len(diag.Diagnoses) == 0 {
+ continue
+ }
+ // level 7
+ ctr7++
+ slog.Debug("\t\t\t" + diag.Name + " : " + diag.Desc)
+ }
+ }
+ }
+ }
+ }
+ }
+ slog.Info("#ICD10 codes/descriptors per level:",
+ slog.Int("Lvl-1", ctr1),
+ slog.Int("Lvl-2", ctr2),
+ slog.Int("Lvl-3", ctr3),
+ slog.Int("Lvl-4", ctr4),
+ slog.Int("Lvl-5", ctr5),
+ slog.Int("Lvl-6", ctr6),
+ slog.Int("Lvl-7", ctr7),
+ )
+}
+
+type cdcICD10HierarchyXML struct {
+ XMLName xml.Name `xml:"ICD10CM.tabular"`
+}
+
+// diag captures the lowest levels of the ICD10 code
+type diag struct {
+ Name string `xml:"name"` //Unique diagnosis ID (DID) in ICD10 encoding
+ Desc string `xml:"desc"` //A medical name/description for a DID
+ Diagnoses []diag `xml:"diag"` //A diagnosis can be split into more detailed diagnoses descriptors.
+}
+
+// section captures the second level of the ICD10 code
+type section struct {
+ Desc string `xml:"desc"` //A medical name/description for a DID.
+ Id string `xml:"id,attr"` //Unique diagnosis ID (DID) in ICD10 encoding
+ Diagnoses []diag `xml:"diag"` //A diagnosis can be split into more detailed diagnoses descriptors.
+}
+
+// chapter captures the first (highest) level of the ICD10 code
+type chapter struct {
+ XmlName xml.Name `xml:"chapter"`
+ Desc string `xml:"desc"`
+ Sections []section `xml:"section"`
+}
+
+// icd10Hierarchy contains the full xml table with the ICD10 code hierarchy.
+type icd10Hierarchy struct {
+ XmlName xml.Name `xml:"ICD10CM.tabular"`
+ Chapters []chapter `xml:"chapter"`
+}
+
+// WHO ICD10
+type SuperClass struct {
+ XMLName xml.Name `xml:"SuperClass"`
+ Code string `xml:"code,attr"`
+}
+
+type whoRubric struct {
+ XMLName xml.Name `xml:"Rubric"`
+ Label string `xml:"Label"`
+}
+
+type whoClass struct {
+ XmlName xml.Name `xml:"Class"`
+ Kind string `xml:"kind,attr"`
+ Code string `xml:"code,attr"`
+ Rubrics []whoRubric `xml:"Rubric"`
+ SuperClass SuperClass `xml:"SuperClass"`
+}
+
+func (class whoClass) isChapter() bool {
+ return class.Kind == "chapter"
+}
+
+func (class whoClass) isBlock() bool {
+ return class.Kind == "block"
+}
+
+func (class whoClass) isCategory() bool {
+ return class.Kind == "category"
+}
+
+type whoIcd10Hierarchy struct {
+ XmlName xml.Name `xml:"ClaML"` //not being parsed??
+ Chapters []whoClass `xml:"Class"`
+}
+
+type whoICD10HierarchyXML struct {
+ XmlName xml.Name `xml:"Class"`
+}
+
+func ParseWhoIcd10HierarchyFromXml(file string) whoIcd10Hierarchy {
+ slog.Info("Parsing who ICD10 code hierarchy from XML", slog.String("file", file))
+ //open file
+ xmlFile, err := os.Open(file)
+ if err != nil {
+ panic(err)
+ }
+ defer xmlFile.Close()
+ xmlFileBytes, _ := io.ReadAll(xmlFile)
+ //unmarshall
+ whoICD10Hierarchy := whoIcd10Hierarchy{}
+ xml.Unmarshal(xmlFileBytes, &whoICD10Hierarchy)
+ return whoICD10Hierarchy
+}
+
+// CheckIcd10HierarchyXMLFile checks if an xml file contains the CDC ICD10 hierarchy or the WHO ICD10 hierarchy
+// Returns a string "cdc" or "who" respectively
+func CheckIcd10HierarchyXMLFile(file string) string {
+ //open file
+ xmlFile, err := os.Open(file)
+ if err != nil {
+ panic(err)
+ }
+ defer xmlFile.Close()
+ xmlFileBytes, _ := io.ReadAll(xmlFile)
+ cdcHierarchy := cdcICD10HierarchyXML{}
+ xml.Unmarshal(xmlFileBytes, &cdcHierarchy)
+ if cdcHierarchy.XMLName.Local == "ICD10CM.tabular" {
+ return "cdc"
+ }
+ whoHierarchy := whoICD10HierarchyXML{}
+ xml.Unmarshal(xmlFileBytes, &whoHierarchy)
+ if whoHierarchy.XmlName.Local == "Class" {
+ return "who"
+ }
+ slog.Warn("Unknown WHO-ICD10",
+ slog.String("hierarchy", fmt.Sprint(whoHierarchy)))
+ return "unknown"
+}
+
+// ICD10 structures
+// icd10Name is a struct for containing a medical name + level + the categories of a DID in ICD10 encoding.
+type icd10Name struct {
+ name string //medical name for a DID in ICD10 encoding
+ categories [6]string //the names of the ICD10 encoding higher and lower in the hierarchy.
+ level int //the ICD10 hierarchy level of this name.
+}
+
+type icd10Table struct {
+ namesMap map[string]icd10Name //maps ICD10 DID to a medical name, level, and categories to which it belongs.
+}
+
+// selectParentCategory returns the ICD10 name of the parent category to which a DID belongs.
+func selectParentCategory(name icd10Name) string {
+ if name.level > 0 {
+ return name.categories[name.level-1]
+ }
+ return "None"
+}
+
+func printIcd10NameMap(table map[string]icd10Name) {
+ slog.Info("ICD10 Name map: ")
+ for id, name := range table {
+ slog.Debug("%s : %s", id, name)
+ }
+}
+
+// initializeIcd10NameMap initializes a name map for ICD10 DID -> medical name, level, and categories it belongs to.
+func initializeIcd10NameMap(file string) map[string]icd10Name {
+ target := CheckIcd10HierarchyXMLFile(file)
+ if target == "who" {
+ return InitializeWhoIcd10NameMap(file)
+ }
+ if target == "cdc" {
+ return initializeCdcIcd10NameMap(file)
+ }
+ panic("Unknown file format ICD10 hierarchy")
+}
+
+func initializeCdcIcd10NameMap(file string) map[string]icd10Name {
+ icd10NameMap := map[string]icd10Name{} //maps ICD10 DID to a medical name, level, and categories to which it belongs.
+ icd10Hierarchy := parseIcd10HierarchyFromXml(file)
+ for _, chap := range icd10Hierarchy.Chapters {
+ category0 := chap.Desc
+ for _, section := range chap.Sections {
+ category1 := section.Desc
+ // manually unrolled loop since we know hierarchy is max 7 levels, otherwise recursive code
+ for _, diag := range section.Diagnoses {
+ if len(diag.Diagnoses) == 0 {
+ icd10Name := icd10Name{name: diag.Desc,
+ categories: [6]string{category0, category1, "NONE", "NONE", "NONE", "NONE"}, level: 2}
+ icd10NameMap[diag.Name] = icd10Name
+ continue
+ }
+ category2 := diag.Desc
+ for _, diag := range diag.Diagnoses {
+ if len(diag.Diagnoses) == 0 {
+ icd10Name := icd10Name{name: diag.Desc,
+ categories: [6]string{category0, category1, category2, "NONE", "NONE", "NONE"},
+ level: 3}
+ icd10NameMap[diag.Name] = icd10Name
+ continue
+ }
+ category3 := diag.Desc
+ for _, diag := range diag.Diagnoses {
+ if len(diag.Diagnoses) == 0 {
+ ICD10Name := icd10Name{name: diag.Desc,
+ categories: [6]string{category0, category1, category2, category3, "NONE", "NONE"},
+ level: 4}
+ icd10NameMap[diag.Name] = ICD10Name
+ continue
+ }
+ category4 := diag.Desc
+ for _, diag := range diag.Diagnoses {
+ if len(diag.Diagnoses) == 0 {
+ ICD10Name := icd10Name{name: diag.Desc,
+ categories: [6]string{category0, category1, category2, category3, category4, "NONE"},
+ level: 5}
+ icd10NameMap[diag.Name] = ICD10Name
+ continue
+ }
+ category5 := diag.Desc
+ for _, diag := range diag.Diagnoses {
+ ICD10Name := icd10Name{name: diag.Desc,
+ categories: [6]string{category0, category1, category2, category3, category4, category5},
+ level: 6}
+ icd10NameMap[diag.Name] = ICD10Name
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return icd10NameMap
+}
+
+func InitializeWhoIcd10NameMap(file string) map[string]icd10Name {
+ icd10NameMap := map[string]icd10Name{} //maps ICD10 DID to a medical name, level, and categories to which it belongs.
+ whoIcd10Hierarchy := ParseWhoIcd10HierarchyFromXml(file)
+ //map code to chapter
+ chapters := map[string]whoClass{}
+ blocks := map[string]whoClass{}
+ categories := map[string]whoClass{}
+
+ for _, class := range whoIcd10Hierarchy.Chapters {
+ if class.isChapter() {
+ chapters[class.Code] = class
+ continue
+ }
+ if class.isBlock() {
+ blocks[class.Code] = class
+ continue
+ }
+ if class.isCategory() {
+ categories[class.Code] = class
+ }
+ }
+
+ getParentCategory := func(id string) (whoClass, bool) {
+ v, ok := categories[id]
+ if ok {
+ return v, true
+ }
+ v, ok = blocks[id]
+ if ok {
+ return v, true
+ }
+ v, ok = chapters[id]
+ if ok {
+ return v, true
+ }
+ return whoClass{}, false
+ }
+
+ for _, cat := range categories {
+ parent, ok := getParentCategory(cat.SuperClass.Code)
+ pcats := []whoClass{parent}
+ //grab category parents
+ for ok {
+ parent, ok = getParentCategory(parent.SuperClass.Code)
+ if ok {
+ pcats = append(pcats, parent)
+ }
+ }
+ entry := [6]string{"NONE", "NONE", "NONE", "NONE", "NONE", "NONE"}
+ pctr := 0
+ slices.Reverse(pcats)
+ for _, pcat := range pcats {
+ entry[pctr] = pcat.Rubrics[0].Label
+ pctr++
+ }
+ icd10Name := icd10Name{name: cat.Rubrics[0].Label, categories: entry, level: pctr}
+ icd10NameMap[cat.Code] = icd10Name
+ }
+ return icd10NameMap
+}
+
+// initializeIcd10AnalysisIDMap creates a map ICD10 DID -> analysis DID and a map analysis ID -> medical name. This is
+// useful to remap diagnosis codes used in the input to a higher level in the ICD10 hierarchy. E.g "typhoid fever" and
+// "cholera" are both "infectuous intestinal diseases", so they could both be identified as such during the analysis.
+// This can be interesting to obtain more global patient trajectories/clusters.
+func intializeIcd10AnalysisMaps(icd10NameMap map[string]icd10Name, level int,
+ icd10ToExclude map[string]bool) (map[string]int, map[int]string, int) {
+ analysisIdMap := map[string]int{} // maps icd 10 code to analysis ID
+ analysisNameMap := map[int]string{} // maps analysis ID to a medical name
+ nameToAnalysisIdMap := map[string]int{} // maps medical name to analysis ID
+ ctr := 0 //serves as analysis ID generator
+ for icd10Code, icd10Name := range icd10NameMap {
+ if _, ok := icd10ToExclude[icd10Name.categories[0]]; ok {
+ // code to exclude from analysis
+ continue
+ }
+ var name string
+ if level == icd10Name.level || level > icd10Name.level {
+ name = icd10Name.name
+ } else {
+ name = icd10Name.categories[level]
+ }
+ // may already have seen name, because of level
+ newID, ok := nameToAnalysisIdMap[name]
+ if !ok {
+ newID = ctr
+ ctr++
+ analysisNameMap[newID] = name
+ nameToAnalysisIdMap[name] = newID
+ }
+ analysisIdMap[icd10Code] = newID
+ }
+ extra := getNonICD10CodesToAddToAnalysis()
+ for code, name := range extra {
+ analysisNameMap[ctr] = name
+ nameToAnalysisIdMap[name] = ctr
+ analysisIdMap[code] = ctr
+ ctr++
+ }
+ slog.Info("Mapped ICD10 codes",
+ slog.Int("N", len(icd10NameMap)),
+ slog.Int("level", level),
+ slog.Int("analysis-IDs", ctr))
+ return analysisIdMap, analysisNameMap, ctr
+}
+
+// ccsrCategory is a struct for containing CCSR categories, encoding medically meaningful names for a DID in ICD10
+// encoding.
+type ccsrCategory struct {
+ name string //default CCSR category/medical name
+ id string //CCSR ID for default category
+ categories map[string]string //Up to 6 different CCSR categories an ICD10 code is mapped to
+}
+
+type icd10ToCCSRTable map[string]ccsrCategory //maps ICD10 DID to its CCSR categories
+
+// ccsrIcd10ToProperIcd10 transforms the ICD10 code from a ccsr file into a proper ICD10 code. The ICD10 codes in the
+// ccsr file are stored without the ".", so this needs to be added to be able to compare to any other data that uses
+// ICD10 codes. Also removes superfluous quotes stored in the ccsr file.
+func ccsrIcd10ToProperIcd10(code string) string {
+ return code[1:4] + "." + code[4:len(code)-1]
+}
+
+// initializeIcd10NameMapFromCCSR initializes a name map for ICD10 DID -> CCSR categories (medical names)
+func initializeIcd10ToCCSRMap(file string) map[string]ccsrCategory {
+ //map to collect data
+ icd10ToCCSRTable := map[string]ccsrCategory{}
+ //open file
+ csvFile, err := os.Open(file)
+ if err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := csvFile.Close(); err != nil {
+ panic(err)
+ }
+ }()
+ //parse file
+ reader := csv.NewReader(csvFile)
+ //the header is 'ICD-10-CM CODE','ICD-10-CM CODE DESCRIPTION','Default CCSR CATEGORY IP','
+ //Default CCSR CATEGORY DESCRIPTION IP','Default CCSR CATEGORY OP','Default CCSR CATEGORY DESCRIPTION OP','
+ //CCSR CATEGORY 1','CCSR CATEGORY 1 DESCRIPTION','CCSR CATEGORY 2','CCSR CATEGORY 2 DESCRIPTION',
+ //'CCSR CATEGORY 3','CCSR CATEGORY 3 DESCRIPTION','CCSR CATEGORY 4','CCSR CATEGORY 4 DESCRIPTION',
+ //'CCSR CATEGORY 5','CCSR CATEGORY 5 DESCRIPTION','CCSR CATEGORY 6','CCSR CATEGORY 6 DESCRIPTION'
+ // skip header
+ reader.Read()
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ //create CSSR category, set default category
+ category := ccsrCategory{name: record[2], id: record[3], categories: map[string]string{}}
+ //fill in unique CSSR alternative categories, up to 6 possible
+ for i := 6; i <= 17; i = i + 2 {
+ catID := record[i]
+ catName := record[i+1]
+ if catName == "" || catID == "' '" {
+ continue
+ }
+ if _, ok := category.categories[catID]; !ok {
+ category.categories[catID] = catName
+ }
+ }
+ //add category to result
+ icd10Code := ccsrIcd10ToProperIcd10(record[0])
+ icd10ToCCSRTable[icd10Code] = category
+ }
+ return icd10ToCCSRTable
+}
+
+// printIcd10ToCSSRTable is a simple function to print the map from iCD10 code to ccsr category. Useful for debugging.
+func printIcd10ToCCSRTable(tab map[string]ccsrCategory) {
+ slog.Info("ICD10 to CCSR table")
+ ctr := 0
+ for icd10Code, ccsr := range tab {
+ ctr++
+ slog.Debug("%s : %s", icd10Code, ccsr.categories)
+ if ctr >= 1000 {
+ return
+ }
+ }
+}
+
+// initializeIcd10AnalysisMapsCCSR creates a map ICD10 DID -> [analysis DID] and a map analysis ID -> medical name,
+// starting from a CCSR mapping, which maps ICD10 codes onto medical meaningful categories.
+// Each icd10 code can be mapped to multiple ccsr categories, and therefore to multiple analysis IDs.
+// TO DO: exclude specific ICD10 codes from the analysis.
+func initializeIcd10AnalysisMapsCCSR(icd10ToCssrMap map[string]ccsrCategory,
+ icd10ToExclude map[string]bool) (map[string][]int, map[int]string, int) {
+ analysisIdMap := map[string][]int{} // maps icd 10 code to analysis IDs
+ analysisNameMap := map[int]string{} // maps analysis ID to a medical name
+ ccsrIDMap := map[string]int{}
+ ctr := 0 //serves as analysis ID generator
+ for icd10Code, ccsr := range icd10ToCssrMap {
+ if _, ok := icd10ToExclude[icd10Code[0:1]]; ok {
+ continue
+ }
+ ids := []int{}
+ for id, name := range ccsr.categories {
+ var ccsrID int
+ var ok bool
+ if ccsrID, ok = ccsrIDMap[id]; !ok {
+ ccsrID = ctr
+ analysisNameMap[ctr] = name
+ ccsrIDMap[id] = ccsrID
+ ctr++
+ }
+ ids = append(ids, ccsrID)
+ }
+ analysisIdMap[icd10Code] = ids
+ }
+ extra := getNonICD10CodesToAddToAnalysis()
+ for code, name := range extra {
+ analysisNameMap[ctr] = name
+ analysisIdMap[code] = []int{ctr}
+ ctr++
+ }
+ slog.Info("Mapped ICD10 codes",
+ slog.Int("N", len(icd10ToCssrMap)),
+ slog.Int("analysis-IDs", ctr))
+ return analysisIdMap, analysisNameMap, ctr
+}
+
+type icd10AnalysisMapsFromCCSR struct {
+ NameMap map[int]string // map analysis DID -> medical name
+ NofDiagnosisCodes int // nr of different diagnosis codes
+ DIDMap map[string][]int // maps ICD10 Code onto multiple DIDs
+}
+
+type icd10AnalysisMapsFromXML struct {
+ NameMap map[int]string // map analysis DID -> medical name
+ NofDiagnosisCodes int // nr of different diagnosis codes
+ DIDMap map[string]int // map ICD10 Code -> DID
+}
+
+func (analysisMap icd10AnalysisMapsFromXML) getDID(icd10DID string) int {
+ if v, ok := analysisMap.DIDMap[icd10DID]; ok {
+ return v
+ } else {
+ return -1
+ }
+}
+
+func (analysisMap icd10AnalysisMapsFromCCSR) getDID(icd10DID string) []int {
+ if v, ok := analysisMap.DIDMap[icd10DID]; ok {
+ return v
+ }
+ return nil
+}
+
+func (analysisMap icd10AnalysisMapsFromXML) GetICDCode(did int) string {
+ for icd10Code, didCode := range analysisMap.DIDMap {
+ if didCode == did {
+ return icd10Code
+ }
+ }
+ return ""
+}
+
+func (analysisMap icd10AnalysisMapsFromCCSR) GetICDCode(did int) string {
+ for icd10Code, didCodes := range analysisMap.DIDMap {
+ for _, didCode := range didCodes {
+ if didCode == did {
+ return icd10Code
+ }
+ }
+ }
+ return ""
+}
+
+func (analysisMap icd10AnalysisMapsFromXML) getIdMap() map[int]string {
+ res := map[int]string{}
+ for icd10Code, didCode := range analysisMap.DIDMap {
+ res[didCode] = icd10Code
+ }
+ return res
+}
+
+func (analysisMap icd10AnalysisMapsFromCCSR) getIdMap() map[int]string {
+ res := map[int]string{}
+ for icd10Code, didCodes := range analysisMap.DIDMap {
+ for _, didCode := range didCodes {
+ res[didCode] = icd10Code
+ }
+ }
+ return res
+}
+
+// AnalysisMaps represent maps extracted from the input that map analysis IDs onto medical terms and vice versa. This is
+// an interface that defines several methods. getICDCode returns for a did the original id in the input for the
+// diagnostic event. fillInPatientDiagnoses creates for a given diagnosis identifier from the input a Diagnosis object
+// and adds it to a patient's list of diagnoses.
+type AnalysisMaps interface {
+ fillInPatientDiagnoses(patient *trajectory.Patient, DidString string, date trajectory.DiagnosisDate) int
+ fillInNonICDPatientDiagnoses(patient *trajectory.Patient, infoMap map[string]*TreatmentInfo) int
+ GetICDCode(did int) string
+ getIdMap() map[int]string
+}
+
+func (analysisMap icd10AnalysisMapsFromXML) fillInPatientDiagnoses(patient *trajectory.Patient, DIDString string,
+ date trajectory.DiagnosisDate) int {
+ DID := analysisMap.getDID(DIDString)
+ if DID == -1 {
+ return 1 // icd10 diagnosis excluded from analysis
+ }
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: DID, Date: date}
+ trajectory.AddDiagnosis(patient, diagnosis)
+ return 0
+}
+
+func (analysisMap icd10AnalysisMapsFromCCSR) fillInPatientDiagnoses(patient *trajectory.Patient, DIDString string,
+ date trajectory.DiagnosisDate) int {
+ DIDs := analysisMap.getDID(DIDString)
+ if DIDs == nil {
+ return 1 // icd10 code excluded from analysis
+ }
+ for _, DID := range DIDs {
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: DID, Date: date}
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ return 0
+}
+
+func (analysisMap icd10AnalysisMapsFromXML) fillInNonICDPatientDiagnoses(patient *trajectory.Patient,
+ infoMap map[string]*TreatmentInfo) int {
+ nonIcd := 0
+ if info, ok := infoMap[patient.PIDString]; ok {
+ if info.RCDate != nil {
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: analysisMap.DIDMap["C98"], Date: *info.RCDate}
+ nonIcd = 1
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ if info.MVACDate != nil {
+ nonIcd = 1
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: analysisMap.DIDMap["C99"], Date: *info.MVACDate}
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ if info.IVTDate != nil {
+ nonIcd = 1
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: analysisMap.DIDMap["C100"], Date: *info.IVTDate}
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ }
+ return nonIcd
+}
+
+func (analysisMap icd10AnalysisMapsFromCCSR) fillInNonICDPatientDiagnoses(patient *trajectory.Patient,
+ infoMap map[string]*TreatmentInfo) int {
+ nonIcd := 0
+ if info, ok := infoMap[patient.PIDString]; ok {
+ if info.RCDate != nil {
+ dids := analysisMap.DIDMap["C98"]
+ for _, did := range dids {
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: did, Date: *info.RCDate}
+ nonIcd = 1
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ }
+ if info.MVACDate != nil {
+ dids := analysisMap.DIDMap["C99"]
+ for _, did := range dids {
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: did, Date: *info.MVACDate}
+ nonIcd = 1
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ }
+ if info.IVTDate != nil {
+ dids := analysisMap.DIDMap["C100"]
+ for _, did := range dids {
+ nonIcd = 1
+ diagnosis := &trajectory.Diagnosis{PID: patient.PID, DID: did, Date: *info.IVTDate}
+ trajectory.AddDiagnosis(patient, diagnosis)
+ }
+ }
+ }
+ return nonIcd
+}
+
+// initializeIcd10AnalysisMaps returns a map ICD10 DID -> internal analysis DID and a map analysis DID ->
+// medical name for an ICD10 Hierarchy passed as xml file and a requested hierarchy level.
+func initializeIcd10AnalysisMapsFromXML(file string, level int, icd10ToExclude map[string]bool) icd10AnalysisMapsFromXML {
+ icd10NameMapFromXml := initializeIcd10NameMap(file) // map ICD10 DID -> ICD 10 Name (medical desc, categories, level)
+ analysisIdMap, analysisNameMap, ctr := intializeIcd10AnalysisMaps(icd10NameMapFromXml, level, icd10ToExclude)
+ return icd10AnalysisMapsFromXML{DIDMap: analysisIdMap, NameMap: analysisNameMap, NofDiagnosisCodes: ctr}
+}
+
+// initializeIcd10AnalysisMapsFromCCSR returns a map ICD10 -> []{internal analysis DID} and map analysis DID -> medical
+// name for ICD10 CCSR categorization passed as a csv file.
+func initializeIcd10AnalysisMapsFromCCSR(file string, icd10ToExclude map[string]bool) icd10AnalysisMapsFromCCSR {
+ icd10ToCssrMap := initializeIcd10ToCCSRMap(file) // map ICD10 Code -> CCSR Name
+ analysisIdMap, analysisNameMap, ctr := initializeIcd10AnalysisMapsCCSR(icd10ToCssrMap, icd10ToExclude)
+ return icd10AnalysisMapsFromCCSR{DIDMap: analysisIdMap, NameMap: analysisNameMap, NofDiagnosisCodes: ctr}
+}
diff --git a/app/parseTriNetXData.go b/app/parseTriNetXData.go
new file mode 100644
index 0000000..7d67fca
--- /dev/null
+++ b/app/parseTriNetXData.go
@@ -0,0 +1,569 @@
+// PTRA: Patient Trajectory Analysis Library
+// Copyright (c) 2022 imec vzw.
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as
+// published by the Free Software Foundation, either version 3 of the
+// License, or (at your option) any later version, and Additional Terms
+// (see below).
+
+// This program is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+// Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public
+// License and Additional Terms along with this program. If not, see
+// .
+
+package app
+
+import (
+ "bytes"
+ "encoding/csv"
+ "encoding/json"
+ "fmt"
+ "io"
+ "log/slog"
+ "math"
+ "os"
+ "path/filepath"
+ "ptra/trajectory"
+ "ptra/utils"
+ "sort"
+ "strconv"
+ "strings"
+)
+
+// getIcd10DescToExcludeFromAnalysis returns a map that lists ICD10 categories to be excluded from analysis by mapping
+// the ICD10 category description (string) onto a boolean.
+func getIcd10DescToExcludeFromTriNetXAnalysis() map[string]bool {
+ exclude := map[string]bool{}
+ exclude["Pregnancy, childbirth and the puerperium (O00-O9A)"] = true
+ exclude["Certain conditions originating in the perinatal period (P00-P96)"] = true
+ exclude["Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified (R00-R99)"] = true
+ exclude["Injury, poisoning and certain other consequences of external causes (S00-T88)"] = true
+ exclude["External causes of morbidity (V00-Y99)"] = true
+ exclude["Factors influencing health status and contact with health services (Z00-Z99)"] = true
+ return exclude
+}
+
+// getIcd10CodesToExcludeFromAnalysis returns the first letters of ICD10 codes to exclude from analysis.
+func getIcd10CodesToExcludeFromTriNetXAnalysis() map[string]bool {
+ exclude := map[string]bool{}
+ exclude["O"] = true
+ exclude["P"] = true
+ exclude["R"] = true
+ exclude["S"] = true
+ exclude["T"] = true
+ exclude["V"] = true
+ exclude["X"] = true
+ exclude["Y"] = true
+ exclude["Z"] = true
+ return exclude
+}
+
+// getNonICD10CodesToAddToAnalysis returns a set of mockup ICD10 codes to be able to introduce non ICD codes to be
+// included for analysis. It returns a map from mockup ICD10 code (string) to description string. It introduces "C98" for
+// "Radical custectomy (bladder cancer)", "C99" for "MVAC Chemotherapy (bladder cancer)", and "C100" for "Intravesical
+// therapy (bladder cancer)".
+func getNonICD10CodesToAddToAnalysis() map[string]string {
+ return map[string]string{
+ "C98": "Radical cystectomy (bladder cancer)",
+ "C99": "MVAC Chemotherapy (bladder cancer)",
+ "C100": "Intravesical therapy (bladder cancer)",
+ }
+}
+
+// Parsing patient information.
+// parseTriNetXPatientData parses a file with patient information from the TriNetX database. Input: a patient file in csv
+// format, a desired number of age groups to initialize cohorts. Diagnoses of the patient need to be filled in after
+// parsing the diagnoses file.
+func parseTriNetXPatientData(file string, nofCohortAges int) (*trajectory.PatientMap, int) {
+ //open file
+ csvFile, err := os.Open(file)
+ if err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := csvFile.Close(); err != nil {
+ panic(err)
+ }
+ }()
+ patientMap := &trajectory.PatientMap{PIDMap: map[int]*trajectory.Patient{}, PIDStringMap: map[string]int{}}
+ maxYOB := 1850
+ minYOB := 2021
+ deathCr := 0
+ regions := map[string]int{} //counts per region
+ regionIds := map[string]int{}
+ //parse file
+ reader := csv.NewReader(csvFile)
+ //the header is omitted from the TriNetX file, but is should be: patient_id, sex, race, ethnicity, year_of_birth,
+ //age_at_death, patient_regional_location, postal_code, marital_status, reason_yob_missing, month_year_death,
+ //source_id
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ var yob int
+ if yob, err = strconv.Atoi(record[4]); err != nil {
+ continue //skip patients without year of birth
+ }
+ pidString := record[0]
+ patientMap.Ctr++ // avoid using 0 as PID
+ pid := patientMap.Ctr //analysis ID
+ var sex int
+ if record[1] == "M" {
+ sex = trajectory.Male
+ patientMap.MaleCtr++
+ }
+ if record[1] == "F" {
+ sex = trajectory.Female
+ patientMap.FemaleCtr++
+ }
+ dateOfDeathString := record[10]
+ var dateOfDeath *trajectory.DiagnosisDate
+ if len(dateOfDeathString) == 6 {
+ year, err := strconv.Atoi(dateOfDeathString[0:4])
+ if err == nil {
+ month, err := strconv.Atoi(dateOfDeathString[4:6])
+ if err == nil {
+ deathCr++
+ dateOfDeath = &trajectory.DiagnosisDate{
+ Year: year,
+ Month: month,
+ Day: 1, //unknown, default to 1
+ }
+ }
+ }
+ }
+ region := record[6]
+ if _, ok := regions[region]; !ok {
+ regions[region] = 0
+ regionIds[region] = len(regionIds)
+ } else {
+ regions[region]++
+ }
+ patient := trajectory.Patient{
+ PID: pid,
+ PIDString: pidString,
+ YOB: yob,
+ CohortAge: 0,
+ Sex: sex,
+ Diagnoses: []*trajectory.Diagnosis{},
+ DeathDate: dateOfDeath,
+ Region: regionIds[region],
+ }
+ patientMap.PIDMap[pid] = &patient
+ patientMap.PIDStringMap[pidString] = pid
+ maxYOB = utils.MaxInt(yob, maxYOB)
+ minYOB = utils.MinInt(yob, minYOB)
+ }
+ // initialize patient age groups
+ ageRange := float64(maxYOB-minYOB) / float64(nofCohortAges)
+ ageRange = math.Ceil(ageRange)
+ if nofCohortAges > 1 {
+ for _, p := range patientMap.PIDMap {
+ p.CohortAge = int(math.Floor(float64(p.YOB-minYOB) / float64(ageRange)))
+ }
+ }
+
+ var buffer bytes.Buffer
+ for region, nr := range regions {
+ buffer.WriteString(fmt.Sprintf("[%s: %d] ", region, nr))
+ }
+ slog.Info("Parsed patient data",
+ slog.Int("#regions", len(regions)),
+ )
+ slog.Debug("Parsed patient data",
+ slog.String("Regions", buffer.String()),
+ )
+ slog.Info("Parsed patients with year of birth known",
+ slog.Int("N", patientMap.Ctr),
+ slog.Int("females", patientMap.FemaleCtr),
+ slog.Int("males", patientMap.MaleCtr),
+ slog.Int("#deaths", deathCr),
+ )
+ slog.Info("Year of birth",
+ slog.Int("oldest", minYOB),
+ slog.Int("youngest", maxYOB),
+ )
+
+ return patientMap, len(regions)
+}
+
+//Parsing patient diagnoses
+
+// parseTriNetXDiagnosisDate turns a TriNetX date string into DiagnosisDate object.
+func parseTriNetXDiagnosisDate(date string) trajectory.DiagnosisDate {
+ year, err := strconv.Atoi(date[0:4])
+ if err != nil {
+ panic(err)
+ }
+ month, err := strconv.Atoi(date[5:7])
+ if err != nil {
+ panic(err)
+ }
+ day, err := strconv.Atoi(date[8:10])
+ if err != nil {
+ panic(err)
+ }
+ return trajectory.DiagnosisDate{Year: year, Month: month, Day: day}
+}
+
+// TriNetXEventOfInterest checks if the ICD10 code is related to bladder cancer
+func TriNetXEventOfInterest(icd10ID string) bool {
+ if icd10ID == "Z85.1" {
+ return true
+ }
+ if len(icd10ID) >= 3 && icd10ID[0:3] == "C67" {
+ return true
+ }
+ return false
+}
+
+// TreatmentInfo implements a structure for storing the dates of certain bladder cancer treatments.
+type TreatmentInfo struct {
+ RCDate *trajectory.DiagnosisDate //Date of radical cystectomy
+ MVACDate *trajectory.DiagnosisDate //Date of MVAC chemotherapy
+ IVTDate *trajectory.DiagnosisDate //Date of intravesical therapy
+}
+
+// parseTriNetXTreatmentFile parses a csv file that contains information of patient's treatments at different time stamps.
+// It returns a map from PID -> TreatmentInfo.
+func parseTriNetXTreatmentFile(fileName string) map[string]*TreatmentInfo {
+ result := map[string]*TreatmentInfo{}
+ file, err := os.Open(fileName)
+ if err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := file.Close(); err != nil {
+ panic(err)
+ }
+ }()
+ reader := csv.NewReader(file)
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ PIDString := record[0]
+ var rcDate, mvacDate, ivtDate *trajectory.DiagnosisDate
+ if len(record[10]) == 10 { // valid date
+ d := parseTriNetXDiagnosisDate(record[10])
+ rcDate = &d
+ }
+ if len(record[11]) == 10 {
+ d := parseTriNetXDiagnosisDate(record[11])
+ mvacDate = &d
+ }
+ if len(record[13]) == 10 {
+ d := parseTriNetXDiagnosisDate(record[13])
+ rcDate = &d
+ }
+ result[PIDString] = &TreatmentInfo{RCDate: rcDate, MVACDate: mvacDate, IVTDate: ivtDate}
+ }
+ return result
+}
+
+// parseTrinetXPatientDiagnoses parses a csv file containing patient diagnoses. It fills in those diagnoses for the given
+// patients. It uses the icd10AnalysisMap to assign internal analysis DID to the diagnoses.
+// TO DO: Handle ICD09 diagnoses.
+func parseTrinetXPatientDiagnoses(diagnosesFile, treatmentInfoFile string, patients *trajectory.PatientMap, icd10AnalysisMap AnalysisMaps, icd9ToIcd10Map map[string]string) {
+ file, err := os.Open(diagnosesFile)
+ if err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := file.Close(); err != nil {
+ panic(err)
+ }
+ }()
+ reader := csv.NewReader(file)
+ ctr := 0 //for counting the number of parsed diagnoses
+ ctrID09 := 0
+ ctrExcl := 0
+ EOICtr := 0
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ ctr++
+ PIDString := record[0]
+ patient, ok := trajectory.GetPatient(PIDString, patients)
+ if !ok {
+ continue //skip unknown patients
+ }
+ DIDCodeSystem := record[2]
+ DIDString := record[3]
+ if (DIDCodeSystem != "ICD-10-CM") && (DIDCodeSystem != "ICD-10-WHO") {
+ // try to remap ICD9 code to ICD10 codes
+ if DIDString, ok = icd9ToIcd10Map[DIDString]; !ok {
+ continue // skip unkown ICD9 codes
+ }
+ ctrID09++
+ }
+ date := parseTriNetXDiagnosisDate(record[7])
+
+ nr := icd10AnalysisMap.fillInPatientDiagnoses(patient, DIDString, date)
+ if nr > 0 {
+ ctrExcl++
+ continue
+ }
+ //Check if diagnosis is event of interest.
+ if patient.EOIDate == nil && TriNetXEventOfInterest(DIDString) {
+ EOICtr++
+ patient.EOIDate = &date // mark first event of interest (e.g. bladder cancers diagnosis)
+ }
+ }
+ var nonICD10DiagnosesMap map[string]*TreatmentInfo
+ nonICDCtr := 0
+ if treatmentInfoFile != "" {
+ nonICD10DiagnosesMap = parseTriNetXTreatmentFile(treatmentInfoFile)
+ for _, patient := range patients.PIDMap {
+ //fill in non ICD10 diagnoses derived from procedure info
+ r := icd10AnalysisMap.fillInNonICDPatientDiagnoses(patient, nonICD10DiagnosesMap)
+ nonICDCtr = nonICDCtr + r
+ }
+ }
+ for _, patient := range patients.PIDMap {
+ trajectory.SortDiagnoses(patient)
+ trajectory.CompactDiagnoses(patient)
+ }
+
+ slog.Info("Parsed diagnosis data",
+ slog.Int("diagnoses", ctr),
+ slog.Int("ICD09", ctrID09),
+ slog.Int("ICD10", ctr-ctrID09),
+ slog.Int("excluded", ctrExcl),
+ slog.Int("events-of-interest", EOICtr),
+ slog.Int("non-ICD", nonICDCtr),
+ )
+}
+
+func ParseTriNetXData(name, patientFile, diagnosisFile, diagnosisInfoFile, treatmentInfoFile string, nofCohortAges,
+ level int, icd9ToIcd10File string, filters []trajectory.PatientFilter) (*trajectory.Experiment, *trajectory.PatientMap) {
+ // parse data
+ // fill in patients
+ patients, nofRegions := parseTriNetXPatientData(patientFile, nofCohortAges)
+ // fill in icd10 to analysis map
+ var analysisMaps AnalysisMaps
+ var nofDiagnosisCodes int
+ var nameMap map[int]string
+ var idMap map[int]string
+ if filepath.Ext(diagnosisInfoFile) == ".xml" {
+ maps := initializeIcd10AnalysisMapsFromXML(diagnosisInfoFile, level, getIcd10DescToExcludeFromTriNetXAnalysis())
+ analysisMaps = maps
+ nofDiagnosisCodes = maps.NofDiagnosisCodes
+ nameMap = maps.NameMap
+ idMap = maps.getIdMap()
+ }
+ if filepath.Ext(diagnosisInfoFile) == ".csv" || filepath.Ext(diagnosisInfoFile) == ".CSV" {
+ maps := initializeIcd10AnalysisMapsFromCCSR(diagnosisInfoFile, getIcd10CodesToExcludeFromTriNetXAnalysis())
+ analysisMaps = maps
+ nofDiagnosisCodes = maps.NofDiagnosisCodes
+ nameMap = maps.NameMap
+ idMap = maps.getIdMap()
+ }
+ icd9ToIcd10Map := map[string]string{}
+ if icd9ToIcd10File != "" {
+ icd9ToIcd10Map = parseIcd9ToIcd10Mapping(icd9ToIcd10File)
+ }
+ // fill in diagnoses for patients
+ parseTrinetXPatientDiagnoses(diagnosisFile, treatmentInfoFile, patients, analysisMaps, icd9ToIcd10Map)
+ // Apply patient filter
+ patients = trajectory.ApplyPatientFilters(filters, patients)
+ slog.Info("Filtered down to", slog.Int("patients", len(patients.PIDMap)))
+ // create cohorts
+ cohorts := trajectory.InitializeCohorts(patients, nofCohortAges, nofRegions, nofDiagnosisCodes)
+ mergedCohort := trajectory.MergeCohorts(cohorts)
+ exp := trajectory.Experiment{
+ NofAgeGroups: nofCohortAges,
+ Level: level,
+ NofDiagnosisCodes: nofDiagnosisCodes,
+ DxDRR: trajectory.MakeDxDRR(nofDiagnosisCodes),
+ DxDPatients: trajectory.MakeDxDPatients(nofDiagnosisCodes),
+ DPatients: mergedCohort.DPatients,
+ Cohorts: cohorts,
+ Name: name,
+ NameMap: nameMap,
+ NofRegions: nofRegions,
+ IdMap: idMap,
+ FCtr: patients.FemaleCtr,
+ MCtr: patients.MaleCtr,
+ }
+ return &exp, patients
+}
+
+// opening json file with ICD09 -> ICD10 mapping
+
+func parseIcd9ToIcd10Mapping(file string) map[string]string {
+ jsonFile, err := os.Open(file)
+ if err != nil {
+ panic(err)
+ }
+ defer jsonFile.Close()
+ slog.Info("Parsing ICD9 to ICD10 mapping from a json file.")
+ jsonBytes, _ := io.ReadAll(jsonFile)
+ var mapping map[string]string
+ json.Unmarshal(jsonBytes, &mapping)
+ return mapping
+}
+
+// TumorInfo is a struct for storing bladder cancer tumor information concerning: tumor size, tumor lymph nodes, tumor
+// metastasis
+type TumorInfo struct {
+ TStage, NStage, MStage, Stage string
+ Date trajectory.DiagnosisDate
+}
+
+// getTumorStage converts tumor size, number of lymph nodes, and metastatis level into an overall cancer stage.
+// T stages: Ta,T1,Tis,T2,T3,T4
+// N stages: N0,N1,N2,N3
+// M stages: M0,M1
+// Stage 0a: Ta,N0,M0
+// Stage 0is:Tis,N0,M0 known as carcinoma in situ (CIS)
+// Stage I: T1,N0,M0
+// Stage II: T2,N0,M0
+// Stage IIIA: T3a,T3b, or T4a,N0,M0 --or-- T1 to T4a,N1,M0
+// Stage IIIB: T1 to T4a, N2 or N3, M0
+// Stage IVA: T4b,any N,M0 or any T, any N, M1a
+// Stage IVB: any T, any N, M1b
+func getTumorStage(tStage, nStage, mStage string) string {
+ if nStage == "N0" && mStage == "M0" {
+ switch tStage {
+ case "Ta":
+ return "0a"
+ case "Tis":
+ return "0is"
+ case "T1":
+ return "I"
+ case "T2":
+ return "II"
+ case "T3a", "T3b", "T4a":
+ return "IIIA"
+ }
+ }
+ if nStage == "N1" && mStage == "M0" {
+ switch tStage {
+ case "T1", "T1a", "T1b", "T2", "T2a", "T2b", "T3", "T3a", "T3b", "T4a":
+ return "IIIA"
+ }
+ }
+ if (nStage == "N2" || nStage == "N3") && mStage == "M0" {
+ switch tStage {
+ case "T1", "T1a", "T1b", "T2", "T2a", "T2b", "T3", "T3a", "T3b", "T4", "T4a":
+ return "IIIB"
+ }
+ }
+ if tStage == "T4b" && mStage == "M0" {
+ return "IVA"
+ }
+ if mStage == "M1a" {
+ return "IVA"
+ }
+ if mStage == "M1b" {
+ return "IVB"
+ }
+ return tStage + nStage + mStage
+}
+
+// tumorIsCISStage checks if tumor is flat or carcinoma in situ (CIS).
+func tumorIsCISStage(tumor *TumorInfo) bool {
+ return tumor.Stage == "0is"
+}
+
+// parsetTriNetXTumorData parses the tumor data from a csv file and returns a map PIDString -> []*TumorInfo.
+func ParsetTriNetXTumorData(fileName string) map[string][]*TumorInfo {
+ file, err := os.Open(fileName)
+ if err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := file.Close(); err != nil {
+ panic(err)
+ }
+ }()
+ result := map[string][]*TumorInfo{}
+ reader := csv.NewReader(file)
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ tumorSite := strings.Split(record[4], ".")
+ if tumorSite[0] == "C67" { //only record bladder cancer information
+ PIDString := record[0]
+ date := parseTriNetXDiagnosisDate(record[1])
+ tumorSizeInfo := strings.Split(record[10], "_")
+ numberOfLymphNodesInfo := strings.Split(record[11], "_")
+ metastaticInfo := strings.Split(record[12], "_")
+ if len(tumorSizeInfo) == 1 || len(numberOfLymphNodesInfo) == 1 || len(metastaticInfo) == 1 {
+ continue
+ }
+ tumor := &TumorInfo{Date: date, TStage: tumorSizeInfo[1], NStage: numberOfLymphNodesInfo[1],
+ MStage: metastaticInfo[1]}
+ tumor.Stage = getTumorStage(tumorSizeInfo[1], numberOfLymphNodesInfo[1], metastaticInfo[1])
+ if ts, ok := result[PIDString]; ok {
+ result[PIDString] = append(ts, tumor)
+ } else {
+ result[PIDString] = []*TumorInfo{tumor}
+ }
+ }
+ }
+ printTumorInfoSummary(result)
+ return result
+}
+
+func printTumorInfoSummary(tumorInfo map[string][]*TumorInfo) {
+ slog.Info("Parsed tumor info. Found info for", slog.Int("patients", len(tumorInfo)))
+ ctr := map[string]int{}
+ for _, tumors := range tumorInfo {
+ for _, tumor := range tumors {
+ ctr[tumor.TStage]++
+ ctr[tumor.NStage]++
+ ctr[tumor.MStage]++
+ }
+ }
+ stages := []string{}
+ for stage, _ := range ctr {
+ stages = append(stages, stage)
+ }
+ sort.Strings(stages)
+ for _, stage := range stages {
+ slog.Debug("tumor info",
+ slog.String("stage", stage),
+ slog.Int("entries", ctr[stage]),
+ )
+ }
+}
+
+func printTumorInfo(tumorInfo map[int][]*TumorInfo) {
+ slog.Info("Tumor Info", slog.Int("#patients", len(tumorInfo)))
+ for pid, infos := range tumorInfo {
+ for _, info := range infos {
+ slog.Debug(" Patient",
+ slog.Int("pid", pid),
+ slog.String("TStage", info.TStage),
+ slog.String("NStage", info.NStage),
+ slog.String("MStage", info.MStage),
+ slog.String("Global", info.Stage),
+ )
+ }
+ }
+}
diff --git a/app/test_exports.go b/app/test_exports.go
index 0593b22..f76e90c 100644
--- a/app/test_exports.go
+++ b/app/test_exports.go
@@ -27,3 +27,5 @@ var ParseTrinetXPatientDiagnoses = parseTrinetXPatientDiagnoses
var ParseIcd10HierarchyFromXml = parseIcd10HierarchyFromXml
var PrintIcd10Hierarchy = printIcd10Hierarchy
var PrintIcd10NameMap = printIcd10NameMap
+var GetIcd10DescToExcludeFromTriNetXAnalysis = getIcd10DescToExcludeFromTriNetXAnalysis
+var GetIcd10CodesToExcludeFromTriNetXAnalysis = getIcd10CodesToExcludeFromTriNetXAnalysis
diff --git a/cluster/clustering-directly.go b/cluster/clustering-directly.go
index a2c16b9..add8893 100644
--- a/cluster/clustering-directly.go
+++ b/cluster/clustering-directly.go
@@ -24,6 +24,7 @@ import (
"fmt"
"io"
"log"
+ "log/slog"
"os"
"os/exec"
"path/filepath"
@@ -101,21 +102,27 @@ func convertTrajectoriesToAbcFormat(exp *trajectory.Experiment, name string) {
// It does a pairwise comparison of all trajectories by calculating the jaccard similarity coefficients. Subsequently,
// MCL clustering is used to group the trajectories by jaccard similarity into clusters.
func ClusterTrajectoriesDirectly(exp *trajectory.Experiment, granularities []int, path, pathToMcl string) {
- fmt.Println("Clustering trajectories directly with MCL")
+ slog.Info("Clustering trajectories directly with MCL")
// convert trajectories to abc format for the mcl tool
- dirName := fmt.Sprintf("%s-clusters-directly/", exp.Name)
+ dirName := fmt.Sprintf("%s-clustered-trajectories/", exp.Name)
workingDir := filepath.Join(path, dirName) + string(filepath.Separator)
- fmt.Println("Working path becomes: ", workingDir)
derr := os.MkdirAll(workingDir, 0777)
if derr != nil {
panic(derr)
}
+ mclWorkingDir := filepath.Join(workingDir, "mclOuput/")
+ fmt.Println("MCL working dir: ", mclWorkingDir)
+ derr = os.MkdirAll(mclWorkingDir, 0777)
+ if derr != nil {
+ panic(derr)
+ }
+ slog.Info("Working path becomes: " + workingDir)
// change working dir cause mcl program dumps files into working dir
- os.Chdir(workingDir)
- abcFileName := fmt.Sprintf("%s%s.abc", workingDir, exp.Name)
+ os.Chdir(mclWorkingDir)
+ abcFileName := fmt.Sprintf("%s/%s.abc", mclWorkingDir, exp.Name)
convertTrajectoriesToAbcFormat(exp, abcFileName)
- tabFileName := fmt.Sprintf("%s%s.tab", workingDir, exp.Name)
- mciFileName := fmt.Sprintf("%s%s.mci", workingDir, exp.Name)
+ tabFileName := fmt.Sprintf("%s/%s.tab", mclWorkingDir, exp.Name)
+ mciFileName := fmt.Sprintf("%s/%s.mci", mclWorkingDir, exp.Name)
mcxloadCmd := fmt.Sprintf("%smcxload", pathToMcl)
cmd := exec.Command(mcxloadCmd, "-abc", abcFileName, "--stream-mirror", "-write-tab", tabFileName, "-o", mciFileName)
var out bytes.Buffer
@@ -126,7 +133,7 @@ func ClusterTrajectoriesDirectly(exp *trajectory.Experiment, granularities []int
if err != nil {
panic(err)
}
- fmt.Println("Output: ", out.String(), serr.String())
+ slog.Info("Output: " + out.String() + serr.String())
// run the clusterings with different granularities
for _, gran := range granularities {
mcl_cmd := fmt.Sprintf("%smcl", pathToMcl)
@@ -135,7 +142,7 @@ func ClusterTrajectoriesDirectly(exp *trajectory.Experiment, granularities []int
var serr2 bytes.Buffer
cmd.Stdout = &out2
cmd.Stderr = &serr2
- fmt.Println("Output: ", out2.String(), serr2.String())
+ slog.Info("Output: " + out2.String() + serr2.String())
err := cmd.Run()
if err != nil {
panic(err)
@@ -143,30 +150,41 @@ func ClusterTrajectoriesDirectly(exp *trajectory.Experiment, granularities []int
}
// convert the clusterings to readable format
clusterFileName := fmt.Sprintf("out.%s.mci", exp.Name)
- outFileName := fmt.Sprintf("dump.%s.mci", exp.Name)
+ outFileName := fmt.Sprintf("%s.mci", exp.Name)
mcxdumpCmd := fmt.Sprintf("%smcxdump", pathToMcl)
for _, gran := range granularities {
cmd := exec.Command(mcxdumpCmd, "-icl", fmt.Sprintf("%s.I%d", clusterFileName, gran), "-tabr", tabFileName, "-o", fmt.Sprintf("%s.I%d", outFileName, gran))
- fmt.Println(mcxdumpCmd, "-icl", fmt.Sprintf("%s.I%d", clusterFileName, gran), "-tabr", tabFileName, "-o", fmt.Sprintf("%s.I%d", outFileName, gran))
+ slog.Info(fmt.Sprint(mcxdumpCmd, "-icl", fmt.Sprintf("%s.I%d", clusterFileName, gran), "-tabr", tabFileName, "-o", fmt.Sprintf("%s.I%d", outFileName, gran)))
var out1 bytes.Buffer
var serr1 bytes.Buffer
cmd.Stdout = &out1
cmd.Stderr = &serr1
err := cmd.Run()
- fmt.Println("Output: ", out1.String(), serr1.String())
+ slog.Info("Output: " + out1.String() + serr1.String())
if err != nil {
panic(err)
}
}
// convert the clusterings generated by mcl tool to gml format
for _, gran := range granularities {
- dumpFileName := fmt.Sprintf("%s.I%d", outFileName, gran)
- convertToDirectTrajectoryClusterGraphs(exp, dumpFileName, fmt.Sprintf("%s.trajectories.gml", dumpFileName))
- convertToDirectTrajectoryClusterGraphsRR(exp, dumpFileName, fmt.Sprintf("%s.trajectories.RR.gml", dumpFileName))
- trajectory.PrintClusteredTrajectoriesToFile(exp, fmt.Sprintf("%s.clustered.trajectories.tab", dumpFileName))
- trajectory.PrintClustersToCSVFiles(exp, fmt.Sprintf("%s.clustered.patients.csv", dumpFileName),
- fmt.Sprintf("%s.clustered.clusters.csv", dumpFileName))
+ granDirName := fmt.Sprintf("%s-clusters-I%d/", exp.Name, gran)
+ outPath := filepath.Join(workingDir, granDirName) + string(filepath.Separator)
+ derr := os.MkdirAll(outPath, 0777)
+ if derr != nil {
+ panic(derr)
+ }
+ inputFileName := fmt.Sprintf("%s.I%d", outFileName, gran)
+ convertToDirectTrajectoryClusterGraphs(exp, inputFileName, fmt.Sprintf("%s%s.trajectories.gml", outPath, inputFileName))
+ convertToDirectTrajectoryClusterGraphsDot(exp, inputFileName, fmt.Sprintf("%s%s.trajectories.dot", outPath, inputFileName))
+ convertToDirectTrajectoryClusterGraphsRR(exp, inputFileName, fmt.Sprintf("%s%s.trajectories.RR.gml", outPath, inputFileName))
+ convertToDirectTrajectoryClusterGraphsRRDot(exp, inputFileName, fmt.Sprintf("%s%s.trajectories.RR.dot", outPath, inputFileName))
+ trajectory.PrintClusteredTrajectoriesToFile(exp, fmt.Sprintf("%s%s.clustered.trajectories.tab", outPath, inputFileName))
+ trajectory.PrintClustersToCSVFiles(exp, fmt.Sprintf("%s%s.clustered.patients.csv", outPath, inputFileName),
+ fmt.Sprintf("%s%s.clustered.clusters.csv", outPath, inputFileName))
}
+ mout := fmt.Sprintf("%s.ext.mod.scores", exp.Name)
+ moutpath := filepath.Join(workingDir, mout)
+ computeAndPlotExtendedModularityScores(exp, granularities, moutpath)
}
// collectTrajectoriesFromClusterData looks up trajectories associated with a given list of trajectory ids and assigns
@@ -268,8 +286,10 @@ func convertToDirectTrajectoryClusterGraphs(exp *trajectory.Experiment, input, o
}
fmt.Fprintf(ofile, "]\n")
}
- fmt.Println("For ", output)
- fmt.Println("Collected ", nofClusters, " clusters")
+ slog.Info("For "+output,
+ slog.Int("Clusters", nofClusters),
+ slog.Int("Trajectories", len(exp.Trajectories)),
+ )
}
// percentMalesFemales computes for a given list of patients the percentage of males and females wrt to the total number
@@ -327,7 +347,7 @@ func transitionInformation(exp *trajectory.Experiment, t *trajectory.Trajectory,
// convertToDirectTrajectoryClusterGraphsRR converts MCL cluster output - a file with for each cluster id a list of
// trajectory ids - to a GML output file that plots the trajectories as graphs. Each cluster is plotted as a separate
// subgraph, with diagnosis codes used as nodes and trajectory transitions used as edges. The edges are annotated with
-// the relatitive risk score (RR) associated with the diagnosis pair that the edge represents.
+// the relative risk score (RR) associated with the diagnosis pair that the edge represents.
func convertToDirectTrajectoryClusterGraphsRR(exp *trajectory.Experiment, input, output string) {
file, err := os.Open(input)
if err != nil {
@@ -411,6 +431,463 @@ func convertToDirectTrajectoryClusterGraphsRR(exp *trajectory.Experiment, input,
}
fmt.Fprintf(ofile, "]\n")
}
- fmt.Println("For ", output)
- fmt.Println("Collected ", nofClusters, " clusters")
+ slog.Info("For "+output,
+ slog.Int("Clusters", nofClusters),
+ slog.Int("Trajectories", len(exp.Trajectories)),
+ )
+}
+
+// convertToDirectTrajectoryClusterGraphsRRDot converts MCL cluster output - a file with for each cluster id a list of
+// trajectory ids - to a DOT output file that plots the trajectories as graphs using GraphViz. Each cluster is plotted
+// as a separate subgraph, with diagnosis codes used as nodes and trajectory transitions used as edges. The edges are
+// annotated with the relative risk score (RR) associated with the diagnosis pair that the edge represents.
+func convertToDirectTrajectoryClusterGraphsRRDot(exp *trajectory.Experiment, input, output string) {
+ file, err := os.Open(input)
+ if err != nil {
+ panic(err)
+ }
+ ofile, oerr := os.Create(output)
+ if oerr != nil {
+ panic(oerr)
+ }
+ defer func() {
+ if err := file.Close(); err != nil {
+ panic(err)
+ }
+ if oerr := ofile.Close(); oerr != nil {
+ panic(oerr)
+ }
+ }()
+ // trajectories to assign to clusters
+ nofClusters := 0
+
+ // parse file
+ reader := csv.NewReader(file)
+ reader.Comma = '\t'
+ reader.FieldsPerRecord = -1
+ reader.LazyQuotes = true
+
+ // print dot directional-graph header, and some formatting
+ fmt.Fprintf(ofile, "digraph {\n")
+ fmt.Fprintf(ofile, " node [fontsize=24 fillcolor=lightskyblue style=filled color=navy]")
+ fmt.Fprintf(ofile, " edge [fontcolor=red fontsize=30]")
+
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ // collect codes in the cluster
+ var codes []int
+ for _, rcode := range record {
+ code, err := strconv.Atoi(rcode)
+ if err != nil {
+ panic(err)
+ }
+ codes = append(codes, code)
+ }
+ // print the trajectories in the cluster
+ collected := collectTrajectoriesFromClusterData(exp, codes, nofClusters)
+ nofClusters++
+ // print this cluster
+ // print header
+ fmt.Fprintf(ofile, fmt.Sprintf(" subgraph cluster_%d {\n", nofClusters-1))
+
+ nodePrinted := map[int]bool{}
+ // print nodes
+ for _, t := range collected {
+ for _, node := range t.Diagnoses {
+ if _, ok := nodePrinted[node]; !ok {
+ fmt.Fprintf(ofile, fmt.Sprintf(" c%d_%d [label=\"%s\"]\n", nofClusters-1, node, utils.WrapText(exp.NameMap[node], 25)))
+ nodePrinted[node] = true
+ }
+ }
+ }
+ // print edges
+ edgePrinted := make([][]bool, exp.NofDiagnosisCodes)
+ for i, _ := range edgePrinted {
+ edgePrinted[i] = make([]bool, exp.NofDiagnosisCodes)
+ }
+ for _, t := range collected {
+ d1 := t.Diagnoses[0]
+ tctr := 0
+ for i := 1; i < len(t.Diagnoses); i++ {
+ d2 := t.Diagnoses[i]
+ if !edgePrinted[d1][d2] {
+ edgePrinted[d1][d2] = true
+ RR := strconv.FormatFloat(exp.DxDRR[d1][d2], 'f', 2, 64)
+ label := RR + " (" + strconv.FormatFloat(exp.DxDRREmpProb[d1][d2], 'f', 2, 64) + ")"
+ fmt.Fprintf(ofile,
+ fmt.Sprintf(" c%d_%d -> c%d_%d [label=\"%s\" penwidth=%s weight=%s]\n", nofClusters-1, d1, nofClusters-1, d2, label, RR, RR))
+ }
+ d1 = d2
+ tctr++
+ }
+ }
+ fmt.Fprintf(ofile, " }\n")
+ }
+ // close the DOT digraph
+ fmt.Fprintf(ofile, "}\n")
+
+ slog.Info("For "+output,
+ slog.Int("Clusters", nofClusters),
+ slog.Int("Trajectories", len(exp.Trajectories)),
+ )
+}
+
+// convertToDirectTrajectoryClusterGraphsRRDot converts MCL cluster output - a file with for each cluster id a list of
+// trajectory ids - to a DOT output file that plots the trajectories as graphs using GraphViz. Each cluster is plotted
+// as a separate subgraph, with diagnosis codes used as nodes and trajectory transitions used as edges. The edges are
+// annotated with the relative risk score (RR) associated with the diagnosis pair that the edge represents.
+func convertToDirectTrajectoryClusterGraphsDot(exp *trajectory.Experiment, input, output string) {
+ file, err := os.Open(input)
+ if err != nil {
+ panic(err)
+ }
+ ofile, oerr := os.Create(output)
+ if oerr != nil {
+ panic(oerr)
+ }
+ defer func() {
+ if err := file.Close(); err != nil {
+ panic(err)
+ }
+ if oerr := ofile.Close(); oerr != nil {
+ panic(oerr)
+ }
+ }()
+ // trajectories to assign to clusters
+ nofClusters := 0
+
+ // parse file
+ reader := csv.NewReader(file)
+ reader.Comma = '\t'
+ reader.FieldsPerRecord = -1
+ reader.LazyQuotes = true
+
+ // print dot directional-graph header, and some formatting
+ fmt.Fprintf(ofile, "digraph {\n")
+ fmt.Fprintf(ofile, " node [fontsize=24 fillcolor=lightskyblue style=filled color=navy]")
+ fmt.Fprintf(ofile, " edge [fontcolor=red fontsize=30]")
+
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ // collect codes in the cluster
+ var codes []int
+ for _, rcode := range record {
+ code, err := strconv.Atoi(rcode)
+ if err != nil {
+ panic(err)
+ }
+ codes = append(codes, code)
+ }
+ // print the trajectories in the cluster
+ collected := collectTrajectoriesFromClusterData(exp, codes, nofClusters)
+ nofClusters++
+ // print this cluster
+ // print header
+ fmt.Fprintf(ofile, fmt.Sprintf(" subgraph cluster_%d {\n", nofClusters-1))
+
+ nodePrinted := map[int]bool{}
+ // print nodes
+ for _, t := range collected {
+ for _, node := range t.Diagnoses {
+ if _, ok := nodePrinted[node]; !ok {
+ fmt.Fprintf(ofile, fmt.Sprintf(" c%d_%d [label=\"%s\"]\n", nofClusters-1, node, utils.WrapText(exp.NameMap[node], 25)))
+ nodePrinted[node] = true
+ }
+ }
+ }
+ // print edges
+ edgePrinted := make([][]bool, exp.NofDiagnosisCodes)
+ for i, _ := range edgePrinted {
+ edgePrinted[i] = make([]bool, exp.NofDiagnosisCodes)
+ }
+ for _, t := range collected {
+ d1 := t.Diagnoses[0]
+ tctr := 0
+ for i := 1; i < len(t.Diagnoses); i++ {
+ d2 := t.Diagnoses[i]
+ if !edgePrinted[d1][d2] {
+ edgePrinted[d1][d2] = true
+ v := float64(t.PatientNumbers[i-1]) / 10.0
+ n := strconv.FormatInt(int64(t.PatientNumbers[i-1]), 10)
+ ns := strconv.FormatFloat(v, 'f', 2, 64)
+ label := n + " (" + strconv.FormatFloat(exp.DxDRREmpProb[d1][d2], 'f', 2, 64) + ")"
+ fmt.Fprintf(ofile,
+ fmt.Sprintf(" c%d_%d -> c%d_%d [label=\"%s\" penwidth=%s weight=%s]\n", nofClusters-1, d1, nofClusters-1, d2, label, ns, ns))
+ }
+ d1 = d2
+ tctr++
+ }
+ }
+ fmt.Fprintf(ofile, " }\n")
+ }
+ // close the DOT digraph
+ fmt.Fprintf(ofile, "}\n")
+
+ slog.Info("For "+output,
+ slog.Int("Clusters", nofClusters),
+ slog.Int("Trajectories", len(exp.Trajectories)),
+ )
+}
+
+// clusterGraph is a graph representation of a list of trajectory clusters.
+type clusterGraph struct {
+ nodes []int //list of nodes in the graph. A node is a diagnosis code that occurs in one or more trajectories
+ edges map[int][]int //adjacency list of edges
+ clusterMembership map[int][]int //per node/diagnosis code the clusters it belongs to
+ weight int
+ degrees map[int]int
+ similarities map[int]map[int]float64
+}
+
+// createClusterGraph creates a new clusterGraph object
+func createClusterGraph() clusterGraph {
+ //clusterMembership and graph can only be initialised when the nodes
+ return clusterGraph{nodes: []int{}, edges: map[int][]int{},
+ clusterMembership: map[int][]int{},
+ degrees: map[int]int{},
+ similarities: map[int]map[int]float64{}}
+}
+
+// addClusterMembership registers that a diagnosis code occurs in a specific cluster.
+func (graph *clusterGraph) addClusterMembership(did int, cid int) {
+ entries, ok := graph.clusterMembership[did]
+ if !ok {
+ entries = []int{cid}
+ graph.clusterMembership[did] = entries
+ return
+ }
+ if !(utils.MemberInt(cid, entries)) { //add membership only once
+ graph.clusterMembership[did] = append(entries, cid)
+ }
+}
+
+// addEdge adds a diagnosis pair to the graph as a unique edge.
+func (graph *clusterGraph) addEdge(d1 int, d2 int) {
+ entries, ok := graph.edges[d1]
+ if !ok {
+ entries = []int{d2}
+ graph.edges[d1] = entries
+ return
+ }
+ if !(utils.MemberInt(d2, entries)) { //add edge only once
+ graph.edges[d1] = append(entries, d2)
+ }
+}
+
+// addNode adds a diagnosis code to the graph as a unique node.
+func (graph *clusterGraph) addNode(d int) {
+ ok := utils.MemberInt(d, graph.nodes) //add node only once
+ if !ok {
+ graph.nodes = append(graph.nodes, d)
+ graph.degrees[d] = 0
+ }
+}
+
+// addTrajectories adds all nodes and edges from a list of trajectories to the graph
+func (graph *clusterGraph) addTrajectories(trajectories []*trajectory.Trajectory, cid int) {
+ for _, t := range trajectories {
+ d1 := t.Diagnoses[0]
+ graph.addNode(d1)
+ graph.addClusterMembership(d1, cid)
+ for _, d2 := range t.Diagnoses[1:] {
+ graph.addNode(d2)
+ graph.addClusterMembership(d2, cid)
+ graph.addEdge(d1, d2)
+ d1 = d2
+ }
+ }
+}
+
+// calculateWeightsAndDegrees calculates:
+// - the graph weight = the total number of edges
+// - the node degrees = the number of edges per node
+func (graph *clusterGraph) calculateWeightsAndDegrees() {
+ //calculate weight
+ for d, ds := range graph.edges {
+ graph.weight += len(ds)
+ _, ok := graph.degrees[d]
+ if !ok {
+ graph.degrees[d] = len(ds)
+ } else {
+ graph.degrees[d] += len(ds)
+ }
+ }
+}
+
+func (graph *clusterGraph) clustersShared(d1, d2 int) int {
+ c1s := graph.clusterMembership[d1]
+ c2s := graph.clusterMembership[d2]
+ ctr := 0
+ for _, c1 := range c1s {
+ if utils.MemberInt(c1, c2s) {
+ ctr++
+ }
+ }
+ return ctr
+}
+
+func (graph *clusterGraph) nClusters(d int) int {
+ v, ok := graph.clusterMembership[d]
+ if !ok {
+ panic("Disease not part of cluster. Should not happen.")
+ }
+ return len(v)
+}
+
+func (graph *clusterGraph) calculateEdgeSimilarities() {
+ for d1, ds := range graph.edges {
+ if _, ok := graph.similarities[1]; !ok {
+ graph.similarities[d1] = map[int]float64{} //init similarity scores
+ }
+ for _, d2 := range ds {
+ sharedN := graph.clustersShared(d1, d2)
+ d1N := graph.nClusters(d1)
+ d2N := graph.nClusters(d2)
+ s := float64(sharedN) / float64(d1N*d2N)
+ graph.similarities[d1][d2] = s
+ }
+ }
+}
+
+func (graph *clusterGraph) print() {
+ fmt.Println("Graph: ")
+ fmt.Println("Nodes: ")
+ fmt.Println(graph.nodes)
+ fmt.Println("Degrees: ")
+ fmt.Println(graph.degrees)
+ fmt.Println("Edges: ")
+ fmt.Println(graph.edges)
+ fmt.Println("Weight: ")
+ fmt.Println(graph.weight)
+ fmt.Println("Similarties: ")
+ fmt.Println(graph.similarities)
+ fmt.Println("Cluster membership: ")
+ fmt.Println(graph.clusterMembership)
+}
+
+func (graph *clusterGraph) ExtendedModularityMetric() float64 {
+ graph.calculateWeightsAndDegrees()
+ graph.calculateEdgeSimilarities()
+ m2 := float64(2 * graph.weight)
+ contributions := 0.0
+ for d1, ds := range graph.edges {
+ for _, d2 := range ds {
+ w := 1.0
+ degree1 := graph.degrees[d1]
+ degree2 := graph.degrees[d2]
+ e := float64(degree1*degree2) / m2
+ contributions += (w - e) * graph.similarities[d1][d2]
+ }
+ }
+ return contributions / m2
+}
+
+// parseMCLGraph takes as input a file with a mcl clustering. The mcl file contains per cluster ID a list of trajectory
+// IDs that belong to that cluster. Create an clusterGraph object with the following information that can be used to
+// calculate an extended modularity metric of a graph:
+// the list of diagnosis codes in the mcl graph
+// the list of edges in the mcl graph
+// per diagnosis code the list of clusters it occurs in
+func ParseMCLGraph(exp *trajectory.Experiment, graph string) clusterGraph {
+
+ file, err := os.Open(graph)
+ if err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := file.Close(); err != nil {
+ panic(err)
+ }
+ }()
+
+ // trajectories to assign to clusters
+ nofClusters := 0
+
+ // parse file
+ reader := csv.NewReader(file)
+ reader.Comma = '\t'
+ reader.FieldsPerRecord = -1
+ reader.LazyQuotes = true
+
+ mclGraph := createClusterGraph()
+
+ for {
+ record, err := reader.Read()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ panic(err)
+ }
+ // collect the trajectory codes in the cluster
+ var tids []int
+ for _, c := range record {
+ tid, err := strconv.Atoi(c)
+ if err != nil {
+ panic(err)
+ }
+ tids = append(tids, tid)
+ }
+ // print the trajectories in the cluster
+ trajectories := collectTrajectoriesFromClusterData(exp, tids, nofClusters)
+ //add the trajectory info to the graph
+ mclGraph.addTrajectories(trajectories, nofClusters)
+ nofClusters++
+ }
+ return mclGraph
+}
+
+// computeAndPlotExtendedModularityMetrics takes as input:
+// * exp: the experiment
+// * grans: the list of granularities chosen for different clusterings
+// * output: the file to which to write the extended modularity metrics for the different clusterings
+// The function creates an output file with 3 entries:
+// 1) The cluster granularity, 2) Metric, 3) a 'x' if it is the best score
+func computeAndPlotExtendedModularityScores(exp *trajectory.Experiment, grans []int, output string) {
+ //create file for outputting the extended modularity scores
+ var metricOutput *os.File
+ var err error
+ if metricOutput, err = os.Create(output); err != nil {
+ panic(err)
+ }
+ defer func() {
+ if err := metricOutput.Close(); err != nil {
+ panic(err)
+ }
+ }()
+ //for each clustering, calculated the extended modularity score
+ scores := make([]float64, len(grans))
+ best := 0
+ for i, gran := range grans {
+ clusterFile := fmt.Sprintf("%s.mci.I%d", exp.Name, gran)
+ graph := ParseMCLGraph(exp, clusterFile)
+ m := graph.ExtendedModularityMetric()
+ scores[i] = m
+ if m > scores[best] {
+ best = i
+ }
+ }
+ //log all extended modularity scores + mark the best one with "x"
+ fmt.Fprintf(metricOutput, "Granularity \t Extended Modularity Score \t Best \n")
+ for i, score := range scores {
+ if i == best {
+ fmt.Fprintf(metricOutput,
+ fmt.Sprintf("%d \t %s \t x \n", grans[i], strconv.FormatFloat(score, 'f', -1, 64)))
+ } else {
+ fmt.Fprintf(metricOutput,
+ fmt.Sprintf(" %d \t %s \n", grans[i], strconv.FormatFloat(score, 'f', -1, 64)))
+ }
+ }
}
diff --git a/cluster/clustering.go b/cluster/clustering.go
index d26efc8..6c63f30 100644
--- a/cluster/clustering.go
+++ b/cluster/clustering.go
@@ -24,6 +24,7 @@ import (
"fmt"
"io"
"log"
+ "log/slog"
"os"
"os/exec"
"path/filepath"
@@ -112,11 +113,11 @@ func convertTrajectoryPairsToAbcFormat(exp *trajectory.Experiment, name string)
}
func ClusterTrajectories(exp *trajectory.Experiment, granularities []int, path, pathToMcl string) {
- fmt.Println("Clustering trajectories with MCL")
+ slog.Info("Clustering trajectories with MCL")
// convert trajectories to abc format for the mcl tool
dirName := fmt.Sprintf("%s-clusters/", exp.Name)
workingDir := filepath.Join(path, dirName) + string(filepath.Separator)
- fmt.Println("Working path becomes: ", workingDir)
+ slog.Info("Working path becomes: " + workingDir)
derr := os.MkdirAll(workingDir, 0777)
if derr != nil {
panic(derr)
@@ -137,7 +138,7 @@ func ClusterTrajectories(exp *trajectory.Experiment, granularities []int, path,
if err != nil {
panic(err)
}
- fmt.Println("Output: ", out.String(), serr.String())
+ slog.Info("Output: " + out.String() + serr.String())
// run the clusterings with different granularities
for _, gran := range granularities {
mcl_cmd := fmt.Sprintf("%smcl", pathToMcl)
@@ -146,7 +147,7 @@ func ClusterTrajectories(exp *trajectory.Experiment, granularities []int, path,
var serr2 bytes.Buffer
cmd.Stdout = &out2
cmd.Stderr = &serr2
- fmt.Println("Output: ", out2.String(), serr2.String())
+ slog.Info("Output: " + out2.String() + serr2.String())
err := cmd.Run()
if err != nil {
panic(err)
@@ -157,14 +158,15 @@ func ClusterTrajectories(exp *trajectory.Experiment, granularities []int, path,
outFileName := fmt.Sprintf("dump.%s.mci", exp.Name)
mcxdumpCmd := fmt.Sprintf("%smcxdump", pathToMcl)
for _, gran := range granularities {
+ fullcmd := fmt.Sprint(mcxdumpCmd, "-icl", fmt.Sprintf("%s.I%d", clusterFileName, gran), "-tabr", tabFileName, "-o", fmt.Sprintf("%s.I%d", outFileName, gran))
+ slog.Info(fullcmd)
cmd := exec.Command(mcxdumpCmd, "-icl", fmt.Sprintf("%s.I%d", clusterFileName, gran), "-tabr", tabFileName, "-o", fmt.Sprintf("%s.I%d", outFileName, gran))
- fmt.Println(mcxdumpCmd, "-icl", fmt.Sprintf("%s.I%d", clusterFileName, gran), "-tabr", tabFileName, "-o", fmt.Sprintf("%s.I%d", outFileName, gran))
var out1 bytes.Buffer
var serr1 bytes.Buffer
cmd.Stdout = &out1
cmd.Stderr = &serr1
err := cmd.Run()
- fmt.Println("Output: ", out1.String(), serr1.String())
+ slog.Info("Output: " + out1.String() + serr1.String())
if err != nil {
panic(err)
}
@@ -378,7 +380,10 @@ func convertToTrajectoryClusterGraphs(exp *trajectory.Experiment, input, output
}
fmt.Fprintf(ofile, "]\n")
}
- fmt.Println("For ", output)
- fmt.Println("Collected ", nofClusters, " clusters and ", len(trajectories), " not clustered trajectories.")
- fmt.Println("Clustered ", len(exp.Trajectories)-len(trajectories), " out of ", len(exp.Trajectories), " trajectories.")
+ slog.Info("For "+output,
+ slog.Int("Clusters", nofClusters),
+ slog.Int("Traj-Clustered", len(exp.Trajectories)-len(trajectories)),
+ slog.Int("Traj-Not-clustered", len(trajectories)),
+ slog.Int("Traj-Total", len(exp.Trajectories)),
+ )
}
diff --git a/cmd/main.go b/cmd/main.go
new file mode 100644
index 0000000..5e0340a
--- /dev/null
+++ b/cmd/main.go
@@ -0,0 +1,465 @@
+// PTRA: Patient Trajectory Analysis Library
+// Copyright (c) 2022 imec vzw.
+
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as
+// published by the Free Software Foundation, either version 3 of the
+// License, or (at your option) any later version, and Additional Terms
+// (see below).
+
+// This program is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+// Affero General Public License for more details.
+
+// You should have received a copy of the GNU Affero General Public
+// License and Additional Terms along with this program. If not, see
+// .
+
+package main
+
+import (
+ "bytes"
+ "io"
+ "log"
+ "log/slog"
+ "ptra/app"
+ "ptra/cluster"
+ "ptra/trajectory"
+ "ptra/utils"
+ "strconv"
+ "strings"
+
+ //"bytes"
+ "flag"
+ "fmt"
+ "path/filepath"
+
+ //"log"
+ "os"
+ "runtime"
+)
+
+/*
+Ptra is a tool for patient trajectory analysis.
+
+Usage:
+ ptra pfile ifile dfile path [flags]
+
+Example:
+ ptra ICD10 patient.csv icd10cm_tabular_2022.xml diagnosis.csv ./MIBC_tfiltered/ --nofAgeGroups 10 --lvl 2
+ --maxYears 5 --minYears 0.001 --minPatients 50 --maxTrajectoryLength 5 --minTrajectoryLength 3 --name MICB_tfiltered
+ --ICD9ToICD10File ICD_9_to_10.json --iter 400 --RR 1.0 --saveRR MIBC_tfiltered.csv --cluster
+ --mclPath /home/caherzee/tools/mcl/ --clusterGranularities 40,60,80,100
+
+The flags are:
+
+--nofAgeGroups nr
+ The number of age groups to consider when dividing the population into cohorts based on age. The tool automatically
+ detects the oldest and youngest patients from input. The number of age groups is used to divide the minumum birth
+ year and maximum birth year into age ranges. E.g. min birth year: 1900, max birth year: 2020, and nr of age groups:
+ 10, will create 10 cohorts where age ranges from [0,12],[12,22],...[108,120].
+--lvl nr
+ Sets the ICD10 level for the diagnosis input [0-6]. The tool maps all ICD10 codes to a medical meaningful term based on
+ this chosen level. ICD10 codes of lower levels may be combined into the same code of a higher level. E.g. A00.0
+ Cholera due to Vibrio cholerae 01, biovar cholerae and A00.1 Cholera due to Vibrio cholerae 01, biovar eltor are lvl
+ 3 codes and may be collapsed to A00 Cholera in lvl 2, or A00-A09 Intestinal infectious diseases in lvl 1, or A00-B99
+ Certain infectious and parasitic diseases in lvl 0.
+--minPatients nr
+ Sets the minimum required number of patients in a trajectory.
+--maxYears nr
+ Sets the maximum number of years between subsequent diagnoses to be considered for inclusion in a trajectory. E.g.
+ 0.5 for half a year.
+--minYears nr
+ Sets the minimum number of years between subsequent diagnoses to be considered for inclusion in a trajectory. E.g.
+ 0.5 for half a year.
+--maxTrajectoryLength nr
+ Sets the maximum length of trajectories to be included in the output. E.g. 5 for trajectories with maximum 5
+ diagnoses.
+--minTrajectoryLength nr
+ Sets the minimum length of trajectories to be included in the output. E.g. 3 for trajectories with minimum 3
+ diagnoses.
+--name string
+ Sets the name of the experiment. This name is used to generate names for output files.
+--cluster
+ If this flag is passed, the computed trajectories are clustered and the clusters are outputted to file.
+--clusterGranularities nr, nr, .., nr
+ The cluster granularities to try. This impacts how many clusters the algorithm tries to find. Default: 40,60,80,100
+--mclPath
+ Sets the path where the mcl binaries can be found.
+--iter nr
+ Sets the number of iterations to be used in the sampling experiments for calculating relative risk ratios. If iter
+ is 400, the calculated p-values are within 0.05 of the true p-values. For iter = 10000, the true p-values are within
+ 0.01 of the true p-values. The higher the number of iterations, the higher the runtime.
+--saveRR file
+ Save the RR matrix, a matrix that represents the RR calculated from the population for each possible combination of
+ ICD10 diagnosis pairs. This matrix can be loaded in other ptra runs to avoid recalculating the RR scores. This can
+ be useful if parameters want to be explored that do not impact the RR calculation itself. Only iter, maxYears and
+ minYears, and filters influence RR calculation. Variations of other parameters for constructing trajectories from RR
+ scores, such as maxTrajectoryLenght, minTrajectoryLength, minPatients, RR etc might be explored in other runs.
+--loadRR file
+ Load the RR matrix from file. Such a file must be created by a previous run of ptra with the --saveRR flag.
+--pfilters age+:nr | age-:nr | sex:male | sex:female | control:yes | control:no
+ A list of filters for selecting patients from which to derive trajectories. The form of a filter is tag:value. E.g.
+ sex:male says to select only male patients.
+--eoi icd10,icd10,...icd10
+ A list of icd10 codes for identifying the event of interest. The event of interest can be used for analyzing or
+ filtering trajectories
+--tfilters cat:[neoplasm | bc], code:icd10;icd10;...icd10
+ A list of filters for reducing the output of trajectories. The form to specify a filter is tag:value.
+ E.g. cat:neoplasm only outputs trajectories where there is at least one diagnosis related to cancer. cat:bc only
+ outputs trajectories where at least one diagnosis is related to bladder cancer. The code: tag can be used to list
+ a number of ICD10 codes of which at least one should occur in the trajectory.
+--loglvl
+ Specify the log level (error, warning, info, debug, all).
+*/
+
+const (
+ programVersion = 0.11
+ programName = "ptra"
+)
+
+func programMessage() string {
+ return fmt.Sprint(programName, " version ", programVersion, " compiled with ", runtime.Version())
+}
+
+const ptraHelp = "\nptra parameters:\n" +
+ "ptra patientInfoFile diagnosisInfoFile diagnosesFile outputPath \n" +
+ "[--nofAgeGroups nr]\n" +
+ "[--lvl nr]\n" +
+ "[--minPatients nr]\n" +
+ "[--maxYears nr]\n" +
+ "[--minYears nr]\n" +
+ "[--maxTrajectoryLength nr]\n" +
+ "[--minTrajectoryLength nr]\n" +
+ "[--name string]\n" +
+ "[--eoid icd10,icd10,...,icd10\n" +
+ "[--cluster]\n" +
+ "[--clusterGranularities nr,nr,...,nr]\n" +
+ "[--mclPath string]\n" +
+ "[--iter nr]\n" +
+ "[--saveRR file]\n" +
+ "[--loadRR file]\n" +
+ "[--pfilters age+:nr,age-:nr,sex:[male | female],control:[yes | no] ]\n" +
+ "[--tfilters cat:[neoplasms | bc],code:icd10;...;icd10]\n" +
+ "[--nrOfThreads nr]\n" +
+ "[--actFile file]\n" +
+ "[--loglvl error | warning | info | debug | all]\n"
+
+func parseFlags(flags flag.FlagSet, requiredArgs int, help string) {
+ if len(os.Args) < requiredArgs {
+ fmt.Fprintln(os.Stderr, "Incorrect number of parameters.")
+ fmt.Fprint(os.Stderr, help)
+ os.Exit(1)
+ }
+ flags.SetOutput(io.Discard)
+ if err := flags.Parse(os.Args[requiredArgs:]); err != nil {
+ x := 0
+ if err != flag.ErrHelp {
+ fmt.Fprint(os.Stderr, err)
+ }
+ fmt.Fprint(os.Stderr, help)
+ os.Exit(x)
+ }
+ if flags.NArg() > 0 {
+ fmt.Fprint(os.Stderr, "Cannot parse remaining parameters:", flags.Args())
+ fmt.Fprint(os.Stderr, help)
+ os.Exit(1)
+ }
+}
+
+func getFileName(s, help string) string {
+ switch s {
+ case "-h", "--h", "-help", "--help":
+ fmt.Fprint(os.Stderr, help)
+ os.Exit(1)
+ }
+ return s
+}
+
+func getPatientFilter(s string) trajectory.PatientFilter {
+ id := func(p *trajectory.Patient) bool { return true }
+ switch s {
+ case "id":
+ return id
+ case "EOI-":
+ return trajectory.EOIAfterFilter()
+ case "EOI+":
+ return trajectory.EOIBeforeFilter()
+ }
+ fs := strings.Split(s, ":")
+ if len(fs) > 1 {
+ switch fs[0] {
+ case "age+":
+ {
+ age, err := strconv.Atoi(fs[1])
+ if err != nil {
+ panic("Age passed to age+ not a well-formed integer: " + fs[1])
+ }
+ return trajectory.AgeAboveAggregator(age)
+ }
+ case "age-":
+ {
+ age, err := strconv.Atoi(fs[1])
+ if err != nil {
+ panic("Age passe to age- not a well-formed integer: " + fs[1])
+ }
+ return trajectory.AgeLessThanAggregator(age)
+ }
+ case "sex":
+ {
+ gender := fs[1]
+ if gender == "male" {
+ return trajectory.FemaleFilter()
+ }
+ if gender == "female" {
+ return trajectory.MaleFilter()
+ }
+ panic("Unknown gender: " + gender)
+ }
+ case "control":
+ {
+ target := fs[1]
+ if target == "yes" {
+ return trajectory.ControlFilter(true)
+ }
+ if target == "no" {
+ return trajectory.ControlFilter(false)
+ }
+ panic("Unknown control option: " + target)
+ }
+ default:
+ panic("Unknown patient filter: " + s)
+ }
+ } else {
+ switch s {
+ case "eoi-":
+ return trajectory.EOIAfterFilter()
+ case "eoi+":
+ return trajectory.EOIBeforeFilter()
+ case "id":
+ return id
+ default:
+ panic("Unknown patient filter: " + s)
+ }
+ }
+}
+
+func getPatientFilters(f string) []trajectory.PatientFilter {
+ fs := strings.Split(f, ",")
+ result := []trajectory.PatientFilter{}
+ for _, f := range fs {
+ result = append(result, getPatientFilter(f))
+ }
+ return result
+}
+
+func getTrajectoryFilter(s string, exp *trajectory.Experiment) trajectory.TrajectoryFilter {
+ id := func(t *trajectory.Trajectory) bool { return true }
+ ss := strings.Split(s, ":")
+ switch ss[0] {
+ case "cat":
+ switch ss[1] {
+ case "neoplasm":
+ return app.CancerTrajectoryFilter(exp)
+ case "bc":
+ return app.BladderCancerTrajectoryFilter(exp)
+ default:
+ panic("Unknown category for trajectory filter: cat:" + ss[1])
+ }
+ case "code":
+ codes := strings.Split(ss[1], ";")
+ return app.Icd10TrajectoryFilter(exp, codes)
+ case "id":
+ return id
+ default:
+ panic("Unknown trajectory filter: " + ss[0])
+ }
+}
+
+func getTrajectoryFilters(f string, exp *trajectory.Experiment) []trajectory.TrajectoryFilter {
+ fs := strings.Split(f, ",")
+ result := []trajectory.TrajectoryFilter{}
+ for _, f := range fs {
+ result = append(result, getTrajectoryFilter(f, exp))
+ }
+ return result
+}
+
+func main() {
+ var (
+ // required parameters
+ patientInfo string //The file with patient information (ID, gender," + birthyear, etc)
+ diagnosisInfo string //The file with diagnosis information (ID,descriptor, hierarchy, etc)
+ patientDiagnoses string //The file with patient diagnoses.
+ outputPath string //The path where output files are written.
+ // optional flags
+ nofAgeGroups int
+ lvl int
+ maxYears float64
+ minYears float64
+ minPatients int
+ maxTrajectoryLength int
+ minTrajectoryLength int
+ name string
+ clust bool
+ mclPath string
+ clusterGranularities string
+ iter int
+ rr float64
+ saveRR string
+ loadRR string
+ pfilters string
+ tfilters string
+ tumorInfo string
+ treatmentInfo string
+ nrOfThreads int
+ eoid string
+ actFile string
+ loglvl string
+ )
+ var flags flag.FlagSet
+ // options for the ptra command
+ flags.IntVar(&nofAgeGroups, "nofAgeGroups", 6, "The population data is divided in cohorts in"+
+ "terms of age groups to calculate relative risk ratios of diagnosis pairs. This parameters configures how"+
+ "many age groups to use")
+ flags.IntVar(&nrOfThreads, "nrOfThreads", 0, "The number of threads ptra uses.")
+ flags.IntVar(&lvl, "lvl", 3, "Diagnosis codes are organised in a hierarchy of diagnosis "+
+ "descriptors. The level says which descriptor in the hiearchy to use for trajectory building.")
+ flags.Float64Var(&maxYears, "maxYears", 5.0, "The maximum number of years between diagnosis "+
+ "A and B to consider the diagnosis pair A->B in a trajectory.")
+ flags.Float64Var(&minYears, "minYears", 0.5, "The minimum number of years between diagnisis "+
+ "A and B to consider the diagnosis pair A->B in a trajectory.")
+ flags.IntVar(&minPatients, "minPatients", 1000, "The minimum number of patients for the last "+
+ "diagnosis in a trajectory")
+ flags.IntVar(&maxTrajectoryLength, "maxTrajectoryLength", 5, "The maximum number of diagnoses"+
+ " in a trajectory")
+ flags.IntVar(&minTrajectoryLength, "minTrajectoryLength", 3, "The minimum number of "+
+ "diagnoses in a trajectory")
+ flags.StringVar(&name, "name", "exp1", "The name of the run. This is used to generate the "+
+ "names of the output files.")
+ flags.StringVar(&eoid, "eoid", "", "The icd10 codes of the events of interest to track")
+ flags.BoolVar(&clust, "cluster", false, "Cluster the trajectories using MCL and output "+
+ "the results")
+ flags.StringVar(&mclPath, "mclPath", "", "The path to the mcl binary.")
+ flags.StringVar(&clusterGranularities, "clusterGranularities", "40,60,80,100", "The "+
+ "granularities used for the mcl clustering step.") // recommended 14,20,40,60
+ flags.IntVar(&iter, "iter", 10000, "The minimum number of sampling iterations "+
+ "diagnosis in a trajectory")
+ flags.Float64Var(&rr, "RR", 1.0, "The minimum RR score for considering pairs.")
+ flags.StringVar(&saveRR, "saveRR", "", "Save the RR matrix to a file so it can be loaded for "+
+ "later runs")
+ flags.StringVar(&loadRR, "loadRR", "", "Load the RR matrix from a given file instead of "+
+ "calculating it from scratch.")
+ flags.StringVar(&pfilters, "pfilters", "id", "A list of pfilters to restrict analysis on specific "+
+ "patients.")
+ flags.StringVar(&tumorInfo, "tumorInfo", "", "A file with information about the tumor stages.")
+ flags.StringVar(&treatmentInfo, "treatmentInfo", "", "A file with information about patient cancer stages.")
+ flags.StringVar(&tfilters, "tfilters", "id", "A list of filters to restrict output of trajectories."+
+ "A list of options of the form \"code:icd;icd;icd\" or \"cat:[neoplasm | bc]\"")
+ flags.StringVar(&actFile, "actFile", "", "A files with ACT codes")
+ flags.StringVar(&loglvl, "loglvl", "info", "Specify the log level (error, warning, info, debug, all).")
+ // parse optional arguments
+ parseFlags(flags, 5, ptraHelp)
+ // configure logger level from CLI
+ var logLevel slog.LevelVar
+ var logerr = logLevel.UnmarshalText([]byte(loglvl))
+ if logerr != nil {
+ panic(logerr)
+ }
+ slog.SetLogLoggerLevel(logLevel.Level())
+ slog.Debug("LOGGER::",
+ slog.String("loglvl", loglvl),
+ slog.String("level", logLevel.String()),
+ )
+ // parse required arguments
+ patientInfo = getFileName(os.Args[1], ptraHelp)
+ diagnosisInfo = getFileName(os.Args[2], ptraHelp)
+ patientDiagnoses = getFileName(os.Args[3], ptraHelp)
+ outputPath, _ = filepath.Abs(getFileName(os.Args[4], ptraHelp))
+ outputPath = outputPath + string(filepath.Separator)
+ slog.Info("Output", slog.String("path", outputPath))
+ // create output directory
+ err := os.MkdirAll(filepath.Dir(outputPath), 0700)
+ if err != nil {
+ panic(err)
+ }
+ // build an output command line
+ var command bytes.Buffer
+ fmt.Fprint(&command, os.Args[0], " ", patientInfo, " ", diagnosisInfo, " ", patientDiagnoses,
+ " ", outputPath)
+ fmt.Fprint(&command, " --nofAgeGroups ", nofAgeGroups)
+ fmt.Fprint(&command, " --lvl ", lvl)
+ fmt.Fprint(&command, " --maxYears ", maxYears)
+ fmt.Fprint(&command, " --minYears ", minYears)
+ fmt.Fprint(&command, " --minPatients ", minPatients)
+ fmt.Fprint(&command, " --maxTrajectoryLength ", maxTrajectoryLength)
+ fmt.Fprint(&command, " --minTrajectoryLength ", minTrajectoryLength)
+ fmt.Fprint(&command, " --name ", name)
+ fmt.Fprint(&command, " --iter ", iter)
+ fmt.Fprint(&command, " --RR ", rr)
+ if saveRR != "" {
+ fmt.Fprint(&command, " --saveRR ", saveRR)
+ }
+ if loadRR != "" {
+ fmt.Fprint(&command, " --loadRR ", loadRR)
+ }
+ if clust {
+ fmt.Fprint(&command, " --cluster")
+ fmt.Fprint(&command, " --mclPath ", mclPath)
+ fmt.Fprint(&command, " --clusterGranularities ", clusterGranularities)
+ }
+ fmt.Fprint(&command, " --pfilters ", pfilters)
+ fmt.Fprint(&command, " --tfilters ", tfilters)
+ if nrOfThreads > 0 {
+ runtime.GOMAXPROCS(nrOfThreads)
+ fmt.Fprint(&command, " --nrOfThreads ", nrOfThreads)
+ }
+ if eoid != "" {
+ fmt.Fprint(&command, " --eoid ", eoid)
+ }
+ if actFile != "" {
+ fmt.Fprint(&command, " --actFile ", actFile)
+ }
+ fmt.Fprint(&command, " --loglvl ", loglvl)
+ // start execution
+ log.Println(programMessage())
+ log.Println("Executing command:\n", command.String())
+ //1. Parse inputs into experiment
+ exp, patients := app.ParseData("exp1", patientInfo, patientDiagnoses, diagnosisInfo, actFile,
+ nofAgeGroups, lvl, getPatientFilters(pfilters), strings.Split(eoid, ","))
+ //2. Initialise relative risk ratios or load them from file from a previous run
+ if loadRR != "" {
+ trajectory.LoadRRMatrix(exp, loadRR)
+ trajectory.LoadDxDPatients(exp, patients, fmt.Sprintf("%s.patients.csv", loadRR))
+ } else {
+ trajectory.InitializeExperimentRelativeRiskRatios(exp, minYears, maxYears, iter)
+ }
+ if saveRR != "" { //save RR matrix to file + DPatients
+ trajectory.SaveRRMatrix(exp, saveRR)
+ trajectory.SaveDxDPatients(exp, fmt.Sprintf("%s.patients.csv", saveRR))
+ }
+ // assist the gc and nil some exp data that is no longer needed after initializing RR
+ exp.Cohorts = nil
+ exp.DPatients = nil
+ //3. Build the trajectories
+ trajectory.BuildTrajectories(exp, minPatients, maxTrajectoryLength, minTrajectoryLength, minYears, maxYears, rr,
+ getTrajectoryFilters(tfilters, exp))
+ //4. Plot trajectories to file
+ trajectory.PrintTrajectoriesToFile(exp, outputPath)
+ slog.Info("Collected trajectories")
+ for i := 0; i < utils.MinInt(len(exp.Trajectories), 100); i++ {
+ trajectory.PrintTrajectory(exp.Trajectories[i], exp)
+ }
+ //5. Perform clustering
+ if clust {
+ var clusterGranularityList []int
+ for _, g := range strings.Split(clusterGranularities, ",") {
+ gi, _ := strconv.ParseInt(g, 10, 0)
+ clusterGranularityList = append(clusterGranularityList, int(gi))
+ }
+ slog.Info("MCL Clustering")
+ //ClusterTrajectories(exp, clusterGranularityList, outputPath, mclPath)
+ cluster.ClusterTrajectoriesDirectly(exp, clusterGranularityList, outputPath, mclPath)
+ }
+}
diff --git a/main.go b/cmd/main_trinetx.go
similarity index 91%
rename from main.go
rename to cmd/main_trinetx.go
index 3e5a6c1..042cb86 100644
--- a/main.go
+++ b/cmd/main_trinetx.go
@@ -21,6 +21,7 @@ package main
import (
"bytes"
"log"
+ "log/slog"
"ptra/app"
"ptra/cluster"
"ptra/trajectory"
@@ -115,15 +116,15 @@ The flags are:
*/
const (
- programVersion = 0.1
- programName = "ptra"
+ triNetXProgramVersion = 0.1
+ triNetXProgramName = "ptra - trinetx"
)
-func programMessage() string {
- return fmt.Sprint(programName, " version ", programVersion, " compiled with ", runtime.Version())
+func triNetXProgramMessage() string {
+ return fmt.Sprint(triNetXProgramName, " version ", triNetXProgramVersion, " compiled with ", runtime.Version())
}
-const ptraHelp = "\nptra parameters:\n" +
+const triNetXPtraHelp = "\nptra parameters:\n" +
"ptra patientInfoFile diagnosisInfoFile diagnosesFile outputPath \n" +
"[--nofAgeGroups nr]\n" +
"[--lvl nr]\n" +
@@ -146,7 +147,7 @@ const ptraHelp = "\nptra parameters:\n" +
"[--treatmentInfo file]\n" +
"[--nrOfThreads nr]\n"
-func parseFlags(flags flag.FlagSet, requiredArgs int, help string) {
+func triNetXParseFlags(flags flag.FlagSet, requiredArgs int, help string) {
if len(os.Args) < requiredArgs {
fmt.Fprintln(os.Stderr, "Incorrect number of parameters.")
fmt.Fprint(os.Stderr, help)
@@ -168,7 +169,7 @@ func parseFlags(flags flag.FlagSet, requiredArgs int, help string) {
}
}
-func getFileName(s, help string) string {
+func triNetXGetFileName(s, help string) string {
switch s {
case "-h", "--h", "-help", "--help":
fmt.Fprint(os.Stderr, help)
@@ -177,7 +178,7 @@ func getFileName(s, help string) string {
return s
}
-func getPatientFilter(s string, tinfo map[string][]*app.TumorInfo) trajectory.PatientFilter {
+func triNetXPatientFilter(s string, tinfo map[string][]*app.TumorInfo) trajectory.PatientFilter {
id := func(p *trajectory.Patient) bool { return true }
switch s {
case "id":
@@ -229,16 +230,16 @@ func getPatientFilter(s string, tinfo map[string][]*app.TumorInfo) trajectory.Pa
}
}
-func getPatientFilters(f string, tinfo map[string][]*app.TumorInfo) []trajectory.PatientFilter {
+func triNetXGetPatientFilters(f string, tinfo map[string][]*app.TumorInfo) []trajectory.PatientFilter {
fs := strings.Split(f, ",")
result := []trajectory.PatientFilter{}
for _, f := range fs {
- result = append(result, getPatientFilter(f, tinfo))
+ result = append(result, triNetXPatientFilter(f, tinfo))
}
return result
}
-func getTrajectoryFilter(s string, exp *trajectory.Experiment) trajectory.TrajectoryFilter {
+func triNetXGetTrajectoryFilter(s string, exp *trajectory.Experiment) trajectory.TrajectoryFilter {
id := func(t *trajectory.Trajectory) bool { return true }
switch s {
case "neoplasm":
@@ -250,11 +251,11 @@ func getTrajectoryFilter(s string, exp *trajectory.Experiment) trajectory.Trajec
}
}
-func getTrajectoryFilters(f string, exp *trajectory.Experiment) []trajectory.TrajectoryFilter {
+func triNetXGetTrajectoryFilters(f string, exp *trajectory.Experiment) []trajectory.TrajectoryFilter {
fs := strings.Split(f, ",")
result := []trajectory.TrajectoryFilter{}
for _, f := range fs {
- result = append(result, getTrajectoryFilter(f, exp))
+ result = append(result, triNetXGetTrajectoryFilter(f, exp))
}
return result
}
@@ -329,14 +330,14 @@ func main() {
flags.StringVar(&treatmentInfo, "treatmentInfo", "", "A file with information about patient cancer stages.")
flags.StringVar(&tfilters, "tfilters", "id", "A list of pfilters to restrict output of trajectories")
// parse optional arguments
- parseFlags(flags, 5, ptraHelp)
+ triNetXParseFlags(flags, 5, triNetXPtraHelp)
// parse required arguments
- patientInfo = getFileName(os.Args[1], ptraHelp)
- diagnosisInfo = getFileName(os.Args[2], ptraHelp)
- patientDiagnoses = getFileName(os.Args[3], ptraHelp)
- outputPath, _ = filepath.Abs(getFileName(os.Args[4], ptraHelp))
+ patientInfo = triNetXGetFileName(os.Args[1], triNetXPtraHelp)
+ diagnosisInfo = triNetXGetFileName(os.Args[2], triNetXPtraHelp)
+ patientDiagnoses = triNetXGetFileName(os.Args[3], triNetXPtraHelp)
+ outputPath, _ = filepath.Abs(triNetXGetFileName(os.Args[4], triNetXPtraHelp))
outputPath = outputPath + string(filepath.Separator)
- fmt.Println("Output path: ", outputPath)
+ slog.Info("Output", slog.String("path", outputPath))
// create output directory
err := os.MkdirAll(filepath.Dir(outputPath), 0700)
if err != nil {
@@ -377,7 +378,7 @@ func main() {
fmt.Fprint(&command, " --nrOfThreads ", nrOfThreads)
}
// start execution
- log.Println(programMessage())
+ log.Println(triNetXProgramMessage())
log.Println("Executing command:\n", command.String())
//1. Parse inputs into experiment
// Parse Tumor info
@@ -386,7 +387,7 @@ func main() {
tinfo = app.ParsetTriNetXTumorData(tumorInfo) // need parsed patients to be able to parse tumor data file
}
exp, patients := app.ParseTriNetXData("exp1", patientInfo, patientDiagnoses, diagnosisInfo,
- treatmentInfo, nofAgeGroups, lvl, minYears, maxYears, ICD9ToICD10File, getPatientFilters(pfilters, tinfo))
+ treatmentInfo, nofAgeGroups, lvl, ICD9ToICD10File, triNetXGetPatientFilters(pfilters, tinfo))
//2. Initialise relative risk ratios or load them from file from a previous run
if loadRR != "" {
trajectory.LoadRRMatrix(exp, loadRR)
@@ -403,10 +404,10 @@ func main() {
exp.DPatients = nil
//3. Build the trajectories
trajectory.BuildTrajectories(exp, minPatients, maxTrajectoryLength, minTrajectoryLength, minYears, maxYears, rr,
- getTrajectoryFilters(tfilters, exp))
+ triNetXGetTrajectoryFilters(tfilters, exp))
//4. Plot trajectories to file
trajectory.PrintTrajectoriesToFile(exp, outputPath)
- fmt.Println("Collected trajectories: ")
+ slog.Info("Collected trajectories: ")
for i := 0; i < utils.MinInt(len(exp.Trajectories), 100); i++ {
trajectory.PrintTrajectory(exp.Trajectories[i], exp)
}
@@ -417,7 +418,7 @@ func main() {
gi, _ := strconv.ParseInt(g, 10, 0)
clusterGranularityList = append(clusterGranularityList, int(gi))
}
- fmt.Println("MCL Clustering:")
+ slog.Info("MCL Clustering:")
//ClusterTrajectories(exp, clusterGranularityList, outputPath, mclPath)
cluster.ClusterTrajectoriesDirectly(exp, clusterGranularityList, outputPath, mclPath)
}
diff --git a/go.mod b/go.mod
index 81cc0f4..600c697 100644
--- a/go.mod
+++ b/go.mod
@@ -1,25 +1,10 @@
module ptra
-go 1.17
+go 1.22
require (
- gioui.org v0.0.0-20210308172011-57750fc8a0a6 // indirect
- github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527 // indirect
- github.com/exascience/pargo v1.1.0 // indirect
- github.com/fogleman/gg v1.3.0 // indirect
- github.com/go-fonts/liberation v0.2.0 // indirect
- github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81 // indirect
- github.com/go-pdf/fpdf v0.5.0 // indirect
- github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
- github.com/valyala/fastrand v1.1.0 // indirect
- github.com/yuin/goldmark v1.4.1 // indirect
- golang.org/x/exp v0.0.0-20191002040644-a1355ae1e2c3 // indirect
- golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d // indirect
- golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 // indirect
- golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f // indirect
- golang.org/x/sys v0.0.0-20211019181941-9d821ace8654 // indirect
- golang.org/x/text v0.3.7 // indirect
- golang.org/x/tools v0.1.11 // indirect
- gonum.org/v1/plot v0.10.0 // indirect
- rsc.io/pdf v0.1.1 // indirect
+ github.com/exascience/pargo v1.1.0
+ github.com/valyala/fastrand v1.1.0
)
+
+require gonum.org/v1/gonum v0.9.3 // indirect
diff --git a/go.sum b/go.sum
index 35e24cb..71632d3 100644
--- a/go.sum
+++ b/go.sum
@@ -1,54 +1,37 @@
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
-gioui.org v0.0.0-20210308172011-57750fc8a0a6 h1:K72hopUosKG3ntOPNG4OzzbuhxGuVf06fa2la1/H/Ho=
gioui.org v0.0.0-20210308172011-57750fc8a0a6/go.mod h1:RSH6KIUZ0p2xy5zHDxgAM4zumjgTw83q2ge/PI+yyw8=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw=
-github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527 h1:NImof/JkF93OVWZY+PINgl6fPtQyF6f+hNUtZ0QZA1c=
-github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw=
github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
-github.com/boombuler/barcode v1.0.1/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/exascience/pargo v1.1.0 h1:pBKDhJYoH2ekBehnPCErSIDoi9DqiWL1V70s5kGZANI=
github.com/exascience/pargo v1.1.0/go.mod h1:8GeMktPA5KycHMfqXXOfiQzlazfbFSURzGZIJUO0tfk=
github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k=
-github.com/fogleman/gg v1.3.0 h1:/7zJX8F6AaYQc57WQCyN9cAIz+4bCJGO9B+dyW29am8=
github.com/fogleman/gg v1.3.0/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k=
github.com/go-fonts/dejavu v0.1.0/go.mod h1:4Wt4I4OU2Nq9asgDCteaAaWZOV24E+0/Pwo0gppep4g=
github.com/go-fonts/latin-modern v0.2.0/go.mod h1:rQVLdDMK+mK1xscDwsqM5J8U2jrRa3T0ecnM9pNujks=
github.com/go-fonts/liberation v0.1.1/go.mod h1:K6qoJYypsmfVjWg8KOVDQhLc8UDgIK2HYqyqAO9z7GY=
-github.com/go-fonts/liberation v0.2.0 h1:jAkAWJP4S+OsrPLZM4/eC9iW7CtHy+HBXrEwZXWo5VM=
-github.com/go-fonts/liberation v0.2.0/go.mod h1:K6qoJYypsmfVjWg8KOVDQhLc8UDgIK2HYqyqAO9z7GY=
github.com/go-fonts/stix v0.1.0/go.mod h1:w/c1f0ldAUlJmLBvlbkvVXLAD+tAMqobIIQpmnUIzUY=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-latex/latex v0.0.0-20210118124228-b3d85cf34e07/go.mod h1:CO1AlKB2CSIqUrmQPqA0gdRIlnLEY0gK5JGjh37zN5U=
-github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81 h1:6zl3BbBhdnMkpSj2YY30qV3gDcVBGtFgVsV3+/i+mKQ=
-github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81/go.mod h1:SX0U8uGpxhq9o2S/CELCSUxEWWAuoCUcVCQWv7G2OCk=
-github.com/go-pdf/fpdf v0.5.0 h1:GHpcYsiDV2hdo77VTOuTF9k1sN8F8IY7NjnCo9x+NPY=
-github.com/go-pdf/fpdf v0.5.0/go.mod h1:HzcnA+A23uwogo0tp9yU+l3V+KXhiESpt1PMayhOh5M=
-github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 h1:DACJavvAHhabrF08vX0COfcOBJRhZ8lUbR+ZWIs0Y5g=
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0/go.mod h1:E/TSTwGwJL78qG/PmXZO1EjYhfJinVAhrmmHX6Z8B9k=
github.com/jung-kurt/gofpdf v1.0.0/go.mod h1:7Id9E/uU8ce6rXgefFLlgrJj/GYY22cpxn+r32jIOes=
github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5/go.mod h1:7Id9E/uU8ce6rXgefFLlgrJj/GYY22cpxn+r32jIOes=
github.com/phpdave11/gofpdf v1.4.2/go.mod h1:zpO6xFn9yxo3YLyMvW8HcKWVdbNqgIfOOp2dXMnm1mY=
github.com/phpdave11/gofpdi v1.0.12/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
-github.com/phpdave11/gofpdi v1.0.13/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/ruudk/golang-pdf417 v0.0.0-20181029194003-1af4ab5afa58/go.mod h1:6lfFZQK844Gfx8o5WFuvpxWRwnSoipWe/p622j1v06w=
-github.com/ruudk/golang-pdf417 v0.0.0-20201230142125-a7e3863a1245/go.mod h1:pQAZKsJ8yyVxGRWYNEm9oFB8ieLgKFnamEyDmSA0BRk=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/valyala/fastrand v1.1.0 h1:f+5HkLW4rsgzdNoleUOB69hyT9IlD2ZQh9GyDMfb5G8=
github.com/valyala/fastrand v1.1.0/go.mod h1:HWqCzkrkg6QXT8V2EXWvXCoow7vLwOFN002oeRzjapQ=
-github.com/yuin/goldmark v1.4.1 h1:/vn0k+RBvwlxEmP5E7SZMqNxPhfMVFEJiykr15/0XKM=
-github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
-golang.org/x/exp v0.0.0-20191002040644-a1355ae1e2c3 h1:n9HxLrNxWWtEb1cA950nuEEj3QnKbtsCJ6KjcgisNUs=
golang.org/x/exp v0.0.0-20191002040644-a1355ae1e2c3/go.mod h1:NOZ3BPKG0ec/BKJQgnvsSFpcKLM5xXVWnvZS97DWHgE=
golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
@@ -59,46 +42,28 @@ golang.org/x/image v0.0.0-20200430140353-33d19683fad8/go.mod h1:FeLwcggjj3mMvU+o
golang.org/x/image v0.0.0-20200618115811-c13761719519/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.0.0-20201208152932-35266b937fa6/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.0.0-20210216034530-4410531fe030/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
-golang.org/x/image v0.0.0-20210607152325-775e3b0c77b9/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
-golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d h1:RNPAfi2nHY7C2srAV8A49jpsYr0ADedCk1wq6fTMTvs=
-golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
-golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
-golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f h1:OfiFi4JbukWwe3lzw+xunroH1mnC1e2Gy5cxNJApiSY=
-golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210304124612-50617c2ba197 h1:7+SpRyhoo46QjKkYInQXpcfxx3TYFEYkn131lwGE9/0=
golang.org/x/sys v0.0.0-20210304124612-50617c2ba197/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20211019181941-9d821ace8654 h1:id054HUawV2/6IGm2IV8KZQjqtwAOo2CYlOToYqa0d0=
-golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
-golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
-golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190206041539-40960b6deb8e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190927191325-030b2cf1153e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
-golang.org/x/tools v0.1.11 h1:loJ25fNOEhSXfHrpoGj91eCUThwdNX6u24rO1xnNteY=
-golang.org/x/tools v0.1.11/go.mod h1:SgwaegtQh8clINPpECJMqnxLv9I09HLqnW3RMqW0CA4=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.0.0-20180816165407-929014505bf4/go.mod h1:Y+Yx5eoAFn32cQvJDxZx5Dpnq+c3wtXuadVZAcxbbBo=
gonum.org/v1/gonum v0.7.0/go.mod h1:L02bwd0sqlsvRv41G7wGWFCsVNZFv/k1xzGIxeANHGM=
gonum.org/v1/gonum v0.8.2/go.mod h1:oe/vMfY3deqTw+1EZJhuvEW2iwGF1bW9wwu7XCu0+v0=
+gonum.org/v1/gonum v0.9.3 h1:DnoIG+QAMaF5NvxnGe/oKsgKcAc6PcUyl8q0VetfQ8s=
gonum.org/v1/gonum v0.9.3/go.mod h1:TZumC3NeyVQskjXqmyWt4S3bINhy7B4eYwW69EbyX+0=
gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw=
gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b/go.mod h1:Wt8AAjI+ypCyYX3nZBvf6cAIx93T+c/OS2HFAYskSZc=
gonum.org/v1/plot v0.9.0/go.mod h1:3Pcqqmp6RHvJI72kgb8fThyUnav364FOsdDo2aGW5lY=
-gonum.org/v1/plot v0.10.0 h1:ymLukg4XJlQnYUJCp+coQq5M7BsUJFk6XQE4HPflwdw=
-gonum.org/v1/plot v0.10.0/go.mod h1:JWIHJ7U20drSQb/aDpTetJzfC1KlAPldJLpkSy88dvQ=
-rsc.io/pdf v0.1.1 h1:k1MczvYDUvJBe93bYd7wrZLLUEcLZAuF824/I4e5Xr4=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
diff --git a/plot/plot.jl b/plot/plot.jl
index a2501c6..2641678 100644
--- a/plot/plot.jl
+++ b/plot/plot.jl
@@ -16,6 +16,14 @@
# License and Additional Terms along with this program. If not, see
#