Reads delimited and fixed-width record files and maps them to FHIR JSON resources. Column mapping, code translation and resource shaping are declared in a JSON data contract, so onboarding a new source extract is a configuration change, not a patch.
cargo build --release # target/release/fhir-utils
cargo build --release --target x86_64-unknown-linux-musl # fully static binary
cargo test # unit, integration, end-to-endCommand |
Purpose |
|
validate a data contract |
|
convert every input under |
|
convert one file |
Flag |
Meaning |
Required |
|
contract file for |
yes for |
|
base directory holding |
one of |
|
directory holding the contract |
with |
|
output directory, created when missing |
yes for |
|
fail on a task error, an empty group key or a row that builds nothing |
no |
-d and -f are mutually exclusive; -f without -c reads the contract from
MAPPING_CONFIG_DIRECTORY, and -d requires both
<base>/input and <base>/config. Exit code 0 on success, 1 on an invalid
contract, a missing path or a row conversion failure.
Source extract input/patients.csv with headers that do not match the internal model:
MRN,LastName,FirstName,Sex,DOB,SSN 1234567,Smith,Ann,F,1980-01-02,123-45-6789 7654321,Jones,Bob,M,1975-11-30,
Code map config/sex.csv, two columns, header names fixed:
source_value,target_value F,female M,male
Contract config/data-contract.json:
{
"general": {
"timeZone": "America/New_York",
"tenantId": "hospital-a",
"streamType": "live"
},
"fileDefinitions": {
"patients": {
"fileType": "csv",
"resourceType": "Patient",
"groupByKey": "MRN",
"tasks": [
{
"task": "rename_columns",
"column_map": {
"MRN": "mrn",
"LastName": "nameLast",
"FirstName": "nameFirst",
"Sex": "gender",
"DOB": "birthDate",
"SSN": "ssn"
}
},
{ "task": "map_codes", "code_map": { "gender": "sex.csv" } }
]
}
}
}Run:
fhir-utils validate -f config/data-contract.json
fhir-utils convert -f input/patients.csv -c config -o out
fhir-utils convert --strict -f input/patients.csv -c config -o outResult:
out/1234567/1234567-Patient-patients-00001.json out/7654321/7654321-Patient-patients-00001.json
The key patients matches the file name patients.csv, groupByKey names the source
column because grouping happens before tasks, and sex.csv resolves against the
configuration directory. Target column names come from the record fields below; a name
outside that list travels through the pipeline but never reaches the resource.
Key |
Description |
Required |
|
IANA zone used to localize naive datetimes |
yes |
|
tenant identifier injected into every row |
yes |
|
|
yes |
|
identifier system for PI, MR, AN, VN, RI, RXN |
no |
|
values read as null |
no |
|
match file definition keys as regular expressions |
no |
|
|
no |
|
one of the resource keys |
yes |
|
source column that groups the output |
yes |
|
zero-based line numbers dropped before the header |
no |
|
column names, or names with widths |
no |
|
ordered tasks appended after the default chain |
no |
-
Every
generalkey is injected into each row as a constant, including keys beyond the ones listed above, so a contract can carry its own values into tasks. -
A key matches when it is a substring of the input file name without its extension; longest key first, then alphabetically.
-
Fixed-width files need a width form of
headers; otherwise the first remaining line supplies the column names. -
Input files matching no definition are skipped and named in the summary.
-
Repeated column names keep the last column of the row.
Patient, AllergyIntolerance, Condition, Encounter, Immunization, Observation,
Location, Organization, Practitioner, Procedure, MedicationUse,
MedicationAdministration, MedicationRequest, MedicationStatement,
DocumentReference, DiagnosticReport, Unstructured, Basic.
Task output must use these names; anything else is ignored by the converter. Every row
also carries filePath and rowNum. ssn is normalized to nine digits without
dashes and repeated-digit values are rejected. a → b renames a to b.
Model |
Fields |
Patient |
|
AllergyIntolerance |
|
Condition |
|
Encounter |
|
Immunization |
|
Location |
|
MedicationUse |
|
Observation |
|
Organization |
|
Practitioner |
|
Procedure |
|
Basic |
|
Unstructured |
|
Default chain, in order: add_row_num, set_nan_to_none,
remove_whitespace_from_columns, copy_columns into groupByKey, then add_constant
per general key plus filePath and configResourceType. User tasks run after it.
Task |
Parameters |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+ |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
none |
|
|
|
|
|
none |
|
|
|
|
|
|
-
Tasks run in order over one chunk at a time; a failing task is logged and the chain continues.
-
List valued cells are strings joined with
|. -
condition_mapandcode_maptake an inline object or a CSV file withsource_valueandtarget_valuecolumns; anullor empty target clears the cell. -
A value the code map does not list passes through unchanged, which is reported once per file and column; add a
defaultentry to control it. -
Unknown tasks and unknown or missing parameters are rejected when the contract loads, as is an unknown key inside a file definition or a missing
fileDefinitionsblock. -
A task target that neither a later task nor the resource reads is reported by
validateand warned about duringconvert;--strictmakes it fail the run. It catches a misspelled field name, which otherwise drops the value silently.
-
Path:
<out>/<groupByKey>/<groupByKey>-<resourceType>-<safeFileId>-<nnnnn>.json. -
The counter runs per group key, resource type and source file.
-
JSON is pretty printed with sorted keys.
-
safeFileIdkeeps[A-Za-z0-9-]and maps every other character to_. -
Group key directories keep letters, digits, spaces and dots; path separators, control characters and
\ : * ? " < > |become, and an all-dot key becomes. -
Re-running overwrites matching file names and leaves earlier files in place.
-
The summary counts files, rows read, resources written and rows that produced none.
-
A row whose record carries no usable code produces no resource; those rows are counted in the summary, listed by row number on standard error, and fatal with
--strict. -
Resource ids are generated per run unless the contract maps
resourceInternalId, so a reload creates new ids; theurn:id:extIDidentifier and theurn:id:source-file-idextension, which carries<file>:<rowNum>, stay stable and are the keys to match on. -
The summary goes to standard output; warnings and errors go to standard error.
-
Warnings print by default;
RUST_LOG=infoadds a line per file naming the definition that matched it and, once converted, its row and resource counts.
FHIR requires some fields the source extract may not carry. When a column is absent the converter fills these so the output loads into a server:
Resource |
Field |
Default |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
data absent |
|
|
|
Map the real column whenever the source has one; the defaults keep output valid, they do not carry clinical meaning.
Class |
Behaviour |
Contract or validation error |
run stops naming the key or the accepted values |
Missing file definition |
file skipped |
Task failure |
warning, pipeline continues; fatal with |
Task target no field reads |
warning at load; fatal with |
Rows without a group key |
warning once per file; fatal with |
Rows producing no resource |
counted in the summary and listed; fatal with |
Unlisted code map value |
warning once per file and column, value passes through |
Input that is not UTF-8 |
run stops naming the file and the re-encoding command |
Row conversion error |
run stops with the group key and file |
Variable |
Meaning |
Default |
|
rows per chunk; |
|
|
contract directory used when |
|
|
contract file name |
|
Settings are read once per process. Contracts, external file definitions, mapping files
and join sources open through one pluggable opener: local paths and file:// URIs work
out of the box, other schemes need a backend registered with opener::set_opener. The
opener is scoped to the calling thread, so each worker thread installs its own.