US DICOM anonymisation (incl. burned-in pixel clean up)

[tomaroberts]

Description

• Adds functionality to clean burned-in areas (potentially containing PII) of US DICOM files
• Implements DICOM recipe functionality (based on deid pixel cleaning) for removing burned-in data (potentially containing patient identifiable data)

Example of before (left) and after pixel cleaning (using pydicom-data US test dataset):

• More broadly, this PR implements ImageOperations class, analogous to TagOperations class. Currently ImageOperations purely works with a single deid recipe file, but the class could be extended in future to perform other operations on DICOM image data, e.g. removing burned-in data from x-rays, or, not using the deid library at all

Type of change

Please delete options accordingly to the description.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Suggested Checklist

• I have performed a self-review of my own code.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have commented my code, particularly in hard-to-understand areas.
• I have passed on my local host device. (see further details at the CONTRIBUTING document)
• Make sure your branch is up-to-date with main branch. See CONTRIBUTING for a general example to syncronise your branch with the main branch.
• I have requested review to this PR.
• I have addressed and marked as resolved all the review comments in my PR.
• Finally, I have selected squash and merge

[codecov]

Codecov Report

❌ Patch coverage is 93.18182% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.50%. Comparing base (7f5399f) to head (3c3a2e6).

Files with missing lines	Patch %	Lines
...l_core/src/core/project_config/image_operations.py	85.71%	4 Missing ⚠️
..._core/src/core/project_config/pixl_config_model.py	84.61%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #635      +/-   ##
==========================================
+ Coverage   87.79%   88.50%   +0.71%     
==========================================
  Files          78       79       +1     
  Lines        3735     3776      +41     
==========================================
+ Hits         3279     3342      +63     
+ Misses        456      434      -22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tomaroberts]

Unsure of what to do for destination: section of us-cts.yaml

[tomaroberts]

@stefpiatek – see CI error here.

I need to add deid to PIXL's packages, however PIXL sets its pydicom version to 2.4.5, whereas deid requires pydicom>=3.0.0,<4.0.0.

It looks like this applies across all of the PIXL modules (core, export, imaging, etc.). Can update the various pyproject.toml files across the software, but that seems like quite a bulk significant change. Thoughts?

PS: I've got a local validation error to do with PixlConfig which is making the tests very angry, but let's start with the above.

[stefpiatek]

I need to add deid to PIXL's packages, however PIXL sets its pydicom version to 2.4.5, whereas deid requires pydicom>=3.0.0,<4.0.0.

It looks like this applies across all of the PIXL modules (core, export, imaging, etc.). Can update the various pyproject.toml files across the software, but that seems like quite a bulk significant change. Thoughts?

Yeah let's go for an upgrade of pydicom, tests should give enough coverage to know if that'll cause issues. Let me know if it needs to update the DICOM validation library with that change as well. IIRC if that updates we need to work out the new way to pre-download the validation schema

[tomaroberts]

@stefpiatek – ready for review. I haven't thrown lots of DICOMs at it yet, but be good to get your eyes on this now I think.

Also just realised... I'm not sure if the functionality I've implemented us actually integrated into the PIXL workflow. Think I might've made a function and test, but suspect it's not linked in?

[tomaroberts]

Tested clean_dicom_image_pixels() on the first .dcm files from 8 patient datasets – all images cleaned, which is encouraging.

[stefpiatek]

Nice! Thanks Tom, couple of suggestions to keep it clean and as simple as possible. Did you want to make those changes and then we test out on some live data?

[tomaroberts]

@stefpiatek – thanks for the review; resolved your requests.

[stefpiatek]

Nice, thanks Tom. Had to do some tweaks to avoid compiling on the GAE but tested out in orthanc and there's de-identified ultrasounds 🎉