BoxToMaskD and MaskToBoxD Transpose Results

[oseymour]

Describe the bug
The outputs from BoxToMaskD and MaskToBoxD are transposed in the spatial dims relative to the inputs.

To Reproduce

from monai.apps.detection.transforms.dictionary import BoxToMaskD, MaskToBoxD
import numpy as np
import matplotlib.pyplot as plt

ins = {
    "image": np.zeros([1,10,10]),
    "boxes": np.array([
        [1, 1, 3, 6],  # xmin, ymin, xmax, ymax
    ]),
    "labels": np.array([1,]),
}

box2mask_out = BoxToMaskD(
    box_keys="boxes", label_keys="labels", box_ref_image_keys="image",
    box_mask_keys="box_masks", min_fg_label=1, ellipse_mask=False
)(ins)

mask2box_out = MaskToBoxD(
    box_keys="boxes", box_mask_keys="box_masks", label_keys="label",
    min_fg_label=1,
)(box2mask_out)

plt.figure(figsize=[8,12])

plt.subplot(3,2,1)
plt.title("Input Box")
plt.imshow(ins["image"].squeeze())
plt.hlines(y=[ins["boxes"][0,1], ins["boxes"][0,3]], xmin=ins["boxes"][0,0], xmax=ins["boxes"][0,2])
plt.vlines(x=[ins["boxes"][0,0], ins["boxes"][0,2]], ymin=ins["boxes"][0,1], ymax=ins["boxes"][0,3])

plt.subplot(3,2,3)
plt.title("Box from BoxToMaskD")
plt.imshow(box2mask_out["image"].squeeze())
plt.hlines(y=[box2mask_out["boxes"][0,1], box2mask_out["boxes"][0,3]], xmin=box2mask_out["boxes"][0,0], xmax=box2mask_out["boxes"][0,2])
plt.vlines(x=[box2mask_out["boxes"][0,0], box2mask_out["boxes"][0,2]], ymin=box2mask_out["boxes"][0,1], ymax=box2mask_out["boxes"][0,3])

plt.subplot(3,2,4)
plt.title("Mask from BoxToMaskD")
plt.imshow(box2mask_out["image"].squeeze())
plt.imshow(box2mask_out["box_masks"].squeeze())

plt.subplot(3,2,5)
plt.title("Box from MaskToBoxD")
plt.imshow(mask2box_out["image"].squeeze())
plt.hlines(y=[mask2box_out["boxes"][0,1], mask2box_out["boxes"][0,3]], xmin=mask2box_out["boxes"][0,0], xmax=mask2box_out["boxes"][0,2])
plt.vlines(x=[mask2box_out["boxes"][0,0], mask2box_out["boxes"][0,2]], ymin=mask2box_out["boxes"][0,1], ymax=mask2box_out["boxes"][0,3])

plt.subplot(3,2,6)
plt.title("Mask from MaskToBoxD")
plt.imshow(mask2box_out["image"].squeeze())
plt.imshow(mask2box_out["box_masks"].squeeze())

plt.show()

Expected behavior
I would expect the masks and the boxes to be oriented in the same direction.

Screenshots
Plot from example code

Environment

Ensuring you use the relevant python executable, please paste the output of:

================================
Printing MONAI config...
================================
MONAI version: 1.5.2
Numpy version: 2.4.4
Pytorch version: 2.5.1+cu121
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: d18565fb3e4fd8c556707f91ac280a2dc3f681c1
MONAI __file__: C:\Users\<username>.HCAD\Documents\GitHub\eval-iu-dino\.venv\Lib\site-packages\monai\__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: NOT INSTALLED or UNKNOWN VERSION.
scikit-image version: NOT INSTALLED or UNKNOWN VERSION.
scipy version: 1.18.0
Pillow version: 12.2.0
Tensorboard version: 2.20.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.20.1+cu121
tqdm version: 4.66.5
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 7.2.2
pandas version: NOT INSTALLED or UNKNOWN VERSION.
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Windows
Win32 version: ('11', '10.0.22631', 'SP0', 'Multiprocessor Free')
Win32 edition: Enterprise
Platform: Windows-11-10.0.22631-SP0
Processor: Intel64 Family 6 Model 186 Stepping 3, GenuineIntel
Machine: AMD64
Python version: 3.12.9
Process name: python.exe
Command: ['C:\\Users\\223113289.HCAD\\AppData\\Roaming\\uv\\python\\cpython-3.12.9-windows-x86_64-none\\python.exe', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='C:\\Windows\\System32\\en-US\\KernelBase.dll.mui', fd=-1)]
Num physical CPUs: 10
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [0.6, 0.0, 1.3, 0.0, 49.1, 30.2, 5.4, 0.3, 24.9, 4.7, 0.0, 0.9]
CPU freq. (MHz): 1472
Load avg. in last 1, 5, 15 mins (%): [0.0, 0.0, 0.0]
Disk usage (%): 88.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.4
Available memory (GB): 10.3
Used memory (GB): 21.1

================================
Printing GPU config...
================================
Num GPUs: 0
Has CUDA: False
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100

Additional context
Add any other context about the problem here.

[aymuos15]

Thanks for the repro!

The transforms aren't transposing tho actually. The box round-trips exactly: [1, 1, 3, 6] goes through BoxToMaskD then MaskToBoxD and comes back as [1., 1., 3., 6.]. The mask sits exactly where the box says it should:

Box coordinate	Value	Array axis it indexes	Mask foreground
xmin, xmax	1, 3	axis 0, which imshow draws vertically	1:3
ymin, ymax	1, 6	axis 1, which imshow draws horizontally	1:6

So x is the first spatial array axis, not the horizontal one, and a box [xmin, ymin, xmax, ymax] covers image[..., xmin:xmax, ymin:ymax]. Your hlines/vlines calls treat coordinate 0 as horizontal, so the outline and the mask get drawn perpendicular to each other even though the data agrees:

Swapping the arguments lines them up:

b = box2mask_out["boxes"][0] - 0.5   # imshow centres pixel i at i
plt.imshow(box2mask_out["box_masks"].squeeze())
plt.hlines(y=[b[0], b[2]], xmin=b[1], xmax=b[3])   # x -> vertical
plt.vlines(x=[b[1], b[3]], ymin=b[0], ymax=b[2])   # y -> horizontal

That said, the docs never say which axis xmin indexes, and "xyxy" means the opposite in torchvision.

@ericspod does this warrant a doc change?

[oseymour]

Is the near-universal standard for axes not that X is the horizontal axis (spatial axis 1) and Y is vertical (spatial axis 0)? So if MONAI's standard box format is xyxy and you're saying y=b[0] in your plt.hlines() call then you're saying y is equal to an x coordinate.

Also, if my [xmin, ymin, xmax, ymax] is [1, 1, 3, 6] then the box should be 2 pixels wide and 5 pixels tall. So it should be taller than it is wide.

[aymuos15]

I do realise that's the general comp vision norm but medical imaging follows the opposite as seen here (NIfTI maps voxel (i,j,k) to world (x,y,z) in order, so the first array axis is x). MONAI does too, e.g. MetaTensor defaults to RAS space and flip_boxes on axis 0 matches Flipd(spatial_axis=0).

So the transpose look is cos imshow draws axis 0 vertically.