RF-DETR inappropriate resolution causing slowdown

[isaacrob]

Hi! Looks like you’re running all instance seg models at 512. RF-DETR models are optimized for different resolutions via neural architecture search. Running nano models at 512 will result in significantly higher latency. For example, seg nano is optimized for 312 and that particular model’s compute scales with the 4th power of resolution, so running it so much higher will be hugely slower! :) instead of using a higher resolution for nano models we recommend using the larger model variants which, among other things, scale resolution

[NorbertKlockiewicz]

Hi, thank you for this insight, we will look into this issue and hopefully push the performance even further 😄

[msluszniak]

Hi @isaacrob, from where do you conclude that this model utilizes images of size 512? I checked our export script for RF-DETR seg nano, and the resolution is set using this:

    from rfdetr import RFDETRSegNano

    model = RFDETRSegNano()
    actual = model.model.model.cpu()
    actual.eval()
    actual.export()
    resolution = model.model.resolution

and manual check in terminal shows that:

python -c "from rfdetr import RFDETRSegNano; m = RFDETRSegNano(); print(m.model.resolution)"

outputs 312.

I will investigate it further and try to pass dummy tensor of size 312 x 312 and check if this will work.

[isaacrob]

From your benchmark page here https://docs.swmansion.com/react-native-executorch/docs/benchmarks/inference-time

[msluszniak]

Yeah, that's is most probably incorrect information. Also, we don't mention anywhere that the actual size used for instance segmentation in case of RF-DETR is 312. We need to fix these two things in documentation. Thank you for once again for your help. Getting inside directly from the model creator is a huge thing for us ❤️

[isaacrob]

😊

So those latencies are with yolo at 512 and RF-DETR at 312?

[msluszniak]

So those latencies are with yolo at 512 and RF-DETR at 312?

Yes, that seems to be correct.

Also, just for the record, I check both sizes 312x312 and 512x512. Cloned fresh repository, checkout on @ms/make-apps-more-intuitive branch for correct demo app, then changed line:

react-native-executorch/packages/react-native-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp

Line 75 in 46ea659

cv::Size modelInputSize(shape[shape.size() - 2], shape[shape.size() - 1]);

to cv::Size modelInputSize(312, 312); // TEST: hardcoded to force size mismatch (and analogically for 512x512). Then
go to apps > computer vision > run yarn android and go to instance segmentation example.

The results for 312x312 (the same as for the current state of the code):

The result for 512x512:

Normally, the size is taken from metadata of exported model, so it is set to correct 312 value :))

Changing tags to documentation since it is the actual problem.