Skip to content

Fix ends of the encoding LUT of LUT-style NLTs - #2

Open
palemieux wants to merge 1 commit into
masterfrom
issues/fix-nlt-encoding-lut-endpoints
Open

palemieux wants to merge 1 commit into
masterfrom
issues/fix-nlt-encoding-lut-endpoints

Conversation

@palemieux

Copy link
Copy Markdown

nlt_rec::prepare_for_encoding() builds the table that inverts the LUT of a type 2 or type 4 NLT. Each entry of that table is a position in the domain the LUT is indexed over, but the first and the last entries were set to ft_min and ft_max, which are the values of the first and last entries of the LUT, i.e. positions in its range. The two only coincide when the LUT entries span the same interval as the LUT is indexed over.

When they do not, samples that fall in the first or the last cell of the encoding table are interpolated towards the wrong end point and are encoded far from where they belong. For example, with a LUT indexed over [0, 2^32) whose entries span [0.22, 0.78] of it, samples within about 9% of the largest value (65504 in a float32 transfer function LUT) came back from a round trip with an error of up to 99%.

Set the end points to fd_min and fd_max, the positions of the first and last entries of the LUT, in the 1, 2 and 4 byte code paths.

Add a test that round trips a ramp through a type 2 and a type 4 LUT whose entries only cover the middle half of the range they are indexed over. It fails without this change (error of 2^29 on the last sample) and passes with it.

nlt_rec::prepare_for_encoding() builds the table that inverts the LUT
of a type 2 or type 4 NLT. Each entry of that table is a position in the
domain the LUT is indexed over, but the first and the last entries were
set to ft_min and ft_max, which are the values of the first and last
entries of the LUT, i.e. positions in its range. The two only coincide
when the LUT entries span the same interval as the LUT is indexed over.

When they do not, samples that fall in the first or the last cell of the
encoding table are interpolated towards the wrong end point and are
encoded far from where they belong. For example, with a LUT indexed over
[0, 2^32) whose entries span [0.22, 0.78] of it, samples within about
9% of the largest value (65504 in a float32 transfer function LUT) came
back from a round trip with an error of up to 99%.

Set the end points to fd_min and fd_max, the positions of the first and
last entries of the LUT, in the 1, 2 and 4 byte code paths.

Add a test that round trips a ramp through a type 2 and a type 4 LUT
whose entries only cover the middle half of the range they are indexed
over. It fails without this change (error of 2^29 on the last sample) and
passes with it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@michaeldsmith

Copy link
Copy Markdown

It would be nice to also fix the comment for members ft_min and ft_max on line 893 OpenJPH\src\core\codestream\ojph_params_local.h

float ft_min, ft_max; // float d_min and d_max

to

float ft_min, ft_max; // float first and last LUT points

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants