Skip to content

[ntuple] Precision loss for big integer values #416

Description

@undefined-panda

When reading RNTuple data, Int64/UInt64 values above Number.MAX_SAFE_INTEGER or below Number.MIN_SAFE_INTEGER lose precision because they're cast to Number in rntuple.mjs:

// FIXME: let process BigInt in the TTree::Draw
obj[this.name] = Number(this.view.getBigInt64(this.o, LITTLE_ENDIAN));

The FIXME comment suggests this is known. Since Number and BigInt are distinct types in JS, would it make sense to return a BigInt only when the value doesn't fit in a safe integer, and a Number otherwise? Something like:

const num = this.view.getBigInt64(this.o, LITTLE_ENDIAN);
const MIN = BigInt(Number.MIN_SAFE_INTEGER);
const MAX = BigInt(Number.MAX_SAFE_INTEGER);
obj[this.name] = (num >= MIN && num <= MAX) ? Number(num) : num;

I could open a PR if this direction makes sense, though I'm not sure how it interacts with TTree::Draw.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions