Skip to content

CTS tests for length and distance expect abs-based calculation for 1D, but spec is ambiguous #4571

@andyleiserson

Description

@andyleiserson

The CTS tests for length and distance expect an abs-based calculation to be used when the argument is 1D (i.e. a scalar). This avoids overflows that are possible in the $\sqrt{\sum e_i^2}$ calculation.

However, the spec is a bit confusing on this point. It does say that length:

evaluates to the absolute value of e if T is scalar.

But it also says in a note that:

the scalar case may be evaluated as sqrt(e * e), which may unnecessarily overflow or lose accuracy.

The floating-point accuracy table says of length:

Inherited from sqrt(dot(x, x)) in the vector case, and sqrt(x*x) in the scalar case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions