Skip to content

Fix clamped_relu's value_inference for a negative beta - #2797

Open
LeSingh1 wants to merge 1 commit into
apple:mainfrom
LeSingh1:clamped-relu-negative-beta
Open

Fix clamped_relu's value_inference for a negative beta#2797
LeSingh1 wants to merge 1 commit into
apple:mainfrom
LeSingh1:clamped-relu-negative-beta

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Problem

clamped_relu is documented as "If x >= 0 return elementwise min(beta, x), otherwise return min(beta, alpha * x)". value_inference computes that by splitting x into a positive and a negative half, clamping each, and adding them back:

x = np.minimum(np.maximum(self.x.val, 0), self.beta.val)
y = np.minimum(np.minimum(self.x.val, 0) * self.alpha.val, self.beta.val)
return x + y

The split-and-add trick relies on exactly one of the two terms being 0. That holds only while beta >= 0. Once beta is negative, np.minimum(..., beta) clamps both terms to beta and the sum is 2 * beta:

mb.clamped_relu(x=np.array([-2.0, 0.0, 3.0], dtype=np.float32), alpha=0.5, beta=-1.0)
value
got [-2., -2., -2.]
correct [-1., -1., -1.]

Negative beta is a supported, backend-tested configuration. The op lowers via backend/nn/op_mapping.py to add_clamped_relu, documented in models/neural_network/builder.py as f(x) = min((x >= 0 ? x : alpha * x), beta), and its backend test already parametrizes beta over [7.0, -8.0] with expected np.minimum(beta, np.where(x >= 0, x, x * alpha)) (test/neural_network/test_numpy_nn_layers.py::test_clamped_relu_cpu). So the constant folder and the runtime disagree on a configuration the runtime explicitly supports.

Fix

Apply the documented formula directly rather than reconstructing it from two clamped halves.

I verified the new folding against that same backend expectation for every combination of alpha in [0.0, 2.0, -3.0] and beta in [7.0, -8.0] over x = np.arange(-20, 20), and confirmed the result dtype stays float32.

Tests

TestClampedReLU::test_builder_eval_negative_beta — checks the folded value against np.minimum(beta, np.where(x >= 0, x, x * alpha)) for the same alpha/beta grid the neural network backend test uses. Fails on main, passes with the fix.

The existing test_builder_eval is untouched and still passes: it uses beta=1.0, and for a non-negative beta the old split-and-add expression and the correct formula agree — which is why this was never caught.

I ran all of ops/tests/iOS14/test_activation.py before and after; the pre-existing failure set is identical, 188 either way (this environment cannot load CoreML.framework, so the run_compare_builder tests fail there regardless).

clamped_relu is documented as "If x >= 0 return elementwise min(beta, x),
otherwise return min(beta, alpha * x)". value_inference computes it by splitting
x into a positive and a negative half, clamping each, and adding them:

    x = np.minimum(np.maximum(self.x.val, 0), self.beta.val)
    y = np.minimum(np.minimum(self.x.val, 0) * self.alpha.val, self.beta.val)
    return x + y

That relies on exactly one of the two terms being 0, which stops holding once
beta is negative: then both terms clamp to beta and the sum is 2 * beta.

    mb.clamped_relu(x=np.array([-2., 0., 3.], dtype=np.float32),
                    alpha=0.5, beta=-1.0)

    got     [-2. -2. -2.]
    correct [-1. -1. -1.]

Negative beta is a supported configuration: the op lowers to the neural network
add_clamped_relu layer, documented as f(x) = min((x >= 0 ? x : alpha * x), beta),
whose backend test already parametrizes beta over [7.0, -8.0]
(test_numpy_nn_layers.py::test_clamped_relu_cpu).

Apply the documented formula directly. Verified against that same backend
expectation for every alpha in [0.0, 2.0, -3.0] and beta in [7.0, -8.0].
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant