Fix clamped_relu's value_inference for a negative beta - #2797
Open
LeSingh1 wants to merge 1 commit into
Open
Conversation
clamped_relu is documented as "If x >= 0 return elementwise min(beta, x),
otherwise return min(beta, alpha * x)". value_inference computes it by splitting
x into a positive and a negative half, clamping each, and adding them:
x = np.minimum(np.maximum(self.x.val, 0), self.beta.val)
y = np.minimum(np.minimum(self.x.val, 0) * self.alpha.val, self.beta.val)
return x + y
That relies on exactly one of the two terms being 0, which stops holding once
beta is negative: then both terms clamp to beta and the sum is 2 * beta.
mb.clamped_relu(x=np.array([-2., 0., 3.], dtype=np.float32),
alpha=0.5, beta=-1.0)
got [-2. -2. -2.]
correct [-1. -1. -1.]
Negative beta is a supported configuration: the op lowers to the neural network
add_clamped_relu layer, documented as f(x) = min((x >= 0 ? x : alpha * x), beta),
whose backend test already parametrizes beta over [7.0, -8.0]
(test_numpy_nn_layers.py::test_clamped_relu_cpu).
Apply the documented formula directly. Verified against that same backend
expectation for every alpha in [0.0, 2.0, -3.0] and beta in [7.0, -8.0].
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
clamped_reluis documented as "Ifx >= 0return elementwisemin(beta, x), otherwise returnmin(beta, alpha * x)".value_inferencecomputes that by splittingxinto a positive and a negative half, clamping each, and adding them back:The split-and-add trick relies on exactly one of the two terms being
0. That holds only whilebeta >= 0. Oncebetais negative,np.minimum(..., beta)clamps both terms tobetaand the sum is2 * beta:[-2., -2., -2.][-1., -1., -1.]Negative
betais a supported, backend-tested configuration. The op lowers viabackend/nn/op_mapping.pytoadd_clamped_relu, documented inmodels/neural_network/builder.pyasf(x) = min((x >= 0 ? x : alpha * x), beta), and its backend test already parametrizesbetaover[7.0, -8.0]with expectednp.minimum(beta, np.where(x >= 0, x, x * alpha))(test/neural_network/test_numpy_nn_layers.py::test_clamped_relu_cpu). So the constant folder and the runtime disagree on a configuration the runtime explicitly supports.Fix
Apply the documented formula directly rather than reconstructing it from two clamped halves.
I verified the new folding against that same backend expectation for every combination of
alphain[0.0, 2.0, -3.0]andbetain[7.0, -8.0]overx = np.arange(-20, 20), and confirmed the result dtype staysfloat32.Tests
TestClampedReLU::test_builder_eval_negative_beta— checks the folded value againstnp.minimum(beta, np.where(x >= 0, x, x * alpha))for the same alpha/beta grid the neural network backend test uses. Fails onmain, passes with the fix.The existing
test_builder_evalis untouched and still passes: it usesbeta=1.0, and for a non-negative beta the old split-and-add expression and the correct formula agree — which is why this was never caught.I ran all of
ops/tests/iOS14/test_activation.pybefore and after; the pre-existing failure set is identical, 188 either way (this environment cannot load CoreML.framework, so therun_compare_buildertests fail there regardless).