Dual-option value arbitrage: first vs. last vs. most explicit policy

[kdeldycke]

WIP PR to address value arbitrage regressions from the 8.3.x series, as reported in #3403.

Explanation

Since Click inception, and up to the 8.2.x series, we did ctx.params[self.name] = value. Options sharing the same parameter ID were processed sequentially, and always overwriting each other. Which means the last write was always winning. This is an accident: a policy that was never enforced, explained or checked against.

In #3030, I changed consume_value/get_default so that the absence of a value became a real manageable UNSET state.

But then the naive dual-options tests started failing: #3030 (comment) . The less intrusive fix for these tests was implemented and made the first of a dual option win. Which changed the implicit behavior.

The underlying policy or principle governing the dual options arbitrage was never noticed, debated or documented. Because it's been a silent side-effect of the code for 12 years.

The need for arbitraging dual-options were more or less known since the unit tests that were flapping in #3030 existed. But they were never deep enough to highlight and enforce the implicit last write wins policy. The regression escaped us because the test were not covering boolean flag_values.

Solution: enforce policy

Re-introduce the last write win policy from pre-8.3.x series.

On tie, the most explicit source wins. For competing defaults, an explicit default beats an auto-derived one.

For the implementation, we can not rely on the natural processing order of dual-options. We need to actively manage and track their state so we have a place to explicitly implement the policies.

Plan

1. Add a shit-load of tests along several combinations of variations:
   • boolean and non-boolean flag_values
   • explicit and non-explicit default
   • explicit and auto-derived defaults
   • competing user-provided dual-options in different orders
   • duplicate competing options provided
   • 2+ competing options sharing the same parameter ID
2. Expect the tests to conform to the policy
3. Make all the tests happy
4. Document, explain and illustrate the policy in the public Sphinx documentation

Context

The lack of distinction between the absence of value and value of absence, that was fixed by #3030, also add semantic implication for defaults on boolean flags.

It was anticipated by @janluke in #1992 (comment) which proposed stricter validation.

Even @ldionne independently proposed to raise an error on the nonsensical flag_value=False + default=True combinations in its original #3111 report that started the chain.

In both cases I deferred these aggressive changes to a future 9.x release to not change Click internals too much.

[kdeldycke]

This PR is ready to be reviewed.