Composite score mixes speed with coverage

[nigrosimone]

The composite is a sum over the scored profiles, so a row goes up both when a server gets faster and when its maintainer implements one more profile. Those are two different properties and one number cannot separate them.

The clearest case in the published data is aspnet-minimal against fulmine, 726 against 669, so the board puts aspnet-minimal above. On the 13 profiles both of them run, fulmine is faster on 10, with a median ratio of 1.37. The difference in the composite comes from aspnet-minimal running 23 profiles and fulmine 14, not from throughput.

I am on the wrong side of this as well (https://github.com/nigrosimone/fulmine.js), which is what made me write it. This week I subscribed my own entry to gateway-64 and gateway-h3 by putting a stock Caddy in front of it. Caddy terminates the TLS and speaks h2 and h3, my server did not change a line. That is up to 200 points my framework did not earn.

I know the sum is deliberate and the docs say so, and the goal behind it is right: an entry that wins one profile should not top the board. The problem is only that the same number now carries two things at once.

Three ideas, cheapest first:

1. Show the mean over the profiles an entry runs, and the count, next to the sum. Both are already computed and it is one column each. Sorting can stay on the sum. Then a reader can tell whether a row is high because it is fast or because it is complete.
2. Average inside a family, then sum the families. Today gRPC is four profiles and upload is one, so being good at gRPC counts four times as much, and that ratio comes from how many variants exist rather than from a decision about what matters.
3. Treat gateway and production-stack the way pipelined and fortunes are already treated. Those are reference-only because they measure something other than the framework. The gateway profiles measure a proxy plus a framework, and the rules require the proxy to be stock third-party software, so the same argument seems to apply.

Happy to send a PR for the first one if you think it is worth having.

[MDA2AV]

I agree we can have an extra column with the average on all tests the framework subscribes, I also agree that the default should still be the current composite sum.

A few things, composite scores are per "section" so there is a score for h1, h2, h3 etc meaning that a framework that only subscribes to h1 tests can still hit # 1 on the h1 section, this way there is no penalization on not subscribing to gRPC or Gateway tests which are a niche.

A note on why composite score works as it does is historical, the first big reason is to incentive framework developers to subscribe to all tests and avoid having everyone just ultra focus on baseline or json for example (which happened on the TechEmpower benchmarks). The second reason is because a framework that covers all tests sometimes will need to make compromises and cannot simply optimize its config or settings for a specific profile.

Currently we are still focusing on the h1 section but soon will move to h2 and h3, adding more tests.

On your point 3. the gateway section is still a far cry of what we want it to be, so the results are somewhat not very meaningful, however they do not interfere in the composite score on the other sections.