[RF] HS3 export: continuous pdf paired with binned dataset has unrecoverable evaluation semantics

[kratsg]

Context: we are implementing an independent HS3 consumer (pyhs3) and validating it against quickFit NLL values on an ATLAS diHiggs (bbγγ) workspace exported via RooFit's HS3 JSON export. This is a request to make the evaluation semantics of binned-data-against-continuous-pdf recoverable from the exported file.

cc @cburgard @Phmonski

The situation

The exported datasets are binned:

{ "name": "AsimovData_0_Run2HM_1", "type": "binned",
  "axes": [{ "name": "atlas_invMass_Run2HM_1",
             "min": 105.0, "max": 160.0, "nbins": 220 }],
  "contents": [0.344, 0.342, "..."] }   // Σ ≈ 40.7 events

while the paired channel pdf is a continuous mixture_dist over the same variable. This is presumably faithful to the original workspace (a binned Asimov RooDataHist), so not an export bug per se — but the file does not say how RooFit evaluates this pairing:

(A) bin centers:    log L = Σ_b c_b · log pdf(x_b^center)
(B) bin integrals:  log L = Σ_b c_b · log ∫_bin_b pdf(x) dx

(cf. IntegrateBins / binned-likelihood attributes). The two differ by a parameter-dependent amount, so an independent consumer reproducing the NLL curve cannot know which to implement. We have filed a matching HS3-spec issue asking for the semantics to be definable: hep-statistics-serialization-standard/hep-statistics-serialization-standard#93.

Suggestions (increasing order of usefulness)

• document the evaluation convention the export assumes;
• include the relevant evaluation options (e.g. IntegrateBins, binned-likelihood attributes) in the export when they are set;
• optionally support exporting the dataset in the representation the fit actually used.

Related question

For absolute-NLL comparisons: quickFit/RooFit NLLs include data-only constants (e.g. the −log N! of the extended term and constraint normalization constants) that are not part of the serialized model. A short statement in the RooFitHS3 docs of which constants RooFit's createNLL includes would make cross-tool validation much less archaeological.

[cburgard]

I know that in the last years, @guitargeek has worked a bit on related issues (i.e. "when and where does ROOT decide to evaluate unbinned distributions over binned datasets"). Hopefully, he will have a good idea of how to handle this.

[guitargeek]

Thanks for starting the discussion on this! I don't have a definite suggestion, but can provide more context on how RooFit does things.

By default, RooFit always evaluates the pdf at the bin center when the dataset is binned. If the model is continuous, this is indeed a crude approximation. That's why the integration over bins was introduced later, and there are two ways to hit this path:

• Specify it at the model level: wrap the pdf in a RooBinSamplingPdf. This is useful for model serialization in the RooWorkspace, because the RooBinSamplingPdf wrapper is part of the RooAbsArg computation graph that gets serialized.
• Specify it at the createNLL stage: add the IntegrateBins() command argument to createNLL() or fitTo(). Has to be done in the fitting script after opening the workspace, so it's not serializable.
• It's not possible to specify this in any other way. With the ModelConfig, not via the dataset, and not with string attributes on the model (like the slightly related "BinnedLikelihood" optimization).

So RooFit doesn't help in setting a good precedent in serializing the semantics, as I think the wrapper pdf is not really elegant. The RooBinSamplingPdf is actually considered by the RooFitHS3 exporters and written as a "binsampling" distribution. For example, to wrap a Gaussian:

    "distributions": [
        {
            "epsilon": 0.0001,
            "name": "pdf",
            "observable": "x",
            "pdf": "gauss",
            "type": "binsampling"
        },
        {
            "mean": "mu",
            "name": "gauss",
            "sigma": "sigma",
            "type": "gaussian_dist",
            "x": "x"
        }
    ],

I find this "wrap-to-change-behavior" pattern unelegant, and indeed it is also not part of the official HS3 spec (I didn't find any hits for binsampling). It also encodes numerical hyperparameters, which I think was not desirable in HS3.

Maybe the likelihoods field is more appropriate to encode what the evaluation is? I think this might me more appropriate, because the bin center approximation is a property of the model-data pairing in the likelihood, not the model or data on their own.

In any case, the only thing that i have a strong opinion is this: I think the the integration over the bin should be the default, because the evaluation at the bin center is an approximation. Here is how the likelihood field for a single-channel model looks like now:

    "likelihoods": [
        {
            "data": ["genData"],
            "distributions": ["gauss"],
            "name": "gauss_genData"
        }
    ],

How to include the metadata on the bin center approximation? That's what needs to be discussed I think.

@kratsg, @cburgard, do you agree with this assessment, or you think this information should go somewhere else?

[guitargeek]

Related question

For absolute-NLL comparisons: quickFit/RooFit NLLs include data-only constants (e.g. the −log N! of the extended term and constraint normalization constants) that are not part of the serialized model. A short statement in the RooFitHS3 docs of which constants RooFit's createNLL includes would make cross-tool validation much less archaeological.

The question is indeed related, but factorizable. Can you maybe open a separate issue about it if you think it's important? I think we can address this very well in RooFit by writing the mathematical formulae explicitly into the docs.