111 create yaml template for problem submission

[kvdblom]

Draft yaml template for review

[Dvermetten]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

[CIGbalance]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

[CIGbalance]

I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?

[kvdblom]

Yes, it would be nice to have problems separately, I have also prototyped how it might work to have both the suite and component problems, see here:

OPL/problems.yaml

Line 1107 in 84b34b6

problems:

I am not sure about separate templates for suites or problems, but we can think about this and discuss what makes sense. I would imagine, e.g., the BBOB sphere still has a bunch of "varies", because it is scalable in the number of variables for example. For a suite, I would want an exhaustive list (e.g., noise: yes, no / optional, or something like that) of the options, rather than a vague "varies". I will describe this in a comment in the template.

[kvdblom]

TODO: create a new issue for this.

[bn12]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

• name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  • name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `

[kvdblom]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

Yes we should do that.

[kvdblom]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

We should split between

• A template
• An example

[Dvermetten]

To check what fields from the other info (based on the form) should be added to the template (taken from a different problem):
other info:
name: null
partial evaluations: Not Present
full name: Electric Motor Design Optimization
constraint properties: Hard Constraints, Soft Constraints, Box Constraints
number of constraints: '12'
type of dynamicism: ''
form of noise model: ''
type of noise space: ''
other noise properties: ''
description of multimodality: Constraints are multimodal
key challenges / characteristics: Time-consuming solution evaluation, highly-constrained
problem
scientific motivation: Challenging to find good solutions in a limited time
limitations: 'Unavailability, even if available, it wouldn''t be helpful to use
for benchmarking due taking a long time to evaluate a single solution '
implementation languages: Python
links to implementations: Implementation not freely available
approximate evaluation time: 8 minutes
links to usage examples: ''
general: This is not an available problem, but could be interesting to show to
researchers which difficulties appear in real-world problems

[kvdblom]

To check what fields from the other info (based on the form) should be added to the template (taken from a different problem): other info: name: null partial evaluations: Not Present full name: Electric Motor Design Optimization constraint properties: Hard Constraints, Soft Constraints, Box Constraints number of constraints: '12' type of dynamicism: '' form of noise model: '' type of noise space: '' other noise properties: '' description of multimodality: Constraints are multimodal key challenges / characteristics: Time-consuming solution evaluation, highly-constrained problem scientific motivation: Challenging to find good solutions in a limited time limitations: 'Unavailability, even if available, it wouldn''t be helpful to use for benchmarking due taking a long time to evaluate a single solution ' implementation languages: Python links to implementations: Implementation not freely available approximate evaluation time: 8 minutes links to usage examples: '' general: This is not an available problem, but could be interesting to show to researchers which difficulties appear in real-world problems

I wonder if we should separate this into numbers for soft and hard constraints:

• number of constraints: '12'

These are currently not yet included

• form of noise model: ''
• type of noise space: ''
• description of multimodality: Constraints are multimodal

[kvdblom]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

* name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  
  * name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `

This should be included now

[kvdblom]

TODO:

• Definitely still needs a consistency check
• When we are happy (to avoid double work), split into two files: (a) template.yaml and (b) example.yaml

[bn12]

I'm unsure whether this information is needed. I would always say "yes" since I can't imagine a situation where there are no dependencies but I guess there surely are. Anyhow, I wonder whether this has any value?!?

[bn12]

If the number is a range (or "any"), it's scalable, if not, it's not.
Isn't it that simple?
And if not, is the field really required or just for experts?

[bn12]

General comment:
I'd prefer to have the number of fields as small as possible to just not discourage people from filling the template. That's why I question every entry and try to keep the template as easy as possible.
I also think the template should not address optimisation experts but domain experts who might not be aware of our terminology.

[bn12]

If the number is 0 then we expect a "no", if not, we expect "yes", or?
Do we need the field?

[bn12]

Can fields "soft" and "hard" be merged to "hardness" or the like?
Possible entries would then be "hard", "soft", or maybe "both"?

[bn12]

Again I think the field might be economised ;)
If the type is empty, presence would be "no" if not, "yes".

[bn12]

Same as for "dynamic".

[bn12]

It's complex ...
What do we need this information for? Do we need it?

I agree that modality is important but there is way more to it than just "uni" or "multi" and just this doesn't really help IMO.

[bn12]

Or "some" in case of different groups of objectives who share evaluation properties?

[bn12]

a possible value could be "any" in the sense of "any of the above" but leaving this field empty is confusing. (please don't contact us?)

[bn12]

Plural? Can people leave multiple implementations? Do we want this?

[bn12]

Is it clear that we refer to a single fitness/objective function evaluation time here? (not a whole opt. run)

And I guess we expect the maximum in case of "independent objectives" (different evaluation times) above? Maybe better put a field "time" in the group of "evaluations"? I don't know ...

[bn12]

We don't expect any entry here, right?
Can't we just put a comment to the two (sub-) fields following like
" in case of "real-world" or "real-world-like" "
?