latest example .ort file does not validate against latest JSON schema

[bmaranville]

I just uploaded a schema validator at https://www.reflectometry.org/projects/file_formats/tools/validator.html ... it is using the schema at https://raw.githubusercontent.com/reflectivity/orsopy/main/orsopy/fileio/schema/refl_header.schema.json

I used it to try to validate the example file at https://github.com/reflectivity/orsopy/blob/main/tests/test_example2.ort and it is finding errors with the date-time objects in the example header, all of which are missing timezone information (which is required for it to be a valid date-time string by RFC3339)

Do we want to require this specificity, or should we make a separate validation regex for the JSON schema that accepts dates without timezone information?

The experiment.date field is also failing, as it is a date range and not at all a date-time string.

[andyfaff]

Nice, I love the validator webpage. Does it grab the schema as and when used, or does it need to be regularly updated?

[bmaranville]

fetches it from the url every time. If we start having versions of the schema there could be a drop-down, or it could get the one specified by "version" in the first line of the header to use for validation. I am reading the version number, and I should display it with the validation result.

[andyfaff]

test_example2.ort is supposed to be checked here.
I am guessing test_example.ort should also fail for the same reasons.

The checking is done in _validate_header_data. There I've noted:

# d contains datetime.datetime objects, which would fail the jsonschema validation, so force those to be strings.

There is a kludge sending them to json and back again to force them to be strings, which will be why the validation doesn't fail. It should be easy for a computer to write a compliant string, but much harder for a person.

• Can the example ORT files be modified to make them compliant with the schema? I'd like to get rid of the kludge.
• I'm happy with timezone info being included in the file. If we make that decision then the specification on relevant parts needs to be updated to be more specific. The specification is currently lax on issues like this.
• there are various locations in the file that use date/timestamp/time etc. I'd like for all of those to be changed to one of those, my preference is timestamp. This isn't suitable for a date range, but the multiplicity of options is confusing.
• Is there a specification for compliant date ranges in RFC3339? What should the experiment.date type be? Or would this require a regex?

Please get hacking on the orsopy code, it'd be good for other coders to be refining what we've currently got. I'd like to hold another meeting soon (doesn't need to be everyone). The main issue that needs to be discussed is specification <-> orsopy agreement. e.g.:

• the specification doesn't say which entries are mandatory in a file, and which are optional.
• there is no Orso.creator mentioned in the spec, but we have such an entry in orsopy.
• the specification isn't nearly clear enough on typing of entries. e.g. the example ORT file had a list of strings for data_source.sample.description (plus some of them had yaml forbidden text such as ':'), the specification said it should be free text, but orsopy thought it should be a string. It might be clearer if the specification included the Python types, e.g. Union[str, List[str]].

[andyfaff]

The kludge I mentioned before is because yaml automatically loads a datetime entry in the file as a datetime object:

>>> dct['creator']
{'name': 'G. User',
  'affiliation': 'PSI',
  'timestamp': datetime.datetime(2020, 4, 6, 13, 21, 18, tzinfo=datetime.timezone(datetime.timedelta(seconds=34200)))}

Automatically loading a datetime entry like this is great when loading a file back into an OrsoDataset. However, the jsonschema validator is expecting a string, it can't deal with the datetime.datetime objects directly.

[bmaranville]

Yes - the opinionated behavior (that is often quite convenient) for the yaml library to read date-time strings into datetime objects was an issue, and is why in the validator python script I uploaded a while back (don't remember where it is in the repo) I added a custom hook to the yaml loader so that it wouldn't do that.

The javascript library used in the web validator kindly offers a mode where it will deserialize to only JSON-safe objects (like strings for date-time)

And to answer your question, yes the dates can be modified to be compliant with RFC3339 - adding "Z" at the very end makes them into UTC date-time, or adding an offset e.g. "+01:00" makes them compliant.

For the date-range object, I think that will take more doing, both in the python deserializer and in the validator. We could allow an object {"date_range": [start date-time string, end date-time string]} or separately specify start_date and end_date with the same schema as all the other dates. I think I like the second option better.

[bmaranville]

I found my version of the kludge, for patching the yaml loader:

import yaml

# block loading of datetime objects, for validation only (json schema can not contain datetime objects, only formatted string)
def timestamp_constructor(loader, node):
    return str(node.value)

class StringDatetimeLoader(yaml.Loader):
    pass

StringDatetimeLoader.add_constructor(u'tag:yaml.org,2002:timestamp', timestamp_constructor)

[andyfaff]

See reflectivity/orsopy#38

[bmaranville]

test_example2.ort is supposed to be checked here.
I am guessing test_example.ort should also fail for the same reasons.

The checking is done in _validate_header_data. There I've noted:

# d contains datetime.datetime objects, which would fail the jsonschema validation, so force those to be strings.

There is a kludge sending them to json and back again to force them to be strings, which will be why the validation doesn't fail. It should be easy for a computer to write a compliant string, but much harder for a person.

* Can the example ORT files be modified to make them compliant with the schema? I'd like to get rid of the kludge.

* I'm happy with timezone info being included in the file. If we make that decision then the specification on relevant parts needs to be updated to be more specific. The specification is currently lax on issues like this.

* there are various locations in the file that use `date/timestamp/time` etc. I'd like for all of those to be changed to one of those, my preference is `timestamp`. This isn't suitable for a date range, but the multiplicity of options is confusing.

* Is there a specification for compliant date ranges in RFC3339? What should the `experiment.date` type be? Or would this require a regex?

Please get hacking on the orsopy code, it'd be good for other coders to be refining what we've currently got. I'd like to hold another meeting soon (doesn't need to be everyone). The main issue that needs to be discussed is specification <-> orsopy agreement. e.g.:

* the specification doesn't say which entries are mandatory in a file, and which are optional.

* there is no `Orso.creator` mentioned in the spec, but we have such an entry in orsopy.

* the specification isn't nearly clear enough on typing of entries. e.g. the example ORT file had a list of strings for `data_source.sample.description` (plus some of them had yaml forbidden text such as ':'), the specification said it should be free text, but orsopy thought it should be a string. It might be clearer if the specification included the Python types, e.g. `Union[str, List[str]]`.

JSON schema does allow a "duration", but unfortunately not a "period" yet (the type of date range listed is more like a period than a duration in RFC3339). A start_date + duration or start_date, end_date pair are possible.