Clip and Image Metadata

[fxtech]

Open Effects Proposal for Standard Change

Please read the contribution guidelines first.

Standard Change Workflow

• Create proposal as issue (you're doing this now!)
• Tag this issue with standard change tag
• Identify subcommittee: at least one plug-in vendor, and at least one host
• Discuss the idea in this issue
• Write new or updated code and doc
• Publish updates as a pull request (ideally on a feature/PROPOSAL-NAME branch)
  • Make sure that PR references this issue number to keep them in sync
  • Discuss and review code in the PR
  • Meet all requirements below for accepting PR
• When subcommittee signs off and other members don't have any further review comments,
  maintainer merges PR to master which closes PR and issue

Requirements for accepting a standard change:

• Header files updated
• Documentation updated
• Release notes added
• Compatibility review completed
• Working code demonstrated with at least one host and one plugin
• At least two members sign off
• No further changes requested from membership

Summary

This proposal aims to add metadata read/write capability to plugins.

Motivation

Some plugins want to be able to get the file path and frame number of the media that sourced the current clip and/or frame.
Other plugins may want to query information about the lens used, or the stack of previous effects applied. The ability to add metadata to the output may also be useful.

Problem

Currently there is no way to query anything about input images to an effect came from, such as the name or path of the source clip.

Impact

This is a new feature, embodied as a new set of properties. It dos not affect any existing suites or properties.

This feature is entirely optional. A host that does not implement it will simply appear to have no meta data to an effect. There is no requirement for an effect to make use of it at all.

Documentation Impact

The documentation for this feature could be embodied entirely in the header file additions.

Stakeholders

Plugin writers should benefit the most from this change.

Discussion

[fxtech]

Some thoughts on this:

1. This API should be contained entirely in a new file: ofxMetadata.h
2. It would be useful for metadata to be available at the Clip and Image levels. For example, it would be meaningful to query a clip to see what the file path is, but you'd need to query the input image to learn the frame number or timecode it came from.
3. It would be useful to be able to add or modify input metadata and pass it on to the output.
4. Metadata should be encapsulated in PropertySet, since it already acts like a dictionary.
5. There needs to be a mechanism to enumerate all of the metadata available in the PropertySet.
6. There should be a well-defined set of standard metadata keys and value types, possibly derived from OTIO conventions.
7. Hosts should be free to implement the metadata API with lazy evaluation. There should not be a performance penalty if a plugin never uses it.

In order to best handle point 2 above, I propose the API is handled with a new OfxMetadataSuiteV1.

The workflow would be to first get a property set from either a Clip or an Image using the existing suites for doing so, then obtain the metadata property set from that, using metadataGet(). This is likely to be handle around some possibly-expensive implementation details, so must be released with metadataRelease(). Property keys can be enumerated with metadataEnumerate().
Note that using the OfxPropertySuiteV1 to modify metadata properties can change them immediately.
It should not be necessary to copy metadata from input clips or images to the output clip or images - the host will do that on its own. But it should be possible to add to or override the metadata in the output clip and/or image.

typedef OfxStatus (OfxMetadataEnumationFuncV1)(OfxPropertySetHandle metadataPropSet, const char *property, void *userData);

typedef struct OfxMetadataSuiteV1 {  
  OfxStatus (*metadataGet)(OfxPropertySetHandle clipOrImagePropertySetHandle, OfxPropertySetHandle *metadataPropHandle);
  OfxStatus (*metadataRelease)(OfxPropertySetHandle metadataPropHandle);
  OfxStatus (*metadataEnumerate)(OfxPropertySetHandle metadataPropHandle, OfxMetadataEnumationFuncV1 callback, void *userData);
};

[fxtech]

Some standard metadata:

• media file path (as a URL)
• source frame number
• source timecode (OTIO uses HH:MM:SS;FRAME or HH:MM:SS.MS)

Vendor-specific metadata keys should be prefixed with the vendor ID (ie. com.borisfx)
File-format-specific metadata should be prefixed with the file format ID if there is one. (ie. openexr)

[AdrianEddy]

Some additional useful metadata:

• source timestamp (pts), i.e. first frame is 0, second is 40ms (in case of 25fps), etc. This is useful when video is VFR and each frame can have a different timestamp, so the current frame number is not enough to calculate the frame time
• frame rate
• media duration (in seconds or milliseconds)
• number of frames
• media resolution
• media rotation angle (cameras nowadays allow to record vertically, and they set 90 degree rotation tag)
• pixel aspect ratio

I'm strongly in favor of this feature, the media file path and accurate timestamp are essential for my use case and OpenFX currenty doesn't provide them

[barretpj]

On Sat, 10 Feb 2024 at 22:01, Paul Miller ***@***.***> wrote: Some standard metadata: - media file path (as a URL) Should we define how to represent a per-clip path for file-per-frame

(sequence) media? printf-style ("%07d" with %% escaping)? ####### style?

- source frame number - source timecode (OTIO uses HH:MM:SS;FRAME or HH:MM:SS.MS) Where ";FRAME" implies drop-frame, presumably, and ":FRAME" non-drop? Vendor-specific metadata keys should be prefixed with the vendor ID (ie. com.borisfx)

Where "vendor-specific" means everything that's not defined in the suite header file?

File-format-specific metadata should be prefixed with the file format ID if there is one. (ie. openexr)

File-format prefixes are unlikely to be possible in my host I fear. Do we need to define how OpenEXR metadata types with non-obvious mapping to OFX string values are to be handled (e.g. chromaticities, box2i, m44f etc)? Do we need to define what the prefixes are for some well-known file formats? Or is it just "copy Nuke"? There's ambiguity between what's vendor-specific and what's file-format-specific too, unless we do say "if it's at all different from Nuke then it's vendor-specific"?

[revisionfx]

prefixes are unlikely to be possible in my host ETC

I was thinking for frame meta-data we would pack chunks of meta-data inside a named group of data - then EXR or CameraX "values" can just be inside that without prefixes...

group "EXR" { list... } - easier to search if you are looking just for something in particular then scanning all

[garyo]

@revisionfx I'm not sure what you're getting at with your grouping idea. Are you thinking the actual metadata values would be a JSON string or something, represented as a single OpenFX string property? OpenFX doesn't have hierarchy/groups for properties otherwise.

[revisionfx]

Phil used the example of OpenEXR
Notice the new attributes (long list):
https://openexr.com/en/latest/StandardAttributes.html
I am suggesting that we could use a generic meta-data suite and "invent" such grouping and call it maybe "schema" ?
What would be in the "EXR" schema is a list of Properties (the attributes above). The Meta-Data suite would allow one to request the "EXR" meta-data handle... as opposed to try to parse a long list of stuff I might not know anything about.

[revisionfx]

DO you need a PropReason for change so frame meta-data change upstream if a node is a client get rerender?
Making sure caching based on parameter changes are not skipping effect then

[fxtech]

We agreed that metadata property keys should be prefixed with 'where it came from', and define a common set with the ofx prefix.

[fxtech]

In order to support "unknown/binary blob" types of metadata we should add a try "raw data" type to PropertSet and while we're at it, parameter support. I'll work up a separate issue for that.

[MrKepzie]

Just as a side-note, I had implemented this in openfx host support 3 years back, with also support for time varying metadata.
I also had to add Bytes support to property types
spec looked like this:

typedef struct OfxMetaDataSuiteV1
{
  /*  @brief Get the time-invariant metadata property set corresponding to the given key.
   @param inArgs The current action inArgs that were passed by the host to avoid lost
   of context by the host.
   @param clip The clip on which to get the metadata from
   @param desiredKeys Optional array of metadata key that are needed. Each metadata desired key is an array
   representing the hierarchy of metadata containers leading to the desired metadata name, e.g: ["temp",project1","scene1","metadata"]
   If NULL, the host must return a property set containing all metadata available on the clip.
   @param nGrouping For each metadata in desiredKeys, the number of items in the grouping
   @param nDesiredKeys The number of keys in desiredKeys
   @returns kOfxStatOK - The metadata property set was successfully fetched and returned
   in the handle
   kOfxStatFailed - The metadata property set could not be fetched because
   no metadata matches the corresponding key
   kOfxStatErrBadHandle - The inArgs handle or clip handle was invalid
   kOfxStatErrMemory - The host had not enough memory to complete the operation
   plug-in should abort whatever it was doing
   */
  OfxStatus (*clipGetMetadata) (OfxPropertySetHandle inArgs,
                                OfxImageClipHandle clip,
                                const char*** desiredKeys,
                                int* nGrouping,
                                int nDesiredKeys,
                                OfxPropertySetHandle* propertySetHandle);

  /*  @brief Get the time-variant metadata property set corresponding to the given key.
   @param inArgs The current action inArgs that were passed by the host to avoid lost
   of context by the host.
   @param clip The clip on which to get the metadata from
   @param time The time at which to sample the metadata
   @param view The view on which to sample the metadata (to support the multi-view suite)
   @param desiredKeys Optional array of metadata key that are needed. Each metadata desired key is an array
   representing the hierarchy of metadata containers leading to the desired metadata name, e.g: ["temp",project1","scene1","metadata"]
   If NULL, the host must return a property set containing all metadata available on the clip.
   @param nGrouping For each metadata in desiredKeys, the number of items in the grouping
   @param nDesiredKeys The number of keys in desiredKeys
   @returns kOfxStatOK - The metadata property set was successfully fetched and returned
   in the handle
   kOfxStatFailed - The metadata property set could not be fetched because
   no metadata matches the corresponding key
   kOfxStatErrBadHandle - The inArgs handle or clip handle was invalid
   kOfxStatErrMemory - The host had not enough memory to complete the operation
   plug-in should abort whatever it was doing
   */
  OfxStatus (*clipGetMetadataAtTime) (OfxPropertySetHandle inArgs,
                                      OfxImageClipHandle clip,
                                      OfxTime time,
                                      int view,
                                      const char*** desiredKeys,
                                      int* nGrouping,
                                      int nDesiredKeys,
                                      OfxPropertySetHandle* propertySetHandle);

} OfxMetaDataSuiteV1;

/*
Action to fetch time invariant metadata on an image effect instance.
This action may only be called from the main thread. This action generalizes and replaces
the old kOfxImageEffectActionGetClipPreferences action, and provides a similar behavior.

inArgs:

outArgs:

A property set where the plug-in can create properties of type int, double or string:
these properties are the metadatas encoded by the plug-in.

A string property may contain binary data, so it should be handled correctly by the host and the plug-in.
The property suite has been modified accordingly.

The effect may only return time invariant metadata from this action.

- For each clip, a property whose name is the clip name appended with kOfxImageEffectPropPassThroughMetadata (see declaration below)

- kOfxImageEffectPropMetadataPassThroughClip (see declaration below)
 */
#define kOfxImageEffectActionGetTimeInvariantMetadata "OfxImageEffectActionGetTimeInvariantMetadata"

/*
 Action to fetch time/view variant metadata on an image effect instance. May be called
 on a render thread.

 The image effect may only return time/view variant  metadatas from this action, such as a timecode.

 inArgs:

 — kOfxPropTime double x1, The time at which to sample the metadata
 - kFnOfxImageEffectPropView, int x1 (only if view aware)


 outArgs:

 A property set where the plug-in can create properties of type int, double or string:
 these properties are the metadatas encoded by the plug-in.

 A string property may contain binary data, so it should be handled correctly by the host and the plug-in.
 The property suite has been modified accordingly.

 The effect may only return time invariant metadata from this action.

 - kOfxImag- For each clip, a property whose name is the clip name appended with kOfxImageEffectPropPassThroughMetadata (see declaration below)
eEffectPropPassThroughMetadata (see below)

 - kOfxImageEffectPropMetadataPassThroughClip (see below)
 */
#define kOfxImageEffectActionGetTimeVariantMetadata "OfxImageEffectActionGetTimeVariantMetadata"


/* @brief string x N property
 Property Set: outArgs of the kOfxImageEffectActionGetTimeInvariantMetadata or kOfxImageEffectActionGetTimeVariantMetadata action

 The plug-in may indicate into it the metadata key(s) of all metadata inherited from a given clip (whose name
 must be prepended to this string) that it does not create.
 Omitted metadata are considered deleted and should not be made available on downstream image effects by the host.
 */
#define kOfxImageEffectPropPassThroughMetadata "_PassThroughMetadata"


/* @brief int x1
 Property Set: Parameter descriptor (read/write), parameter instance (read only)
 Valid values - 0 or 1
 Default value - 0
 Indicates whether a change to this parameter may involve a change to the metadata.

 If set to 1, the host should not only call the kOfxActionInstanceChanged action on the corresponding image effect,
 but also call this action with a type kOfxTypeClip with argument kOfxImageEffectOutputClipName recursively downstream on all image effects.

 This enables a plug-in to update its parameters when a metadata is changed upstream.
 */
#define kOfxParamPropMetadataSlave "OfxParamPropMetadataSlave"

typedef struct OfxPropertySuiteV2 {


  /// Inherited from OfxPropertySuiteV1
  OfxPropertySuiteV1 suiteV1;
  /// End Inherited from OfxPropertySuiteV1

  /* @brief Creates an integer property with the given name and the given
    @param property The name of the property
    @param dimension The number of dimensions
    @param defaultValue An array of size dimension containing the default values
    @returns - kOfxStatFailed: The host declines creation of this property
             - kOfxStatErrExists: A property with the same name already exists in the property set
             - kOfxStatErrUnsupported:  The host does not support properties creation
   */
  OfxStatus (*propCreateInt) (OfxPropertySetHandle properties, const char* property, int dimension, int defaultValue);

  /* @brief Same as other propCreateInt, except for double properties
   */
  OfxStatus (*propCreateDouble) (OfxPropertySetHandle properties, const char* property, int dimension, double defaultValue);

  /* @brief Same as other propCreateInt, except for c-string properties.
   The host should copy the default string values so that their lifetime outlive
   the action within which the plug-in is currently calling this function.
   */
  OfxStatus (*propCreateString) (OfxPropertySetHandle properties, const char* property, int dimension, const char* defaultValue);

  /* @brief Same as other propCreateInt, except for pointer properties.
   The pointers given in defaultValue should outlive the action from which the plug-in is currently calling
   this function until the next action is called on the corresponding image effect.
   */
  OfxStatus (*propCreatePointer) (OfxPropertySetHandle properties, const char* property, int dimension, void* defaultValue);

  /* @brief Same as other propCreateString, except that it also takes a nBytes array
   of size 'dimension' indicating for each c-string its size in bytes.
   */
  OfxStatus (*propCreateBytes) (OfxPropertySetHandle properties, const char* property, int dimension, int nBytes, const unsigned char* defaultValue);

  /* @brief Creates a new property that acts as a container for other properties.
   This allows to create recursive property sets.
   @param property The name of the property
   @returns
     - \ref kOfxStatFailed: The host declines creation of this property
     - \ref kOfxStatErrExists: A property with the same name already exists in the property set
     - \ref kOfxStatErrUnsupported:  The host does not support properties creation
   */
  OfxStatus (*propCreatePropertySet) (OfxPropertySetHandle properties, const char* property, OfxPropertySetHandle* createdSetHandle);

  /* @brief Get a handle to a property of type property set.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param setHandle is the property set handle
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   */
  OfxStatus (*propGetPropertySet) (OfxPropertySetHandle properties, const char* property, OfxPropertySetHandle* setHandle);


  /* @brief Set a single value in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param index is for multidimenstional properties and is dimension of the one we are setting
   @param nBytes The number of bytes to copy from the "value" buffer.
   @param value is the buffer we are setting. The host should make a copy of this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   - ::kOfxStatErrValue
   */
  OfxStatus (*propSetBytes) (OfxPropertySetHandle properties, const char* property, int index, int nBytes, const unsigned char* value);


  /* @brief Set a single value in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param index is for multidimenstional properties and is dimension of the one we are setting
   @param nBytes The number of bytes to copy onto the "value" buffer.
   @param value is the buffer we are getting. The buffer should already be allocated of nBytes.
   The host should make a copy to this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   */
  OfxStatus (*propGetBytes) (OfxPropertySetHandle properties, const char* property, int index, int *nBytes, unsigned char** value);

  /* @brief Set a single value in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param index is for multidimenstional properties and is dimension of the one we are setting
   @param nBytes The number of bytes to copy onto the "value" buffer.
   @param value is the buffer we are getting. The buffer should already be allocated of nBytes.
   The host should make a copy to this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   */
  OfxStatus (*propGetNumBytes) (OfxPropertySetHandle properties, const char* property, int index, int *nBytes);

  /* @brief Set all values at once in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param count is for multidimensional properties and is number of dimensions of the property
   @param nBytes The number of bytes to copy from the "value" buffer. This is an array of size "count" containing
   for each dimension the size in bytes of the corresponding buffer in "value".
   @param value is an array of size "count" containing for each dimension
   the buffer we are setting. The host should make a copy of this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   - ::kOfxStatErrValue
   */
  OfxStatus (*propSetBytesN) (OfxPropertySetHandle properties, const char* property, int count, const int *nBytes, const unsigned char** value);


  /* @brief Set all values at once in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param count is for multidimensional properties and is number of dimensions of the property
   @param nBytes The number of bytes to copy to the "value" buffer. This is an array of size "count" containing
   for each dimension the size in bytes of the corresponding buffer in "value".
   @param value is an array of size "count" containing for each dimension
   the buffer we are setting. Each buffer is already allocated of at least nBytes.
   The host should make a copy to this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   - ::kOfxStatErrValue
   */
  OfxStatus (*propGetBytesN) (OfxPropertySetHandle properties, const char* property, int count, int *nBytes, unsigned char** value);


  /** @brief Get the number of properties in a property set.
   @param numProperties In output, this is the number of properties in the set.
   @returns - kOfxStatFailed  The host did not manage to figure out the number of properties
   in the set or does not want the plug-in to be able to iterate over the properties.
   - kOfxStatOk The host successfully returned the number of properties in the property set.
   - kOfxStatErrBadHandle The properties handle is invalid
   */
  OfxStatus (*propGetNumProperties) (OfxPropertySetHandle properties, int* numProperties);

  /** @brief Get the name and type of a property in a property set
   @param index The index of the property to fetch in the set. This must be a number ranging
   from 0 to the number of properties returned by the propGetNumProperties function.
   @param property In output, a UTF-8 encoded c-string containing the name of the property.
   This name can be used to call any other function of the property suite. The string is
   valid until the action ends.
   @param typeName In output, a UTF-8 encoded c-string containing the name of the type of the property.
   This can either be "int", "double", "pointer", "string", "bytes", "set".
   @returns - kOfxStatFailed  The host did not manage to figure out the property name
   in the set or does not want the plug-in to be able to iterate over the properties.
   - kOfxStatOk The host successfully returned the property name
   - kOfxStatErrBadIndex The index is invalid
   - kOfxStatErrBadHandle The properties handle is invalid
   */
  OfxStatus (*propGetPropertyNameAndType) (OfxPropertySetHandle properties, int index, char** property, char** typeName);

} OfxPropertySuiteV2;