From 4a7f695c4eb2b2aba20d7cfce0c51f4e2b83eb00 Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Thu, 4 Jun 2026 13:17:19 +0100 Subject: [PATCH 1/2] Add initial docs on vetoes for data streaming --- .../datastreaming/Datastreaming-Topics.md | 11 ++++++ .../datastreaming/Datastreaming-vetos.md | 34 +++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100644 doc/specific_iocs/datastreaming/Datastreaming-vetos.md diff --git a/doc/specific_iocs/datastreaming/Datastreaming-Topics.md b/doc/specific_iocs/datastreaming/Datastreaming-Topics.md index 117430480..47c74e8c4 100644 --- a/doc/specific_iocs/datastreaming/Datastreaming-Topics.md +++ b/doc/specific_iocs/datastreaming/Datastreaming-Topics.md @@ -111,3 +111,14 @@ partitions: 1 This is the last known forwarder configuration, sent by {ref}`bskafka`. This is for if the forwarder crashes, then it can quickly retrieve its last configuration. Flatbuffers schemas in this topic: - [`fc00` - Forwarder Configuration](https://github.com/ISISComputingGroup/streaming-data-types/tree/master/schemas/fc00_forwarder_config.fbs) + + +{#vetoconfigtopic} +## `_vetoConfig` + +partitions: 1 + +This contains updates for the veto configuration for hard or soft vetos as set by `kafka_dae_control`. Thse are published whenever a veto is enabled or whenever a run is started or stopped. + +Flatbuffers schemas in this topic: +- [`TBD!`]() diff --git a/doc/specific_iocs/datastreaming/Datastreaming-vetos.md b/doc/specific_iocs/datastreaming/Datastreaming-vetos.md new file mode 100644 index 000000000..82afaa65a --- /dev/null +++ b/doc/specific_iocs/datastreaming/Datastreaming-vetos.md @@ -0,0 +1,34 @@ +# Data streaming - Vetoes + +Vetoes are essentially a way of telling either the hardware (with a `hard` veto) or downstream consumers of event data (`soft` veto) to ignore a set of events in a frame conditionally based on a signal (whether that's an internal state, electrical signal or a software-based condition). + +Configuring these is handled with a bit mask where 1 signifies to ignore if `x` condition happens and 0 means always do something with the events. + +In the data streaming system, there are three states an individual veto can configured with: +- `ignored` - the veto is not active and therefore all events are considered "good". +- `soft` - events will be streamed from the hardware, but downstream consumers should ignore these events if respecting vetoes. +- `hard` - events will not be streamed from the hardware at all. + +The only difference between soft and hard from a controls point of view is whether or not the corresponding bit is set on the hardware (the streaming control board) itself. With both `soft` and `hard` vetoes, these vetoes are emitted as a (TBD!) schema on the {ref}`veto config` topic. + +For every frame, there exists [a field](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/pu00_pulse_metadata.fbs#L12) in the header (which is always sent, even if no events are streamed) which is the frame's "active" veto information. + + +## Veto bit mask +This is the same for: +1) the configured vetoes in (TBD!) schema, emitted by `kafka_dae_control` +2) the streamed UDP frame packet's header in word 9, and in turn the `pu00` [active vetoes](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/pu00_pulse_metadata.fbs#L12) + +Bits: +- **Bit 0**: FIFO veto +- **Bit 1**: SMP veto +- **Bit 2**: TS2 pulse veto +- **Bit 3**: Wrong pulse veto +- **Bit 4**: Unused +- **Bit 5**: ISIS slow +- **Bits 6..=9**: External vetoes +- **Bits 10..=13**: Fast chopper vetoes +- **Bits 14..=15**: Reserved vetoes +- **Bits 16..=23**: Unused +- **Bits 24..=31**: Frame repeat number +TODO: this isn't quite right, and we should probably link these up with the docs on event_udp_to_kafka so there's only one place to change if we need to update them. From af4f3fb3ac0888e4c2ee6045323854f269cda9e1 Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Thu, 4 Jun 2026 13:27:47 +0100 Subject: [PATCH 2/2] add link to vc00 schema --- .../datastreaming/Datastreaming-Topics.md | 2 +- .../datastreaming/Datastreaming-vetos.md | 11 +++++------ 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/doc/specific_iocs/datastreaming/Datastreaming-Topics.md b/doc/specific_iocs/datastreaming/Datastreaming-Topics.md index 47c74e8c4..167b5a370 100644 --- a/doc/specific_iocs/datastreaming/Datastreaming-Topics.md +++ b/doc/specific_iocs/datastreaming/Datastreaming-Topics.md @@ -121,4 +121,4 @@ partitions: 1 This contains updates for the veto configuration for hard or soft vetos as set by `kafka_dae_control`. Thse are published whenever a veto is enabled or whenever a run is started or stopped. Flatbuffers schemas in this topic: -- [`TBD!`]() +- [`vc00`](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/vc00_veto_configuration.fbs) diff --git a/doc/specific_iocs/datastreaming/Datastreaming-vetos.md b/doc/specific_iocs/datastreaming/Datastreaming-vetos.md index 82afaa65a..5294a06c6 100644 --- a/doc/specific_iocs/datastreaming/Datastreaming-vetos.md +++ b/doc/specific_iocs/datastreaming/Datastreaming-vetos.md @@ -1,6 +1,6 @@ # Data streaming - Vetoes -Vetoes are essentially a way of telling either the hardware (with a `hard` veto) or downstream consumers of event data (`soft` veto) to ignore a set of events in a frame conditionally based on a signal (whether that's an internal state, electrical signal or a software-based condition). +Vetoes are essentially a way of telling either the detector acquisition hardware (with a `hard` veto) or downstream consumers of event data (`soft` veto) to ignore a set of events in a frame conditionally based on a signal (whether that's an internal state, electrical signal or a software-based condition). Configuring these is handled with a bit mask where 1 signifies to ignore if `x` condition happens and 0 means always do something with the events. @@ -9,15 +9,14 @@ In the data streaming system, there are three states an individual veto can conf - `soft` - events will be streamed from the hardware, but downstream consumers should ignore these events if respecting vetoes. - `hard` - events will not be streamed from the hardware at all. -The only difference between soft and hard from a controls point of view is whether or not the corresponding bit is set on the hardware (the streaming control board) itself. With both `soft` and `hard` vetoes, these vetoes are emitted as a (TBD!) schema on the {ref}`veto config` topic. - -For every frame, there exists [a field](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/pu00_pulse_metadata.fbs#L12) in the header (which is always sent, even if no events are streamed) which is the frame's "active" veto information. +The only difference between soft and hard from a controls point of view is whether or not the corresponding bit is set on the hardware (the streaming control board) itself. With both `soft` and `hard` vetoes, these vetoes are emitted as a [`vc00`](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/vc00_veto_configuration.fbs) blob on the {ref}`veto config` topic. +For every frame, there exists [a field](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/pu00_pulse_metadata.fbs#L12) in the header (which is always sent, even if no events are streamed) which is the frame's "active" veto information. This is whether or not a frame has been vetoed due to the condition. ## Veto bit mask This is the same for: -1) the configured vetoes in (TBD!) schema, emitted by `kafka_dae_control` -2) the streamed UDP frame packet's header in word 9, and in turn the `pu00` [active vetoes](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/pu00_pulse_metadata.fbs#L12) +1) the configured vetoes in the [`vc00`](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/vc00_veto_configuration.fbs) schema, emitted by `kafka_dae_control` +2) the streamed UDP frame packet's header in word 9, and in turn the `pu00` [active vetoes](https://github.com/ISISComputingGroup/streaming-data-types/blob/master/schemas/pu00_pulse_metadata.fbs#L12) that are sent every frame regardless of whether or not events are being vetoed. Bits: - **Bit 0**: FIFO veto