From 93e0008d8fe305ef170b379a486122046ea173ec Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 10 Mar 2026 14:41:03 -0700 Subject: [PATCH 01/22] CDA-74 Created ADR for timeseries csv formatting --- .../decisions/timeseries-csv-format.rst | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/source/decisions/timeseries-csv-format.rst diff --git a/docs/source/decisions/timeseries-csv-format.rst b/docs/source/decisions/timeseries-csv-format.rst new file mode 100644 index 000000000..f44e22086 --- /dev/null +++ b/docs/source/decisions/timeseries-csv-format.rst @@ -0,0 +1,85 @@ +##### +CSV Format for TimeSeries +##### + + +Summary +======= + +This ADR defines a standardized CSV representation for TimeSeries. It specifies a row-per-record CSV format that preserves essential metadata and ensures consistent ingestion by analytics, automation, and warehousing systems. + + +Opinions +======== + +Opinion 1 +--------- + +@brysonspilman + +Summary: +Since the intended use of the CSV format is for retrieval only, a customized format that follows standardize csv practices is appropriate. + +Key points: + +- CSV will be serialized via the jackson api, with either a custom csv DTO, or csv annotations on the existing DTO on specific fields to include. +- Timeseries CSV will include the date-time, value, and units fields as well as optionally the time-series id, office, quality, and version-date. +- By default, optional fields will not be included. +- A flag to include optional fields as metadata comments at the top of the file may be added. This would be a commented line indicating the amount of metadata followed by commented metadata rows using key:value pairs. See example below. +- Column names use kebab-case for consistency with JSON and XML. +- Units will either be included in metadata comments or as a column. Recommend not including this in the value column-header, as column names should match DTO fields. A custom serializer could be created if units need to be included in the value-column-header. +- date-time values will be serialized as ISO-8601 strings. NOTE this differs from JSON and XML for date-time which are serialized as epoch-millis. +- Null values are empty fields. Missing values use quality-code = 5, for consistency with JSON and XML. +- UTF-8 encoding, comma delimiter, LF line endings, header always included. +- One row is produced per Record. +- Multi-retrieve never includes multiple time-series IDs. + +Example CSVs: + +1) All optionals turned off, and no metadata comments: +date-time, value, units +2021-06-21T00:00:00Z, 0.0, ft +2021-06-22T00:00:00Z, 1.0, ft +2021-06-23T00:00:00Z, 2.0, ft +2021-06-24T00:00:00Z, 3.0, ft + +2) All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: +date-time, value (ft) +2021-06-21T00:00:00Z, 0.0 +2021-06-22T00:00:00Z, 1.0 +2021-06-23T00:00:00Z, 2.0 +2021-06-24T00:00:00Z, 3.0 + +3) All optionals turned on, with metadata-as-comments turned on: +# metadata-count: 5 +# time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI +# office-id: SWT +# version-date: 2021-06-21T00:00:00Z +# quality-code: 1 +# units: ft +date-time, value +2021-06-21T00:00:00Z, 0.0 +2021-06-22T00:00:00Z, 1.0 +2021-06-23T00:00:00Z, 2.0 +2021-06-24T00:00:00Z, 3.0 + +4) All optionals turned on, with metadata-as-comments not turned on: +time-series-id, office-id, date-time, value, units, version-date, quality-code +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 + +(Note that if we go with option 2 for units, then the units column would not be included in this example in 4), instead it would be included in the value column header) + +Decision Status +=============== + +(Status: proposed) + + +References +========== + +Related Types: cwms.cda.data.dto.TimeSeries, TimeSeries.Record +Issue/Discussion: https://github.com/USACE/cwms-data-api/issues/1525 \ No newline at end of file From a39233f2b2e1de5683a79d9400f33d207f433e88 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Wed, 11 Mar 2026 09:15:57 -0700 Subject: [PATCH 02/22] CDA-74 Updated ADR for timeseries csv to include doc number and added to index.rst --- ...{timeseries-csv-format.rst => 0008-timeseries-csv-format.rst} | 0 docs/source/decisions/index.rst | 1 + 2 files changed, 1 insertion(+) rename docs/source/decisions/{timeseries-csv-format.rst => 0008-timeseries-csv-format.rst} (100%) diff --git a/docs/source/decisions/timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst similarity index 100% rename from docs/source/decisions/timeseries-csv-format.rst rename to docs/source/decisions/0008-timeseries-csv-format.rst diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst index cc4e5d92e..0dc16c28f 100644 --- a/docs/source/decisions/index.rst +++ b/docs/source/decisions/index.rst @@ -24,3 +24,4 @@ Some decisions may also be a proposal and marked appropriately. Authorization Middleware <./0005-data-authorization-middleware.md> CDA Authorization Filtering <./0006-cda-authorization-filtering.md> Access Management Clients <./0007-access-management-clients.md> + Timeseries CSV Format <./0008-timeseries-csv-format.rst> From 4823b58c0cca0c1ca6942d91170a069e713d96d1 Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Thu, 12 Mar 2026 15:00:37 -0700 Subject: [PATCH 03/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Mike Neilson --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index f44e22086..5464bb34a 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -18,7 +18,7 @@ Opinion 1 @brysonspilman Summary: -Since the intended use of the CSV format is for retrieval only, a customized format that follows standardize csv practices is appropriate. +Since the intended use of the CSV format is for retrieval only, a customized format that follows standardized csv practices is appropriate. Key points: From 24e8b63790c494ce56ef866083c9c6720bef5b6f Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:24:17 -0700 Subject: [PATCH 04/22] Updated numbering and sectioning of csv ADR --- .../decisions/0008-timeseries-csv-format.rst | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 5464bb34a..253ee4bc4 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -17,10 +17,12 @@ Opinion 1 @brysonspilman -Summary: +Summary +~~~~~~~ Since the intended use of the CSV format is for retrieval only, a customized format that follows standardized csv practices is appropriate. -Key points: +Key points +~~~~~~~~~~ - CSV will be serialized via the jackson api, with either a custom csv DTO, or csv annotations on the existing DTO on specific fields to include. - Timeseries CSV will include the date-time, value, and units fields as well as optionally the time-series id, office, quality, and version-date. @@ -34,23 +36,24 @@ Key points: - One row is produced per Record. - Multi-retrieve never includes multiple time-series IDs. -Example CSVs: +Example CSVs +~~~~~~~~~~~~ -1) All optionals turned off, and no metadata comments: +1. All optionals turned off, and no metadata comments: date-time, value, units 2021-06-21T00:00:00Z, 0.0, ft 2021-06-22T00:00:00Z, 1.0, ft 2021-06-23T00:00:00Z, 2.0, ft 2021-06-24T00:00:00Z, 3.0, ft -2) All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: +2. All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: date-time, value (ft) 2021-06-21T00:00:00Z, 0.0 2021-06-22T00:00:00Z, 1.0 2021-06-23T00:00:00Z, 2.0 2021-06-24T00:00:00Z, 3.0 -3) All optionals turned on, with metadata-as-comments turned on: +3. All optionals turned on, with metadata-as-comments turned on: # metadata-count: 5 # time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI # office-id: SWT @@ -63,14 +66,14 @@ date-time, value 2021-06-23T00:00:00Z, 2.0 2021-06-24T00:00:00Z, 3.0 -4) All optionals turned on, with metadata-as-comments not turned on: +4. All optionals turned on, with metadata-as-comments not turned on: time-series-id, office-id, date-time, value, units, version-date, quality-code ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 -(Note that if we go with option 2 for units, then the units column would not be included in this example in 4), instead it would be included in the value column header) +(Note that if we go with option 2 for units, then the units column would not be included in this example in 4., instead it would be included in the value column header) Decision Status =============== From 59611f945d0bcb88f4b382ccaf27c878dd8428d3 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:31:39 -0700 Subject: [PATCH 05/22] Updated csv to be in code block --- .../decisions/0008-timeseries-csv-format.rst | 64 +++++++++++-------- 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 253ee4bc4..c5b0d1929 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -40,38 +40,50 @@ Example CSVs ~~~~~~~~~~~~ 1. All optionals turned off, and no metadata comments: -date-time, value, units -2021-06-21T00:00:00Z, 0.0, ft -2021-06-22T00:00:00Z, 1.0, ft -2021-06-23T00:00:00Z, 2.0, ft -2021-06-24T00:00:00Z, 3.0, ft + + .. code-block:: text + + date-time, value, units + 2021-06-21T00:00:00Z, 0.0, ft + 2021-06-22T00:00:00Z, 1.0, ft + 2021-06-23T00:00:00Z, 2.0, ft + 2021-06-24T00:00:00Z, 3.0, ft 2. All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: -date-time, value (ft) -2021-06-21T00:00:00Z, 0.0 -2021-06-22T00:00:00Z, 1.0 -2021-06-23T00:00:00Z, 2.0 -2021-06-24T00:00:00Z, 3.0 + + .. code-block:: text + + date-time, value (ft) + 2021-06-21T00:00:00Z, 0.0 + 2021-06-22T00:00:00Z, 1.0 + 2021-06-23T00:00:00Z, 2.0 + 2021-06-24T00:00:00Z, 3.0 3. All optionals turned on, with metadata-as-comments turned on: -# metadata-count: 5 -# time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI -# office-id: SWT -# version-date: 2021-06-21T00:00:00Z -# quality-code: 1 -# units: ft -date-time, value -2021-06-21T00:00:00Z, 0.0 -2021-06-22T00:00:00Z, 1.0 -2021-06-23T00:00:00Z, 2.0 -2021-06-24T00:00:00Z, 3.0 + + .. code-block:: text + + # metadata-count: 5 + # time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI + # office-id: SWT + # version-date: 2021-06-21T00:00:00Z + # quality-code: 1 + # units: ft + date-time, value + 2021-06-21T00:00:00Z, 0.0 + 2021-06-22T00:00:00Z, 1.0 + 2021-06-23T00:00:00Z, 2.0 + 2021-06-24T00:00:00Z, 3.0 4. All optionals turned on, with metadata-as-comments not turned on: -time-series-id, office-id, date-time, value, units, version-date, quality-code -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 + + .. code-block:: text + + time-series-id, office-id, date-time, value, units, version-date, quality-code + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 (Note that if we go with option 2 for units, then the units column would not be included in this example in 4., instead it would be included in the value column header) From 2e47552726595551371334a9d6263b996e7c04f6 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:37:36 -0700 Subject: [PATCH 06/22] Removed option to include units in headers. While possible to achieve, it breaks the standard of column to field naming used by jackson. --- .../decisions/0008-timeseries-csv-format.rst | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index c5b0d1929..4ddc054bc 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -49,17 +49,7 @@ Example CSVs 2021-06-23T00:00:00Z, 2.0, ft 2021-06-24T00:00:00Z, 3.0, ft -2. All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: - - .. code-block:: text - - date-time, value (ft) - 2021-06-21T00:00:00Z, 0.0 - 2021-06-22T00:00:00Z, 1.0 - 2021-06-23T00:00:00Z, 2.0 - 2021-06-24T00:00:00Z, 3.0 - -3. All optionals turned on, with metadata-as-comments turned on: +2. All optionals turned on, with metadata-as-comments turned on: .. code-block:: text @@ -75,7 +65,7 @@ Example CSVs 2021-06-23T00:00:00Z, 2.0 2021-06-24T00:00:00Z, 3.0 -4. All optionals turned on, with metadata-as-comments not turned on: +3. All optionals turned on, with metadata-as-comments not turned on: .. code-block:: text @@ -85,8 +75,6 @@ Example CSVs ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 -(Note that if we go with option 2 for units, then the units column would not be included in this example in 4., instead it would be included in the value column header) - Decision Status =============== From 458186635c5712605b27e8e0af258f1ac335f7e9 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:40:18 -0700 Subject: [PATCH 07/22] Updated to not include units comment if optionals turned off --- docs/source/decisions/0008-timeseries-csv-format.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 4ddc054bc..1aab6a4bf 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -43,11 +43,11 @@ Example CSVs .. code-block:: text - date-time, value, units - 2021-06-21T00:00:00Z, 0.0, ft - 2021-06-22T00:00:00Z, 1.0, ft - 2021-06-23T00:00:00Z, 2.0, ft - 2021-06-24T00:00:00Z, 3.0, ft + date-time, value + 2021-06-21T00:00:00Z, 0.0 + 2021-06-22T00:00:00Z, 1.0 + 2021-06-23T00:00:00Z, 2.0 + 2021-06-24T00:00:00Z, 3.0 2. All optionals turned on, with metadata-as-comments turned on: From dc5ec4aecaa851468c1ac4b6d55789978d42c857 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Thu, 19 Mar 2026 15:17:22 -0700 Subject: [PATCH 08/22] Updated key points to reflect examples better. --- docs/source/decisions/0008-timeseries-csv-format.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 1aab6a4bf..5bf16cadc 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -24,8 +24,8 @@ Since the intended use of the CSV format is for retrieval only, a customized for Key points ~~~~~~~~~~ -- CSV will be serialized via the jackson api, with either a custom csv DTO, or csv annotations on the existing DTO on specific fields to include. -- Timeseries CSV will include the date-time, value, and units fields as well as optionally the time-series id, office, quality, and version-date. +- CSV will be serialized via the jackson api +- Timeseries CSV will include the date-time and value fields as well as optionally the time-series id, office, quality, units, and version-date. - By default, optional fields will not be included. - A flag to include optional fields as metadata comments at the top of the file may be added. This would be a commented line indicating the amount of metadata followed by commented metadata rows using key:value pairs. See example below. - Column names use kebab-case for consistency with JSON and XML. From 00a7c248479946926b6c137d6512f8edc501c2af Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 25 Mar 2026 13:31:27 -0700 Subject: [PATCH 09/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 5bf16cadc..0e3ea6c7d 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -24,7 +24,7 @@ Since the intended use of the CSV format is for retrieval only, a customized for Key points ~~~~~~~~~~ -- CSV will be serialized via the jackson api +- CSV will be serialized via the jackson api for consistency with JSON and XML serialization. - Timeseries CSV will include the date-time and value fields as well as optionally the time-series id, office, quality, units, and version-date. - By default, optional fields will not be included. - A flag to include optional fields as metadata comments at the top of the file may be added. This would be a commented line indicating the amount of metadata followed by commented metadata rows using key:value pairs. See example below. From 63531823ba6545c7d11911eb7261fb1aaa6e734a Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Wed, 25 Mar 2026 13:33:14 -0700 Subject: [PATCH 10/22] CDA-74 - ADR revisions based on feedback --- docs/source/decisions/0008-timeseries-csv-format.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 5bf16cadc..3ea51492c 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -27,14 +27,16 @@ Key points - CSV will be serialized via the jackson api - Timeseries CSV will include the date-time and value fields as well as optionally the time-series id, office, quality, units, and version-date. - By default, optional fields will not be included. +- Comments are indicated by a leading # character. This is not an RFC 4180 standard, but is a common convention already used by some CWMS systems, including the existing office and location-group endpoints which return csv. - A flag to include optional fields as metadata comments at the top of the file may be added. This would be a commented line indicating the amount of metadata followed by commented metadata rows using key:value pairs. See example below. - Column names use kebab-case for consistency with JSON and XML. -- Units will either be included in metadata comments or as a column. Recommend not including this in the value column-header, as column names should match DTO fields. A custom serializer could be created if units need to be included in the value-column-header. +- Units will either be included in metadata comments or as a column. Recommend not including this in the value column-header, as column names should match DTO fields. - date-time values will be serialized as ISO-8601 strings. NOTE this differs from JSON and XML for date-time which are serialized as epoch-millis. - Null values are empty fields. Missing values use quality-code = 5, for consistency with JSON and XML. -- UTF-8 encoding, comma delimiter, LF line endings, header always included. -- One row is produced per Record. -- Multi-retrieve never includes multiple time-series IDs. +- UTF-8 encoding, comma delimiter, LF line endings. Comma-only CSV follows RFC 4180 compliance +- Column headers are always present +- One row is produced per Record. Record is defined as a single date-time and value pair +- A payload never includes multiple time-series IDs. Example CSVs ~~~~~~~~~~~~ From ac9bde9ff2d8f022826648644371347be9a88495 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Thu, 26 Mar 2026 12:54:16 -0700 Subject: [PATCH 11/22] CDA-74 - Updated decisions to use a table with justification --- .../decisions/0008-timeseries-csv-format.rst | 73 +++++++++++++------ 1 file changed, 49 insertions(+), 24 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index a39ffb2b6..3c5d68350 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -24,19 +24,46 @@ Since the intended use of the CSV format is for retrieval only, a customized for Key points ~~~~~~~~~~ -- CSV will be serialized via the jackson api for consistency with JSON and XML serialization. -- Timeseries CSV will include the date-time and value fields as well as optionally the time-series id, office, quality, units, and version-date. -- By default, optional fields will not be included. -- Comments are indicated by a leading # character. This is not an RFC 4180 standard, but is a common convention already used by some CWMS systems, including the existing office and location-group endpoints which return csv. -- A flag to include optional fields as metadata comments at the top of the file may be added. This would be a commented line indicating the amount of metadata followed by commented metadata rows using key:value pairs. See example below. -- Column names use kebab-case for consistency with JSON and XML. -- Units will either be included in metadata comments or as a column. Recommend not including this in the value column-header, as column names should match DTO fields. -- date-time values will be serialized as ISO-8601 strings. NOTE this differs from JSON and XML for date-time which are serialized as epoch-millis. -- Null values are empty fields. Missing values use quality-code = 5, for consistency with JSON and XML. -- UTF-8 encoding, comma delimiter, LF line endings. Comma-only CSV follows RFC 4180 compliance -- Column headers are always present -- One row is produced per Record. Record is defined as a single date-time and value pair -- A payload never includes multiple time-series IDs. ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Decision | Justification | ++============================+=====================================================================================================================================================================================================================================================================================================================================+ +| Serialization: Jackson API | Maintain consistency with JSON and XML serialization infrastructure and object-mapper settings already used across the API. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Required columns | The following are always present: date-time and value. Units are always included in the value column header as parentheses, e.g., ``value (ft)``. Units must exist in exactly one canonical location in all modes. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Optional columns | The following columns are optional and off by default: time-series-id, office-id, version-date, data-entry-date, quality. Everything except date-time and value (with units in the header) is optional. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Meta-data fields | The following fields are considered metadata and may be included at the top of the payload, rather than as columns: time-series-id, office-id, version-date. Meta-data fields are optional and are turned off by default. This is useful if we want metadata shown but don't want repeated data in rows. | +| | These fields (if included via an accept-header param ``metadata-format=comment``), would start with a line indicating how many metadata comment lines follow - to aid in parsing (e.g.``#metadata-count: 3``) | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Units location | Units are always expressed in the value column header via parentheses (e.g., ``value (m^3/s)``) and are not repeated elsewhere. | +| | We do not include units as a separate column nor in metadata comments. We avoid the anti-pattern of dual representation of the same semantic field; units must exist in exactly one canonical location in all modes. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Version-date encoding | Serialized in the CSV column as: ``base`` for the special value 1111-11-11T11:11, ``aggregate`` for aggregate versions, an ISO-8601 timestamp for actual version dates, and omitted entirely if unversioned. | +| | This requires custom serialization handling. It matches CWMS-VUE behavior. Alternative (a separate CSV column per case) is not adopted because there is no compelling use-case and it would bloat the schema. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Column headers | Column headers are always present. RFC 4180 allows headers and doing so keeps the format scalable if additional optional columns are introduced later; clients do not have to rely on fixed column indices. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Comments | Lines beginning with ``#`` are treated as comments. While not part of RFC 4180, this convention is already used by CWMS endpoints (e.g., office and location-group) that return CSV, and it provides backward compatibility and human readability. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Column naming | Kebab-case column names for consistency with JSON and XML. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Accept Header for format | Default CSV serialization uses ISO-8601 strings. Clients may request alternate formats via the HTTP Accept header parameter ``date-format``, e.g.: | +| and columns | - ``text/csv;date-format=ISO8601-Instant`` (default) | +| | - ``text/csv;date-format=epoch-millis`` | +| | Use of Accept header parameters may be included to turn off/on optional quality and data-entry-date columns(e.g. ``quality=provided``, ``data-entry-date=provided``). | +| | If these were query params instead, that would allow for easier toggling of columns within the browser. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Quality representation | ``quality`` (aka quality-code) is an optional integer column. A bitmask (integer) is preferred over a byte[] because it compactly represents multiple boolean flags in a single scalar value with fast, native bitwise operations, whereas a byte[] adds overhead without improving expressiveness for fixed flag sets. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Nulls and missing values | Null field values are rendered as empty CSV fields. Missing values use quality-code = 5 for consistency with JSON and XML. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Encoding and delimiters | UTF-8, comma delimiter, LF line endings. Comma-only CSV follows RFC 4180 compliance. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Record structure | One row per record, where a record is defined as a single date-time and value pair with quality and data-entry-date optionally included as columns | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Single TS per payload | A payload never includes multiple time-series IDs. | ++----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Example CSVs ~~~~~~~~~~~~ @@ -45,7 +72,7 @@ Example CSVs .. code-block:: text - date-time, value + date-time, value (cfs) 2021-06-21T00:00:00Z, 0.0 2021-06-22T00:00:00Z, 1.0 2021-06-23T00:00:00Z, 2.0 @@ -55,13 +82,11 @@ Example CSVs .. code-block:: text - # metadata-count: 5 + # metadata-count: 3 # time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI # office-id: SWT - # version-date: 2021-06-21T00:00:00Z - # quality-code: 1 - # units: ft - date-time, value + # version-date: aggregate + date-time, value (cfs) 2021-06-21T00:00:00Z, 0.0 2021-06-22T00:00:00Z, 1.0 2021-06-23T00:00:00Z, 2.0 @@ -71,11 +96,11 @@ Example CSVs .. code-block:: text - time-series-id, office-id, date-time, value, units, version-date, quality-code - ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 - ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 - ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 - ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 + time-series-id, office-id, date-time, value (cfs), version-date, data-entry-date, quality-code + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, aggregate, 2021-06-21T00:05:00Z, 5 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, aggregate, 2021-06-22T00:05:00Z, 5 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, aggregate, 2021-06-23T00:05:00Z, 5 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, aggregate, 2021-06-24T00:05:00Z, 5 Decision Status =============== From 8439daffa2936a4adb7bf5caab469970ba8935e0 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Mon, 30 Mar 2026 10:54:46 -0700 Subject: [PATCH 12/22] CDA-74 - Updates to use list-table --- .../decisions/0008-timeseries-csv-format.rst | 81 ++++++++++--------- 1 file changed, 41 insertions(+), 40 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 3c5d68350..fc8ce6275 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -24,46 +24,47 @@ Since the intended use of the CSV format is for retrieval only, a customized for Key points ~~~~~~~~~~ -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Decision | Justification | -+============================+=====================================================================================================================================================================================================================================================================================================================================+ -| Serialization: Jackson API | Maintain consistency with JSON and XML serialization infrastructure and object-mapper settings already used across the API. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Required columns | The following are always present: date-time and value. Units are always included in the value column header as parentheses, e.g., ``value (ft)``. Units must exist in exactly one canonical location in all modes. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Optional columns | The following columns are optional and off by default: time-series-id, office-id, version-date, data-entry-date, quality. Everything except date-time and value (with units in the header) is optional. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Meta-data fields | The following fields are considered metadata and may be included at the top of the payload, rather than as columns: time-series-id, office-id, version-date. Meta-data fields are optional and are turned off by default. This is useful if we want metadata shown but don't want repeated data in rows. | -| | These fields (if included via an accept-header param ``metadata-format=comment``), would start with a line indicating how many metadata comment lines follow - to aid in parsing (e.g.``#metadata-count: 3``) | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Units location | Units are always expressed in the value column header via parentheses (e.g., ``value (m^3/s)``) and are not repeated elsewhere. | -| | We do not include units as a separate column nor in metadata comments. We avoid the anti-pattern of dual representation of the same semantic field; units must exist in exactly one canonical location in all modes. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Version-date encoding | Serialized in the CSV column as: ``base`` for the special value 1111-11-11T11:11, ``aggregate`` for aggregate versions, an ISO-8601 timestamp for actual version dates, and omitted entirely if unversioned. | -| | This requires custom serialization handling. It matches CWMS-VUE behavior. Alternative (a separate CSV column per case) is not adopted because there is no compelling use-case and it would bloat the schema. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Column headers | Column headers are always present. RFC 4180 allows headers and doing so keeps the format scalable if additional optional columns are introduced later; clients do not have to rely on fixed column indices. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Comments | Lines beginning with ``#`` are treated as comments. While not part of RFC 4180, this convention is already used by CWMS endpoints (e.g., office and location-group) that return CSV, and it provides backward compatibility and human readability. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Column naming | Kebab-case column names for consistency with JSON and XML. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Accept Header for format | Default CSV serialization uses ISO-8601 strings. Clients may request alternate formats via the HTTP Accept header parameter ``date-format``, e.g.: | -| and columns | - ``text/csv;date-format=ISO8601-Instant`` (default) | -| | - ``text/csv;date-format=epoch-millis`` | -| | Use of Accept header parameters may be included to turn off/on optional quality and data-entry-date columns(e.g. ``quality=provided``, ``data-entry-date=provided``). | -| | If these were query params instead, that would allow for easier toggling of columns within the browser. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Quality representation | ``quality`` (aka quality-code) is an optional integer column. A bitmask (integer) is preferred over a byte[] because it compactly represents multiple boolean flags in a single scalar value with fast, native bitwise operations, whereas a byte[] adds overhead without improving expressiveness for fixed flag sets. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Nulls and missing values | Null field values are rendered as empty CSV fields. Missing values use quality-code = 5 for consistency with JSON and XML. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Encoding and delimiters | UTF-8, comma delimiter, LF line endings. Comma-only CSV follows RFC 4180 compliance. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Record structure | One row per record, where a record is defined as a single date-time and value pair with quality and data-entry-date optionally included as columns | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Single TS per payload | A payload never includes multiple time-series IDs. | -+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Decision + - Justification + * - Serialization: Jackson API + - Maintain consistency with JSON and XML serialization infrastructure and object-mapper settings already used across the API. + * - Required columns + - The following are always present: date-time and value. Units are always included in the value column header as parentheses, e.g., ``value (ft)``. Units must exist in exactly one canonical location in all modes. + * - Optional columns + - The following columns are optional and off by default: time-series-id, office-id, version-date, data-entry-date, quality. Everything except date-time and value (with units in the header) is optional. + * - Meta-data fields + - The following fields are considered metadata and may be included at the top of the payload, rather than as columns: time-series-id, office-id, version-date. Meta-data fields are optional and are turned off by default. This is useful if we want metadata shown but don't want repeated data in rows. If included via an Accept header parameter ``metadata-format=comment``, the payload starts with a line indicating how many metadata comment lines follow to aid parsing (e.g., ``# metadata-count: 3``). + * - Units location + - Units are always expressed in the value column header via parentheses (e.g., ``value (m^3/s)``) and are not repeated elsewhere. We do not include units as a separate column nor in metadata comments. We avoid the anti-pattern of dual representation of the same semantic field; units must exist in exactly one canonical location in all modes. + * - Version-date encoding + - Serialized in the CSV column as: ``base`` for the special value 1111-11-11T11:11, ``aggregate`` for aggregate versions, an ISO-8601 timestamp for actual version dates, and omitted entirely if unversioned. This requires custom serialization handling and matches CWMS-VUE behavior. The alternative (a separate CSV column per case) is not adopted because there is no compelling use-case and it would bloat the schema. + * - Column headers + - Column headers are always present. RFC 4180 allows headers and doing so keeps the format scalable if additional optional columns are introduced later; clients do not have to rely on fixed column indices. + * - Comments + - Lines beginning with ``#`` are treated as comments. While not part of RFC 4180, this convention is already used by CWMS endpoints (e.g., office and location-group) that return CSV, and it provides backward compatibility and human readability. + * - Column naming + - Kebab-case column names for consistency with JSON and XML. + * - Accept header for format and columns + - Default CSV serialization uses ISO-8601 strings. Clients may request alternate formats via the HTTP Accept header parameter ``date-format``, for example: + + - ``text/csv;date-format=ISO8601-Instant`` (default) + - ``text/csv;date-format=epoch-millis`` + + Use Accept header parameters to turn on optional columns as needed (e.g., ``quality=provided``, ``data-entry-date=provided``). If these were query params instead, it would allow easier toggling of columns within the browser. + * - Quality representation + - ``quality`` (aka quality-code) is an optional integer column. A bitmask (integer) is preferred over a byte[] because it compactly represents multiple boolean flags in a single scalar value with fast, native bitwise operations, whereas a byte[] adds overhead without improving expressiveness for fixed flag sets. + * - Nulls and missing values + - Null field values are rendered as empty CSV fields. Missing values use quality-code = 5 for consistency with JSON and XML. + * - Encoding and delimiters + - UTF-8, comma delimiter, LF line endings. Comma-only CSV follows RFC 4180 compliance. + * - Record structure + - One row per record, where a record is defined as a single date-time and value pair with quality and data-entry-date optionally included as columns. + * - Single TS per payload + - A payload never includes multiple time-series IDs. Example CSVs ~~~~~~~~~~~~ From d982b4a30a0e19b5bb9e15437c8823f52b968a6a Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Mon, 30 Mar 2026 11:15:16 -0700 Subject: [PATCH 13/22] CDA-74 - Adds note about why headers are always included and clients not relying on indices for optional columns --- docs/source/decisions/0008-timeseries-csv-format.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index fc8ce6275..4d2ed8a61 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -35,11 +35,11 @@ Key points * - Required columns - The following are always present: date-time and value. Units are always included in the value column header as parentheses, e.g., ``value (ft)``. Units must exist in exactly one canonical location in all modes. * - Optional columns - - The following columns are optional and off by default: time-series-id, office-id, version-date, data-entry-date, quality. Everything except date-time and value (with units in the header) is optional. + - The following columns are optional and off by default: time-series-id, office-id, version-date, data-entry-date, quality. Everything except date-time and value (with units in the header) is optional. Because headers are always included, optional columns can be easily toggled on and off without breaking parsing logic. Clients should rely on column names, not indices, to access fields. * - Meta-data fields - The following fields are considered metadata and may be included at the top of the payload, rather than as columns: time-series-id, office-id, version-date. Meta-data fields are optional and are turned off by default. This is useful if we want metadata shown but don't want repeated data in rows. If included via an Accept header parameter ``metadata-format=comment``, the payload starts with a line indicating how many metadata comment lines follow to aid parsing (e.g., ``# metadata-count: 3``). * - Units location - - Units are always expressed in the value column header via parentheses (e.g., ``value (m^3/s)``) and are not repeated elsewhere. We do not include units as a separate column nor in metadata comments. We avoid the anti-pattern of dual representation of the same semantic field; units must exist in exactly one canonical location in all modes. + - Units are always expressed in the value column header via parentheses (e.g., ``value (m^3/s)``) and are not repeated elsewhere. We do not include units as a separate column nor in metadata comments. We avoid the anti-pattern of dual representation of the same semantic field; units must exist in exactly one canonical location in all modes. Note this will require custom deserialization handling to extract units from the header, but it is worth it to avoid the bloat and confusion of multiple unit representations. * - Version-date encoding - Serialized in the CSV column as: ``base`` for the special value 1111-11-11T11:11, ``aggregate`` for aggregate versions, an ISO-8601 timestamp for actual version dates, and omitted entirely if unversioned. This requires custom serialization handling and matches CWMS-VUE behavior. The alternative (a separate CSV column per case) is not adopted because there is no compelling use-case and it would bloat the schema. * - Column headers From b7399ac1f9fb5dd3640473a43a187924a7d7ea0d Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 31 Mar 2026 12:51:13 -0700 Subject: [PATCH 14/22] CDA-74 - Splits up table to actual decision made --- .../decisions/0008-timeseries-csv-format.rst | 57 +++++++++++-------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 4d2ed8a61..fc96d13bc 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -26,45 +26,56 @@ Key points .. list-table:: :header-rows: 1 - :widths: 25 75 + :widths: 20 25 55 - * - Decision + * - Topic + - Decision - Justification - * - Serialization: Jackson API + * - Serialization + - We will utilize the Jackson API - Maintain consistency with JSON and XML serialization infrastructure and object-mapper settings already used across the API. * - Required columns - - The following are always present: date-time and value. Units are always included in the value column header as parentheses, e.g., ``value (ft)``. Units must exist in exactly one canonical location in all modes. + - Always include ``date-time`` and ``value``; include units in the value column header as parentheses (e.g., ``value (ft)``) + - Units must exist in exactly one canonical location in all modes. * - Optional columns - - The following columns are optional and off by default: time-series-id, office-id, version-date, data-entry-date, quality. Everything except date-time and value (with units in the header) is optional. Because headers are always included, optional columns can be easily toggled on and off without breaking parsing logic. Clients should rely on column names, not indices, to access fields. - * - Meta-data fields - - The following fields are considered metadata and may be included at the top of the payload, rather than as columns: time-series-id, office-id, version-date. Meta-data fields are optional and are turned off by default. This is useful if we want metadata shown but don't want repeated data in rows. If included via an Accept header parameter ``metadata-format=comment``, the payload starts with a line indicating how many metadata comment lines follow to aid parsing (e.g., ``# metadata-count: 3``). + - Optional (off by default): ``time-series-id``, ``office-id``, ``version-date``, ``data-entry-date``, ``quality`` + - Everything except ``date-time`` and ``value`` (with units in the header) is optional. Because headers are always included, optional columns can be toggled without breaking parsing. Clients should rely on column names, not indices. + * - Metadata fields + - May be emitted as top-of-payload comments (``metadata-format=comment``) or as actual columns (``metadata-format=column``) + - The following fields can be treated as metadata comments at top-of-payload rather than columns: ``time-series-id``, ``office-id``, ``version-date``. These are optional (off by default). When included as comments, the payload starts with a line indicating count (e.g., ``# metadata-count: 3``) to aid parsing. * - Units location - - Units are always expressed in the value column header via parentheses (e.g., ``value (m^3/s)``) and are not repeated elsewhere. We do not include units as a separate column nor in metadata comments. We avoid the anti-pattern of dual representation of the same semantic field; units must exist in exactly one canonical location in all modes. Note this will require custom deserialization handling to extract units from the header, but it is worth it to avoid the bloat and confusion of multiple unit representations. + - Express units only in the value column header via parentheses (e.g., ``value (cfs)``) + - Do not include units as a separate column or in metadata comments. This avoids the anti-pattern of dual representation; units live in exactly one canonical location. Custom deserialization may be required to extract units from the header, which is preferable to duplicate representations. * - Version-date encoding - - Serialized in the CSV column as: ``base`` for the special value 1111-11-11T11:11, ``aggregate`` for aggregate versions, an ISO-8601 timestamp for actual version dates, and omitted entirely if unversioned. This requires custom serialization handling and matches CWMS-VUE behavior. The alternative (a separate CSV column per case) is not adopted because there is no compelling use-case and it would bloat the schema. + - Use ``base`` for 1111-11-11T11:11, ``aggregate`` for aggregate versions, ISO-8601 timestamp for actual version dates, and omit the field if unversioned + - Matches CWMS-VUE behavior. A separate CSV column per case was rejected due to lack of use-cases and schema bloat. Note this requires custom serialization handling. * - Column headers - - Column headers are always present. RFC 4180 allows headers and doing so keeps the format scalable if additional optional columns are introduced later; clients do not have to rely on fixed column indices. + - Always include headers + - RFC 4180 allows headers; including them keeps the format scalable if optional columns are introduced later and prevents reliance on fixed column indices. We will include a header param of ``headers=present`` in the Accept header to explicitly indicate that headers are included, even though they will always be present. This allows for future flexibility if we ever need to emit headerless CSV for some reason. * - Comments - - Lines beginning with ``#`` are treated as comments. While not part of RFC 4180, this convention is already used by CWMS endpoints (e.g., office and location-group) that return CSV, and it provides backward compatibility and human readability. + - Treat lines beginning with ``#`` as comments + - While not part of RFC 4180, this convention is already used by CWMS endpoints (e.g., office and location-group) that return CSV, and is human-readable. * - Column naming - - Kebab-case column names for consistency with JSON and XML. + - Kebab-case names + - Keeps naming consistent with JSON and XML. * - Accept header for format and columns - - Default CSV serialization uses ISO-8601 strings. Clients may request alternate formats via the HTTP Accept header parameter ``date-format``, for example: - - - ``text/csv;date-format=ISO8601-Instant`` (default) - - ``text/csv;date-format=epoch-millis`` - - Use Accept header parameters to turn on optional columns as needed (e.g., ``quality=provided``, ``data-entry-date=provided``). If these were query params instead, it would allow easier toggling of columns within the browser. + - Use HTTP Accept header parameters to select date format and optional columns + - Default CSV serialization uses ISO-8601 strings. Examples: ``text/csv;date-format=ISO8601-Instant`` (default), ``text/csv;date-format=epoch-millis``. Use Accept header parameters to enable optional columns (e.g., ``quality=present``, ``data-entry-date=present``). If these were query params instead, toggling would be easier in a browser, but Accept keeps content negotiation consistent. * - Quality representation - - ``quality`` (aka quality-code) is an optional integer column. A bitmask (integer) is preferred over a byte[] because it compactly represents multiple boolean flags in a single scalar value with fast, native bitwise operations, whereas a byte[] adds overhead without improving expressiveness for fixed flag sets. + - ``quality`` (aka quality-code) is an optional integer bitmask + - A bitmask (integer) compactly represents multiple boolean flags with fast native bitwise operations; a ``byte[]`` adds overhead without improving expressiveness for fixed flag sets. * - Nulls and missing values - - Null field values are rendered as empty CSV fields. Missing values use quality-code = 5 for consistency with JSON and XML. + - Render nulls as empty fields; use ``quality-code = 5`` for missing values + - Keeps behavior consistent with JSON and XML. * - Encoding and delimiters - - UTF-8, comma delimiter, LF line endings. Comma-only CSV follows RFC 4180 compliance. + - UTF-8, comma delimiter, LF line endings + - Comma-only CSV follows RFC 4180 compliance. * - Record structure - - One row per record, where a record is defined as a single date-time and value pair with quality and data-entry-date optionally included as columns. + - One row per record + - A record is a single date-time and value pair; ``quality-code`` and ``data-entry-date`` may be included as optional columns. * - Single TS per payload - - A payload never includes multiple time-series IDs. + - Do not mix multiple time-series IDs in one payload + - Ensures a payload represents exactly one time-series. Example CSVs ~~~~~~~~~~~~ From a893c890cc53d2174425b1167ca8fd9952efb2fa Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 1 Apr 2026 08:27:18 -0700 Subject: [PATCH 15/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index fc96d13bc..518354cba 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -36,7 +36,7 @@ Key points - Maintain consistency with JSON and XML serialization infrastructure and object-mapper settings already used across the API. * - Required columns - Always include ``date-time`` and ``value``; include units in the value column header as parentheses (e.g., ``value (ft)``) - - Units must exist in exactly one canonical location in all modes. + - Units should exist in exactly one canonical location in all modes. Conditionally adding them as metadata comments will cause confusion over the inconsistency * - Optional columns - Optional (off by default): ``time-series-id``, ``office-id``, ``version-date``, ``data-entry-date``, ``quality`` - Everything except ``date-time`` and ``value`` (with units in the header) is optional. Because headers are always included, optional columns can be toggled without breaking parsing. Clients should rely on column names, not indices. From c0c8c229a424d659400b76b0667c6c5d6b3354b5 Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 1 Apr 2026 08:30:55 -0700 Subject: [PATCH 16/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 518354cba..3d0d7b581 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -39,7 +39,7 @@ Key points - Units should exist in exactly one canonical location in all modes. Conditionally adding them as metadata comments will cause confusion over the inconsistency * - Optional columns - Optional (off by default): ``time-series-id``, ``office-id``, ``version-date``, ``data-entry-date``, ``quality`` - - Everything except ``date-time`` and ``value`` (with units in the header) is optional. Because headers are always included, optional columns can be toggled without breaking parsing. Clients should rely on column names, not indices. + - Everything except ``date-time`` and ``value`` (with units in the header) is optional. Because headers are always included, optional columns can be toggled without breaking parsing. Clients should rely on column names, not indices. Given units are in the `value` header, clients will need to handle this appropriately to determine the correct column index. * - Metadata fields - May be emitted as top-of-payload comments (``metadata-format=comment``) or as actual columns (``metadata-format=column``) - The following fields can be treated as metadata comments at top-of-payload rather than columns: ``time-series-id``, ``office-id``, ``version-date``. These are optional (off by default). When included as comments, the payload starts with a line indicating count (e.g., ``# metadata-count: 3``) to aid parsing. From 2962df48b5d812da751c16522e3037d501315bf0 Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 1 Apr 2026 10:12:46 -0700 Subject: [PATCH 17/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 3d0d7b581..90df01e51 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -65,7 +65,7 @@ Key points - ``quality`` (aka quality-code) is an optional integer bitmask - A bitmask (integer) compactly represents multiple boolean flags with fast native bitwise operations; a ``byte[]`` adds overhead without improving expressiveness for fixed flag sets. * - Nulls and missing values - - Render nulls as empty fields; use ``quality-code = 5`` for missing values + - Missing values will be represented with an empty value field (null) and will have ``quality-code = 5``. Constants will not be used to represent missing values. - Keeps behavior consistent with JSON and XML. * - Encoding and delimiters - UTF-8, comma delimiter, LF line endings From 5e2f45d6645515173ac27f29e6b884b6c55c3009 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Wed, 1 Apr 2026 11:01:40 -0700 Subject: [PATCH 18/22] CDA-74 - removed metadata-count line decision, added version-date to record definition --- docs/source/decisions/0008-timeseries-csv-format.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 90df01e51..74ea32c50 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -41,8 +41,8 @@ Key points - Optional (off by default): ``time-series-id``, ``office-id``, ``version-date``, ``data-entry-date``, ``quality`` - Everything except ``date-time`` and ``value`` (with units in the header) is optional. Because headers are always included, optional columns can be toggled without breaking parsing. Clients should rely on column names, not indices. Given units are in the `value` header, clients will need to handle this appropriately to determine the correct column index. * - Metadata fields - - May be emitted as top-of-payload comments (``metadata-format=comment``) or as actual columns (``metadata-format=column``) - - The following fields can be treated as metadata comments at top-of-payload rather than columns: ``time-series-id``, ``office-id``, ``version-date``. These are optional (off by default). When included as comments, the payload starts with a line indicating count (e.g., ``# metadata-count: 3``) to aid parsing. + - May be emitted as top-of-payload comments (``metadata-format=comments``) or as actual columns (``metadata-format=columns``) + - The following fields can be treated as metadata comments at top-of-payload rather than columns: ``time-series-id``, ``office-id``, ``version-date``. These are optional (off by default). It is assumed that the only comments in the payload will be metadata comments, and as such, clients can parse out metadata by reading comment lines until the first non-comment line is reached. * - Units location - Express units only in the value column header via parentheses (e.g., ``value (cfs)``) - Do not include units as a separate column or in metadata comments. This avoids the anti-pattern of dual representation; units live in exactly one canonical location. Custom deserialization may be required to extract units from the header, which is preferable to duplicate representations. @@ -72,7 +72,7 @@ Key points - Comma-only CSV follows RFC 4180 compliance. * - Record structure - One row per record - - A record is a single date-time and value pair; ``quality-code`` and ``data-entry-date`` may be included as optional columns. + - A record is a single date-time and value pair; ``quality-code`` and ``data-entry-date`` may be included as optional columns. ``version-date`` is also an attribute of the record, but is not expected to vary within a payload and can be treated as metadata. * - Single TS per payload - Do not mix multiple time-series IDs in one payload - Ensures a payload represents exactly one time-series. @@ -94,7 +94,6 @@ Example CSVs .. code-block:: text - # metadata-count: 3 # time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI # office-id: SWT # version-date: aggregate From 14f439b2dc1998206fdc184f6785aedfdeb0ba44 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Wed, 1 Apr 2026 11:51:37 -0700 Subject: [PATCH 19/22] CDA-74 - Removed serialization api decision as it doesn't need to be made at this level --- docs/source/decisions/0008-timeseries-csv-format.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 74ea32c50..e5f76162e 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -31,9 +31,6 @@ Key points * - Topic - Decision - Justification - * - Serialization - - We will utilize the Jackson API - - Maintain consistency with JSON and XML serialization infrastructure and object-mapper settings already used across the API. * - Required columns - Always include ``date-time`` and ``value``; include units in the value column header as parentheses (e.g., ``value (ft)``) - Units should exist in exactly one canonical location in all modes. Conditionally adding them as metadata comments will cause confusion over the inconsistency From c3f964ec1faa2992e3d4c7629b470a8cbbffba3f Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 1 Apr 2026 11:52:35 -0700 Subject: [PATCH 20/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index e5f76162e..a7b050c57 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -48,7 +48,7 @@ Key points - Matches CWMS-VUE behavior. A separate CSV column per case was rejected due to lack of use-cases and schema bloat. Note this requires custom serialization handling. * - Column headers - Always include headers - - RFC 4180 allows headers; including them keeps the format scalable if optional columns are introduced later and prevents reliance on fixed column indices. We will include a header param of ``headers=present`` in the Accept header to explicitly indicate that headers are included, even though they will always be present. This allows for future flexibility if we ever need to emit headerless CSV for some reason. + - RFC 4180 allows headers; including them keeps the format scalable if optional columns are introduced later and prevents reliance on fixed column indices. We will include a header param of ``header=present`` in the Accept header to explicitly indicate that headers are included, even though they will always be present. This allows for future flexibility if we ever need to emit headerless CSV for some reason. * - Comments - Treat lines beginning with ``#`` as comments - While not part of RFC 4180, this convention is already used by CWMS endpoints (e.g., office and location-group) that return CSV, and is human-readable. From 27a08d5ce77d2e99dba099dcf63471c64b6ca589 Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 1 Apr 2026 11:52:55 -0700 Subject: [PATCH 21/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index a7b050c57..4ed53f3d6 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -66,7 +66,7 @@ Key points - Keeps behavior consistent with JSON and XML. * - Encoding and delimiters - UTF-8, comma delimiter, LF line endings - - Comma-only CSV follows RFC 4180 compliance. + - Comma-only CSV follows RFC 4180 compliance. Tab/Pipe/semicolon delimiters will not be supported. * - Record structure - One row per record - A record is a single date-time and value pair; ``quality-code`` and ``data-entry-date`` may be included as optional columns. ``version-date`` is also an attribute of the record, but is not expected to vary within a payload and can be treated as metadata. From 90815a0744b4f8e11da9f83d4344dccff903c185 Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Wed, 1 Apr 2026 11:54:02 -0700 Subject: [PATCH 22/22] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 4ed53f3d6..43ca18db8 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -69,7 +69,7 @@ Key points - Comma-only CSV follows RFC 4180 compliance. Tab/Pipe/semicolon delimiters will not be supported. * - Record structure - One row per record - - A record is a single date-time and value pair; ``quality-code`` and ``data-entry-date`` may be included as optional columns. ``version-date`` is also an attribute of the record, but is not expected to vary within a payload and can be treated as metadata. + - A record is a single date-time and value pair; ``quality-code`` and ``data-entry-date`` may be included as optional columns. ``version-date`` is also an attribute of the record but is covered under the optional metadata comments. * - Single TS per payload - Do not mix multiple time-series IDs in one payload - Ensures a payload represents exactly one time-series.