Amend specification so that dap4 data responses with checksums are identifiable as such

[ndp-opendap]

If a client asks a dap4 server for a response with checksums using the dap4.checksum query parameter the requesting client can then "know" to process the checksums when deserializing the response.

However, if the requesting client asks for checksums and then write the returned dap4 data response to a file there is nothing in that file to indicate that checksums are present. That is a problem!

I think we should amend the specification to so that a bit in the preamble byte(s) of the dap4 data response is utilized to indicate that the response contains checksum information.

[Mikejmnez]

I also think that it should be absolutely clear which version of the server (tds/hyrax) supports this behavior. So for backwards compatibility, the client should assume that, if not extra query parameter is added to the url :

tds: by default does not return checksums

hyrax: by default, returns checksums

[DennisHeimbigner]

I support this proposal of adding a flag in the leading chunk header.

But for completeness, and if I recall correctly, when checksumming is enabled, then the dmr should contain a checksum attribute. So if that attribute
is missing, it should mean that checksum is off. I think I actually tried to implement this, but it was too painful, so I abandoned it.

[ndp-opendap]

But for completeness, and if I recall correctly, when checksumming is enabled, then the dmr should contain a checksum attribute. So if that attribute
is missing, it should mean that checksum is off. I think I actually tried to implement this, but it was too painful, so I abandoned it.

I believe that! I think to require that the dap4 data response dmr contain the checksum values as dap attributes implies that the service must do the checksumming and then add those values to the head of the response. I think this would break the ability to stream dap4 data responses and force the result to be completely constructed on the server before transmission begins. Much like netcdf.

[jhrg]

In https://opendap.github.io/dap4-specification/DAP4.html#_checksums, the document says the following (my emphasis added):

For a dmr response the checksum is made visible to the client by the addition of an Attribute to each top-level variable in the DMR. This attribute is named “_DAP4_Checksum_CRC32”.

If the request to the server is a data request, then the checksum value will follow the value of the variable in the data part of the response. The computed checksum is appended to the serialized representation for transmission to the client. Note that in this case, the client is expected to add the “_DAP4_Checksum_CRC32” attribute to the DMR.

So, for a data response, the attribute is not added by the server, but by the receiving client. Not perfect, and maybe in need of alteration, but not as bad as, build the entire data response, then finish the DMR.

For the DMR response, there is no need to hold the data response in memory, but the server would have to read the data.

When we make checksums by-request-only, this makes more sense.