Goal
Decide whether COMMAND_PROTOCOL_V1 is base-cli's cross-language record format or a legacy shim,
and either extend it to carry real operational data or position NDJSON as its successor.
Background
command_protocol.py is the one wire format in the package written deliberately for non-Python
readers. The code says so directly (lib/python/base_cli/command_protocol.py:202-206):
The wire framing is LF-delimited. str.splitlines() also accepts CR, vertical tab, form feed, and
Unicode separators, which would make the Python decoder more permissive than the Bash and Zsh readers.
The framing is well designed for shell consumption: line-delimited, hex-encoded string payloads
(so no quoting or delimiter problems), explicit record counts and end markers, canonical integer
parsing, and NUL rejection.
Two problems limit it as a platform surface.
1. Only two value types exist. FieldSpec.value_type accepts "string" or "boolean"
(_validate_and_store_schema(), line 291). There are no integers, floats, timestamps, lists, or
nested structures. Real operational records need exit codes, durations, byte counts, counts,
timestamps, and tag lists. Every one of those must be encoded as a string by the producer and
re-parsed by each consumer, in each language, with no schema-declared type — which defeats the point
of having a typed schema.
2. Decoding is not streaming, with a high cap. loads_records() takes the whole payload as one
str, does payload.split("\n") to materialize every line, and builds a complete list, with
MAX_RECORD_COUNT = 1_000_000. A large record set is fully resident twice. There is no iterator form.
It is also unpositioned: no page under docs/ explains when to use the command protocol versus
NDJSON, and no reader implementation for another language ships with the repository — so the
Bash/Zsh readers the code is tuned for are not actually verifiable here.
Scope
Choose one and document it:
- Extend to v2. Add
integer, number, timestamp (RFC 3339), and a list type, keeping the
hex encoding for strings and the LF framing. Add a streaming iter_records(). Publish a JSON
Schema alongside docs/schemas/v1/command-protocol.schema.json and a reference reader in at
least one other language (Bash is the natural choice given the existing tuning and the sibling
base-bash-libs repository).
- Position NDJSON as the successor. Declare
COMMAND_PROTOCOL_V1 frozen and shell-oriented,
document its exact niche (shell readers that cannot depend on a JSON parser), point everything
else at the NDJSON contract, and add the migration note to docs/output-contracts.md.
Option 2 is cheaper and may well be right — jq is near-universal now — but the current silence
leaves adopters unable to choose.
Acceptance criteria
- One page states which format to use for which consumer, with the trade-off made explicit.
- If option 1: the new types round-trip through Python and at least one non-Python reader, with
golden fixtures validated by both, and iter_records() decodes a 1M-record payload without
materializing it.
- If option 2:
command_protocol is documented as frozen with its niche named, and the
string/boolean limitation is stated rather than implied.
- Either way,
docs/schemas/v1/command-protocol.schema.json and the public docs agree with the code.
Validation
Round-trip fixtures in Python plus one other language, consistent with the existing
scripts/validate_contract_fixtures.py / .mjs pattern.
Non-goals
- Do not break the existing v1 framing for current consumers.
- Do not add a binary encoding.
Goal
Decide whether
COMMAND_PROTOCOL_V1is base-cli's cross-language record format or a legacy shim,and either extend it to carry real operational data or position NDJSON as its successor.
Background
command_protocol.pyis the one wire format in the package written deliberately for non-Pythonreaders. The code says so directly (
lib/python/base_cli/command_protocol.py:202-206):The framing is well designed for shell consumption: line-delimited, hex-encoded string payloads
(so no quoting or delimiter problems), explicit record counts and end markers, canonical integer
parsing, and NUL rejection.
Two problems limit it as a platform surface.
1. Only two value types exist.
FieldSpec.value_typeaccepts"string"or"boolean"(
_validate_and_store_schema(), line 291). There are no integers, floats, timestamps, lists, ornested structures. Real operational records need exit codes, durations, byte counts, counts,
timestamps, and tag lists. Every one of those must be encoded as a string by the producer and
re-parsed by each consumer, in each language, with no schema-declared type — which defeats the point
of having a typed schema.
2. Decoding is not streaming, with a high cap.
loads_records()takes the whole payload as onestr, doespayload.split("\n")to materialize every line, and builds a complete list, withMAX_RECORD_COUNT = 1_000_000. A large record set is fully resident twice. There is no iterator form.It is also unpositioned: no page under
docs/explains when to use the command protocol versusNDJSON, and no reader implementation for another language ships with the repository — so the
Bash/Zsh readers the code is tuned for are not actually verifiable here.
Scope
Choose one and document it:
integer,number,timestamp(RFC 3339), and a list type, keeping thehex encoding for strings and the LF framing. Add a streaming
iter_records(). Publish a JSONSchema alongside
docs/schemas/v1/command-protocol.schema.jsonand a reference reader in atleast one other language (Bash is the natural choice given the existing tuning and the sibling
base-bash-libsrepository).COMMAND_PROTOCOL_V1frozen and shell-oriented,document its exact niche (shell readers that cannot depend on a JSON parser), point everything
else at the NDJSON contract, and add the migration note to
docs/output-contracts.md.Option 2 is cheaper and may well be right —
jqis near-universal now — but the current silenceleaves adopters unable to choose.
Acceptance criteria
golden fixtures validated by both, and
iter_records()decodes a 1M-record payload withoutmaterializing it.
command_protocolis documented as frozen with its niche named, and thestring/boolean limitation is stated rather than implied.
docs/schemas/v1/command-protocol.schema.jsonand the public docs agree with the code.Validation
Round-trip fixtures in Python plus one other language, consistent with the existing
scripts/validate_contract_fixtures.py/.mjspattern.Non-goals