Skip to content

Compile root cfg from --override values without a config.yaml - #80

Merged
Volv-G merged 1 commit into
masterfrom
piforge/tangle-pipeline-crud/cfg-override-only-compile-withou-7ec21bc
Oct 13, 2026
Merged

Volv-G merged 1 commit into
masterfrom
piforge/tangle-pipeline-crud/cfg-override-only-compile-withou-7ec21bc

Conversation

@Volv-G

@Volv-G Volv-G commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

A root @pipeline that loads config can now compile from --override values when the implicit default config.yaml is absent. This enables compile-time fan-out without a config file:

@pipeline("Sharded")
def sharded(cfg) -> None:
    for shard in cfg.shards:
        process.named(f"Process {shard}")(shard=shard)
tangle sdk pipelines compile sharded.py -o sharded.yaml --override 'shards=[us, eu]'

Behavior

Case Result
Root, no config=, default config.yaml missing, ≥1 --override (cfg param and/or propagate_config=True) Compiles from {} + typed overrides; warning names override keys only
Same, but no overrides Error (unchanged), now hints at --override KEY=VALUE
Explicit config= missing (with or without overrides) Error (unchanged)
Nested child missing its own config Error (unchanged)
Overrides, no cfg and no propagate_config Error (unchanged)

A missing default is modelled exactly as an empty config.yaml (raw_cfg={}, no config identity), so the compile key and output bytes match an empty file. For a propagate_config root the overrides become the broadcast payload; children still take only keys they declare.

Tests

New tests in tests/test_pipeline_compiler.py: list override → distinct named tasks (API + CLI), typed int → range(), missing-vs-empty key/bytes equivalence, strict cases above, propagate-only root broadcast. Full uv run pytest passes locally.

A root @pipeline that loads config (a `cfg` parameter and/or
propagate_config=True), declares no explicit `config=`, and receives at
least one --override now compiles from an empty base config when the
implicit default config.yaml is absent, exactly as if an empty
config.yaml existed (raw_cfg={}, no config identity, identical compile
key and output). This enables compile-time fan-out such as
`for shard in cfg.shards` with `--override 'shards=[us, eu]'`.

A warning names only the override keys. Still strict: missing explicit
`config=`, missing default with no overrides (error now hints at
--override), a nested child's missing config, and overrides on a
pipeline with neither `cfg` nor propagate_config.
@Volv-G
Volv-G requested a review from Ark-kun as a code owner October 13, 2026 22:29
@Volv-G
Volv-G merged commit d031e0f into master Oct 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant