Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/extending/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ plain defaults that make any subclass a valid node; `Asset` overrides them with
| Member | Default | Asset |
|--------|---------|-------|
| `id`, `kind`, `key`, `qualified_key` | from `Component` | qualified with the source key |
| `materializable` | `True` | field |
| `enabled` | `True` | field |
| `relations` | `{}` | every relation the class declares, by name |
| `upstream_relations()` | `{}` | the relations whose kind is `asset`: the graph's edges |
| `bound(name)` | from `Component` | what is bound to one relation |
Expand Down
2 changes: 1 addition & 1 deletion docs/extending/runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ state.mark_canceled(operation)
```

Completing an operation promotes its dependents to ready; failing one cancels everything
downstream. Non-materializable operations start as `SKIPPED` and count as satisfied
downstream. Non-enabled operations start as `SKIPPED` and count as satisfied
predecessors. Each transition emits the matching `OPERATION_*` event with a deterministic id
(`emit=False` skips the emission for runners whose child process emits it itself). All
mutations happen on the event loop thread, so no locking is needed.
Expand Down
6 changes: 3 additions & 3 deletions docs/guide/assets.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ An instance carries the runtime state a definition does not know about:
| `destinations` | Destination instances to write to. A single destination is accepted and wrapped in a list. |
| `dataset` | Namespace (schema, folder) the asset materializes into. Defaults to the source's. |
| `default_destination_key` | With several destinations, the one downstream readers load from. |
| `materializable` | `False` turns the asset into a read-only dependency: it is skipped by runners but its stored output is still readable. |
| `enabled` | `False` turns the asset into a read-only dependency: it is skipped by runners but its stored output is still readable. |
| `materialization_strategy` | How strictly the data is checked against the schema. |
| `normalizer` | The normalizer applied before conform. |
| `id` | Instance identity, a UUID by default. |
Expand All @@ -151,7 +151,7 @@ Set them at construction, or derive a reconfigured copy by calling an existing i
```py
asset = users(destinations=il.CSVDestination(base_path="./data"), dataset="shop")

read_only = asset(materializable=False)
read_only = asset(enabled=False)
strict = asset(materialization_strategy=il.MaterializationStrategy.STRICT)
bare = asset(normalizer=None) # None explicitly clears the normalizer
```
Expand All @@ -170,7 +170,7 @@ Unknown keyword arguments raise `TypeError` rather than being silently dropped.
| Call | Effect |
|------|--------|
| `asset.run(partition, dag, metadata)` | Execute, normalize, conform. Return the data. Write nothing. |
| `asset.materialize(partition, dag, metadata)` | Everything `run` does, then write to every destination. Returns the data, or `None` when the asset is not materializable. |
| `asset.materialize(partition, dag, metadata)` | Everything `run` does, then write to every destination. Returns the data, or `None` when the asset is disabled. |
| `await asset.run_async(...)`, `await asset.materialize_async(...)` | The same, for async callers. |

`partition` is required for partitioned assets and ignored (with a warning) for unpartitioned
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/backfilling.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ data in place rather than the ancient tail.

## One run for the whole window

When every materializable partitioned asset in the DAG declares `allow_window=True`, pass the
When every enabled partitioned asset in the DAG declares `allow_window=True`, pass the
window itself. Each asset receives the range through `context.window` and fetches it in one
call; destinations split the write per partition:

Expand Down
2 changes: 1 addition & 1 deletion docs/guide/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Run and operation events flow through the logging stack, sharing one format and
ordinary log lines on stderr:

```
19:39:52.368 INFO Running DAG with 3 materializable operation(s) (3 total) using AsyncRunner
19:39:52.368 INFO Running DAG with 3 enabled operation(s) (3 total) using AsyncRunner
19:39:52.370 INFO RUN_STARTED - Run started (3 operations)
19:39:52.370 INFO OPERATION_STARTED users Operation 'users' started
19:39:52.371 INFO OPERATION_STARTED orders Operation 'orders' started
Expand Down
8 changes: 4 additions & 4 deletions docs/guide/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,11 @@ running them, the same explicit-bind pattern as any other relation:

```py
matcher.campaign_matches.bind("campaigns", fb.campaigns)
dag = il.DAG(matcher) # fb.campaigns joins the DAG as materializable=False
dag = il.DAG(matcher) # fb.campaigns joins the DAG as enabled=False
```

`examples/campaign_matcher.yaml` is the same wiring written as a manifest: two connectors each
override their `campaigns` asset to `materializable: false`, and `campaign_matches` names both by
override their `campaigns` asset to `enabled: false`, and `campaign_matches` names both by
`{ref: ...}` since an asset always travels under its own source. See [Specs](specs.md) for the
manifest format.

Expand Down Expand Up @@ -245,13 +245,13 @@ included.

## Running one asset with its parents

A bound upstream the run does not materialize joins the DAG anyway, as a non-materializable copy
A bound upstream the run does not materialize joins the DAG anyway, as a disabled copy
under the same id: it is a live instance with its own destinations, so it can be read without
being run. That is the whole mechanism behind running one asset with its parents:

```py
mini = dag.mini_dag(fin.revenue.id)
[(op.qualified_key, op.materializable) for op in mini.operations]
[(op.qualified_key, op.enabled) for op in mini.operations]
# [("finance.revenue", True), ("shop.orders", False)]
```

Expand Down
6 changes: 3 additions & 3 deletions docs/guide/execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ dag.get_successors(asset.id) # downstream ids
dag.mini_dag(asset.id) # one node plus read-only parents
```

Non-materializable nodes stay in the graph as dependencies but never execute and never appear
Non-enabled nodes stay in the graph as dependencies but never execute and never appear
in the generations.

## Materializing
Expand Down Expand Up @@ -104,7 +104,7 @@ print(result)
`ExecutionInfo` carries `component_id`, `component_key`, `status`, `start_time`, `end_time`,
`execution_time`, `error`, `traceback`, and `effects` (what the operation asked the platform to
persist). `ExecutionStatus` is `QUEUED`, `READY`, `RUNNING`, `COMPLETED`, `FAILED`, `SKIPPED`
(non-materializable), `CANCELED` (downstream of a failure).
(disabled), `CANCELED` (downstream of a failure).

## Failure handling

Expand All @@ -114,7 +114,7 @@ traceback, its dependents are canceled, and the run continues or stops depending
so events and results are complete before it surfaces. A failure of the walk machinery itself
(a deadlock, an invalid graph) raises `RunnerError`.

Before anything executes, the runner validates the scope against every materializable
Before anything executes, the runner validates the scope against every enabled
operation: partitioned operations without a scope, windows against operations that forbid them,
and time-partition mismatches fail the whole run up front.

Expand Down
2 changes: 1 addition & 1 deletion docs/guide/partitioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ Passing a window to a run is then a single execution covering the whole range. A
partition still works and is presented to the asset as a one-partition window. Destinations
split window writes per partition, so storage looks identical either way.

A run given a window fails as a whole when any materializable partitioned asset in the DAG does
A run given a window fails as a whole when any enabled partitioned asset in the DAG does
not allow windows.

## What the context exposes
Expand Down
4 changes: 2 additions & 2 deletions docs/guide/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,10 @@ Calling an instance returns a deep copy with overrides applied; the original is

```py
staging = source(destinations=il.CSVDestination(base_path="./staging"))
read_only = source(materializable=False)
read_only = source(enabled=False)
```

Accepted keywords: `dataset`, `default_destination_key`, `materializable` (applied to every
Accepted keywords: `dataset`, `default_destination_key`, `enabled` (applied to every
asset), `normalizer`, `materialization_strategy`, and any relation name. A fixed keyword left
out means "unchanged"; a relation name passed at all changes it, since `None` there clears the
binding. Rebinding a relation also repoints the assets that had received it by trickle, leaving
Expand Down
8 changes: 4 additions & 4 deletions docs/guide/specs.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ nothing else to ask. In the platform, the store resolves it. A reference nobody

A source is the unit of reconstruction. Its spec carries the assets as an **override map** keyed
by asset key rather than as individual specs, which keeps the document compact and lets
per-asset state (its own destinations, `materializable`, its bound upstreams) survive:
per-asset state (its own destinations, `enabled`, its bound upstreams) survive:

```py
Shop(account_id="act_1").to_spec().init
# {"account_id": "act_1", "assets": {"orders": {"id": "...", "materializable": True}, ...}}
# {"account_id": "act_1", "assets": {"orders": {"id": "...", "enabled": True}, ...}}
```

Reconstruction builds each asset class with its overrides. The map is also the list of assets the
Expand Down Expand Up @@ -101,7 +101,7 @@ init:
assets:
campaigns:
id: fb-campaigns
materializable: false
enabled: false
- key: campaign_matcher
init:
assets:
Expand Down Expand Up @@ -141,7 +141,7 @@ dag = il.DAG.from_spec(DAGSpec(**payload))

It is what `MultiProcessRunner` ships to its workers and what `interloper run --format inline`
accepts. The override map is built from the DAG's **actual** asset instances, so a parent the run
only reads travels as its own document carrying that one asset with `materializable: false`,
only reads travels as its own document carrying that one asset with `enabled: false`,
and stays read-only after the round-trip.

## What makes something serializable
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/decorators.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Calling an instance returns a copy; omitted keywords mean "unchanged".
| `id` | |
| `dataset` | `dataset` (re-points assets that inherited the old value) |
| `default_destination_key` | `default_destination_key` |
| `materializable` | `materializable` (applied to every asset) |
| `enabled` | `enabled` (applied to every asset) |
| `materialization_strategy` | `materialization_strategy` |
| `normalizer` (`None` clears) | `normalizer` |
| any relation name (replaced; `None` clears) | any relation name (replaced; `None` clears, and repoints what was trickled) |
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/events.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ and type. Common metadata: `component_id`, `component_kind`, `component_key`, `s

| Type | When |
|------|------|
| `operation_queued` | At run start, for every materializable operation. |
| `operation_queued` | At run start, for every enabled operation. |
| `operation_started` | The operation was submitted. |
| `operation_completed` | `execute()` returned. |
| `operation_failed` | `execute()` raised. Adds `error` and, when the operation captures tracebacks, `traceback`. |
Expand Down
Loading
Loading