-
Notifications
You must be signed in to change notification settings - Fork 0
feat(elt-pipelines): Port statusdisplay to elt-pipelines #470
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
580dcd0
bb19576
0350c3b
eac7a8a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| with source as ( | ||
|
|
||
| select * from {{ source('accelerator_statusdisplay', 'cycles') }} | ||
|
|
||
| ), | ||
|
|
||
| renamed as ( | ||
|
|
||
| select | ||
|
|
||
| _dlt_id as dlt_id, | ||
| label as name | ||
|
|
||
| from source | ||
|
|
||
| ) | ||
|
|
||
| select * from renamed |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| models: | ||
| - name: base_statusdisplay__cycles | ||
| description: > | ||
| Each row contains the cycle id & label along with a primary key referenced | ||
| in the secondary base_statusdisplay__cycles__phases table where | ||
| information describing a particular phase of a cycle is stored. | ||
| columns: | ||
| - name: dlt_id | ||
| description: Primary key - unique identifier for the cycle from the source system | ||
| - name: name | ||
| description: The cycle label/name |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| -- See base_statusdisplay__cycles.sql for a description of why this table exists. | ||
|
|
||
| with source as ( | ||
|
|
||
| select * from {{ source('accelerator_statusdisplay', 'cycles__phases') }} | ||
|
|
||
| ), | ||
|
|
||
| renamed as ( | ||
|
|
||
| select | ||
| type as phase, | ||
| target, | ||
| start as started_at, | ||
| {{ adapter.quote("end") }} as ended_at, | ||
| _dlt_parent_id as dlt_cycles_id | ||
|
|
||
| from source | ||
|
|
||
| ) | ||
|
|
||
| select * from renamed | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| models: | ||
| - name: base_statusdisplay__cycles__phases | ||
| description: > | ||
| Each row in the cycles__phases table corrseponds to a section of time | ||
| within a given cycle. | ||
| columns: | ||
| - name: phase | ||
| description: The name of the phase within the cycle | ||
| - name: target | ||
| description: The target station number that was running during this phase | ||
| - name: started_at | ||
| description: Timestamp when the phase started | ||
| - name: ended_at | ||
| description: Timestamp when the phase ended | ||
| - name: dlt_cycles_id | ||
| description: Foreign key referencing base_statusdisplay__cycles (dlt_id) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| -- Simple join to denormalize the cycles and cycles__phases tables | ||
| -- that are not much use as separated table. | ||
|
|
||
| {# Denormalize cycles & cycles__phases to include the cycle label for each phase #} | ||
|
|
||
| with | ||
|
|
||
| base_cycles as ( | ||
|
|
||
| select * from {{ ref('base_statusdisplay__cycles') }} | ||
|
|
||
| ), | ||
|
|
||
| base_cycles__phases as ( | ||
|
|
||
| select * from {{ ref('base_statusdisplay__cycles__phases') }} | ||
|
|
||
| ), | ||
|
|
||
| join_cycle_labels_and_phases as ( | ||
|
|
||
| select | ||
|
|
||
| {{ adapter.quote('name') }}, | ||
| started_at, | ||
| ended_at, | ||
| phase, | ||
| {{ adapter.quote('target') }} | ||
|
|
||
| from base_cycles | ||
| join base_cycles__phases on base_cycles.dlt_id = base_cycles__phases.dlt_cycles_id | ||
|
|
||
| ) | ||
|
|
||
| select * from join_cycle_labels_and_phases |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| models: | ||
| - name: stg_statusdisplay__cycles | ||
| description: > | ||
| Each record defines a phase of a cycle, including which target was | ||
| running. Multiple records exist for a given period of time if more than | ||
| one target station was running. | ||
|
|
||
| unit_tests: | ||
| - name: test_stg_statusdisplay__cycles | ||
| model: stg_statusdisplay__cycles | ||
| given: | ||
| - input: ref('base_statusdisplay__cycles') | ||
| format: sql | ||
| fixture: base_statusdisplay__cycles | ||
| - input: ref('base_statusdisplay__cycles__phases') | ||
| format: sql | ||
| fixture: base_statusdisplay__cycles__phases | ||
| expect: | ||
| rows: | ||
| - { | ||
| name: "2024/2", | ||
| started_at: "2024-07-09 07:30:00 UTC", | ||
| ended_at: "2024-07-24 07:30:00 UTC", | ||
| target: 1, | ||
| } | ||
|
Comment on lines
+20
to
+25
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
set -e
file='elt-pipelines/facility_ops/transform/models/staging/accelerator/stg_statusdisplay__cycles.yml'
printf '%s\n' '--- target file ---'
cat -n "$file"
printf '%s\n' '--- related references ---'
rg -n -C 5 'stg_statusdisplay__cycles|statusdisplay__cycles|phase|2024/2' elt-pipelines/facility_ops/transform --glob '*.yml' --glob '*.sql' --glob '*.yaml'Repository: ISISNeutronMuon/analytics-data-platform Length of output: 16894 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- dbt version declarations ---'
rg -n -C 3 'dbt(-core)?|dbt_unit_tests|unit_tests' \
--glob 'requirements*.txt' --glob 'pyproject.toml' --glob 'setup.cfg' \
--glob 'Pipfile*' --glob 'poetry.lock' --glob 'uv.lock' \
--glob 'packages.yml' --glob 'dbt_project.yml' --glob '*.yml' --glob '*.yaml' .
printf '%s\n' '--- unit-test expected-row conventions ---'
rg -n -C 3 'expect:\s*$|rows:|phase:' elt-pipelines/facility_ops/transform/models --glob '*.yml' --glob '*.yaml' | head -240Repository: ISISNeutronMuon/analytics-data-platform Length of output: 50395 Assert the The staging model outputs 🤖 Prompt for AI Agents |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use the declared source table name.
accelerator_statusdisplaydeclarescycles_phases, notcycles__phases. This model cannot resolve the source and dbt compilation will fail.Proposed fix
📝 Committable suggestion
🤖 Prompt for AI Agents