Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ repos:
#####
# Python
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.4
rev: v0.16.6
hooks:
# Sort imports
- id: ruff-check
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ This project provides data tools and low friction access to versioned datasets w
3. See which datasets are available:
```python
from cfa.dataops import datacat

print(datacat.__namespace_list__)
```
4. Resolve a dataset reference from a full path or unique suffix:
Expand Down
56 changes: 28 additions & 28 deletions docs/data_developer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,15 +81,13 @@ prefix = "path/to/transformed/data"
```python title="cfa/dataops/datasets/{team_dir}/schemas/{dataset_name}.py"
import pandera.pandas as pa

extract_schema = pa.DataFrameSchema({
"column1": pa.Column(str),
"column2": pa.Column(float)
})

load_schema = pa.DataFrameSchema({
"transformed_col1": pa.Column(str),
"transformed_col2": pa.Column(float)
})
extract_schema = pa.DataFrameSchema(
{"column1": pa.Column(str), "column2": pa.Column(float)}
)

load_schema = pa.DataFrameSchema(
{"transformed_col1": pa.Column(str), "transformed_col2": pa.Column(float)}
)
```

### ETL script
Expand Down Expand Up @@ -156,33 +154,35 @@ import pandas as pd
import pandera.pandas as pa

# Define the schemas for validation
extract_schema = pa.DataFrameSchema({
"date": pa.Column(pd.DatetimeTZDtype(tz='UTC')),
"value": pa.Column(float, checks=pa.Check.greater_than(0)),
"category": pa.Column(str, checks=pa.Check.isin(['A', 'B', 'C']))
})

load_schema = pa.DataFrameSchema({
"date": pa.Column(pd.DatetimeTZDtype(tz='UTC')),
"normalized_value": pa.Column(float),
"category": pa.Column(str)
})
extract_schema = pa.DataFrameSchema(
{
"date": pa.Column(pd.DatetimeTZDtype(tz="UTC")),
"value": pa.Column(float, checks=pa.Check.greater_than(0)),
"category": pa.Column(str, checks=pa.Check.isin(["A", "B", "C"])),
}
)

load_schema = pa.DataFrameSchema(
{
"date": pa.Column(pd.DatetimeTZDtype(tz="UTC")),
"normalized_value": pa.Column(float),
"category": pa.Column(str),
}
)


# Add mock data generation for testing
# prefix with 'extract' or 'load'
def extract_mock_data(output="pandas", size=10) -> pd.DataFrame|pl.DataFrame:
def extract_mock_data(output="pandas", size=10) -> pd.DataFrame | pl.DataFrame:
data = {
"date": pd.date_range(
start="2023-01-01",
periods=size,
tz='UTC'
),
"value": np.random.uniform(1, 100, size),
"category": np.random.choice(['A', 'B', 'C'], size)
"date": pd.date_range(start="2023-01-01", periods=size, tz="UTC"),
"value": np.random.uniform(1, 100, size),
"category": np.random.choice(["A", "B", "C"], size),
}
df = pd.DataFrame(data)
return df if output == "pandas" or output == "pd" else pl.from_pandas(df)


# Validate synthetic data matches schema
if __name__ == "__main__":
test_df = extract_mock_data()
Expand Down
13 changes: 7 additions & 6 deletions docs/data_user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,14 @@ Use `datacat.get_ref(...)` when you want to resolve a dataset first and then reu
The typical way to create a reference is as follows:
```python
from cfa.dataops import datacat

ref = datacat.public.team.data_trends
```

This can be done much simpler using `get_ref`:
```python
from cfa.dataops import datacat

data_ref = datacat.get_ref("data_trends")
```

Expand All @@ -85,8 +87,8 @@ If you want to see which version will be returned before loading the dataframe,
from cfa.dataops import datacat

resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(
version_spec=">=2025-05-01,<2025-06-01",
selection="newest",
version_spec=">=2025-05-01,<2025-06-01",
selection="newest",
)

print(resolved.version)
Expand Down Expand Up @@ -153,7 +155,7 @@ from cfa.dataops import datacat

# newest match in the range (single version)
df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(
version_spec=">=2025-05-01,<2025-06-01"
version_spec=">=2025-05-01,<2025-06-01"
)
```

Expand All @@ -165,7 +167,7 @@ Used version: '2025-05-30T19-55-51'
```python
# oldest match in the same range (selection=oldest)
df_old = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(
version_spec=">=2025-05-01,<2025-06-01", selection = "oldest"
version_spec=">=2025-05-01,<2025-06-01", selection="oldest"
)
```

Expand All @@ -177,8 +179,7 @@ Used version: '2025-05-30T14-50-36'
```python
# Preview the exact version that would be loaded for the same range
resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(
version_spec=">=2025-05-01,<2025-06-01",
selection="newest"
version_spec=">=2025-05-01,<2025-06-01", selection="newest"
)

resolved.version
Expand Down
5 changes: 2 additions & 3 deletions docs/managing_catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,10 @@ datacat.private.my_project.custom_dataset.load.get_dataframe()

# Preview the version that will be loaded
df_meta = datacat.private.scenarios.covid19vax_trends.load.resolve_version(
version_spec=">=2025-05-01,<2025-06-01",
selection="newest",
version_spec=">=2025-05-01,<2025-06-01",
selection="newest",
)
print(df_meta.version)

```

### Listing Available Resources
Expand Down
5 changes: 4 additions & 1 deletion docs/release_notes/v2025.12.10.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,19 +63,22 @@ We're thrilled to announce the first official release of the **CFA DataOps** pro
**List Available Datasets:**
```python
from cfa.dataops import datacat

print(datacat.__namespace_list__)
```

**Load a Dataframe with Version Filtering:**
```python
from cfa.dataops import datacat

df = datacat.public.my_dataset.load.get_dataframe(version=">2024.12.01,<2025.08")
```

**Generate a Report** *(removed in a later release)*:
```python
from cfa.dataops import reportcat # removed in a later release
reportcat.examples.dataset_report_ipynb.nb_to_html_file('report.html')

reportcat.examples.dataset_report_ipynb.nb_to_html_file("report.html")
```

## 📚 Documentation
Expand Down