Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: CI
on:
push:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
verify:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
matrix:
python-version: ['3.11', '3.12']
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install
run: |
python -m pip install -e .
- name: Counterexamples and contracts
run: python -m unittest discover -s tests -v
- name: Replay committed evidence
run: pitbridge verify --out demo
- name: Generate and replay a fresh report
run: |
pitbridge demo --out outputs
pitbridge verify --out outputs
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
__pycache__/
*.py[cod]
*.egg-info/
.venv/
build/
dist/
outputs*/
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 dev-belly

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
68 changes: 67 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,68 @@
# PITBridge
Point-in-time financial feature engineering with DuckDB, revision-safe snapshots, data contracts and source lineage.

**Financial features as they were known when a decision was made.**

[![CI](https://github.com/dev-belly/PITBridge/actions/workflows/ci.yml/badge.svg)](https://github.com/dev-belly/PITBridge/actions/workflows/ci.yml)
[中文说明](README.zh-CN.md) · [Methodology](docs/METHODOLOGY.md) · [Data contract](docs/DATA_CONTRACT.md) · [Interview notes](docs/INTERVIEW.md)

An invoice dated January can be published in February, arrive in March, and be corrected later. Joining on January's date alone can put future knowledge into an earlier credit decision. PITBridge resolves **event time, publication time, ingestion time and observation revisions** before exporting a feature snapshot.

![Decision-time leakage counterexample](docs/evidence.png)

## Run in one minute

```bash
git clone https://github.com/dev-belly/PITBridge.git
cd PITBridge
python -m pip install -e .
pitbridge demo --out outputs
pitbridge verify --out outputs
python -m unittest discover -s tests -v
```

Open `outputs/report.html` in a browser. The report is self-contained, works offline and includes a filter for changed selections. Python 3.11+; **no third-party runtime dependencies**.

```bash
# Bring your own observations, decisions and feature specs:
pitbridge build --inputs demo/inputs.json --out outputs
```

## What the saved example proves

The deliberately small synthetic fixture is hand-auditable: **11 observations, 15 decisions, 45 feature lookups** across tax, bank and utility sources.

| Saved result | Count | Interpretation |
| :--- | ---: | :--- |
| Unsafe selections using future knowledge | 11 | The event-only baseline selected a record unavailable at decision time. |
| Selections changed by the full contract | 16 | Includes freshness and tombstone effects as well as future knowledge. |
| Selected / missing feature rows | 16 / 29 | Missingness is preserved with a reason, never silently replaced by zero. |

Inspect [the comparison](demo/comparison.csv), [record-level lineage](demo/snapshots.csv), [normalized inputs](demo/inputs.json), and [summary](demo/summary.json). These counts demonstrate the fixture; they are not a population leakage rate.

## Selection contract

1. Require `event_at <= decision_at` and `max(published_at, ingested_at) <= decision_at`.
2. For each entity/source/feature/event, select the **highest known revision**, even if a lower revision arrives later.
3. Apply known tombstones to that event. Do not resurrect its earlier revision.
4. Select the latest remaining event within the feature's inclusive freshness window.
5. Preserve a row for every decision/spec pair, including `no_history`, `not_available`, `deleted` and `stale`.

The production path is a SQLite window-function join. A separate Python enumerator checks its results, including randomized histories and append-future invariance. [Read the implementation](src/pitbridge/core.py).

## Evidence you can replay

| Artifact | Purpose |
| :--- | :--- |
| `inputs.json` | Canonical, normalized source records, decisions and feature contracts. |
| `snapshots.csv` | Values, missingness, selected record IDs, revisions and timestamps. |
| `comparison.csv` | Explicitly unsafe event-only baseline; never used as training features. |
| `summary.json`, `report.html` | Derived counts and an offline inspection report. |
| `manifest.json` | SHA-256 hashes of all five artifacts. |

Verification checks hashes **and** reruns both algorithms, rejecting a fabricated summary even if its hash was updated. Hashes are not signatures and cannot establish that an external source is truthful.

## Scope

This is a reference implementation for scalar financial observations. It does not provide streaming ingestion, access control, label generation or aggregated rolling features. Version numbers and trustworthy availability timestamps must come from the upstream source contract. SQLite is intentionally inspectable; distributed-scale performance has not been benchmarked.

PITBridge complements [CreditVintage](https://github.com/dev-belly/CreditVintage)'s application-time model evaluation. The repositories are separate components; no integration is claimed. MIT license.
35 changes: 35 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# PITBridge|金融时点数据与版本溯源

**核心问题:做出信贷决策时,银行当时到底能看到哪个版本的数据?**

例如,一月份开票金额在二月份公开、三月份才进入银行系统,四月份又被修订。直接按照“一月份”去关联历史申请,会把后来的信息放进早期特征,导致回测过于乐观。

PITBridge 同时处理数据发生时间、公开时间、入库时间和修订版本;以 SQLite SQL 生成特征,用独立 Python 枚举结果核验,并保留每一个特征的来源记录。

![保存结果的可复现图表](docs/evidence.png)

## 快速运行

```bash
python -m pip install -e .
pitbridge demo --out outputs
pitbridge verify --out outputs
python -m unittest discover -s tests -v
```

打开 `outputs/report.html` 即可查看离线报告。需要 Python 3.11 或更高版本,无第三方运行依赖。

## 已实现的重点

- **防止未来信息泄漏:** 数据发生时间与真正可用时间都必须不晚于申请时间。
- **正确处理修订:** 选择当时已知的最高版本;迟到的旧版本不会覆盖新版本。
- **撤销与过期:** 撤销记录只影响对应观察时点,不能重新使用该记录旧版本;特征可配置有效天数。
- **显式缺失:** 区分没有历史、尚未可用、已撤销和已过期,保留申请行。
- **逐条溯源:** 导出来源、记录 ID、版本、发生时间、公开时间与入库时间。
- **可重放证据:** 修改报告并更新哈希,仍会被语义重放发现。

合成示例有 11 条源记录、15 次决策、45 次特征查询。错误的“只按发生时间关联”基线有 11 次使用未来信息;完整规则共改变 16 次选择。后者还包括撤销和过期影响,不能把两个数字混为一谈。

[方法与规则](docs/METHODOLOGY.md) · [输入字段](docs/DATA_CONTRACT.md) · [中文面试问答及简历表述](docs/INTERVIEW.md)

这是可以核验的金融数据工程研究组件。演示数据全部为合成;没有宣称接入真实银行、降低实际违约率或处理生产规模数据。
46 changes: 46 additions & 0 deletions demo/comparison.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
decision_id,source,feature,safe_record_id,safe_value,safe_status,unsafe_record_id,unsafe_value,unsafe_uses_future_knowledge,different_selection
SME-A-1,bank,net_inflow,,,no_history,,,False,False
SME-A-1,tax,invoice_revenue,,,not_available,tax-a-v2,180000,True,True
SME-A-1,utility,kwh,power-a,1800.0,selected,power-a,1800,False,False
SME-A-2,bank,net_inflow,cash-a,50000.0,selected,cash-a-delete,,True,True
SME-A-2,tax,invoice_revenue,tax-a-v1,100000.0,selected,tax-a-v2,180000,True,True
SME-A-2,utility,kwh,power-a,1800.0,selected,power-a,1800,False,False
SME-A-3,bank,net_inflow,,,deleted,cash-a-delete,,False,True
SME-A-3,tax,invoice_revenue,tax-a-v1,100000.0,selected,tax-a-v2,180000,True,True
SME-A-3,utility,kwh,power-a,1800.0,selected,power-a,1800,False,False
SME-A-4,bank,net_inflow,,,deleted,cash-a-delete,,False,True
SME-A-4,tax,invoice_revenue,tax-a-v2,180000.0,selected,tax-a-feb,220000,True,True
SME-A-4,utility,kwh,power-a,1800.0,selected,power-a,1800,False,False
SME-A-5,bank,net_inflow,,,deleted,cash-a-delete,,False,True
SME-A-5,tax,invoice_revenue,tax-a-feb,220000.0,selected,tax-a-feb,220000,False,False
SME-A-5,utility,kwh,,,stale,power-a,1800,False,True
SME-B-1,bank,net_inflow,,,no_history,,,False,False
SME-B-1,tax,invoice_revenue,,,not_available,tax-b,90000,True,True
SME-B-1,utility,kwh,,,stale,power-b,1300,True,True
SME-B-2,bank,net_inflow,cash-b-v2,30000.0,selected,cash-b-v2,30000,False,False
SME-B-2,tax,invoice_revenue,,,not_available,tax-b,90000,True,True
SME-B-2,utility,kwh,,,stale,power-b,1300,True,True
SME-B-3,bank,net_inflow,cash-b-v2,30000.0,selected,cash-b-v2,30000,False,False
SME-B-3,tax,invoice_revenue,,,not_available,tax-b,90000,True,True
SME-B-3,utility,kwh,,,stale,power-b,1300,True,True
SME-B-4,bank,net_inflow,cash-b-v2,30000.0,selected,cash-b-v2,30000,False,False
SME-B-4,tax,invoice_revenue,tax-b,90000.0,selected,tax-b,90000,False,False
SME-B-4,utility,kwh,power-b,1300.0,selected,power-b,1300,False,False
SME-B-5,bank,net_inflow,cash-b-v2,30000.0,selected,cash-b-v2,30000,False,False
SME-B-5,tax,invoice_revenue,tax-b,90000.0,selected,tax-b,90000,False,False
SME-B-5,utility,kwh,,,stale,power-b,1300,False,True
SME-C-1,bank,net_inflow,,,no_history,,,False,False
SME-C-1,tax,invoice_revenue,,,no_history,,,False,False
SME-C-1,utility,kwh,,,no_history,,,False,False
SME-C-2,bank,net_inflow,,,no_history,,,False,False
SME-C-2,tax,invoice_revenue,,,no_history,,,False,False
SME-C-2,utility,kwh,,,no_history,,,False,False
SME-C-3,bank,net_inflow,,,no_history,,,False,False
SME-C-3,tax,invoice_revenue,,,no_history,,,False,False
SME-C-3,utility,kwh,,,no_history,,,False,False
SME-C-4,bank,net_inflow,,,no_history,,,False,False
SME-C-4,tax,invoice_revenue,,,no_history,,,False,False
SME-C-4,utility,kwh,,,no_history,,,False,False
SME-C-5,bank,net_inflow,,,no_history,,,False,False
SME-C-5,tax,invoice_revenue,,,no_history,,,False,False
SME-C-5,utility,kwh,,,no_history,,,False,False
Loading
Loading