Skip to content

fix(ingest): normalize Markdown front-matter dates before storage - #253

Merged
codeforstartups merged 4 commits into
codeforstartups:developmentfrom
Shraddhameduri:fix/front-matter-date-normalization
Sep 25, 2026
Merged

codeforstartups merged 4 commits into
codeforstartups:developmentfrom
Shraddhameduri:fix/front-matter-date-normalization

Conversation

@Shraddhameduri

Copy link
Copy Markdown
Contributor

Fixes #246

What

yaml.safe_load() converts unquoted YAML front-matter dates into datetime.date / datetime.datetime objects, which DynamoDB's TypeSerializer rejects at write time (TypeError: Unsupported type "datetime.date"). This PR normalizes parsed front-matter in the ingestion layer before it reaches storage.

Changes

  • src/dynavec/ingest.py: new recursive _normalize_front_matter() helper (date/datetime → .isoformat(), recurses through dicts and lists, scalars untouched). Applied in MarkdownSource._front_matter(), which also covers the S3Source Markdown path since it reuses the same parser.
  • tests/test_markdown_source.py: 5 new tests — unquoted dates → ISO strings, timestamps → ISO strings, nested dates in dicts/lists, quoted strings and scalars untouched, and an end-to-end check that ingested metadata passes boto3's TypeSerializer after _to_dynamo.

Test results

  • pytest tests/test_markdown_source.py — 19 passed
  • pytest tests/test_ingest.py tests/test_s3_source.py — 25 passed

@codeforstartups codeforstartups left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct bug fix for #246 — yaml.safe_load() turns unquoted front-matter dates into datetime.date/datetime objects that DynamoDB's TypeSerializer rejects. _normalize_front_matter recursively converts them to ISO-8601 strings (handling nested dicts/lists too). Verified locally: ruff clean, 19 markdown-source tests pass. Merging — thanks @Shraddhameduri! 🙌

@codeforstartups
codeforstartups merged commit 0f536b5 into codeforstartups:development Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(ingest): Normalize Markdown front-matter dates before storage

2 participants