Daft Examples

Prompt an LLM. Embed a document. Process a video. It's all just a DataFrame.

Daft is a data engine that treats AI operations as first-class citizens. Calling an LLM, generating embeddings, reading a PDF — these aren't afterthoughts bolted onto a DataFrame library. They're column expressions, right in the query plan.

This repo is 90+ runnable examples that show you how.

df = daft.from_pydict({"text": ["Einstein was a brilliant scientist.", "Mozart was a brilliant pianist."]})

df = df.with_column("summary", prompt(df["text"], model="openai/gpt-4.1-mini"))
df = df.with_column("embedding", embed_text(df["text"], model="Qwen/Qwen3-Embedding-0.6B"))

Get started

git clone https://github.com/Eventual-Inc/daft-examples.git
cd daft-examples
make setup    # installs deps, copies .env.example → .env

Add your OpenAI key to .env, then:

uv run quickstart/01_hello_world_prompt.py

That's it. Every script in this repo is self-contained — dependencies are declared inline via PEP 723, so uv run handles everything.

---

Quickstart

Five scripts, each under 30 seconds. Start here.

#	Script	What it does
01	Hello World	Classify text with an LLM prompt
02	Semantic Search	PDF → embeddings → vector search → Turbopuffer
03	Data Enrichment	ETL pipeline with LLM-based enrichment
04	Audio Files	Read audio metadata, resample with daft.File
05	Video Files	Extract video frames and metadata

---

Examples

Small, focused scripts. One concept each.

Prompt — LLM as a column expression

Script	What it shows
prompt.py	Basic classification — one function call
prompt_structured_outputs.py	Pydantic models for type-safe LLM output
prompt_chat_completions.py	Chat-style completions with system personas
prompt_files_images.py	Multimodal — send images and PDFs to the model
prompt_pdfs.py	Feed entire PDFs into the prompt
prompt_openai_web_search.py	Web search tool integration
prompt_qa.py	Synthetic Q&A generation with LLM-as-judge
prompt_session.py	Stateful prompt sessions
prompt_unity_catalog.py	Prompt over Unity Catalog tables
prompt_gemini3_code_review.py	Automated code review with Gemini

Embed — vectors as a column expression

Script	What it shows
embed_text.py	Text embeddings at multiple dimensions
embed_images.py	Image embeddings with Apple AIMv2
embed_text_providers.py	Compare embedding providers side by side
embed_video_frames.py	Embed individual video frames
cosine_similarity.py	Semantic similarity search

Files — audio, video, PDF, code as native types

Script	What it shows
daft_file.py	daft.File basics
daft_audiofile.py	Audio metadata, resampling
daft_audiofile_udf.py	Custom audio processing UDF
daft_videofile.py	Video metadata and keyframes
daft_videofile_stream.py	Streaming video frame extraction
daft_file_pdf.py	PDF parsing and page extraction
daft_file_code.py	Source code analysis
daft_file_knowledge_base.py	Mixed content in one bucket — code, docs, PDFs, media — branch by extension

UDFs — bring your own logic

Script	What it shows
daft_func.py	Simple @daft.func UDF
daft_func_async.py	Async UDFs for I/O-bound work
daft_func_batch.py	Batch-mode UDFs
daft_cls_model.py	@daft.cls — load a model once, run it on every row
daft_cls_with_types.py	Class UDFs with TypedDict and Pydantic
daft_cls_async_client.py	Async class UDFs with persistent clients

SQL & analytics

Script	What it shows
stocks.py	Window functions on real stock data — moving averages, rankings, Golden Cross detection

Classify

Script	What it shows
classify_text.py	Text classification
classify_image.py	Image classification

I/O

Script	What it shows
read_pdfs.py	Discover and read PDFs from remote storage
read_video_files.py	Frame-level video reading with daft.read_video_frames

Common Crawl

Script	What it shows
cc_show.py	Browse Common Crawl data
cc_chunk_embed.py	Chunk and embed web pages
cc_wet_paragraph_dedupe.py	Paragraph-level deduplication at scale

---

Pipelines

End-to-end workflows. These are where things get interesting.

RAG

Pipeline	What it does
rag.py	Minimal RAG — embed, retrieve, generate
full_rag.py	Full RAG — PDF extraction, PyMuPDF UDF, cross-join ranking, generation

Context engineering

Pipeline	What it does
lambda_mapreduce.py	6 long-context reasoning patterns as native query plans (search, summarize, classify, extract, QA, analyze)
chunking_strategies.py	Compare fixed-size, sentence, and paragraph chunking
few_shot_example_selection.py	Embedding-based few-shot selection
llm_judge_elo.py	LLM-as-judge with ELO ranking

Audio & video

Pipeline	What it does
voice_ai_analytics.py	Transcription → summarization → translation → embeddings → RAG over transcripts
key_moments_extraction.py	Extract and clip key moments from audio transcripts
shot_boundary_detection.py	Video scene detection with frame embeddings

Search & recommendations

Pipeline	What it does
ai_search.py	PDF search with Turbopuffer
embed_docs.py	Codebase analysis with SpaCy chunking and embeddings
data_enrichment.py	LLM-powered data enrichment pipeline

Code

Pipeline	What it does
prompt_github.py	Prompt over GitHub repos
cursor.py	Code analysis pipeline

---

Datasets

Processing patterns for real public datasets — not toy data.

Dataset	Scripts	What you'll process
Common Crawl	WARC, WAT, WET parsing, text deduplication, chunk & embed	Billions of web pages
LAION	Image-text pairs, CLIP training data, metadata	5B+ image-text pairs
Open Images	Image loading, processing, vision models	9M annotated images
TPC-H	SQL queries, performance benchmarks	Industry-standard analytical benchmark

---

Setup

Requirements

• Python 3.12
• uv
• FFmpeg (for audio/video examples)

API keys

Most examples need an OpenAI key. Some need more. Copy the example and fill in what you have:

cp .env.example .env

Key	What uses it
OPENAI_API_KEY	Most prompt, embed, and RAG examples
OPENROUTER_API_KEY	Multi-model and structured output examples
TURBOPUFFER_API_KEY	Vector search pipelines
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY	Common Crawl, TPC-H, Open Images
HF_TOKEN	Private HuggingFace datasets

Running any example

uv run quickstart/01_hello_world_prompt.py
uv run examples/prompt/prompt.py
uv run pipelines/rag/full_rag.py

Every script declares its own dependencies. No extras to install.

---

Development

make format        # ruff format + import sort (use this, not `uv format` alone)
make lint          # lint check
make precommit     # lint + format check (runs on git commit)
make test          # run all tests
make test-no-creds # run tests that don't need API keys

uv format only runs Ruff's formatter and does not sort imports. Use make format for both.

See CONTRIBUTING.md for guidelines on adding new examples.

---

License

Apache 2.0