broadcast-join

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.

PySpark ETL & analytics pipeline for taxi trip ETA, partitioned Parquet, windowed aggregations and performance patterns.

This repository showcases how to setup a Scala Spark job on Docker and in Dataproc and execute a Broadcast Join technique.

End- to-End Performance-optimized sales data pipeline using Medallion Architecture with broadcast joins, fact/dimension modeling, Autoloader & incremental processing

Diagnoses Spark data skew from the job's own event log - no cluster, no JVM - and tells key skew apart from a straggler by comparing task time against bytes read. Then measures the fixes against each other on real Spark 4: across 16 cells, salting won 0. Broadcast 6.5x, AQE 1.5x, salting 0.81x.

Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.

Battle-tested Apache Spark tuning patterns with reproducible benchmarks. 10 techniques (partition pruning, broadcast joins, AQE, skew handling, Z-ORDER, and more) — each paired with measured before/after speedups runnable on a laptop.

🚖 Ingest and analyze NYC yellow taxi data with a streamlined ETL pipeline, featuring data cleaning, analytics, and business-ready outputs.