💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.

Python framework for building Google Cloud Composer workflows.

Airflow DAG for sentiment analysis on GCP

Auto run a Cloud Composer DAG when an object is uploaded to a Google Cloud Storage bucket

Welcome to the MiniProjects Playground—an interactive space where learning meets doing! This repository is a collection of hands-on mini-projects that I've crafted after delving into various tech stacks and frameworks. From theory to application, each project is a testament to the practical side of coding.

Hands-on examples for setting up Google Cloud Composer (managed Apache Airflow), resolving IAM issues, and building ETL DAG pipelines from Google Cloud Storage to BigQuery.

airflowser 💠 : Cloud Composer #airflow #dag

🏗️ ⌾ Terraform demo project provisioning a Google Cloud Dataflow job on GCP with configurable streaming and batch pipelines, worker machine type, autoscaling, custom parameters, temp and staging bucket settings, IAM service account bindings, network and subnetwork configuration, and Flex Template support for managed data processing workloads.

This project demonstrates a seamless integration of Apache Airflow, Snowflake, and Google Cloud Composer to create an automated ETL pipeline for fetching, transforming, and storing stock price data. The workflow highlights the power of cloud-native orchestration and scalable data warehousing to handle real-time data processing efficiently

End-to-End data engineering project with Google services