Lakehouse + Streaming Pipeline (in progress)

The plan

A production-grade lakehouse built entirely on free tiers, documented phase by phase:

1. Ingest — public real-time API → Redpanda
2. Store — Apache Iceberg tables on Cloudflare R2 (zero egress cost)
3. Transform — incremental processing with DuckDB + PyIceberg
4. Orchestrate — Dagster, everything as code, CI/CD, data quality checks
5. Serve — live dashboard embedded in this portfolio

Each phase ships with a blog post covering design decisions, trade-offs, and costs.

Phase 1 starting soon — follow along on the blog.