I built a small end-to-end pipeline to simulate a common data engineering scenario: ingesting new files from cloud storage into a data platform automatically. The pipeline: extracts trending songs data from Kworb writes the data as Parquet files uploads them to Google Cloud Storage (GCS) uses Databricks Autoloader to ingest new files incrementally Architecture The flow is straightforward: Extr