Part 5 - Ingestion DAG and Raw Storage 📥 This part continues from the runtime config and looks at the first real Airflow DAG in the chain: dags/api_ingestion_dag.py. What the DAG does The ingestion DAG runs every three minutes. Its job is to: load the station sample, pick one or more stations for the current interval, fetch OpenAQ and OpenWeather payloads, save those payloads as raw JSON, an
Part 5 - Ingestion DAG and Raw Storage 📥
Abdelrahman Adnan·Dev.to··1 min read
D
Continue reading on Dev.to
This article was sourced from Dev.to's RSS feed. Visit the original for the complete story.