Docker Compose for Data Ingestion

Create folder called dags_new.

📒 Notes taken by Abd @ITNA Digital

Feel free to reach out to me on Slack or leave a comment here.

Video Timestamps

1:32 - Docker Compose for Airflow
7:00 - Creating a simple DAG & Bash Operator
10:40 - Parameterize the DAG & Scheduling with cronjob syntax
14:40 - [Troubleshooting] DAG on Airflow, Modifying the Dockerfile & Fixing scheduling issues
18:18 - Triggering the DAG on Airflow
20:00 - Airflow log
21:15 - Modifying Bash operator to download the CSV & Issues with temp directory
24:30 - Running the modified DAG, Installing wget, Using cURL
28:10 - DAG on Airflow, Entering container with exec and seeing the CSV file in the temp dir
31:40 - Parameterizing URL, Jinja templating for bash command, datetime formatting for execution time
41:26 - [Exploring] Putting the Ingestion Script into Airflow; Docker Operator
44:00 - ingest_script, modifying requirements.txt and Dockerfile, integrating ingestion to DAG, Environment Variable, modifying docker compose and connecting to DAG,
1:01:00 - Checking DB with pgCLI, directory of db, checking the container connection
1:06:00 - DAG on Airflow, Troubleshooting, configing ingestion script and DAG, Idempotancy
1:13:00 - Running the scripts