📝 Notes taken by Abd @ITNA Digital

Links

🔗 Link to the Video

🔗 Reference Code (has explainer comments)

Keywords

Table of Contents

Components of Workflow

Components of an Airflow workflow

• DAG ⇒ Species the dependecies between a set of tasks with explcite execution order.
• Task ⇒ Defined unit of tasks (known as opreators). They can used to fetch data, running analysis, triggering other tasks etc.
• Digram ⇒ Individual execution or run of a DAG.
• Task Instance ⇒ Individual run of a single Task. These tasks instances have indicator phases like- running, queued, success, failed, up for retry etc.

DAG Structure

An Airflow DAG is defined in a Python which consists of -

• DAG defintion
• Tasks
  • Operator: Predefined tasks; Can be bash, python or even bigquery stuff (?)
  • Task Instances: Waits for external event to happen
  • Taskflow Decorator: Custom Python function