From the course: Hands-On Introduction: Data Engineering
Join today to access over 24,700 courses taught by industry experts.
Build a 1 Task DAG
From the course: Hands-On Introduction: Data Engineering
Build a 1 Task DAG
- [Instructor] Airflow is designed to construct and manage orchestrated directed acyclic graphs, or DAGs. In other words, Airflow provides a way to programmatically arrange repeatable, monitored, and interdependent operations in a specific order. Our first task is to build and run a simple DAG in Airflow. This particular DAG will consist of a single node that create this file, .txt, with the contents, Hello, LinkedIn Learning. In Airflow DAGs are represented as Python scripts located within a specific directory. To create a new DAG, simply create a new Python script, and place it in this directory. As long as the script is properly formatted and error-free, Airflow scheduler will automatically pick it up during the next heartbeat. Let's switch back to our Code Spaces environment and first check where Airflow is expecting DAGs to be. To do that, you can run the following command. This command reads the Airflow.cfg file and…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.
