Skip to content

Latest commit

 

History

History
28 lines (24 loc) · 1.01 KB

README.md

File metadata and controls

28 lines (24 loc) · 1.01 KB

Airflow_project

Apache airflow ETL

This project uses apache airflow to unzip and treat a zipped folder through a pipeline of tasks in apache airflow.

The python script dowloads the tar file to use.

Before running this script you must start apache airflow with the command

start_airflow

and opening the Ui of airflow. Open a terminal and create a directory structure for staging area as follows: /home/project/airflow/dags/finalassignment/staging.

sudo mkdir -p /{your_desired_path}/airflow/dags/finalassignment/staging

Note: If you face any permission issues in writing to the directories, please execute the following commands:

sudo chown -R 100999 /home/project/airflow/dags/finalassignment
sudo chmod -R g+rw /home/project/airflow/dags/finalassignment  
sudo chown -R 100999 /home/project/airflow/dags/finalassignment/staging
sudo chmod -R g+rw /home/project/airflow/dags/finalassignment/staging

Change to your work directory:

cd /home/project/airflow/dags/finalassignment/staging