Airflow tutorial 2: Set up airflow environment with docker

1 minute read

We will learn how to set up airflow environment using Docker

The GitHub links for this tutorial

Airflow problem

Airflow is built to integrate with all databases, system, cloud environments, …
- Managing and maintaining all of the dependencies changes will be really difficult.
- Takes lots of time to set up, and config Airflow env.
- How to share development and production environments for all developers.

Docker is an open platform for developing, shipping, and running applications.
Docker provides the ability to package and run an application in a loosely isolated environment called a container. The isolation and security allow you to run many containers simultaneously on a given host, regardless of its operating system: Mac, Windows, PC, cloud, data center, …
Containers are lightweight because they don’t need the extra load of a hypervisor, but run directly within the host machine’s kernel. This means you can run more containers on a given hardware combination than if you were using virtual machines.

Docker is freeing us from the task of managing, maintaining all of the Airflow dependencies, and deployment.
Easy to share and deploy different versions and environments.
Keep track through Github tags and releases.
Ease of deployment from testing to production environment.

Fork and clone the below Github repo and follow the instruction to set up