Anything you add to your local container will be added to the directory you connect it with in Docker. Instead, one solution is to use “volumes”, which allow you to share a directory between your local machine with the Docker container. But a more important reason is that jumping in containers and editing them is considered bad practice and “hacky” in Docker, because you can no longer build the image your container runs on from your Dockerfile. One reason is that the minimal version of Linux installed in the container doesn’t even have a text editor. The thing is, you don’t want to jump into your container and add the DAG definition files directly in there. In our case, the directory we need to add DAGs to in the container is: You create a DAG by defining the script and simply adding it to a folder ‘dags’ within the $AIRFLOW_HOME directory. In Airflow, DAGs definition files are python scripts (“configuration as code” is one of the advantages of Airflow). (In my case, my container was automatically named competent_vaughan by Docker) Running a DAG You can jump into your running container’s command line using the command: Once you do that, Airflow is running on your machine, and you can visit the UI by visiting On the command line, you can find the container name by running: Now that you have the image downloaded, you can create a running container with the following command:ĭocker run -d -p 8080:8080 puckel/docker-airflow webserver You can see your image was downloaded by typing: So, all you have to do to get this pre-made container running Apache Airflow is type:Īnd after a few short moments, you have a Docker image installed for running Airflow in a Docker container. You can find the github repo associated with this container here. You can find the documentation for this repo here. We’ll be using the second one: puckel/docker-airflow which has over 10 million pulls and almost 100 stars. Once you do that, go to Docker Hub and search “Airflow” in the list of repositories, which produces a bunch of results. This rest of this post focuses on deploying Airflow with docker and it assumes you are somewhat familiar with Docker or you have read my previous article on getting started with Docker.Īs a first step, you obviously need to have Docker installed and have a Docker Hub account. Deploying Airflow with Docker and Running your First DAGīased on content from: Getting Started with Airflow Using Docker, Mark Nagelberg
0 kommentar(er)
