Airflow Docker

-t hiprabhat/airflow:latest docker push hiprabhat/airflow:latest Replace the name of your docker repo from hiprabhat/airflow Now update the image section in values.yaml to. Dockerfile is used to create image to be used by Airflow web server, scheduler and workers. Docker-compose.yml is used to define and start all the containers. A simple model is proposed to classify famous iris datasets. I have added two DAGs with PythonOperator in the dags folder.

Latest version
  • In order to quick change airflow config inside a docker container,There are many ways. Instead of change airflow.cfg, you can change environment variable directly. In docker container, it can very easy to revise in docker-compose.yml directly. And you can just restart the docker-compose quickly. Here is some common configuration variable.
  • Apache Airflow is an open-source tool to programmatically author, schedule and monitor workflows. Developed back in 2014 by Airbnb, and later released as open source, Airflow has become a very popular solution, with more than 16 000 stars in GitHub.
  • In this blog, we are going to run the sample dynamic DAG using docker.Airflow is a workflow engine which is responsible for managing and scheduling running jobs and data pipelines.

Released:

An opinionated implementation of exclusively using airflow DockerOperators for all Operators

Project description

Description

An opinionated implementation of exclusively using airflowDockerOperators for all Operators.

Default Operator

Default Sensor

Task Code

Branch Operator

Dag Task

Task Code

Short Circuit Operator

Dag Task

Task Code

Context Usage

Dag Task

Task Code

Configuration

The following operator defaults can be set under the airflowdockernamespace:

  • force_pull (boolean true/false)
  • auto_remove (boolean true/false)
  • network_mode

For example, to set force_pull to False by default set the followingenvironment variable like so:

This package works as an airflow plugin as well. When installed andrunning airflow, dags can import like so

Apache

i.e.

Tests

We also ship an airflowdocker/tester image to verify the integrity ofyour DAG definitions before committing them.

One can run the tests against your own dags like so:

or else see theairflow-docker-composeproject which ships with a test subcommand for precisely this purpose.

Release historyRelease notifications RSS feed

2.1.1

2.1.0

2.0.3

2.0.2

2.0.1

2.0.0

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.0

0.5.0

0.4.6

0.4.5

0.4.4

0.4.3

0.4.2

0.4.1

0.4.0

0.3.0

0.2.2

0.2.1

0.2.0

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for airflow-docker, version 2.1.1
Filename, sizeFile typePython versionUpload dateHashes
Filename, size airflow_docker-2.1.1-py2.py3-none-any.whl (20.7 kB) File type Wheel Python version py2.py3 Upload dateHashes
Filename, size airflow-docker-2.1.1.tar.gz (17.5 kB) File type Source Python version None Upload dateHashes
Close

Hashes for airflow_docker-2.1.1-py2.py3-none-any.whl

Hashes for airflow_docker-2.1.1-py2.py3-none-any.whl
AlgorithmHash digest
SHA256f85b85f95c7a675c1159cd1f37f12d13344007eef005e45fb191bee81442eb47
MD544db3da4f67498671c1d16e6c5c8ed9f
BLAKE2-25621ac91322f7b8a284a3f305255dad805f85ccd45365f87203a2fa6eb88f60cea
Close

Hashes for airflow-docker-2.1.1.tar.gz

Hashes for airflow-docker-2.1.1.tar.gz
AlgorithmHash digest
SHA25628149435931b29ded3c196ad48c6a22219870df036cda164eddfb19c69a41f35
MD57a7da8074b41ad0bd43fd41498b8111e
BLAKE2-2562913f54226f8d4d3d0840b661df0f7f7067cbedce39c311cd6572b30961a6c9b

Welcome to Airflow Docker!

What is it?

Airflow Docker is an extension to the open source project Airflow. Specifically it provides a base operator, forked from the existing docker operator, and a number of operators, and sensors on top of it, all that are fundamentally a wrapped docker run command.

Airflow Docker Image

Ok. Its docker. We get it. But why?!?

By standardizing around a single execution pattern, namely everything is a docker operator, a number of benefits fall into place:

  1. All of the normal benefits of docker. Shared layers, immutable artifacts, artifact versioning, etc.
  2. Because everything, from a sensor, to a short circuit operation, to a sql query type operation, to a standard python type operation, we were able to begin to build useful building blocks that augmented this standard run time behavior.
  3. Isolation with respect to the airflow deployment itself - we can feel a lot more confident upgrading airflow or one of its dependencies if that has almost no chance of breaking somene’s tasks.

So what is included?

Docker Airflow Image

  1. airflow-docker: This is the core library. It is inteded to be installed in the same environment as airflow. We publish python packages to the python package index and a couple of purpose build docker images to dockerhub.
  2. airflow-docker-helper: This is a lightweight, pure python library intended to be installed in your docker images that provides useful primitives for interacting with airflow from within the context of a running docker container.

… And more

Airflow Docker Puckel

List of Repos

  • Github Repos: see list
  • Slack: slack