Airflow Fargate

Using Operator ¶. Use the ECSOperator to run a task defined in AWS ECS. In the following example, the task “helloworld” runs hello-world task in c cluster. It overrides the command in the hello-world-container container. Before using ECSOperator, cluster and task definition need to be created. Airflow on AWS Fargate We propose the creation of a new Airflow Executor, called the FargateExecutor, that runs tasks asynchronously on AWS Fargate. The Airflow Scheduler comes up with a command that needs to be executed in some shell. Afctl - A CLI tool that includes everything required to create, manage and deploy airflow projects faster and smoother. Airflow-aws-executors - Run Airflow Tasks directly on AWS Batch, AWS Fargate, or AWS ECS; provisioning less infra is more. Airflow-code-editor - A tool for Apache Airflow that allows you to edit DAGs in browser.

Airflow eks fargateApache airflow fargateAirflow FargateExecutorAirflow
30 minutes to deploying your first task
1-2 days minimum. Need to setup hardware (server, database, message queues), install packages + dependencies
Hosted solution. Zero maintenance of CloudReactor instance required.
Need to set up monitoring on Airflow server. Installing upgrades to Airflow packages and dependencies is time consuming and could lead to a system halt
Code in your existing language. Code is deployed as a task to AWS ECS
Need to learn Airflow-specific concepts such as operators, hooks, DAGs, workers and schedulers. Extensive tutorials available online.
Create workflows in seconds (really) with a UI-based editor
Workflows (DAGs) are defined via code. This allows for extreme flexibility, but also makes creation onerous
Fully supported
Open source: no support available
Runs serverless on AWS ECS Fargate. Scaling is simple
Airflow instances have to be provisioned for peak capacity. Scaling capacity may not be straightforward

Airflow Ecs Fargate

Ensure that an Internet gateway is attached to the Route Table so the client application, Airflow, can access the Fargate cluster and launch the containers in these subnets. Select the Subnet Associations tab, then copy the Subnet IDs.