Airflow Nodejs

Flow-Based Programming (FBP)

NoFlo is a JavaScript implementation of Flow-Based Programming (FBP). Separating the control flow of software from the actual software logic. Helping you organize large applications easier than traditional OOP paradigms, especially when importing and modifying large data sets.

Node.js airflow directed-acyclic-graphs. Improve this question. Follow asked Sep 1 '19 at 11:50. Paul Grimshaw Paul Grimshaw. 14k 5 5 gold badges 32 32 silver badges 54 54 bronze badges. Triggering a DAG with Node.JS and Google Cloud Functions Apache Airflow is designed to run DAGs on a regular schedule, but you can also trigger DAGs in response to events, such as a change in a Cloud Storage bucket or a message pushed to Cloud Pub/Sub.

NoFlo and Node.js

FBP itself is not a new idea: it comes from a term IBM used in the 70s to define the concepts of more manageable business workflows. NoFlo, however, refreshes these concepts and brings them to web and Node.js applications.

NoFlo library

The MIT-licensed NoFlo library can either be used to run full flow-based applications or as a library for making complex workflows or asynchronous processes more manageable. It can also be used as an Extract, Transform, Load (ETL) tool in business applications.

NoFlo components can be written in any language that transpiles into JavaScript including ES6.

I remember the first time I discovered cron. I’ve been obsessed with automation my whole life, and as a fledgling developer cron became my workhorse.

At the time, I worked at an IT help desk. I used cron for email reminders, automated reports, backups: anything I could schedule, I did.


cron is simple: you just tell it what you want to run, when you want to run it:

But it comes with tradeoffs: primarily, you have to operate a server to run it, which costs money and time. If you don’t already have a server where you can run jobs, setting one up just for cron is far from ideal.

Operator

Airflow Nodejs

In the era of the cloud, it’s also outdated. Cron jobs are simple only if you cut your teeth on Linux and can effectively administer a server. This is an inaccessible option for many modern devs who operate far up the stack.

So it’s no surprise that we have other choices for scheduling code in 2020:

  • etc.

These options work well in their intended context: Lambda is great when you’re glueing together AWS resources. Kubernetes CronJobs or ECS Tasks are ideal when your code runs in a container. Airflow is great when you have complex dependencies between jobs, for example in data pipelines.

But when you just want to run a script on a schedule, they’re overkill.

Airflow Nodejs Operator

Enter Pipedream


Pipedream is an integration platform for developers. You run serverless workflows - any Node.js code - triggered by HTTP requests, timers, emails, and more. You can also use pre-built actions to connect to hundreds of APIs and apps - actions are just Node functions that perform common operations against these APIs.

You can sign up and run any Node code on a schedule in less than one minute. Watch this video or follow the step-by-step instructions below:

Airflow Nodejs

First, install the Pipedream CLI:

cd into a directory with a Node script you’d like to run, or just create a simple one-line script:

then deploy that script to Pipedream, running it every 15 seconds:

This step will prompt you to sign up for Pipedream if you haven’t already. The pd CLI will deploy your code to Pipedream, and print logs as the script produces them. You can press Ctrl-C to quit the real-time stream, and listen for new logs later by running:

You can delete this job and all its logs by running:

You can list all running jobs with pd list, update their code or schedule with pd update, and more. See the docs to learn more.

A practical example: send an HTTP request on a schedule

If you already host code at some URL, and just want to trigger a job to run via HTTP request, you can use Pipedream to send that request on a schedule. In this example, we’ll hit the Star Wars API once a day.

First, open your editor and create a file called http.js with the following contents:

Then run

This will make a GET request to SWAPI, once a day at midnight, but you can modify this script to send any HTTP request, at any schedule. See our guide on making HTTP requests in Node for more examples.

See the README on cron jobs or the Pipedream docs to learn more about the platform at large.

Other Features and Feedback

Airflow Run Nodejs

We’re shipping new features every day and eager for your feedback. Check out our roadmap and reach out anytime via email or Slack to let us know what we can improve.