The CLI

What is this tutorial?

It's a hands-on way to learn about the new OpenFn CLI. By following the prompts and "challenges", a developer with a bit of Javascript experience should be able to write, run, and debug complex, multi-step jobs with OpenFn, using nothing but a text editor and their terminal.
The estimated time to finish this developer challenge is 1 to 2 hours (depending on your familiarity with the underlying concepts and tooling)
If you are stuck and need help, please post in community.openfn.org

Intro to the OpenFn CLI

The @openfn/cli is a command line interface for running OpenFn workflows locally. It enables developers to run, build, and test steps in an OpenFn workflow.

This CLI replaces @openfn/devtools and provides a new suite of features and improvements, including:

a new runtime and compiler for executing and creating runnable OpenFn jobs,
customizable logging output,
automatic installation of language adaptors,
and support for the adaptors monorepo (@openfn/adaptors) where all OpenFn adaptor source code and documentation lives.

These features are designed to make it easier and more convenient for developers to use and understand OpenFn.

Looking for a way to execute jobs from OpenFn v1 locally? Use Core!

If you're looking for a way to execute jobs running on the OpenFn v1 platform, please see the documentation for @openfn/core and Devtools.

Prerequisites

Ensure you have a code editor installed on your machine (e.g. VS Code, Sublime)
Install NodeJs v18 is the minimum version required
- To install a specific version of Node.js (in this case, version 18) on Linux, Windows, or macOS, you can use a version manager such as nvm (Node Version Manager) or any multiple runtime version manager eg: asdf. These tools allow you to install and switch between multiple versions of Node.js on the same machine. See below for instructions for different operating systems.
- Read this article to learn how to install NodeJs in your machine kinsta.com/blog/how-to-install-node-js/
Have a basic understanding of OpenFn—check out jobs and adaptors, at least, in the OpenFn Concepts of this site.
Install the OpenFn CLI with npm install -g @openfn/cli

Walkthrough & Challenges

1. Getting started with the CLI

Let's start by running a simple command with the CLI. Type the following into your terminal:

openfn test

The word openfn will invoke the CLI. The word test will invoke the test command.

You should see some output like this:

[CLI] ℹ Versions:
         ▸ node.js     18.12.1
         ▸ cli         0.0.29
         ▸ runtime     0.0.19
         ▸ compiler    0.0.25
[CLI] ℹ Running test job...
[CLI] ✔ Compiled job
[JOB] ℹ Calculating the answer to life, the universe, and everything...
[R/T] ✔ Operation 1 complete in 1ms
[CLI] ✔ Result: 42

What we've just done is executed a JavaScript expression, which we call a job. The output prefixed with [JOB] comes directly from console.log statements in our job code. All other output is the CLI trying to tell us what it is doing.

What is a job?

A job Javascript code which follows a particular set of conventions. Typically a job has one or more operations which perform a particular task (like pulling information from a database, creating a record, etc.) and return state for the next operation to use.

The test job we just ran looks like this:

const fn = () => state => {
  console.log(
    'Calculating the answer to life, the universe, and everything...'
  );
  return state * 2;
};
export default [fn()];

You can see this (and a lot more detail) by running the test command with debug-level logging:

openfn test --log debug

Tasks:

Create a file called hello.js and write the following code.
```
console.log('Hello World!');
```
What is console.log?
console.log is a core JavaScript language function which lets us send messages to the terminal window.
Run the job using the CLI
```
openfn hello.js
```

Note that our console.log statement was printed as [JOB] Hello world!. Using the console like this is helpful for debugging and/or understanding what's happening inside our jobs.

🏆 Challenge: Write a job that prints your name

Modify hello.js to print your name.
Re-run the job by running openfn hello.js -a http.

Validate that you receive the logs below:

[CLI] ✔ Compiled job from hello.js
[JOB] ℹ My name is { YourName }
[R/T] ✔ Operation 1 complete in 0ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 366ms! ✨

2. Using adaptor helper functions

Adaptors are Javascript or Typescript modules that provide OpenFn users with a set of helper functions for simplifying communication with a specific external system. Learn more about adaptors here: docs.openfn.org/adaptors

Basic usage:

Let’s use @openfn/language-http adaptor to fetch a list of forms from https://jsonplaceholder.typicode.com/

Understanding CLI arguments

Use -a to specify the adaptor; use -i to auto-install the necessary adaptor

Run openfn help to see the full list of CLI arguments.

Tasks:

Create a file called getPosts.js and write the following code

get('https://jsonplaceholder.typicode.com/posts');
fn(state => {
  console.log(state.data[0]);
  return state;
});

Run the job by running

openfn getPosts.js -i -a http

Since it is our first time using the http adaptor, we are installing the adaptor using -i argument

See expected CLI logs

[CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 1.072s
[JOB] ℹ {
 userId: 1,
 id: 1,
 title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
 body: 'quia et suscipit\n' +
   'suscipit recusandae consequuntur expedita et cum\n' +
   'reprehenderit molestiae ut ut quas totam\n' +
   'nostrum rerum est autem sunt rem eveniet architecto'
}
[R/T] ✔ Operation 2 complete in 0ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 1.42s! ✨

🏆 Challenge: Get and inspect data via HTTP

Using the https://jsonplaceholder.typicode.com/users API, get a list of users and print the first user object.

Create file called getUsers.js and write your operation to fetch the user.
Run the job using the OpenFn/cli openfn getUsers.js -a http.
Validate that you receive this expected CLI logs:

openfn getUsers.js -a http

Validate that you receive this expected CLI logs:

[CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 581ms
[JOB] ℹ {
  id: 1,
  name: 'Leanne Graham',
  username: 'Bret',
  email: 'Sincere@april.biz',
  address: {
    street: 'Kulas Light',
    suite: 'Apt. 556',
    city: 'Gwenborough',
    zipcode: '92998-3874',
    geo: { lat: '-37.3159', lng: '81.1496' }
  },
  phone: '1-770-736-8031 x56442',
  website: 'hildegard.org',
  company: {
    name: 'Romaguera-Crona',
    catchPhrase: 'Multi-layered client-server neural-net',
    bs: 'harness real-time e-markets'
  }
}
[R/T] ✔ Operation 2 complete in 2ms
[CLI] ✔ Writing output to ./output.json [CLI] ✔ Done in 950ms! ✨

3. Understanding `state`

If a job expression is a set of instructions for a chef (a recipe?) then the initial state is all of the ingredients they need tied up in a perfect little bundle. See "It all starts with state" in the knowledge base for extra context.

It usually looks something like this

{
  "configuration": {
    "hostUrl": "https://moh.kenya.gov.ke/dhis2",
    "username": "someone",
    "password": "something-secret"
  },
  "data": {
    "type": "registration",
    "patient": {
      "age": 24,
      "gender": "M",
      "nationalId": "321cs7"
    }
  }
}

`state.configuration`

This key is where we put credentials which are used to authorize connections to any authenticated system that the job will interact with. (Note that this part of state is usually overwritten at runtime with a real "credential" when using the OpenFn platform, rather than the CLI.)

Important

Note that console.log(state) will display the whole state, including state.configuration elements such as username and password. Remove this log whenever you're done debugging to avoid accidentally exposing sensitive information when the job is successfully deployed on production.

The OpenFn platform has built in protections to "scrub" state from the logs, but when you're using the CLI directly you're on your own!

`state.data`

This key is where we put data related to a specific job run. On the platform, it's the work-order-specific data from a triggering HTTP request or some bit of information that's passed from one job to another.

Using CLI, state.json will be loaded automatically from the current directory

Or you can specify the path to the state file by passing the option -s, --state-path

Specify a path to your state.json file with this command:

openfn hello.js -a http -s tmp/state.json

Expected CLI logs

[CLI] ✔ Compiled job from hello.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 876ms
[R/T] ✔ Operation 2 complete in 0ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 1.222s! ✨

How can we use state?

Each adaptor has a configuration schema that's recommended for use in your state.json. Here is an example of how to set up state.configuration for language-http.

{
  "username": "name@email",
  "password": "supersecret",
  "baseUrl": "https://jsonplaceholder.typicode.com"
}

Tasks:

Update your state.json to look like this:
```
{
  "configuration": {
    "baseUrl": "https://jsonplaceholder.typicode.com"
  }
}
```
Since we have update our configuration in our state.json we can now use get() helper function without the need to specify the baseUrl—i.e get('posts')

Update your getPosts.js job to look like this:

getPosts.js
// Get all posts
get('posts');

fn(state => {
  const posts = state.data;
  console.log(posts[0]);
  return state;
});

Now run the job using the following command

openfn getPosts.js -a http -s tmp/state.json

And validate that you see the expected CLI logs:

[CLI] ✔ Compiled job from job.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 120ms
[JOB] ℹ {
 userId: 1,
 id: 1,
 title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
 body: 'quia et suscipit\n' +
   'suscipit recusandae consequuntur expedita et cum\n' +
   'reprehenderit molestiae ut ut quas totam\n' +
   'nostrum rerum est autem sunt rem eveniet architecto'
}
[R/T] ✔ Operation 2 complete in 0ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 470ms! ✨

🏆 Challenge: Fetch Covid-19 metadata

Using the disease.sh API, write an operation that returns all covid-19 metadata.

tip

https://disease.sh/v3/covid-19/ as your baseUrl in state.configuration

Validate your output: there are a lot of ways you might choose to format or display this data. Share your results with your administrator for feedback.

4. Additional arguments and commands

🏆 Challenge: Practice CLI arguments and commands

Perform these tasks and submit answers to the discussion questions to your administrator for feedback.

Compile a openfn job (hello.js).
What's the difference between the job you wrote and the compiled job?
Run a job without "strict mode" enabled.
What's the difference between the outputs when strict mode is enabled and disabled?
Run a job with the log level set to none, and then run it again with the log level set to debug.
When is it appropriate to use these different log levels?

5. Manipulating data in a sequence of operations

In most cases you need to manipulate, clean, or transform data at some step in your workflow. For example after we get data from the https://jsonplaceholder.typicode.com registry we might need to group the posts by user id. The example below shows how we can:

get all posts and return them in state.data
group returned posts by userId
log posts with userId 1

Example:

getPosts.js
// Get all posts
get('posts');

// Group posts by user id
fn(state => {
  const posts = state.data;

  // Group posts by userId
  const groupPostsByUserId = posts.reduce((acc, post) => {
    const existingValue = acc[post.userId] || [];
    return { ...acc, [post.userId]: [...existingValue, post] };
  }, {});

  // console.log(groupPostsByUserId);
  return { ...state, groupPostsByUserId };
});

// Log posts where userId = 1
fn(state => {
  const { groupPostsByUserId } = state;
  console.log('Post with userId 1', groupPostsByUserId[1]);
  return state;
});

What is array.reduce?

The reduce() method applies a function against an accumulator and each value of the array (from left-to-right) to reduce it to a single value.

Perhaps the easiest-to-understand case for reduce() is to return the sum of all the elements in an array:

JavaScript Demo: `Array.reduce()`

const array1 = [1, 2, 3, 4];

// 0 + 1 + 2 + 3 + 4
const initialValue = 0;
const sumWithInitial = array1.reduce(
  (accumulator, currentValue) => accumulator + currentValue,
  initialValue
);

console.log(sumWithInitial);
// Expected output: 10

You can learn more about array.reduce from this article

Expected CLI logs

[CLI] ✔ Compiled job from getPosts.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 825ms
[R/T] ✔ Operation 2 complete in 0ms
[JOB] ℹ Post with userId 1 [
 //All of posts for userId 1
]
[R/T] ✔ Operation 3 complete in 12ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 1.239s! ✨

🏆 Challenge: extract names & emails

Using https://jsonplaceholder.typicode.com/posts/1/comments API fetch comments for post with id 1 and extract name and email from each comment in that post

Get post all comments for post id 1
Extract name and email from comments
Log the extracted data from comments

Discuss the results with your administrator.

6. Debugging errors

When debugging, it’s interesting to use log to have a visual representation of the content of the manipulated objects (such as state).

When you want to inspect the content of state in between operations, add an fn() block with a console.log:

// firstOperation(...);

fn(state => {
  console.log(state);
  return state;
});

// secondOperation(...);

Create debug.js and paste the code below

debug.js
// Get all posts
get('posts');

// Get post by index helper function
fn(state => {
  // const getPostbyIndex = (index) => dataValue(index)(state);
  console.log(dataValue(1));

  return { ...state };
});

Run openfn debug.js -a http

Expected CLI logs

[CLI] ✘ TypeError: path.match is not a function
    at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26)
    at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22)
    at getPostbyIndex (vm:module(0):5:37)
    at vm:module(0):18:36
    at /tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:241:12
    at file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:288:26
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:269:18)
    at async executeHandler (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/dist/process/runner.js:388:20)

As you can see from our logs that helper function dataValue has a TypeError, To troubleshoot this you can go to the documentation for dataValue -> docs.openfn.org/adaptors/packages/common-docs/#datavaluepath--operation

According to the docs, dataValue take path which is a string type. But in our operation we were passing an integer, that’s why we have a TypeError. You can fix the error by passing a string in dataValue i.e console.log(dataValue(“1”))

Expected CLI logs

[CLI] ✔ Compiled job from debug.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 722ms
[JOB] ℹ [Function (anonymous)]
[R/T] ✔ Operation 2 complete in 1ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 1.102s ✨

If you need more information for debugging you can pass -l debug which will give all information about the run

i.e openfn debug.js -a http -l debug

🏆 Challenge: control error messages

Debug what is causing an error on the following line of code and display the error message

// Get post where id is 180
get('posts/180');

Discuss the results with your administrator.

7. Each and array iteration

We often have to perform the same operation multiple times for items in an array. Most of the helper functions for data manipulation are inherited from @openfn/language-common and are available in most of the adaptors.

Create job.js and add the following codes

// Get all posts
get('posts');

// Group posts by user
fn(state => {
  const posts = state.data;

  // Group posts by userId
  const groupPostsByUserId = posts.reduce((acc, post) => {
    const existingValue = acc[post.userId] || [];
    return { ...acc, [post.userId]: [...existingValue, post] };
  }, {});

  // console.log(groupPostsByUserId);
  return { ...state, groupPostsByUserId };
});

// Log posts where userId = 1
fn(state => {
  const { groupPostsByUserId } = state;
  const posts = groupPostsByUserId[1];

  // console.log("Post with userId 1", groupPostsByUserId[1]);
  return { ...state, posts };
});

each('posts[*]', state => {
  console.log('Post', JSON.stringify(state.data, null, 2));
  return state;
});

Notice how this code uses the each function, a helper function defined in language-common but accessed from this job that is using language-http. Most adaptors import and export many functions from language-common.

Run openfn job.js -a http

Expected CLI logs

[CLI] ✔ Compiled job from job.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 730ms
[R/T] ✔ Operation 2 complete in 0ms
[R/T] ✔ Operation 3 complete in 0ms
[JOB] ℹ Posts [
// Posts
]
[R/T] ✔ Operation 4 complete in 10ms
[CLI] ✔ Writing output to output.json
[CLI] ✔ Done in 1.091s! ✨

🏆 Challenge: Reduce, filter, and map

Using Javascript globals i.e Array.reduce, Array.filter or Array.map, build function that will get posts by user id.

Create a file called job1.js
Add the 1st operation which is get all posts
Add 2nd operation which has a function that filter posts by id
Use the function from 2nd operation to get all post for user id 1

Discuss the results with your administrator.

CLI Usage - Key Commands

You’ll learn about these commands in the following challenges, but please refer to this section for the key commands used in working with the CLI.

Check the version

openfn version

Get help

openfn help

Run a job

openfn path/to/job.js -ia {adaptor-name}

Note: You MUST specify which adaptor to use. Pass the -i flag to auto-install that adaptor (it's safe to do this redundantly).

You can find the list of publicly available adaptors here.

Path is the job to load the job from (a .js file or a dir containing a job.js file) For example openfn execute hello.js Reads hello.js, looks for state and output in foo

-i, --autoinstall Auto-install the language adaptor
-a, --adaptors, --adaptor A language adaptor to use for the job

If an adaptor is already installed by auto install, you can use the command without the -i options. i.e openfn hello.js -a http

Change log level

You can pass -l info or --log info to get more feedback about what's happening, or --log debug for more details than you could ever use. Below is the list of different log levels

openfn hello.js -a http -l none

log level	description
`-l none`	Quiet mode
`-l default`	Top level information of what is happening
`-l info`	Get more feedback on what is happening openfn
`-l debug`	Get information about runtime, cli, compiler and the job

Compilation

The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust, portable, and easier to debug from a pure JS perspective.

openfn compile [path]

Will compile the openfn job and print or save the resulting js.

Learn more about CLI github.com/OpenFn/kit/

Intro to the OpenFn CLI​

Prerequisites​

Walkthrough & Challenges​

1. Getting started with the CLI​

Tasks:​

🏆 Challenge: Write a job that prints your name​

2. Using adaptor helper functions​

Basic usage:​

Tasks:​

🏆 Challenge: Get and inspect data via HTTP​

3. Understanding state​

state.configuration​

state.data​

How can we use state?​

Tasks:​

🏆 Challenge: Fetch Covid-19 metadata​

4. Additional arguments and commands​

🏆 Challenge: Practice CLI arguments and commands​

5. Manipulating data in a sequence of operations​

Example:​

JavaScript Demo: Array.reduce()​

🏆 Challenge: extract names & emails​

6. Debugging errors​

Create debug.js and paste the code below​

Run openfn debug.js -a http​

🏆 Challenge: control error messages​

7. Each and array iteration​

Create job.js and add the following codes​

Run openfn job.js -a http​

🏆 Challenge: Reduce, filter, and map​

CLI Usage - Key Commands​

Check the version​

Get help​

Run a job​

Change log level​

Compilation​

Intro to the OpenFn CLI

Prerequisites

Walkthrough & Challenges

1. Getting started with the CLI

Tasks:

🏆 Challenge: Write a job that prints your name

2. Using adaptor helper functions

Basic usage:

Tasks:

🏆 Challenge: Get and inspect data via HTTP

3. Understanding `state`

`state.configuration`

`state.data`

How can we use state?

Tasks:

🏆 Challenge: Fetch Covid-19 metadata

4. Additional arguments and commands

🏆 Challenge: Practice CLI arguments and commands

5. Manipulating data in a sequence of operations

Example:

JavaScript Demo: `Array.reduce()`

🏆 Challenge: extract names & emails

6. Debugging errors

Create debug.js and paste the code below

Run openfn debug.js -a http

🏆 Challenge: control error messages

7. Each and array iteration

Create job.js and add the following codes

Run openfn job.js -a http

🏆 Challenge: Reduce, filter, and map

CLI Usage - Key Commands

Check the version

Get help

Run a job

Change log level

Compilation