Version: v1.105

The CLI

What is this tutorial?

It's a hands-on way to learn about the new OpenFn CLI. By following the prompts and "challenges", a developer with a bit of Javascript experience should be able to write, run, and debug complex, multi-step jobs with OpenFn, using nothing but a text editor and their terminal.
The estimated time to finish this developer challenge is 1 to 2 hours (depending on your familiarity with the underlying concepts and tooling)
If you are stuck and need help, please post in community.openfn.org

Intro to the OpenFn CLI

The @openfn/cli is a command line interface for running OpenFn workflows locally. It enables developers to run, build, and test steps in an OpenFn workflow.

This CLI replaces @openfn/devtools and provides a new suite of features and improvements, including:

a new runtime and compiler for executing and creating runnable OpenFn jobs,
customizable logging output,
automatic installation of language adaptors,
and support for the adaptors monorepo (@openfn/adaptors) where all OpenFn adaptor source code and documentation lives.

These features are designed to make it easier and more convenient for developers to use and understand OpenFn.

Looking for a way to execute jobs from OpenFn v1 locally? Use Core!

If you're looking for a way to execute jobs running on the OpenFn v1 platform, please see the documentation for @openfn/core and Devtools.

Prerequisites

Ensure you have a code editor installed on your machine (e.g. VS Code, Sublime)
Install NodeJs v18 is the minimum version required
- To install a specific version of Node.js (in this case, version 18) on Linux, Windows, or macOS, you can use a version manager such as nvm (Node Version Manager) or any multiple runtime version manager eg: asdf. These tools allow you to install and switch between multiple versions of Node.js on the same machine. See below for instructions for different operating systems.
- Read this article to learn how to install NodeJs in your machine kinsta.com/blog/how-to-install-node-js/
Have a basic understanding of OpenFn—check out jobs and adaptors, at least, in the OpenFn Concepts of this site.
Install the OpenFn CLI with npm install -g @openfn/cli

Walkthrough & Challenges

1. Getting started with the CLI

Let's start by running a simple command with the CLI. Type the following into your terminal:

openfn test

The word openfn will invoke the CLI. The word test will invoke the test command.

You should see some output like this:

[CLI] ℹ Versions:
        ▸ node.js     18.12.1
        ▸ cli         0.0.39
        ▸ runtime     0.0.24
        ▸ compiler    0.0.32
[CLI] ℹ Running test job...
[CLI] ℹ Workflow object:
[CLI] ℹ {
  "start": "start",
  "jobs": [
    {
      "id": "start",
      "data": {
        "defaultAnswer": 42
      },
      "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()",
      "next": {
        "calculate": "!state.error"
      }
    },
    {
      "id": "calculate",
      "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()",
      "next": {
        "result": true
      }
    },
    {
      "id": "result",
      "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()"
    }
  ]
}

[CLI] ✔ Compilation complete
[R/T] ♦ Starting job start
[JOB] ℹ Starting computer...
[R/T] ℹ Operation 1 complete in 0ms
[R/T] ✔ Completed job start in 1ms
[R/T] ♦ Starting job calculate
[JOB] ℹ Calculating to life, the universe, and everything..
[R/T] ℹ Operation 1 complete in 0ms
[R/T] ✔ Completed job calculate in 1ms
[R/T] ♦ Starting job result
[R/T] ℹ Operation 1 complete in 0ms
[R/T] ✔ Completed job result in 0ms
[CLI] ✔ Result: 42

What we've just done is executed a JavaScript expression, which we call a job. The output prefixed with [JOB] comes directly from console.log statements in our job code. All other output is the CLI trying to tell us what it is doing.

What is a job?

A job is Javascript code which follows a particular set of conventions. Typically a job has one or more operations which perform a particular task (like pulling information from a database, creating a record, etc.) and return state for the next operation to use.

The test job we just ran looks like this:

const fn = () => state => {
  console.log(
    'Calculating the answer to life, the universe, and everything...'
  );
  return state * 2;
};
export default [fn()];

You can see this (and a lot more detail) by running the test command with debug-level logging:

openfn test --log debug

Tasks:

To get started with @openfn/cli

Create a new folder for the repository you'll be working on by running the following command: mkdir devchallenge && cd devchallenge
While you can keep your job scripts anywhere, it's a good practice to store state.json and output.json in a tmp folder. To do this, create a new directory called tmp within your devchallenge folder: mkdir tmp
Since state.json and output.json may contain sensitive configuration information and project data, it's important to never upload them to GitHub. To ensure that GitHub ignores these files, add the tmp directory to your .gitignore file: echo "tmp" >> .gitignore
(Optional) Use the tree command to check that your directory structure looks correct. Running tree -a in your devchallenge folder should display a structure like this:
```
 devchallenge
 ├── .gitignore
 └── tmp
     ├── state.json
     └── output.json
```

Create a file called hello.js and write the following code.
```
console.log('Hello World!');
```
What is console.log?
console.log is a core JavaScript language function which lets us send messages to the terminal window.

Run the job using the CLI

openfn hello.js -o tmp/output.json

View expected output

  [CLI] ⚠ WARNING: No adaptor provided!
  [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg:
            openfn job.js -a common
  [CLI] ✔ Compiled from helo.js
  [R/T] ♦ Starting job job-1
  [JOB] ℹ Hello World!
  [R/T] ✔ Completed job job-1 in 1ms
  [CLI] ✔ State written to tmp/output.json
  [CLI] ✔ Finished in 17ms ✨

Note that our console.log statement was printed as [JOB] Hello world!. Using the console like this is helpful for debugging and/or understanding what's happening inside our jobs.

🏆 Challenge: Write a job that prints your name

Modify hello.js to print your name.
Re-run the job by running openfn hello.js -a common -o tmp/output.json.
Validate that you receive the logs below:

[CLI] ✔ Compiled job from hello.js
[JOB] ℹ My name is { YourName }
[R/T] ✔ Operation 1 complete in 0ms
[CLI] ✔ Writing output to tmp/output.json
[CLI] ✔ Done in 366ms! ✨

2. Using adaptor helper functions

Adaptors are Javascript or Typescript modules that provide OpenFn users with a set of helper functions for simplifying communication with a specific external system. Learn more about adaptors here: docs.openfn.org/adaptors

Basic usage:

Let’s use @openfn/language-http adaptor to fetch a list of forms from https://jsonplaceholder.typicode.com/

Understanding CLI arguments

Use -a to specify the adaptor; use -i to auto-install the necessary adaptor

Run openfn help to see the full list of CLI arguments.

Tasks:

Create a file called getPosts.js and write the following code

getPosts.js
get('https://jsonplaceholder.typicode.com/posts');
fn(state => {
  console.log(state.data[0]);
  return state;
});

Run the job by running

openfn getPosts.js -i -a http -o tmp/output.json

Since it is our first time using the http adaptor, we are installing the adaptor using -i argument

3. See expected CLI logs

  [CLI] ✔ Installing packages...
  [CLI] ✔ Installed @openfn/language-http@4.2.8
  [CLI] ✔ Installation complete in 14.555s
  [CLI] ✔ Compiled from getPosts.js
  [R/T] ♦ Starting job job-1
  GET request succeeded with 200 ✓
  [JOB] ℹ {
    userId: 1,
    id: 1,
    title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
    body: 'quia et suscipit\n' +
      'suscipit recusandae consequuntur expedita et cum\n' +
      'reprehenderit molestiae ut ut quas totam\n' +
      'nostrum rerum est autem sunt rem eveniet architecto'
  }
  [R/T] ✔ Completed job job-1 in 872ms
  [CLI] ✔ State written to tmp/output.json
  [CLI] ✔ Finished in 15.518s ✨

🏆 Challenge: Get and inspect data via HTTP

Using the https://jsonplaceholder.typicode.com/users API, get a list of users and print the first user object.

Create file called getUsers.js and write your operation to fetch the user.
Run the job using the OpenFn/cli openfn getUsers.js -a http -o tmp/output.json.
Validate that you receive this expected CLI logs:

openfn getUsers.js -a http -o tmp/output.json

See expected CLI logs:

[CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 581ms
[JOB] ℹ {
  id: 1,
  name: 'Leanne Graham',
  username: 'Bret',
  email: 'Sincere@april.biz',
  address: {
    street: 'Kulas Light',
    suite: 'Apt. 556',
    city: 'Gwenborough',
    zipcode: '92998-3874',
    geo: { lat: '-37.3159', lng: '81.1496' }
  },
  phone: '1-770-736-8031 x56442',
  website: 'hildegard.org',
  company: {
    name: 'Romaguera-Crona',
    catchPhrase: 'Multi-layered client-server neural-net',
    bs: 'harness real-time e-markets'
  }
}
[R/T] ✔ Operation 2 complete in 2ms
[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨

3. Understanding `state`

If a job expression is a set of instructions for a chef (a recipe?) then the initial state is all of the ingredients they need tied up in a perfect little bundle. See "It all starts with state" in the knowledge base for extra context.

It usually looks something like this

{
  "configuration": {
    "hostUrl": "https://moh.kenya.gov.ke/dhis2",
    "username": "someone",
    "password": "something-secret"
  },
  "data": {
    "type": "registration",
    "patient": {
      "age": 24,
      "gender": "M",
      "nationalId": "321cs7"
    }
  }
}

`state.configuration`

This key is where we put credentials which are used to authorize connections to any authenticated system that the job will interact with. (Note that this part of state is usually overwritten at runtime with a real "credential" when using the OpenFn platform, rather than the CLI.)

Important

Note that console.log(state) will display the whole state, including state.configuration elements such as username and password. Remove this log whenever you're done debugging to avoid accidentally exposing sensitive information when the job is successfully deployed on production.

The OpenFn platform has built in protections to "scrub" state from the logs, but when you're using the CLI directly you're on your own!

`state.data`

This key is where we put data related to a specific job run. On the platform, it's the work-order-specific data from a triggering HTTP request or some bit of information that's passed from one job to another.

Using CLI, state.json will be loaded automatically from the current directory

Or you can specify the path to the state file by passing the option -s, --state-path

Specify a path to your state.json file with this command:

openfn hello.js -a http -s tmp/state.json -o tmp/output.json

Expected CLI logs

[CLI] ✔ Compiled job from hello.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 876ms
[R/T] ✔ Operation 2 complete in 0ms
[CLI] ✔ Writing output to tmp/output.json
[CLI] ✔ Done in 1.222s! ✨

How can we use state?

Each adaptor has a configuration schema that's recommended for use in your state.json. Here is an example of how to set up state.configuration for language-http.

{
  "username": "name@email",
  "password": "supersecret",
  "baseUrl": "https://jsonplaceholder.typicode.com"
}

Tasks:

Update your state.json to look like this:
state.json
```
{
  "configuration": {
    "baseUrl": "https://jsonplaceholder.typicode.com"
  }
}
```
Since we have update our configuration in our state.json we can now use get() helper function without the need to specify the baseUrl—i.e get('posts')

Update your getPosts.js job to look like this:

getPosts.js
// Get all posts
get('posts');

fn(state => {
  const posts = state.data;
  console.log(posts[0]);
  return state;
});

Now run the job using the following command

openfn getPosts.js -a http -s tmp/state.json -o tmp/output.json

And validate that you see the expected CLI logs:

[CLI] ✔ Compiled job from getPosts.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 120ms
[JOB] ℹ {
 userId: 1,
 id: 1,
 title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
 body: 'quia et suscipit\n' +
   'suscipit recusandae consequuntur expedita et cum\n' +
   'reprehenderit molestiae ut ut quas totam\n' +
   'nostrum rerum est autem sunt rem eveniet architecto'
}
[R/T] ✔ Operation 2 complete in 0ms
[CLI] ✔ Writing output to tmp/output.json
[CLI] ✔ Done in 470ms! ✨

🏆 Challenge: Fetch Covid-19 metadata

Using the The Atlantic's COVID Tracking Project API., write an operation that returns all covid-19 metadata.

tip

https://api.covidtracking.com as your baseUrl in state.configuration

Validate your output: there are a lot of ways you might choose to format or display this data. Share your results with your administrator for feedback.

4. Additional arguments and commands

🏆 Challenge: Practice CLI arguments and commands

Perform these tasks and submit answers to the discussion questions to your administrator for feedback.

Compile a openfn job (hello.js).
What's the difference between the job you wrote and the compiled job?
Run a job with the log level set to none, and then run it again with the log level set to debug.
When is it appropriate to use these different log levels?

5. Manipulating data in a sequence of operations

In most cases you need to manipulate, clean, or transform data at some step in your workflow. For example after we get data from the https://jsonplaceholder.typicode.com registry we might need to group the posts by user id. The example below shows how we can:

get all posts and return them in state.data
group returned posts by userId
log posts with userId 1

Example:

getPosts.js
// Get all posts
get('posts');

// Group posts by user id
fn(state => {
  const posts = state.data;

  // Group posts by userId
  const groupPostsByUserId = posts.reduce((acc, post) => {
    const existingValue = acc[post.userId] || [];
    return { ...acc, [post.userId]: [...existingValue, post] };
  }, {});

  // console.log(groupPostsByUserId);
  return { ...state, groupPostsByUserId };
});

// Log posts where userId = 1
fn(state => {
  const { groupPostsByUserId } = state;
  console.log('Post with userId 1', groupPostsByUserId[1]);
  return state;
});

What is array.reduce?

The reduce() method applies a function against an accumulator and each value of the array (from left-to-right) to reduce it to a single value.

Perhaps the easiest-to-understand case for reduce() is to return the sum of all the elements in an array:

JavaScript Demo: `Array.reduce()`

const array1 = [1, 2, 3, 4];

// 0 + 1 + 2 + 3 + 4
const initialValue = 0;
const sumWithInitial = array1.reduce(
  (accumulator, currentValue) => accumulator + currentValue,
  initialValue
);

console.log(sumWithInitial);
// Expected output: 10

You can learn more about array.reduce from this article

Expected CLI logs

[CLI] ✔ Compiled job from getPosts.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 825ms
[R/T] ✔ Operation 2 complete in 0ms
[JOB] ℹ Post with userId 1 [
 //All of posts for userId 1
]
[R/T] ✔ Operation 3 complete in 12ms
[CLI] ✔ Writing output to tmp/output.json
[CLI] ✔ Done in 1.239s! ✨

🏆 Challenge: extract names & emails

Using https://jsonplaceholder.typicode.com/posts/1/comments API fetch comments for post with id 1 and extract name and email from each comment in that post

Get post all comments for post id 1
Extract name and email from comments
Log the extracted data from comments

Discuss the results with your administrator.

6. Debugging errors

When debugging, it’s interesting to use log to have a visual representation of the content of the manipulated objects (such as state).

When you want to inspect the content of state in between operations, add an fn() block with a console.log:

// firstOperation(...);

fn(state => {
  console.log(state);
  return state;
});

// secondOperation(...);

Create debug.js and paste the code below

debug.js
// Get all posts
get('posts');

// Get post by index helper function
fn(state => {
  // const getPostbyIndex = (index) => dataValue(index)(state);
  console.log(dataValue(1));

  return { ...state };
});

Run openfn debug.js -a http

Expected CLI logs

[CLI] ✘ TypeError: path.match is not a function
    at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26)
    at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22)
    at getPostbyIndex (vm:module(0):5:37)
    at vm:module(0):18:36
    at /tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:241:12
    at file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:288:26
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:269:18)
    at async executeHandler (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/dist/process/runner.js:388:20)

As you can see from our logs that helper function dataValue has a TypeError, To troubleshoot this you can go to the documentation for dataValue -> docs.openfn.org/adaptors/packages/common-docs/#datavaluepath--operation

According to the docs, dataValue take path which is a string type. But in our operation we were passing an integer, that’s why we have a TypeError. You can fix the error by passing a string in dataValue i.e console.log(dataValue(“1”))

Expected CLI logs

[CLI] ✔ Compiled job from debug.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 722ms
[JOB] ℹ [Function (anonymous)]
[R/T] ✔ Operation 2 complete in 1ms
[CLI] ✔ Writing output to tmp/output.json
[CLI] ✔ Done in 1.102s ✨

If you need more information for debugging you can pass -l debug which will give all information about the run

i.e openfn debug.js -a http -l debug

🏆 Challenge: control error messages

Debug what is causing an error on the following line of code and display the error message

// Get post where id is 180
get('posts/180');

Discuss the results with your administrator.

7. Each and array iteration

We often have to perform the same operation multiple times for items in an array. Most of the helper functions for data manipulation are inherited from @openfn/language-common and are available in most of the adaptors.

Modify getPosts.js to group posts by user-ID

getPosts.js
// Get all posts
get('posts');

// Group posts by user
fn(state => {
  const posts = state.data;

  // Group posts by userId
  const groupPostsByUserId = posts.reduce((acc, post) => {
    const existingValue = acc[post.userId] || [];
    return { ...acc, [post.userId]: [...existingValue, post] };
  }, {});

  // console.log(groupPostsByUserId);
  return { ...state, groupPostsByUserId };
});

// Log posts where userId = 1
fn(state => {
  const { groupPostsByUserId } = state;
  const posts = groupPostsByUserId[1];

  // console.log("Post with userId 1", groupPostsByUserId[1]);
  return { ...state, posts };
});

each('posts[*]', state => {
  console.log('Post', JSON.stringify(state.data, null, 2));
  return state;
});

Notice how this code uses the each function, a helper function defined in language-common but accessed from this job that is using language-http. Most adaptors import and export many functions from language-common.

Run openfn getPosts.js -a http -o tmp/output.json

Expected CLI logs

[CLI] ✔ Compiled job from getPosts.js
GET request succeeded with 200 ✓
[R/T] ✔ Operation 1 complete in 730ms
[R/T] ✔ Operation 2 complete in 0ms
[R/T] ✔ Operation 3 complete in 0ms
[JOB] ℹ Posts [
// Posts
]
[R/T] ✔ Operation 4 complete in 10ms
[CLI] ✔ Writing output to tmp/output.json
[CLI] ✔ Done in 1.091s! ✨

🏆 Challenge: Reduce, filter, and map

Using Javascript globals i.e Array.reduce, Array.filter or Array.map, build function that will get posts by user id.

Create a file called job1.js
Add the 1st operation which is get all posts
Add 2nd operation which has a function that filter posts by id
Use the function from 2nd operation to get all post for user id 1

Discuss the results with your administrator.

8. Running Workflows

As of v0.0.35 the @openfn/cli supports running not only jobs, but also workflows. Running a workflow allows you to define a list of jobs and rules for executing them. You can use a workflow to orchestrate the flow of data between systems in a structured and automated way.

For example, if you have two jobs in your workflow (GET users from system A & POST users to system B), you can set up your workflow to run all jobs in sequence from start to finish. This imitates the flow trigger patterns on the OpenFn platform where a second job should run after the first one succeeds, respectively, using the data returned from the first job. “

tl;dr

You won't have to assemble the initial state of the next job, the final state of the upstream job will automatically be passed down to the downstream job as the initial state.

Workflow

A workflow is the execution plan for running several jobs in a sequence. It is defined as a JSON object that consists of the following properties:

start (optional): The ID of the job that should be executed first (defaults to jobs[0]).
jobs (required): An array of job objects, each of which represents a specific task to be executed.
- id (required): A job name that is unique to the workflow and helps you ID your job.
- configuration: (optional) Specifies the configuration file associated with the job.
- data (optional): A JSON object that contains the pre-populated data.
- adaptor (required): Specifies the adaptor used for the job (version optional).
- expression (required): Specifies the JavaScript file associated with the job. It can also be a string that contains a JavaScript function to be executed as the job.
- next (optional): An object that specifies which jobs to call next. All edges returning true will run. The object should have one or more key-value pairs, where the key is the ID of the next job, and the value is a boolean expression that determines whether the next job should be executed.If there are no next edges, the workflow will end.

Example of a workflow

Here's an example of a simple workflow that consists of three jobs:

workflow.json
{
  "start": "getPatients",
  "jobs": [
    {
      "id": "getPatients",
      "adaptor": "http",
      "expression": "getPatients.js",
      "configuration": "tmp/http-creds.json",
      "next": {
        "getGlobalOrgUnits": true
      }
    },
    {
      "id": "getGlobalOrgUnits",
      "adaptor": "common",
      "expression": "getGlobalOrgUnits.js",
      "next": {
        "createTEIs": true
      }
    },
    {
      "id": "createTEIs",
      "adaptor": "dhis2",
      "expression": "createTEIs.js",
      "configuration": "tmp/dhis2-creds.json"
    }
  ]
}

tmp/http-creds.json

tmp/http-creds.json
{
  "baseUrl": "https://jsonplaceholder.typicode.com/"
}

tmp/dhis2-creds.json

tmp/dhis2-creds.json
{
  "hostUrl": "https://play.dhis2.org/2.39.1.2",
  "password": "district",
  "username": "admin"
}

getPatients.js

getPatients.js
// Get users from jsonplaceholder
get('users');

// Prepare new users as new patients
fn(state => {
  const newPatients = state.data;
  return { ...state, newPatients };
});

getGlobalOrgUnits.js

getGlobalOrgUnits.js
// Globals: orgUnits
fn(state => {
  const globalOrgUnits = [
    {
      label: 'Njandama MCHP',
      id: 'g8upMTyEZGZ',
      source: 'Gwenborough',
    },
    {
      label: 'Njandama MCHP',
      id: 'g8upMTyEZGZ',
      source: 'Wisokyburgh',
    },
    {
      label: 'Njandama MCHP',
      id: 'g8upMTyEZGZ',
      source: 'McKenziehaven',
    },
    {
      label: 'Njandama MCHP',
      id: 'g8upMTyEZGZ',
      source: 'South Elvis',
    },
    {
      label: 'Ngelehun CHC',
      id: 'IpHINAT79UW',
      source: 'Roscoeview',
    },
    {
      label: 'Ngelehun CHC',
      id: 'IpHINAT79UW',
      source: 'South Christy',
    },
    {
      label: 'Ngelehun CHC',
      id: 'IpHINAT79UW',
      source: 'Howemouth',
    },
    {
      label: 'Ngelehun CHC',
      id: 'IpHINAT79UW',
      source: 'Aliyaview',
    },
    {
      label: 'Baoma Station CHP',
      id: 'jNb63DIHuwU',
      source: 'Bartholomebury',
    },
    {
      label: 'Baoma Station CHP',
      id: 'jNb63DIHuwU',
      source: 'Lebsackbury',
    },
  ];

  return { ...state, globalOrgUnits };
});

createTEIs.js

createTEIs.js
fn(state => {
  const { newPatients, globalOrgUnits } = state;

  const getOrgUnit = city =>
    globalOrgUnits.find(orgUnit => orgUnit.source === city).id;

  const mappedEntities = newPatients.map(patient => {
    const [firstName = 'Patient', lastName = 'Test'] = (
      patient.name || ''
    ).split(' ');

    const orgUnit = getOrgUnit(patient.address.city);

    const attributes = [
      { attribute: 'w75KJ2mc4zz', value: firstName },
      { attribute: 'zDhUuAYrxNC', value: lastName },
      { attribute: 'cejWyOfXge6', value: 'Male' },
    ];

    return { ...patient, attributes: attributes, orgUnit: orgUnit };
  });

  return { ...state, mappedEntities };
});

each(
  'mappedEntities[*]',
  create('trackedEntityInstances', {
    orgUnit: dataValue('orgUnit'),
    trackedEntityType: 'nEenWmSyUEp',
    attributes: dataValue('attributes'),
  })
);

Run openfn [path/to/workflow.json] to execute the workflow.

For example if you created workflow.json in the root of your project directory, This is how your project will look like

    devchallenge
    ├── .gitignore
    ├── getPatients.js
    ├── createTEIs.js
    ├── getGlobalOrgUnits.js
    ├── workflow.json
    └── tmp
        ├── http-creds.json
        ├── dhis2-creds.json
        └── output.json

openfn workflow.json -o tmp/output.json

On execution, this workflow will first run the getPatients.js job. If is successful, getGlobalOrgUnits.js will run using the final state of getPatients.js. If getGlobalOrgUnits.js is successful, createTEIs.js will run using the final state of getGlobalOrgUnits.js.

Note that without the -i flag, you'll need to already have your adaptor installed. To execute the workflow with the adaptor autoinstall option run this command:

openfn workflow.json -i -o tmp/output.json

On execution, this workflow will first auto-install the adaptors then run the workflow

Important

When working with the workflow.json file, it is important to handle sensitive information, such as credentials and initial input data, in a secure manner. To ensure the protection of your sensitive data, please follow the guidelines outlined below:

Configuration Key: In the workflow.json file, specify a path to a git ignored configuration file that will contain necessary credentials that will be used to access the destination system. For example:
```
{
   ...
   "configuration": "tmp/openMRS-credentials.json"
 },
```
Data Key: Incase you need to pass initial data to your job, specify a path to a gitignored data file
```
{
...
 "data": "tmp/initial-data.json",
}
```

CLI Usage - Key Commands

You’ll learn about these commands in the following challenges, but please refer to this section for the key commands used in working with the CLI.

Check the version

openfn version

Get help

openfn help

Run a job

openfn path/to/job.js -ia {adaptor-name}

Note: You MUST specify which adaptor to use. Pass the -i flag to auto-install that adaptor (it's safe to do this redundantly).

You can find the list of publicly available adaptors here.

Path is the job to load the job from (a .js file or a dir containing a job.js file) For example openfn execute hello.js Reads hello.js, looks for state and output in foo

-i, --autoinstall Auto-install the language adaptor
-a, --adaptors, --adaptor A language adaptor to use for the job

If an adaptor is already installed by auto install, you can use the command without the -i options. i.e openfn hello.js -a http

Change log level

You can pass -l info or --log info to get more feedback about what's happening, or --log debug for more details than you could ever use. Below is the list of different log levels

openfn hello.js -a http -l none

log level	description
`-l none`	Quiet mode
`-l default`	Top level information of what is happening
`-l info`	Get more feedback on what is happening openfn
`-l debug`	Get information about runtime, cli, compiler and the job

Compilation

The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust, portable, and easier to debug from a pure JS perspective.

openfn compile [path]

Will compile the openfn job and print or save the resulting js.

Learn more about CLI github.com/OpenFn/kit/

Intro to the OpenFn CLI​

Prerequisites​

Walkthrough & Challenges​

1. Getting started with the CLI​

Tasks:​

🏆 Challenge: Write a job that prints your name​

2. Using adaptor helper functions​

Basic usage:​

Tasks:​

🏆 Challenge: Get and inspect data via HTTP​

3. Understanding state​

state.configuration​

state.data​

How can we use state?​

Tasks:​

🏆 Challenge: Fetch Covid-19 metadata​

4. Additional arguments and commands​

🏆 Challenge: Practice CLI arguments and commands​

5. Manipulating data in a sequence of operations​

Example:​

JavaScript Demo: Array.reduce()​

🏆 Challenge: extract names & emails​

6. Debugging errors​

Create debug.js and paste the code below​

Run openfn debug.js -a http​

🏆 Challenge: control error messages​

7. Each and array iteration​

Modify getPosts.js to group posts by user-ID​

Run openfn getPosts.js -a http -o tmp/output.json​

🏆 Challenge: Reduce, filter, and map​

8. Running Workflows​

Workflow​

Example of a workflow​

CLI Usage - Key Commands​

Check the version​

Get help​

Run a job​

Change log level​

Compilation​

Intro to the OpenFn CLI

Prerequisites

Walkthrough & Challenges

1. Getting started with the CLI

Tasks:

🏆 Challenge: Write a job that prints your name

2. Using adaptor helper functions

Basic usage:

Tasks:

🏆 Challenge: Get and inspect data via HTTP

3. Understanding `state`

`state.configuration`

`state.data`

How can we use state?

Tasks:

🏆 Challenge: Fetch Covid-19 metadata

4. Additional arguments and commands

🏆 Challenge: Practice CLI arguments and commands

5. Manipulating data in a sequence of operations

Example:

JavaScript Demo: `Array.reduce()`

🏆 Challenge: extract names & emails

6. Debugging errors

Create debug.js and paste the code below

Run openfn debug.js -a http

🏆 Challenge: control error messages

7. Each and array iteration

Modify getPosts.js to group posts by user-ID

Run openfn getPosts.js -a http -o tmp/output.json

🏆 Challenge: Reduce, filter, and map

8. Running Workflows

Workflow

Example of a workflow

CLI Usage - Key Commands

Check the version

Get help

Run a job

Change log level

Compilation