common@3.3.3

arrayToString(arr, separator)

as(key, operation)

asData(data, state)

assert(expression, errorMessage)

chunk(array, chunkSize)

combine(operations)

composeNextState(state, response)

cursor(value, options)

dataPath(path)

dataValue(path)

debug(args)

each(dataSource, operation)

field(key, value)

fields(fields)

fn(func)

fnIf(condition, operation)

group(arrayOfObjects, keyPath, callback)

humanProper(str)

index()

join(targetPath, sourcePath, targetKey)

jsonValue(obj, path)

lastReferenceValue(path)

lodash()

log(args)

map(path, callback)

merge(dataSource, fields)

parseCsv(csvData, [parsingOptions], [callback])

referencePath(path)

scrubEmojis(text, replacementChars)

source(path)

sourceValue(path)

splitKeys(obj, keys)

toArray(arg)

validate(schema, data)

This adaptor exports the following namespaced functions:

util.decode(base64Data, options)

util.encode(data, options)

util.uuid()

beta.each(dataSource, operation)

dateFns.format()

dateFns.parse()

http.get(url, options)

http.options(opts)

http.post(url, data, options)

http.request(method, url, options)

Functions

arrayToString

arrayToString(arr, separator) ⇒ string

Turns an array into a string, separated by X.

Param	Type	Description
arr	array	Array of toString'able primatives.
separator	string	Separator string.

Example

field("destination_string__c", function(state) {
  return arrayToString(dataValue("path_of_array")(state), ', ')
})

---

as

as(key, operation) ⇒ Operation

Run an operation and save the result to a custom key in state instead of overwriting state.data.

Param	Type	Description
key	string	The state key to assign the result of the operation to.
operation	function	An operation that returns a new state object with a data property

Example: Fetch cce-data from collections and store them under state.cceData

as('cceData', collections.get('cce-data-dhis2', { key: `*:*:${$.syncedAt}*` }));

---

asData

asData(data, state) ⇒ array

Simple switcher allowing other expressions to use either a JSONPath or object literals as a data source.

• JSONPath referencing a point in state
• Object Literal of the data itself.
• Function to be called with state.

Param	Type	Description
data	String | object | function
state	object	The current state.

Example

asData('$.key'| key | callback)

---

assert

assert(expression, errorMessage) ⇒ operation

Asserts the given expression or function resolves to true, or else throws an exception. Optionally accepts and error message.

Param	Type	Description
expression	any	The expression or function to be evaluated.
errorMessage	string	The error message thrown in case of a failed state.

Example

assert('a' === 'b', '"a" is not equal to "b"')

---

chunk

chunk(array, chunkSize) ⇒ Object

Chunks an array into an array of arrays, each with no more than a certain size.

Param	Type	Description
array	Object	Array to be chunked
chunkSize	Integer	The maxiumum size of each chunks

Example

chunk([1,2,3,4,5], 2)

---

combine

combine(operations) ⇒ Operation

Combines two operations into one

Param	Type	Description
operations	Operations	Operations to be performed.

Example

combine(
  create('foo'),
  delete('bar')
)

---

composeNextState

composeNextState(state, response) ⇒ State

Prepares next state

Param	Type	Description
state	State	state
response	Object	Response to be added

Example

composeNextState(state, response)

---

cursor

cursor(value, options) ⇒ Operation

Sets a cursor property on state. Supports natural language dates like now, today, yesterday, n hours ago, n days ago, and start, which will be converted relative to the environment (ie, the Lightning or CLI locale). Custom timezones are not yet supported. You can provide a formatter to customise the final cursor value, which is useful for normalising different inputs. The custom formatter runs after natural language date conversion. See the usage guide at https://docs.openfn.org/documentation/jobs/job-writing-guide#using-cursors

Param	Type	Description
value	any	the cursor value. Usually an ISO date, natural language date, or page number
options	object	options to control the cursor.
options.key	string	set the cursor key. Will persist through the whole run.
options.defaultValue	any	the value to use if value is falsy
options.format	function	custom formatter for the final cursor value

Example: Use a cursor from state if present, or else use the default value

cursor($.cursor, { defaultValue: 'today' })

Example: Use a pagination cursor

cursor(22)

---

dataPath

dataPath(path) ⇒ string

Ensures a path points at the data.

Param	Type	Description
path	string	JSONPath referencing a point in data.

Example

dataPath('key')

---

dataValue

dataValue(path) ⇒ Operation

Picks out a single value from the source data object—usually state.data. If a JSONPath returns more than one value for the reference, the first item will be returned.

Param	Type	Description
path	String	JSONPath referencing a point in data.

Example

dataValue('key')

---

debug

debug(args) ⇒ Operation

Outputs a message to the console with the debug log level. This is usually filtered out by default. Use this at the top level of your job code, but not inside callbacks.

Param	Type	Description
args	any	A value or message to display in the logs

Example: Log values from state

debug('Patient List::', $.patients);

Example: Use console.debug inside a callback or fn block

fn((state) => {
  console.debug(state.data);
  return state;
})

---

each

each(dataSource, operation) ⇒ Operation

Iterates over an array of items and invokes an operation upon each one, where the state object is scoped so that state.data is the item under iteration. The rest of the state object is untouched and can be referenced as usual. You can pass an array directly, or use lazy state or a JSONPath string to reference a slice of state.

Param	Type	Description
dataSource	DataSource	JSONPath referencing a point in state.
operation	Operation	The operation needed to be repeated.

Example: Using lazy state ($) to iterate over items in state.data and pass each into an "insert" operation

each(
  $.data,
  // Inside the callback operation, `$.data` is scoped to the item under iteration
  insert("patient", {
    patient_name: $.data.properties.case_name,
    patient_id: $.data.case_id,
  })
);

Example: Iterate over items in state.data and pass each one into an "insert" operation

each(
  $.data,
  insert("patient", (state) => ({
    patient_id: state.data.case_id,
    ...state.data
  }))
);

Example: Using JSON path to iterate over items in state.data and pass each one into an "insert" operation

each(
  "$.data[*]",
  insert("patient", (state) => ({
    patient_name: state.data.properties.case_name,
    patient_id: state.data.case_id,
  }))
);

---

field

field(key, value) ⇒ Field

Returns a key, value pair in an array.

Param	Type	Description
key	string	Name of the field
value	Value	The value itself or a sourceable operation.

Example

field('destination_field_name__c', 'value')

---

fields

fields(fields) ⇒ Object

Zips key value pairs into an object.

Param	Type	Description
fields	Fields	a list of fields

Example

fields(list_of_fields)

---

fn

fn(func) ⇒ Operation

Creates a custom step (or operation) for more flexible job writing.

Param	Type	Description
func	function	is the function

Example

fn(state => {
  // do some things to state
  return state;
});

---

fnIf

fnIf(condition, operation) ⇒ Operation

Execute a function only when the condition returns true

Param	Type	Description
condition	Boolean	The condition that returns true
operation	Operation	The operation needed to be executed.

Example

fnIf((state) => state?.data?.name, get("https://example.com"));

---

group

group(arrayOfObjects, keyPath, callback) ⇒ Operation

Groups an array of objects by a specified key path.

Param	Type	Description
arrayOfObjects	Array.<Object>	The array of objects to be grouped.
keyPath	string	The key path to group by.
callback	function	(Optional) Callback function

Example

const users = [
  { name: 'Alice', age: 25, city: 'New York' },
  { name: 'Bob', age: 30, city: 'San Francisco' },
  { name: 'Charlie', age: 25, city: 'New York' },
  { name: 'David', age: 30, city: 'San Francisco' }
];
group(users, 'city');
// state is { data: { 'New York': [/Alice, Charlie/], 'San Francisco': [ /Bob, David / ] }

---

humanProper

humanProper(str) ⇒ string

Substitutes underscores for spaces and proper-cases a string

Param	Type	Description
str	string	String that needs converting

Example

field("destination_string__c", humanProper(state.data.path_to_string))

---

index

index() ⇒ DataSource

Returns the index of the current array being iterated. To be used with each as a data source.

Example

index()

---

join

join(targetPath, sourcePath, targetKey) ⇒ Operation

Adds data from a target object

Param	Type	Description
targetPath	String	Target path
sourcePath	String	Source path
targetKey	String	Target Key

Example

join('$.key','$.data','newKey')

---

jsonValue

jsonValue(obj, path) ⇒ Operation

Picks out a single value from a JSON object. If a JSONPath returns more than one value for the reference, the first item will be returned.

Param	Type	Description
obj	object	A valid JSON object.
path	String	JSONPath referencing a point in given JSON object.

Example

jsonValue({ a:1 }, 'a')

---

lastReferenceValue

lastReferenceValue(path) ⇒ Operation

Picks out the last reference value from source data.

Param	Type	Description
path	String	JSONPath referencing a point in references.

Example

lastReferenceValue('key')

---

lodash

lodash()

Lodash utility library. All lodash v4.17 functions are available on the _ namespace, eg, _.map, _.cloneDeep, etc.

See: https://lodash.com/docs/

Example: Split an array into chunks of 2 items each

fn(state => {
  const items = [1, 2, 3, 4, 5];
  const chunks = lodash.chunk(items, 2);
  return { ...state, chunks };
});

---

log

log(args) ⇒ Operation

Outputs a message, like calling console.log. Use this at the top level of your job code, but not inside callbacks.

Param	Type	Description
args	any	A value or message to display in the logs

Example: Log values from state

log('Patient List::', $.patients);

Example: Use console.log inside a callback or fn block

fn((state) => {
  console.log(state.data);
  return state;
})

---

map

map(path, callback) ⇒ State

Iterates over a collection of items and returns a new array of mapped values, like Javascript's Array.map() function.

Each item in the source array will be passed into the callback function. The returned value will be added to the new array. The callback is passed the original item, the current index in the source array (ie, the nth item number), and the state object.

Writes a new array to state.data with transformed values.c array.

Param	Type	Description
path	string | Array	An array of items or a a JSONPath string which points to an array of items.
callback	function	The mapping function, invoked with (data, index, state) for each item in the array.

Example: Transform an array of items in state

map($.items', (data, index, state) => {
  return {
    id: index + 1,
    name: data.name,
    createdAt: state.cursor,
  };
});

Example: Map items asynchronously (e.g. fetch extra info)

map($.items, async (data, index, state) => {
  const userInfo = await fetchUserInfo(data.userId);
  return {
    id: index + 1,
    name: data.name,
    extra: userInfo,
  };
});

---

merge

merge(dataSource, fields) ⇒ DataSource

Merges fields into each item in an array.

Param	Type	Description
dataSource	DataSource
fields	Object	Group of fields to merge in.

Example

merge(
  "$.books[*]",
  fields(
    field( "publisher", sourceValue("$.publisher") )
  )
)

---

parseCsv

parseCsv(csvData, [parsingOptions], [callback]) ⇒ Operation

Takes a CSV file string or stream and parsing options as input, and returns a promise that resolves to the parsed CSV data as an array of objects. Options for parsingOptions include:

• delimiter {string/Buffer/[string/Buffer]} - Defines the character(s) used to delineate the fields inside a record. Default: ','
• quote {string/Buffer/[string/Buffer]} - Defines the characters used to surround a field. Default: '"'
• escape {Buffer/string/null/boolean} - Set the escape character as one character/byte only. Default: "
• columns {boolean / array / function} - Generates record in the form of object literals. Default: true
• bom {boolean} - Strips the byte order mark (BOM) from the input string or buffer. Default: true
• trim {boolean} - Ignore whitespace characters immediately around the delimiter. Default: true
• ltrim {boolean} - Ignore whitespace characters from the left side of a CSV field. Default: true
• rtrim {boolean} - Ignore whitespace characters from the right side of a CSV field. Default: true
• chunkSize {number} - The size of each chunk of CSV data. Default: Infinity
• skip_empty_lines {boolean} - Ignore empty lines in the CSV file. Default: true

Returns: Operation - The function returns a Promise that resolves to the result of parsing a CSV stringOrStream.

Param	Type	Description
csvData	String | Stream	A CSV string or a readable stream
[parsingOptions]	Object	Optional. Parsing options for converting CSV to JSON.
[callback]	function	(Optional) callback function. If used it will be called state and an array of rows.

---

referencePath

referencePath(path) ⇒ string

Ensures a path points at references.

Param	Type	Description
path	string	JSONPath referencing a point in references.

Example

referencePath('key')

---

scrubEmojis

scrubEmojis(text, replacementChars) ⇒ string

Replaces emojis in a string.

Param	Type	Description
text	string	String that needs to be cleaned
replacementChars	string	Characters that replace the emojis

Example

scrubEmojis('Dove🕊️⭐ 29')

---

source

source(path) ⇒ Array.<(String|Object)>

Picks out a value from source data. Will return whatever JSONPath returns, which will always be an array. If you need a single value use sourceValue instead.

Param	Type	Description
path	String	JSONPath referencing a point in state.

Example

source('$.key')

---

sourceValue

sourceValue(path) ⇒ Operation

Picks out a single value from source data. If a JSONPath returns more than one value for the reference, the first item will be returned.

Param	Type	Description
path	String	JSONPath referencing a point in state.

Example

sourceValue('$.key')

---

splitKeys

splitKeys(obj, keys) ⇒ Array.<Object>

Splits an object into two objects based on a list of keys. The first object contains the keys that are not in the list, and the second contains the keys that are.

Returns: Array.<Object> - - Tuple of objects, first object contains keys not in list, second contains keys that are.

Param	Type	Description
obj	Object	The object to split.
keys	Array.<string>	List of keys to split on.

---

toArray

toArray(arg) ⇒ array

Ensures primitive data types are wrapped in an array. Does not affect array objects.

Param	Type	Description
arg	any	Data required to be in an array

Example

each(function(state) {
  return toArray( dataValue("path_of_array")(state) )
}, ...)

---

validate

validate(schema, data) ⇒ Operation

Validate against a JSON schema. Any errors are written to an array at state.validationErrors. Schema can be passed directly, loaded as a JSON path from state, or loaded from a URL Data can be passed directly or loaded as a JSON path from state. By default, schema is loaded from state.schema and data from state.data.

Param	Type	Description
schema	string | object	The schema, path or URL to validate against
data	string | object	The data or path to validate

Example: Validate state.data with state.schema

validate()

Example: Validate form data at state.form with a schema from a URL

validate("https://www.example.com/schema/record", "form")

Example: Validate the each item in state.records with a schema from a URL

each("records[*]", validate("https://www.example.com/schema/record"))

---

util

These functions belong to the util namespace.

util.decode

decode(base64Data, options) ⇒ string | object

Decodes a Base64 encoded string back to its original format.

Returns: string | object - - The decoded string or JavaScript Object.

Param	Type	Default	Description
base64Data	string		The Base64 encoded string.
options	object		Options.
[options.parseJson]	boolean	true	If false, no attempt will be made to parse the decoded data into a JSON object.

Example: Decode a Base64 string

const decoded = util.decode('SGVsbG8gV29ybGQ=');

Example: Decode a Base64 JSON object to a standard JavaScript object

const decoded = util.decode('eyJuYW1lIjoiSmFuZSBEb2UifQ==');
console.log(decoded); // Output: {name: 'Jane Doe' }

Example: To skip the JSON stringification step

const decodedString = util.decode('Hello World', { parseJson: false })

---

util.encode

encode(data, options) ⇒ string

Encodes a given string or Javascript object into Base64 format.

Returns: string - - The Base64 encoded string.

Param	Type	Default	Description
data	string | object		The string or object to be encoded.
options	object		Options
[options.parseJson]	boolean	true	If false, no attempt will be made to stringiy the data before encoding it.

Example: Encode a string

const encodedString = util.encode('Hello World');
console.log(encoded); // Output: SGVsbG8gV29ybGQ=

Example: Encode an object

const encodedObject = util.encode({name: 'Jane Doe' })
console.log(encodedObject); //output eyJuYW1lIjoiSmFuZSBEb2UifQ==

Example: To skip the JSON stringification step

const encodedObject = util.encode('Hello World', { parseJson: false })

---

util.uuid

uuid() ⇒ string

Generates a UUID (Universally Unique Identifier)

Returns: string - - A newly generated UUID

Example: Generate a UUID

const id = util.uuid();
console.log(id); // Output:'3f4e254e-8f6f-4f8b-9651-1c1c262cc83f'

---

beta

These functions belong to the beta namespace.

beta.each

each(dataSource, operation) ⇒ Operation

Scopes an array of data based on a JSONPath. Useful when the source data has n items you would like to map to an operation. The operation will receive a slice of the data based of each item of the JSONPath provided.

It also ensures the results of an operation make their way back into the state's references.

Param	Type	Description
dataSource	DataSource	JSONPath referencing a point in state.
operation	Operation	The operation needed to be repeated.

Example

each("$.[*]",
   create("SObject",
   field("FirstName", sourceValue("$.firstName")))
 )

---

dateFns

These functions belong to the dateFns namespace.

dateFns.format

format()

The format function from the date-fns library. See https://date-fns.org/v3.6.0/docs/parse

---

dateFns.parse

parse()

The parse function from the date-fns library. See https://date-fns.org/v3.6.0/docs/parse

---

http

These functions belong to the http namespace.

http.get

get(url, options) ⇒ Operation

Make a GET request.

Param	Type	Description
url	string	URL to access
options	CommonRequestOptions	Request options

This operation writes the following keys to state:

State Key	Description
data	the parsed response body
response	the response from the HTTP server, including headers, statusCode, body, etc
references	an array of all previous data objects used in the Job

Example: Request a resource

http.get('https://jsonplaceholder.typicode.com/todos')

Example: Request a resource with basic auth

http.get(
 'https://jsonplaceholder.typicode.com/todos',
 http.options().basic('user', 'pass')
)

Example: Request a resource with oauth

http.get(
 'https://jsonplaceholder.typicode.com/todos',
 http.options().oauth($.configuration.access_token)
)

---

http.options

options(opts) ⇒ OptionsHelpers

Builder function to create request options. Returns an object with helpers to easily add commonly used options. The return object is chainable so you can set as many options as you want. Pass an object to set your own options.

Param	Type	Description
opts	OptionsHelpers	options to pass to the request

Example: Get with a query an oauth token

http.get($.data.url, http.options({ query: $.query }).oauth($.configuration.access_token))

---

http.post

post(url, data, options) ⇒ Operation

Make a POST request.

Param	Type	Description
url	string	URL to access
data	object	Body data to append to the request.
options	CommonRequestOptions	Request options

This operation writes the following keys to state:

State Key	Description
data	the parsed response body
response	the response from the HTTP server, including headers, statusCode, body, etc
references	an array of all previous data objects used in the Job

Example: Post a JSON object (setting the content-type header)

http.post(
  "https://jsonplaceholder.typicode.com/todos",
  $.data,
  http.options().json()
);

---

http.request

request(method, url, options) ⇒ Operation

Make a HTTP request.

Param	Type	Description
method	string	The HTTP method to use.
url	string	URL to resource.
options	CommonRequestOptions	Request options

This operation writes the following keys to state:

State Key	Description
data	the parsed response body
response	the response from the HTTP server, including headers, statusCode, body, etc
references	an array of all previous data objects used in the Job

Example

http.request(
  'GET',
  'https://jsonplaceholder.typicode.com/todos'
)

---

Interfaces

CommonRequestOptions

Options provided to the HTTP request

Properties

Name	Type	Description
errors	object | boolean	Map of errorCodes -> error messages, ie, { 404: 'Resource not found;' }. Pass false to suppress errors.
form	object	Pass a JSON object to be serialised into a multipart HTML form (as FormData) in the body.
query	object	An object of query parameters to be encoded into the URL.
headers	object	An object of headers to append to the request.
parseAs	string	Parse the response body as json, text or stream. By default will use the response headers.
timeout	number	Request timeout in ms. Default: 300 seconds.
tls	object	TLS/SSL authentication options. See https://nodejs.org/api/tls.html#tlscreatesecurecontextoptions

---

OptionsHelpers

Helper functions provided by http.options.

Properties

Name	Type	Description
json	function	Sets the `content-type' header to 'application/json'
basic	function	Sets basic auth on the Authorization header. Pass username and password
bearer	function	Sets a Bearer token on the Authorization header. Pass the token.
oauth	function	Sets a Bearer token on the Authorization header. Pass the oauth token.

---