Skip to main content

common@2.0.2

arrayToString(arr, separator)
asData(data, state)
chunk(array, chunkSize)
combine(operations)
composeNextState(state, response)
cursor(value, options)
dataPath(path)
dataValue(path)
each(dataSource, operation)
expandReferences(value, [skipFilter])
field(key, value)
fields(fields)
fn(func)
fnIf(condition, operation)
group(arrayOfObjects, keyPath, callback)
humanProper(str)
index()
join(targetPath, sourcePath, targetKey)
jsonValue(obj, path)
lastReferenceValue(path)
map(path, operation, state)
merge(dataSource, fields)
parseCsv(csvData, [parsingOptions], [callback])
referencePath(path)
scrubEmojis(text, replacementChars)
source(path)
sourceValue(path)
splitKeys(obj, keys)
toArray(arg)

This adaptor exports the following namespaced functions:

beta.each(dataSource, operation)
dateFns.format()
dateFns.parse()
http.get(url, options)
http.options(options)
http.post(url, options)
http.request(method, url, options)

Functions

arrayToString

arrayToString(arr, separator) ⇒ string

Turns an array into a string, separated by X.

ParamTypeDescription
arrarrayArray of toString'able primatives.
separatorstringSeparator string.

Example

field("destination_string__c", function(state) {
return arrayToString(dataValue("path_of_array")(state), ', ')
})

asData

asData(data, state) ⇒ array

Simple switcher allowing other expressions to use either a JSONPath or object literals as a data source.

  • JSONPath referencing a point in state
  • Object Literal of the data itself.
  • Function to be called with state.
ParamTypeDescription
dataString | object | function
stateobjectThe current state.

Example

asData('$.key'| key | callback)

chunk

chunk(array, chunkSize) ⇒ Object

Chunks an array into an array of arrays, each with no more than a certain size.

ParamTypeDescription
arrayObjectArray to be chunked
chunkSizeIntegerThe maxiumum size of each chunks

Example

chunk([1,2,3,4,5], 2)

combine

combine(operations) ⇒ Operation

Combines two operations into one

ParamTypeDescription
operationsOperationsOperations to be performed.

Example

combine(
create('foo'),
delete('bar')
)

composeNextState

composeNextState(state, response) ⇒ State

Prepares next state

ParamTypeDescription
stateStatestate
responseObjectResponse to be added

Example

composeNextState(state, response)

cursor

cursor(value, options) ⇒ Operation

Sets a cursor property on state. Supports natural language dates like now, today, yesterday, n hours ago, n days ago, and start, which will be converted relative to the environment (ie, the Lightning or CLI locale). Custom timezones are not yet supported. You can provide a formatter to customise the final cursor value, which is useful for normalising different inputs. The custom formatter runs after natural language date conversion. See the usage guide at https://docs.openfn.org/documentation/jobs/job-writing-guide#using-cursors

ParamTypeDescription
valueanythe cursor value. Usually an ISO date, natural language date, or page number
optionsobjectoptions to control the cursor.
options.keystringset the cursor key. Will persist through the whole run.
options.defaultValueanythe value to use if value is falsy
options.formatfunctioncustom formatter for the final cursor value

Example: Use a cursor from state if present, or else use the default value

cursor($.cursor, { defaultValue: 'today' })

Example: Use a pagination cursor

cursor(22)

dataPath

dataPath(path) ⇒ string

Ensures a path points at the data.

ParamTypeDescription
pathstringJSONPath referencing a point in data.

Example

dataPath('key')

dataValue

dataValue(path) ⇒ Operation

Picks out a single value from the source data object—usually state.data. If a JSONPath returns more than one value for the reference, the first item will be returned.

ParamTypeDescription
pathStringJSONPath referencing a point in data.

Example

dataValue('key')

each

each(dataSource, operation) ⇒ Operation

Scopes an array of data based on a JSONPath. Useful when the source data has n items you would like to map to an operation. The operation will receive a slice of the data based of each item of the JSONPath provided.

It also ensures the results of an operation make their way back into the state's references.

ParamTypeDescription
dataSourceDataSourceJSONPath referencing a point in state.
operationOperationThe operation needed to be repeated.

Example

each("$.[*]",
create("SObject",
field("FirstName", sourceValue("$.firstName"))
)
)

expandReferences

expandReferences(value, [skipFilter]) ⇒ Operation

Recursively resolves objects that have resolvable values (functions).

ParamTypeDescription
valueobjectdata
[skipFilter]functiona function which returns true if a value should be skipped

field

field(key, value) ⇒ Field

Returns a key, value pair in an array.

ParamTypeDescription
keystringName of the field
valueValueThe value itself or a sourceable operation.

Example

field('destination_field_name__c', 'value')

fields

fields(fields) ⇒ Object

Zips key value pairs into an object.

ParamTypeDescription
fieldsFieldsa list of fields

Example

fields(list_of_fields)

fn

fn(func) ⇒ Operation

Creates a custom step (or operation) for more flexible job writing.

ParamTypeDescription
funcfunctionis the function

Example

fn(state => {
// do some things to state
return state;
});

fnIf

fnIf(condition, operation) ⇒ Operation

A custom operation that will only execute the function if the condition returns true

ParamTypeDescription
conditionBooleanThe condition that returns true
operationOperationThe operation needed to be executed.

Example

fnIf((state) => state?.data?.name, get("https://example.com"));

group

group(arrayOfObjects, keyPath, callback) ⇒ Operation

Groups an array of objects by a specified key path.

ParamTypeDescription
arrayOfObjectsArray.<Object>The array of objects to be grouped.
keyPathstringThe key path to group by.
callbackfunction(Optional) Callback function

Example

const users = [
{ name: 'Alice', age: 25, city: 'New York' },
{ name: 'Bob', age: 30, city: 'San Francisco' },
{ name: 'Charlie', age: 25, city: 'New York' },
{ name: 'David', age: 30, city: 'San Francisco' }
];
group(users, 'city');
// state is { data: { 'New York': [/Alice, Charlie/], 'San Francisco': [ /Bob, David / ] }

humanProper

humanProper(str) ⇒ string

Substitutes underscores for spaces and proper-cases a string

ParamTypeDescription
strstringString that needs converting

Example

field("destination_string__c", humanProper(state.data.path_to_string))

index

index() ⇒ DataSource

Returns the index of the current array being iterated. To be used with each as a data source.

Example

index()

join

join(targetPath, sourcePath, targetKey) ⇒ Operation

Adds data from a target object

ParamTypeDescription
targetPathStringTarget path
sourcePathStringSource path
targetKeyStringTarget Key

Example

join('$.key','$.data','newKey')

jsonValue

jsonValue(obj, path) ⇒ Operation

Picks out a single value from a JSON object. If a JSONPath returns more than one value for the reference, the first item will be returned.

ParamTypeDescription
objobjectA valid JSON object.
pathStringJSONPath referencing a point in given JSON object.

Example

jsonValue({ a:1 }, 'a')

lastReferenceValue

lastReferenceValue(path) ⇒ Operation

Picks out the last reference value from source data.

ParamTypeDescription
pathStringJSONPath referencing a point in references.

Example

lastReferenceValue('key')

map

map(path, operation, state) ⇒ State

Scopes an array of data based on a JSONPath. Useful when the source data has n items you would like to map to an operation. The operation will receive a slice of the data based of each item of the JSONPath provided.

ParamTypeDescription
pathstringJSONPath referencing a point in state.data.
operationfunctionThe operation needed to be repeated.
stateStateRuntime state.

Example

map("$.[*]",
create("SObject",
field("FirstName", sourceValue("$.firstName"))
)
)

merge

merge(dataSource, fields) ⇒ DataSource

Merges fields into each item in an array.

ParamTypeDescription
dataSourceDataSource
fieldsObjectGroup of fields to merge in.

Example

merge(
"$.books[*]",
fields(
field( "publisher", sourceValue("$.publisher") )
)
)

parseCsv

parseCsv(csvData, [parsingOptions], [callback]) ⇒ Operation

Takes a CSV file string or stream and parsing options as input, and returns a promise that resolves to the parsed CSV data as an array of objects. Options for parsingOptions include:

  • delimiter {string/Buffer/[string/Buffer]} - Defines the character(s) used to delineate the fields inside a record. Default: ','
  • quote {string/Buffer/[string/Buffer]} - Defines the characters used to surround a field. Default: '"'
  • escape {Buffer/string/null/boolean} - Set the escape character as one character/byte only. Default: "
  • columns {boolean / array / function} - Generates record in the form of object literals. Default: true
  • bom {boolean} - Strips the byte order mark (BOM) from the input string or buffer. Default: true
  • trim {boolean} - Ignore whitespace characters immediately around the delimiter. Default: true
  • ltrim {boolean} - Ignore whitespace characters from the left side of a CSV field. Default: true
  • rtrim {boolean} - Ignore whitespace characters from the right side of a CSV field. Default: true
  • chunkSize {number} - The size of each chunk of CSV data. Default: Infinity
  • skip_empty_lines {boolean} - Ignore empty lines in the CSV file. Default: true

Returns: Operation - The function returns a Promise that resolves to the result of parsing a CSV stringOrStream.

ParamTypeDescription
csvDataString | StreamA CSV string or a readable stream
[parsingOptions]ObjectOptional. Parsing options for converting CSV to JSON.
[callback]function(Optional) callback function. If used it will be called state and an array of rows.

referencePath

referencePath(path) ⇒ string

Ensures a path points at references.

ParamTypeDescription
pathstringJSONPath referencing a point in references.

Example

referencePath('key')

scrubEmojis

scrubEmojis(text, replacementChars) ⇒ string

Replaces emojis in a string.

ParamTypeDescription
textstringString that needs to be cleaned
replacementCharsstringCharacters that replace the emojis

Example

scrubEmojis('Dove🕊️⭐ 29')

source

source(path) ⇒ Array.<(String|Object)>

Picks out a value from source data. Will return whatever JSONPath returns, which will always be an array. If you need a single value use sourceValue instead.

ParamTypeDescription
pathStringJSONPath referencing a point in state.

Example

source('$.key')

sourceValue

sourceValue(path) ⇒ Operation

Picks out a single value from source data. If a JSONPath returns more than one value for the reference, the first item will be returned.

ParamTypeDescription
pathStringJSONPath referencing a point in state.

Example

sourceValue('$.key')

splitKeys

splitKeys(obj, keys) ⇒ Array.<Object>

Splits an object into two objects based on a list of keys. The first object contains the keys that are not in the list, and the second contains the keys that are.

Returns: Array.<Object> - - Tuple of objects, first object contains keys not in list, second contains keys that are.

ParamTypeDescription
objObjectThe object to split.
keysArray.<string>List of keys to split on.

toArray

toArray(arg) ⇒ array

Ensures primitive data types are wrapped in an array. Does not affect array objects.

ParamTypeDescription
arganyData required to be in an array

Example

each(function(state) {
return toArray( dataValue("path_of_array")(state) )
}, ...)

beta

These functions belong to the beta namespace.

beta.each

each(dataSource, operation) ⇒ Operation

Scopes an array of data based on a JSONPath. Useful when the source data has n items you would like to map to an operation. The operation will receive a slice of the data based of each item of the JSONPath provided.

It also ensures the results of an operation make their way back into the state's references.

ParamTypeDescription
dataSourceDataSourceJSONPath referencing a point in state.
operationOperationThe operation needed to be repeated.

Example

each("$.[*]",
create("SObject",
field("FirstName", sourceValue("$.firstName")))
)

dateFns

These functions belong to the dateFns namespace.

dateFns.format

format()

The format function from the date-fns library. See https://date-fns.org/v3.6.0/docs/parse


dateFns.parse

parse()

The parse function from the date-fns library. See https://date-fns.org/v3.6.0/docs/parse


http

These functions belong to the http namespace.

http.get

get(url, options) ⇒ Operation

Make a GET request.

ParamTypeDescription
urlstringURL to access
optionsCommonRequestOptionsRequest options

This operation writes the following keys to state:

State KeyDescription
datathe parsed response body
responsethe response from the HTTP server, including headers, statusCode, body, etc
referencesan array of all previous data objects used in the Job

Example: Request a resource

http.get('https://jsonplaceholder.typicode.com/todos')

Example: Request a resource with basic auth

http.get(
'https://jsonplaceholder.typicode.com/todos',
http.options().basic('user', 'pass')
)

Example: Request a resource with oauth

http.get(
'https://jsonplaceholder.typicode.com/todos',
http.options().oauth($.configuration.access_token)
)

http.options

options(options) ⇒ OptionsHelpers

Builder function to create request options. Returns an object with helpers to easily add commonly used options. The return object is chainable so you can set as many options as you want. Pass an object to set your own options.

ParamTypeDescription
optionsCommonRequestOptionsoptions to pass to the request

Example: Get with a query an oath token

get($.data.url, http.options({ query: $.query }).oath($.configuration.access_token)

http.post

post(url, options) ⇒ Operation

Make a POST request.

ParamTypeDescription
urlstringURL to access
optionsCommonRequestOptionsRequest options

This operation writes the following keys to state:

State KeyDescription
datathe parsed response body
responsethe response from the HTTP server, including headers, statusCode, body, etc
referencesan array of all previous data objects used in the Job

Example: Post a JSON object (setting the content-type header)

 http.post(
'https://jsonplaceholder.typicode.com/todos',
$.data,
options().json(),
})

http.request

request(method, url, options) ⇒ Operation

Make a HTTP request.

ParamTypeDescription
methodstringThe HTTP method to use.
urlstringURL to resource.
optionsCommonRequestOptionsRequest options

This operation writes the following keys to state:

State KeyDescription
datathe parsed response body
responsethe response from the HTTP server, including headers, statusCode, body, etc
referencesan array of all previous data objects used in the Job

Example

http.request(
'GET',
'https://jsonplaceholder.typicode.com/todos'
)

Interfaces

OptionsHelpers

Helper functions provided by http.options.

Properties

NameTypeDescription
jsonfunctionSets the `content-type' header to 'application/json'
basicfunctionSets basic auth on the Authorization header. Pass username and password
bearerfunctionSets a Bearer token on the Authorization header. Pass the token.
oauthfunctionSets a Bearer token on the Authorization header. Pass the oauth token.