common@2.1.1
- arrayToString(arr, separator)
- asData(data, state)
- chunk(array, chunkSize)
- combine(operations)
- composeNextState(state, response)
- cursor(value, options)
- dataPath(path)
- dataValue(path)
- each(dataSource, operation)
- expandReferences(value, [skipFilter])
- field(key, value)
- fields(fields)
- fn(func)
- fnIf(condition, operation)
- group(arrayOfObjects, keyPath, callback)
- humanProper(str)
- index()
- join(targetPath, sourcePath, targetKey)
- jsonValue(obj, path)
- lastReferenceValue(path)
- map(path, operation, state)
- merge(dataSource, fields)
- parseCsv(csvData, [parsingOptions], [callback])
- referencePath(path)
- scrubEmojis(text, replacementChars)
- source(path)
- sourceValue(path)
- splitKeys(obj, keys)
- toArray(arg)
This adaptor exports the following namespaced functions:
- helpers.decode(base64Data)
- helpers.encode(data)
- helpers.uuid()
- beta.each(dataSource, operation)
- dateFns.format()
- dateFns.parse()
- http.get(url, options)
- http.options(options)
- http.post(url, options)
- http.request(method, url, options)
Functions
arrayToString
arrayToString(arr, separator) ⇒ string
Turns an array into a string, separated by X.
Param | Type | Description |
---|---|---|
arr | array | Array of toString'able primatives. |
separator | string | Separator string. |
Example
field("destination_string__c", function(state) {
return arrayToString(dataValue("path_of_array")(state), ', ')
})
asData
asData(data, state) ⇒ array
Simple switcher allowing other expressions to use either a JSONPath or object literals as a data source.
- JSONPath referencing a point in
state
- Object Literal of the data itself.
- Function to be called with state.
Param | Type | Description |
---|---|---|
data | String | object | function | |
state | object | The current state. |
Example
asData('$.key'| key | callback)
chunk
chunk(array, chunkSize) ⇒ Object
Chunks an array into an array of arrays, each with no more than a certain size.
Param | Type | Description |
---|---|---|
array | Object | Array to be chunked |
chunkSize | Integer | The maxiumum size of each chunks |
Example
chunk([1,2,3,4,5], 2)
combine
combine(operations) ⇒ Operation
Combines two operations into one
Param | Type | Description |
---|---|---|
operations | Operations | Operations to be performed. |
Example
combine(
create('foo'),
delete('bar')
)
composeNextState
composeNextState(state, response) ⇒ State
Prepares next state
Param | Type | Description |
---|---|---|
state | State | state |
response | Object | Response to be added |
Example
composeNextState(state, response)
cursor
cursor(value, options) ⇒ Operation
Sets a cursor property on state.
Supports natural language dates like now
, today
, yesterday
, n hours ago
, n days ago
, and start
,
which will be converted relative to the environment (ie, the Lightning or CLI locale). Custom timezones
are not yet supported.
You can provide a formatter to customise the final cursor value, which is useful for normalising
different inputs. The custom formatter runs after natural language date conversion.
See the usage guide at https://docs.openfn.org/documentation/jobs/job-writing-guide#using-cursors
Param | Type | Description |
---|---|---|
value | any | the cursor value. Usually an ISO date, natural language date, or page number |
options | object | options to control the cursor. |
options.key | string | set the cursor key. Will persist through the whole run. |
options.defaultValue | any | the value to use if value is falsy |
options.format | function | custom formatter for the final cursor value |
Example: Use a cursor from state if present, or else use the default value
cursor($.cursor, { defaultValue: 'today' })
Example: Use a pagination cursor
cursor(22)
dataPath
dataPath(path) ⇒ string
Ensures a path points at the data.
Param | Type | Description |
---|---|---|
path | string | JSONPath referencing a point in data . |
Example
dataPath('key')
dataValue
dataValue(path) ⇒ Operation
Picks out a single value from the source data object—usually state.data
.
If a JSONPath returns more than one value for the reference, the first
item will be returned.
Param | Type | Description |
---|---|---|
path | String | JSONPath referencing a point in data . |
Example
dataValue('key')
each
each(dataSource, operation) ⇒ Operation
Iterates over an array of items and invokes an operation upon each one, where the state object is scoped so that state.data is the item under iteration. The rest of the state object is untouched and can be referenced as usual. You can pass an array directly, or use lazy state or a JSONPath string to reference a slice of state.
Param | Type | Description |
---|---|---|
dataSource | DataSource | JSONPath referencing a point in state . |
operation | Operation | The operation needed to be repeated. |
Example: Using lazy state ($) to iterate over items in state.data and pass each into an "insert" operation
each(
$.data,
// Inside the callback operation, `$.data` is scoped to the item under iteration
insert("patient", {
patient_name: $.data.properties.case_name,
patient_id: $.data.case_id,
})
);
Example: Iterate over items in state.data and pass each one into an "insert" operation
each(
$.data,
insert("patient", (state) => ({
patient_id: state.data.case_id,
...state.data
}))
);
Example: Using JSON path to iterate over items in state.data and pass each one into an "insert" operation
each(
"$.data[*]",
insert("patient", (state) => ({
patient_name: state.data.properties.case_name,
patient_id: state.data.case_id,
}))
);
expandReferences
expandReferences(value, [skipFilter]) ⇒ Operation
Recursively resolves objects that have resolvable values (functions).
Param | Type | Description |
---|---|---|
value | object | data |
[skipFilter] | function | a function which returns true if a value should be skipped |
field
field(key, value) ⇒ Field
Returns a key, value pair in an array.
Param | Type | Description |
---|---|---|
key | string | Name of the field |
value | Value | The value itself or a sourceable operation. |
Example
field('destination_field_name__c', 'value')
fields
fields(fields) ⇒ Object
Zips key value pairs into an object.
Param | Type | Description |
---|---|---|
fields | Fields | a list of fields |
Example
fields(list_of_fields)
fn
fn(func) ⇒ Operation
Creates a custom step (or operation) for more flexible job writing.
Param | Type | Description |
---|---|---|
func | function | is the function |
Example
fn(state => {
// do some things to state
return state;
});
fnIf
fnIf(condition, operation) ⇒ Operation
A custom operation that will only execute the function if the condition returns true
Param | Type | Description |
---|---|---|
condition | Boolean | The condition that returns true |
operation | Operation | The operation needed to be executed. |
Example
fnIf((state) => state?.data?.name, get("https://example.com"));
group
group(arrayOfObjects, keyPath, callback) ⇒ Operation
Groups an array of objects by a specified key path.
Param | Type | Description |
---|---|---|
arrayOfObjects | Array.<Object> | The array of objects to be grouped. |
keyPath | string | The key path to group by. |
callback | function | (Optional) Callback function |
Example
const users = [
{ name: 'Alice', age: 25, city: 'New York' },
{ name: 'Bob', age: 30, city: 'San Francisco' },
{ name: 'Charlie', age: 25, city: 'New York' },
{ name: 'David', age: 30, city: 'San Francisco' }
];
group(users, 'city');
// state is { data: { 'New York': [/Alice, Charlie/], 'San Francisco': [ /Bob, David / ] }
humanProper
humanProper(str) ⇒ string
Substitutes underscores for spaces and proper-cases a string
Param | Type | Description |
---|---|---|
str | string | String that needs converting |
Example
field("destination_string__c", humanProper(state.data.path_to_string))
index
index() ⇒ DataSource
Returns the index of the current array being iterated.
To be used with each
as a data source.
Example
index()
join
join(targetPath, sourcePath, targetKey) ⇒ Operation
Adds data from a target object
Param | Type | Description |
---|---|---|
targetPath | String | Target path |
sourcePath | String | Source path |
targetKey | String | Target Key |
Example
join('$.key','$.data','newKey')
jsonValue
jsonValue(obj, path) ⇒ Operation
Picks out a single value from a JSON object. If a JSONPath returns more than one value for the reference, the first item will be returned.
Param | Type | Description |
---|---|---|
obj | object | A valid JSON object. |
path | String | JSONPath referencing a point in given JSON object. |
Example
jsonValue({ a:1 }, 'a')
lastReferenceValue
lastReferenceValue(path) ⇒ Operation
Picks out the last reference value from source data.
Param | Type | Description |
---|---|---|
path | String | JSONPath referencing a point in references . |
Example
lastReferenceValue('key')
map
map(path, operation, state) ⇒ State
Scopes an array of data based on a JSONPath.
Useful when the source data has n
items you would like to map to
an operation.
The operation will receive a slice of the data based of each item
of the JSONPath provided.
Param | Type | Description |
---|---|---|
path | string | JSONPath referencing a point in state.data . |
operation | function | The operation needed to be repeated. |
state | State | Runtime state. |
Example
map("$.[*]",
create("SObject",
field("FirstName", sourceValue("$.firstName"))
)
)
merge
merge(dataSource, fields) ⇒ DataSource
Merges fields into each item in an array.
Param | Type | Description |
---|---|---|
dataSource | DataSource | |
fields | Object | Group of fields to merge in. |
Example
merge(
"$.books[*]",
fields(
field( "publisher", sourceValue("$.publisher") )
)
)
parseCsv
parseCsv(csvData, [parsingOptions], [callback]) ⇒ Operation
Takes a CSV file string or stream and parsing options as input, and returns a promise that
resolves to the parsed CSV data as an array of objects.
Options for parsingOptions
include:
delimiter
{string/Buffer/[string/Buffer]} - Defines the character(s) used to delineate the fields inside a record. Default:','
quote
{string/Buffer/[string/Buffer]} - Defines the characters used to surround a field. Default:'"'
escape
{Buffer/string/null/boolean} - Set the escape character as one character/byte only. Default:"
columns
{boolean / array / function} - Generates record in the form of object literals. Default:true
bom
{boolean} - Strips the byte order mark (BOM) from the input string or buffer. Default:true
trim
{boolean} - Ignore whitespace characters immediately around thedelimiter
. Default:true
ltrim
{boolean} - Ignore whitespace characters from the left side of a CSV field. Default:true
rtrim
{boolean} - Ignore whitespace characters from the right side of a CSV field. Default:true
chunkSize
{number} - The size of each chunk of CSV data. Default:Infinity
skip_empty_lines
{boolean} - Ignore empty lines in the CSV file. Default:true
Returns: Operation
- The function returns a Promise that resolves to the result of parsing a CSV stringOrStream
.
Param | Type | Description |
---|---|---|
csvData | String | Stream | A CSV string or a readable stream |
[parsingOptions] | Object | Optional. Parsing options for converting CSV to JSON. |
[callback] | function | (Optional) callback function. If used it will be called state and an array of rows. |
referencePath
referencePath(path) ⇒ string
Ensures a path points at references.
Param | Type | Description |
---|---|---|
path | string | JSONPath referencing a point in references . |
Example
referencePath('key')
scrubEmojis
scrubEmojis(text, replacementChars) ⇒ string
Replaces emojis in a string.
Param | Type | Description |
---|---|---|
text | string | String that needs to be cleaned |
replacementChars | string | Characters that replace the emojis |
Example
scrubEmojis('Dove🕊️⭐ 29')
source
source(path) ⇒ Array.<(String|Object)>
Picks out a value from source data.
Will return whatever JSONPath returns, which will always be an array.
If you need a single value use sourceValue
instead.
Param | Type | Description |
---|---|---|
path | String | JSONPath referencing a point in state . |
Example
source('$.key')
sourceValue
sourceValue(path) ⇒ Operation
Picks out a single value from source data. If a JSONPath returns more than one value for the reference, the first item will be returned.
Param | Type | Description |
---|---|---|
path | String | JSONPath referencing a point in state . |
Example
sourceValue('$.key')
splitKeys
splitKeys(obj, keys) ⇒ Array.<Object>
Splits an object into two objects based on a list of keys. The first object contains the keys that are not in the list, and the second contains the keys that are.
Returns: Array.<Object>
- - Tuple of objects, first object contains keys not in list, second contains keys that are.
Param | Type | Description |
---|---|---|
obj | Object | The object to split. |
keys | Array.<string> | List of keys to split on. |
toArray
toArray(arg) ⇒ array
Ensures primitive data types are wrapped in an array. Does not affect array objects.
Param | Type | Description |
---|---|---|
arg | any | Data required to be in an array |
Example
each(function(state) {
return toArray( dataValue("path_of_array")(state) )
}, ...)
helpers
These functions belong to the helpers namespace.
helpers.decode
decode(base64Data) ⇒ string
Decodes a Base64 encoded string back to its original format.
Returns: string
- - The decoded string.
Param | Type | Description |
---|---|---|
base64Data | string | The Base64 encoded string. |
Example: Decode a Base64 string
const decoded = decode('SGVsbG8gV29ybGQ=');
console.log(decoded); // Output: Hello World
helpers.encode
encode(data) ⇒ string
Encodes a given string into Base64 format.
Returns: string
- - The Base64 encoded string.
Param | Type | Description |
---|---|---|
data | string | The string to be encoded. |
Example: Encode a string
const encoded = encode('Hello World');
console.log(encoded); // Output: SGVsbG8gV29ybGQ=
helpers.uuid
uuid() ⇒ string
Generates a UUID (Universally Unique Identifier).
Returns: string
- - A newly generated UUID.
Example: Generate a UUID
const id = uuid();
console.log(id); // Output:'3f4e254e-8f6f-4f8b-9651-1c1c262cc83f'
beta
These functions belong to the beta namespace.
beta.each
each(dataSource, operation) ⇒ Operation
Scopes an array of data based on a JSONPath.
Useful when the source data has n
items you would like to map to
an operation.
The operation will receive a slice of the data based of each item
of the JSONPath provided.
It also ensures the results of an operation make their way back into the state's references.
Param | Type | Description |
---|---|---|
dataSource | DataSource | JSONPath referencing a point in state . |
operation | Operation | The operation needed to be repeated. |
Example
each("$.[*]",
create("SObject",
field("FirstName", sourceValue("$.firstName")))
)
dateFns
These functions belong to the dateFns namespace.
dateFns.format
format()
The format function from the date-fns library. See https://date-fns.org/v3.6.0/docs/parse
dateFns.parse
parse()
The parse function from the date-fns library. See https://date-fns.org/v3.6.0/docs/parse
http
These functions belong to the http namespace.
http.get
get(url, options) ⇒ Operation
Make a GET request.
Param | Type | Description |
---|---|---|
url | string | URL to access |
options | CommonRequestOptions | Request options |
This operation writes the following keys to state:
State Key | Description |
---|---|
data | the parsed response body |
response | the response from the HTTP server, including headers, statusCode, body, etc |
references | an array of all previous data objects used in the Job |
Example: Request a resource
http.get('https://jsonplaceholder.typicode.com/todos')
Example: Request a resource with basic auth
http.get(
'https://jsonplaceholder.typicode.com/todos',
http.options().basic('user', 'pass')
)
Example: Request a resource with oauth
http.get(
'https://jsonplaceholder.typicode.com/todos',
http.options().oauth($.configuration.access_token)
)
http.options
options(options) ⇒ OptionsHelpers
Builder function to create request options. Returns an object with helpers to easily add commonly used options. The return object is chainable so you can set as many options as you want. Pass an object to set your own options.
Param | Type | Description |
---|---|---|
options | CommonRequestOptions | options to pass to the request |
Example: Get with a query an oath token
get($.data.url, http.options({ query: $.query }).oath($.configuration.access_token)
http.post
post(url, options) ⇒ Operation
Make a POST request.
Param | Type | Description |
---|---|---|
url | string | URL to access |
options | CommonRequestOptions | Request options |
This operation writes the following keys to state:
State Key | Description |
---|---|
data | the parsed response body |
response | the response from the HTTP server, including headers, statusCode, body, etc |
references | an array of all previous data objects used in the Job |
Example: Post a JSON object (setting the content-type header)
http.post(
'https://jsonplaceholder.typicode.com/todos',
$.data,
options().json(),
})
http.request
request(method, url, options) ⇒ Operation
Make a HTTP request.
Param | Type | Description |
---|---|---|
method | string | The HTTP method to use. |
url | string | URL to resource. |
options | CommonRequestOptions | Request options |
This operation writes the following keys to state:
State Key | Description |
---|---|
data | the parsed response body |
response | the response from the HTTP server, including headers, statusCode, body, etc |
references | an array of all previous data objects used in the Job |
Example
http.request(
'GET',
'https://jsonplaceholder.typicode.com/todos'
)
Interfaces
OptionsHelpers
Helper functions provided by http.options
.
Properties
Name | Type | Description |
---|---|---|
json | function | Sets the `content-type' header to 'application/json' |
basic | function | Sets basic auth on the Authorization header. Pass username and password |
bearer | function | Sets a Bearer token on the Authorization header. Pass the token. |
oauth | function | Sets a Bearer token on the Authorization header. Pass the oauth token. |