Both microservice and OpenFn.org (the iPaaS) have some pre-configured limits to ensure smooth operation. Read below for a detailed explanation of those defaults and how to adjust the limits.
If you don't feel like reading much, here's the tl;dr:
- Don't send more than
/inboxAPI in a single request.
- Make sure your
state.jsondon't total more than
10MBwhen you start or finish a run.
- Keep memory usage below
512MBduring the run.
NodeVM memory limits
The default memory limit for a Node process is
512MB and unless you've
customized it for use on core or have agreed to a high-memory plan on
OpenFn.org, job runs will be killed by Node (
when they reach that threshold.
Increase memory limit for an individual execution
1GB limit for an individual job, use:
node --max-old-space-size=1024 core execute ...arguments
Increase memory limit for all jobs on that machine
4GB limit for all jobs, set an environment variable with:
Request body limit on inboxes
The maximum size of the JSON body accepted on the
/inbox/your-uuid endpoint is
8MB. Note that payloads should be kept below this limit in order to ensure
that Job state limits are not hit when a subsequent job run
If you're using OpenFn for bulk data processing/periodic batch jobs, you can either reconfigure these limits on your microservice deployment or contact email@example.com to have special provisions made for your OpenFn.org project.
Job state limits
When a job is to be sent for processing the total size of all job artifacts
expression and the initial
state) cannot exceed
10MB. In other
words, your code (
expression.js) plus your data and configuration
state.json) cannot exceed
10MB in total.
Similarly, when state is saved to cloud storage/disk after execution there is
10MB limit to the size of a job's final
If you're dealing with large amounts of data you've got to (a) keep your total
memory use below
512MB during execution and (b) clean up state so that
whatever you're passing to the next job, whether via FLOW or via saved state for
a CRON job, remains below
Sometimes, this necessitates streaming data from A -> B in the same job. Other times it will necessitate storing data as local variables and then posting data to your inbox in chunks before cleaning up state.