Not sure where to start? Head back to the "Planning" page to think about how you want to scale up your OpenFn automation projects.
Assess your capacity
Use these questions to start assessing capacity and technical resources so that your deployment partner can better estimate your total cost of ownership.
- How do you currently deploy, monitor, and maintain cloud-based applications at your organization/government? All deployment environments and institutions are unique and OpenFn is flexible; based on your current dev-ops processes we will recommend different deployment mechanisms.
- What IT and DevOps staff resources are available to support OpenFn deployment and maintenance? Do they have experience with Docker & Kubernetes? Do they have experience with Postgres databases?
- Will the deployment require high-availability? (i.e., if OpenFn will receive requests in real-time from other applications rather than run cron-based jobs, then at least two instances of OpenFn should be run simultaneously behind a load-balancer, making use of “distributed Erlang” to ensure graceful application redundancy; if OpenFn will not be responsible for receiving requests and will only be responsible for making relatively time-independent outbound requests on a cron schedule, the importance of maintaining a zero-downtime system is slightly reduced.)
|Skill||Relevance and reason|
|Erlang||The OpenFn webapp/orchestration layer is an Erlang OTP application.|
|Postgres||The default database for OpenFn is PostgreSQL|
|Docker||We publish all OpenFn images on Docker Hub. Whether you're streamlining developer setup or using container orchestration technologies, understanding docker and containerized computing is helpful.|
|Kubernetes||For high-availability deployments, Kubernetes services provide load balancing and simplify container management on multiple hosts. They make it easy for an enterprise's apps to have greater scalability and be flexible, portable and more productive.|
Kubernetes is NOT required, but it's recommended for high-availability deployments. Consider docker or bare-metal deployments (Erlang OTP apps work very well on Linux) for a simpler setup.
- Use a scalable SQL service and keeping at least two app nodes running with
the following specs will help prevent unwanted downtime.
- GKE requests: cpu@ "500m", memory@ "1024Mi"
- GKE limits: memory@ "2560Mi"
- For a simple non-Kubernetes/HA deployments, the minimum recommended machines
- Application machine: 2 vCPU (roughly a single core of a 2.6 GHz Intel
Xeon E5) with 3.75 GB memory and 15 gb of storage for the application
- Any linux-based operating system that can run Docker (Ubuntu 20.04+ or Debian 9+).
- Docker (18 or greater).
- Database machine: 2 vCPU (roughly a single core of a 2.6 GHz Intel Xeon E5) with 3.75 GB memory. Storage required for the DB varies by how many days (if any) of message data you’d like to store on the app itself and cannot be determined without estimates for message/run throughput. If scaling physical storage is not difficult for your particular deployment, start at 40gb. 3. A Postgres (at least v14.2) instance (as we run this on a separate server) from the application for greater stability.
- Application machine: 2 vCPU (roughly a single core of a 2.6 GHz Intel Xeon E5) with 3.75 GB memory and 15 gb of storage for the application
- If both the application and database are hosted on the same machine (which is not recommended) that machine should have roughly the sum of the requirements above.
- Note that the application by default provides an HTTP endpoint (no
TLS/SSL). A reverse-proxy/load-balancer is expected to provide both HTTPS
(HTTP2 compliant) and load balancing between instances.
- I.e. the application server provides no encryption for web access, a web server in front of the application needs to be provided; Nginx is a good start, provided with TLS certificates.
- While network architecture is up to the client, we strongly recommend a private subnet for the application servers.
- The OpenFn application does not need to be deployed on the same machine as any other services, however network routing and firewall rules will need to be provided in order for the integration to access the source and destination systems if hosted on different servers.
- For troubleshooting/external support, administrators will need SSH access
to an unrestricted account (
sudofor Ubuntu) if deployment maintenance services are required.
While your deployment strategy should be carefully considered with a DevOps specialist, the following sample configurations may provide useful starting points.
Deploy the application and database on the same machine.
(b) Recommended Minimum
Deploy the application and database on separate machines.