We’ve put together a resource to help other Digital Public Goods and Global Goods achieve just this. You can view and comment on the resource here, or read on to learn why.

Digital health technologies have a critical role to play in both improving the delivery of health services and making health systems more robust and adaptable. Whilst this has been the consensus for a while, various factors such as bureaucracy, lack of capacity, authority/ownership, budget or simply lack of prioritization have hindered the adoption and implementation of such technologies.

However, the COVID-19 pandemic has brought digital health systems back to the top of the agenda, having forced governments to adapt their health systems and processes in a short amount of time and confronted them with the fragility and inflexibility of some of their existing systems. This has led to an increase in funding for both new and existing health technologies.

At the same time, the recent pandemic also saw an increase in data breaches and data leaks in the healthcare sector. Whilst these are a serious matter in any system, the nature of the healthcare technologies means that the data they are storing or managing almost always includes Personally Identifiable Information (PII) which, if leaked, can have dramatic consequences for those they identify.

The higher the number of technologies and quantity of data stored in these, the higher the risk of vulnerabilities and negative impact of a data breach or data leak. If Global Goods are to make the most of this renewed interest and increase in funding for digital health technologies, it is crucial for them to keep security as a top priority to ensure that they ‘do no harm’, a core Digital Public Good principle.

Several organizations have already put some thought into documenting security best practices and standards. However, none situate these security decisions within the wider context of a product development timeline or roadmap. This requires individuals to regularly scan through long lists of security recommendations to pick out and prioritize their implementation, increasing the likelihood of an oversight or early-made product decisions which make it harder to implement such security features down the line.

As product manager at Open Function Group, I myself have recently been reviewing and analyzing numerous resources to identify and prioritize features required to get to fully-secure authentication and authorization for our newest product Lightning–and make sure that nothing falls through the cracks. I’ve also had the opportunity to gather insights from Digital Public Good community forums (OpenMRS, DHIS2, OpenLMIS, etc) and speak to other product managers and engineers from the Digital Public Goods community to ask them about their learnings, and any tips they might have: Austin and Morten from DHIS2, Dev and Biyeun from Dimagi.

To more widely share these learnings, we have drafted this resource documenting the specific security measures that should be considered at each stage of product development _specifically with regards to authentication and authorization. _Our hope is that this article will help other product managers in the Global Good (GG) and Digital Public Good (DPG) communities consider these features into their backlog/roadmap from the outset and make sure they are following a ‘secure by design’ approach. To inform our analysis, we analyzed learnings from the wider DPG community, as well as recommendations from other security experts and international standards, including GovStack, OpenHIE, OWASP, NIST, and more.

The resource can be found here.

What is workflow automation?​

Workflow automation is an approach to making the flow of tasks across platforms run independently. A prescription SMS notification feels seamless, but it probably requires complex automation across multiple systems like clinical registries, pharmacy stock databases, and SMS gateways. Tools like OpenFn run “behind the scenes” to perform calculations and transfer relevant data to/from different systems autonomously, minimising the risk of human error and ultimately saving time and money that an agency would spend on manual data transfers. The goal is to preserve data integrity, uphold data security, and enforce compliance with policies and data standards.

Since 2014, OpenFn has been at the forefront of workflow automation in the social sector, providing a platform to help governments and NGOs focus on the human aspect of their work rather than wasting time trying to exchange data between systems or perform calculations manually.

MyAgro​

MyAgro, an NGO helping West African farmers increase their yield, uses OpenFn to power its SMS-based savings deposit system for small-scale farmers. MyAgro distributes and tracks saving vouchers using the CommCare app. Smallholder farmers purchase pre-paid myAgro saving scratch cards and text the one-time code to secure the voucher credit in their account. Once the amount has been added to their balance, a confirmation notification is sent to the farmer. MyAgro works with the farmers to reach their savings goal, which they then use to pay for seeds, fertilisers, and training courses. OpenFn:

1. Syncs voucher codes from CommCare to Salesforce so the relevant offices can track who purchased the saving vouchers.
2. Matches incoming SMSs against the database of vouchers to deposit correct funds into the individual account.
3. Allocates savings to farmers’ accounts in Salesforce and updates the account status.
4. Sends a confirmation SMS with updated account balances to farmers, so they are aware of their credit.

OpenFn helps myAgro reach more farmers with less administrative overhead by automating these key steps. MyAgro calculated that even at their 2014 levels of scale, OpenFn saved them 260+ hours per year and increased customer confidence as they worked to serve poor farmers.

Sinapis​

Sinapis, an NGO that empowers entrepreneurs through training courses, uses OpenFn to automate its payment collection process. Every time a client’s new M-Pesa payment is registered in Kopo Kopo, OpenFn:

1. Automatically matches incoming payments with existing Sinapis accounts.
2. Logs new payments and updated payment status in Salesforce to track entrepreneurs’ charges.
3. Updates account balances to help staff monitor outstanding balances and overdue fees.

With OpenFn handling the routine data processing, Sinapis has more time to support its entrepreneurs. Precious staff time can be spent delivering training, the essential face-to-face coaching that transforms their clients’ fledgling enterprises. Their team does not have to worry about the accuracy of payments and account balances; the OpenFn implementation replaces error-prone manual data entry with a systematised, intelligible, fully automated process.

DIAL​

The Digital Impact Alliance (DIAL), an organisation that works to overcome digital development barriers, wanted to prototype a “building-blocks based approach” for their upcoming GovStack.global project. They configured an HR workflow solution that connects mobile data collection apps, databases, and payment apps using OpenFn. Every time an organisation registers new workers via ODK Collect, OpenFn:

1. Automatically registers new employees in an iHRIS database to oversee all new hires.
2. Makes payments to workers via Mifos, ensuring all wage transfers are direct.

While it’s not in use at scale, this prototype is a powerful illustration of how automation enables scalable, component-based solutions to become more than the sum of their parts. The processes that DIAL implement with OpenFn help unlock the true potential of these powerful underlying DPGs.

What’s the big picture here?​

OpenFn’s goal is to increase the efficiency and effectiveness of the social sector. Why should health and humanitarian organisations waste precious resources manually moving data across systems when workflow automation can do it more safely and quickly? We want to ensure employees focus on using data to do their work, not transferring data between systems and running rote calculations.

Saving organisations time and money is part of the picture, but we also deliver better program outcomes through efficiency and effectiveness. Suppose OpenFn securely automates the complex digital processes critical health and humanitarian interventions require. Reducing error rates and security vulnerabilities and getting higher quality data into the hands of a doctor faster actually saves lives. Responsible automation leads to better health and humanitarian outcomes, enabling larger-scale intervention delivery via efficiency gains and improving service quality by getting accurate data into the hands of patients, clinicians, and policymakers when they need it.

Call to Action​

Think about your own experiences; have you found yourself in a situation where manual data processing—some rote, mindless task—has gotten in the way of doing the more exciting parts of your job? Whether you are in the health and humanitarian space or just want to make your life easier, OpenFn is here for you. Our open-source toolkit is available for everyone to automate tasks such as sending SMS and email notifications or automating data cleaning and formatting for faster reporting.

Head to docs.openfn.org to learn more about enterprise-grade automations with our DPG and check out community.openfn.org to connect and collaborate with like-minded social-sector integrators and OpenFn staff. We hope to see you soon!

About Open Function Group​

Open Function Group (OFG) is a global team of integration specialists and developers of the OpenFn Integration Toolkit. As a DPG, OpenFn’s core technology is free and open-source. If you want more information about the Toolkit or to sign up for a free plan on our associated SaaS application, visit https://www.openfn.org/.

In this post we'll discuss:

1. Arrow functions
2. Falsy values
3. Other short-hand notation

Keep reading for all the important learnings I have taken from just the first four modules!

What is that little => arrow?​

Javascript allows us to create functions which take arguments and do stuff with them. Most times, function declaration in Javascript looks like this:

function getAge(dateString) {
  if (!dateString) return;

  const today = new Date();
  const birthDate = new Date(dateString);
  var age = today.getFullYear() - birthDate.getFullYear();
  var m = today.getMonth() - birthDate.getMonth();
  if (m < 0 || (m === 0 && today.getDate() < birthDate.getDate())) {
    age--;
  }
  return age;
}

This was simple enough for me to follow. We are declaring a function, getAge(), which has the steps to calculate a person's age and can be easily reused throughout the job.

But other times, function writing looks like this:

const getAge(dateString) => {
  if (!dateString) return;
  // and so on...
}

In OpenFn, you're often writing unnamed functions that take state as their only argument and you use them right away. See how concise writing and using a function can be with the arrow:

upsert('tbl_study', 'study_id', {
  study_id: state => state.studyIDMap[state.formType],
});

These arrow functions confused me when reading through jobs. I've learned that, for the most part, they can be thought of as short-hand notation which removes the need to type out function for every function declaration.

Further, we can make the code even more concise by also removing the parenthesis, curly braces, and the return keyword when certain criteria are met. In the code above, the parenthesis are omitted because there are no parameters, and the curly braces and return keyword are omitted because the function body is composed of a single-line block. As you can see, there are a variety of ways to write functions and this course is helping me better recognize them. Some key terminology here is: function expression, arrow function, concise function, anonymous function, and implicit returns.

Falsy values​

During the mapping phase of integration design, we often discuss how each answer choice for picklist values should map from the source system to the destination system. Sometimes the mapping is simple but other times, there is an extensive list of possible choices that can be found in the message and not all are relevant to the destination system. Then the question is, how should the job handle values which are not explicitly mapped?

Sometimes we hear clients say to "ignore" those values. But what does it really mean to "ignore" a value? Should we set it to 0? An empty string? How about null, undefined or NaN? In Javascript these are all falsy values.

Determining what is really meant here is important and the value selected has different implications depending on the system.

Take this sample mapping for different districts in Conakry. The value left of the colon is from the source system and the value to the right is for the destination system.

const districtMapping = {
  Ratoma: 'RT',
  Kaloum: 'KL',
  Dixinn: 'DX',
  Matam: 'MA',
  Matoto: 'MT',
  Other: undefined,
};

If the destination system is Salesforce, this mapping would not upload "Other" to Salesforce. However if "Other" mapped to an empty string instead, this would upload the empty string to Salesforce. This distinction is especially important in cases where we are overwriting existing data. For instance, if a student previously lived in Ratoma and then moved to an unknown district marked as "Other", undefined would not update the student's district in Salesforce but the empty string would. Both of these are falsy values but have different meanings in Salesforce.

How about if the message includes a value for a district that is not in the mapping? Such as "New York". Should the job default to undefined? Or null?

These questions are just a few examples of how understanding falsy values in Javascript can make it easier to implement the best mapping for the real-world use case.

Short-circuit evaluation, template literals, and all the short hand that used to confuse me​

Short-circuit evaluation

When we are syncing forms to a database, we sometimes expect different versions of the same form with fields present one version but not the other. One way to mitigate this discrepancy is by submitting a dummy value for the field whenever it is missing in the message. I just learned the fastest way to add this code to my jobs! For example, the code below will ensure that household_id always has a value in the destination system: if survey_info/household_id is present in the message this will be the assigned value, otherwise it will assign the dummy value state.data.body._id.

household_id: state.data.body['survey_info/household_id'] || state.data.body._id,

Template literals

Like the arrow discussed above, the "dollar sign" was another symbol that often confused me. But template literals are actually very straightforward. They increase the readability of the code and make it easier to see what the resulting string will be. Wrap your template in the backtick (`) symbols and then each variable in ${}.

const sign = '$';
console.log(`The ${sign} isn't so confusing!`);

Higher-order functions & iterators​

The most challenging module in the course covered higher-order functions. These are defined as functions that accept other functions as arguments and/or return functions as output. But why are these important and how are they used in OpenFn jobs? It turns out we use them quite alot! The code below is an example from an existing integration with the field names replaced.

const participantsToUpdate = state.data.json.filter(data =>
  state.idList.includes(data.id)
);
const participantsToCreate = state.data.json.filter(
  data => !state.idList.includes(data.id)
);

This code is using a built-in JavaScript method that helps us iterate on arrays to manipulate elements and return values. The .filter() method is being used to return a new array after filtering out certain elements from the original array. We have declared two variables to store the new arrays: participantsToUpdate and participantsToCreate. Participants whose ids are existing in the destination system (or who are in the idList array) are added to participantsToUpdate, and any remaining participants are added to participantsToCreate. This filtering helped us perform the correct operations on each participant type. For instance, later in the job, we use the filtered arrays to only overwrite a participant's reportNumber field if it is a new participant. .filter() is just one of many higher-order functions that power OpenFn jobs.

Objects & job mappings​

The final lesson was on objects and key-value pairs--something used in almost every OpenFn job! The result of the design and mapping phase of requirements gathering is almost always a mapping document which includes key-value pairs in plain English. These specifications are then translated to JavaScript via objects. The code below is a snippet of an object which captures the key-value pairs for states in the US.

const stateMapping = {
   AK: 'ALASKA',
   AZ: 'ARIZONA',
   AR: 'ARKANSAS',
   CA: 'CALIFORNIA',
   ....

The value left of the colon is how the state is represented in the source system, and the value right of the colon represents how OpenFn will send the data to the destination system. This mapping process is key to integration design. Learn more about mappings here.

Next steps​

I'm well on my way to becoming a better job reader and writer. Here are some next steps:

1. Understand fn(state) and how state can be manipulated in OpenFn jobs.

2. Explore what's available on the JavaScript docs site.

3. Sign up for the next level JavaScript course.

The Blurred Line​

Because a React app is built on components, the basic UI units, it is natural to think and organise your tests around components! And so unit testing, in this case, would refer to "component testing", which may be confusing at times, especially when the concept of unit testing is again applied to testing functions such as Redux reducers and action creators or any other JavaScript function in your application.

The other challenge that I often faced was whether to write tests for each component in isolation or write a test for a feature that encapsulates a set of related components. The later would be equivalent to writing what I would call "integration tests".

Finally, one would say "well then you could have just written the tests in a way that resemble the way the application is used"! This approach is commonly recommended in the React community, but it quickly becomes really complex to maintain the layers of separation between unit tests, integration tests and end-to-end tests.

What did I learn?​

Given a React/Redux application, here is how I would organise my testing strategy:

Unit Tests​

• In a React app, unit tests will largely apply to testing "helper functions" and not to testing components, as justified in the next section. Helper functions, in this case, would refer to functions that live outside the components and are neither Redux action creators nor reducers. These functions can be used inside components, action creators, reducers or other parts of your application.

• Writing unit tests for "helper functions" would ensure their signatures and expected outputs are protected against regressions. This would also ensure their use across components or other functions is consistent and as expected.

• Where possible, each "helper function" must have its own unit test.

• An example of a unit test would like:

  const sum = require('../../js/sum');
  
  test('adds 1 + 2 to equal 3', () => {
    expect(sum(1, 2)).toBe(3);
  });
  

• Write a thousand of these.

Integration Tests​

• In the context of a React/Redux app, component tests can be thought of as integration tests. This is because React components are built around features such as <Signup />, <Search />, etc. So one React component can be a mix of other components to achieve a UI feature set.

• To test a component, write an integration test that covers the use of a given component for a given UI feature.

• If a component being tested dispatches a Redux action, this is the right place to test those actions and their effect on the UI.

• Pay attention to the concept of feature isolation vs component isolation as it will help you write better integration tests and also easily mock component contexts.

• A classic example of feature isolation is when you have a <UserList /> component which displays a list of users and has a <button /> to add a new user. Writing a test for <UserList /> would be equivalent to testing a feature.

• In this example, one would be tempted to test the action of clicking on the <AddUserButton /> and further test the <NewUser /> form... nope! This is where we draw the line! Only test that the <UserList /> renders the mock users in the list and that the <AddUserButton /> is present/enabled. The <UserList /> feature ends there, otherwise you will be sliding into End-to-End testing :)! The <User /> component, although it is invoked by <UserList /> component, it is isolated enough to be tested in its own integration test.

• Testing components this way would make "context mocking" easier for components.

• Another important benefit for isolating testing context, as in the example above, is that it will be easier to mock the redux actions and/or api calls using tools such as Jest and Mock Service Worker (or "msw") as explained in the Choosing Tools section.

• The value of writing integration tests for components, in this way, ensures that a given component renders the UI consistently, given all possible combinations of contexts and interactions. This will also allow you to ensure redux actions invoked by the component are called as expected and with the correct arguments.

• An example component integration test would look like:

  // ....other imports
  import { setupServer } from 'msw/node';
  // Tell jest to mock the module
  jest.mock('../js/actions/UserActions', () => ({
    ...jest.requireActual('../js/actions/UserActions'),
    saveUser: jest.fn(),
  }));
  
  import { saveUser as mockSaveUser } from '../js/actions/UserActions';
  const server = setupServer(...handlers);
    // Enable API mocking before tests
    beforeAll(() => server.listen());
    // Reset any runtime handlers we may add during the tests
    afterEach(() => server.resetHandlers());
    // Disable API mocking after the tests are done.
    afterAll(() => server.close());
    beforeEach(() => {
      jest.clearAllMocks();
    });
  describe('<AddUser/>', () => {
  
  test('create new user', async () => {
      const {getByPlaceholderText,getByText} = render(<User {...defaultProps} />);
      userEvent.type(getByPlaceholderText('First Name'), 'John');
      userEvent.type(getByPlaceholderText('Last Name'), 'Doe');
      userEvent.click(getByText('Save'));
      expect(mockSaveUser).toHaveBeenCalledTimes(1);
      expect(mockSaveUser).toHaveBeenCalledWith({
        firstName: 'John',
        lastName: 'Doe',
      });
  }
  
  

• Write a good couple of these.

End-to-End (e2e) Tests​

• In a React/Redux App, this would mean testing a full flow of a given feature. end-to-end tests would require launching the entire application, including the backend, to run a given test.

• Note that end-to-end tests are different from integration tests as they require the entire App to run and render the full flow to your component under test.

• With this understanding, consider writing e2e tests per workflow.

• An example e2e workflow is the "Viewing and adding users" workflow.

• The e2e test for this workflow would require a test runner to launch the app, log-in, navigate to the users list page, verify existing users are in the list, click on the Add New User button and confirm that the new user has been added to the list.

• As you can see, e2e tests have more dependencies and require that you setup your testing environment in way that closely simulates your real application usage.

• An example e2e test for a React/Redux App with a Phoenix/Elixir backend, using Hound as a test runner looks like this:

defmodule OpenFn.UsersTest do
  setup do
    user = insert(:user, confirmed_at: DateTime.utc_now())
    {:ok, user: user }
  end

  @tag :integration
  test "Sign-up.", %{user: user} do
    navigate_to("/sign-up")
    form = find_element(:id, "sign_up_form")

    form
    |> find_within_element(:id, "first-name")
    |> fill_field("John")

    form
    |> find_within_element(:id, "last-name")
    |> fill_field("Doe")

    form
    |> find_within_element(:id, "email")
    |> fill_field("doe@gmail.com")

    form
    |> find_within_element(:id, "save-button")
    |> click

    assert page_title() === ~s/Welcome to my page/
    end
end

• Write only a few of these.

Choosing Testing Tools​

There are many testing tools out there, but for a typical React/Redux app the following tools should help you accomplish the above tasks:

1. Jest as test runner for unit and integration tests.
2. React Testing Library used along with Jest as an "assertion library" for integration tests.
3. MSW used along with Jest as a REST API mocking library.
4. Hound as a test runner for e2e tests in Elixir/Phoenix apps. Puppeteer can also be used along with Jest.
   • If Puppeteer is used, it will work seamlessly with Jest but only in headless browser mode. It also reduces on tech stack since you will only need Jest.
   • Hound gives you the ability to run your e2e tests both in headless and browser mode.

Final thoughts and next steps​

Testing a React App can be really hard, but worth it! By building Aria-accessible components ahead of time, you save yourself 💰 and good health! A few more hints would be:

• Build clean, isolated and plugable components for your better testing experience. "God components" can be a pain to test!
• Using test runners such as Jest, that use emulated web browsers (e.g., jsdom) rather than a real browser come with their own challenges in rendering and traversing complex DOM trees, especially if you are using UI libraries such as MUI.
• If using Jest for integration tests, I would recommend the components under test have as few dependencies as possible to avoid the complexity involved in mocking http requests and waiting for asynchronous DOM rendering.

What would I do differently? Here are my few thoughts:

• Organise and document detailed test cases for manual "click testing".
• Identify and clearly isolate components for integration tests.
• Do not delete slow tests, instead re-write your component to be faster. Respect the linter's advice, always!
• Use a commonly supported frontend testing stack such as Jest, Msw, or Puppeteer for easier setup and community support.
• Setup your test runner to use a test database. It helps, especially during e2e testing.
• Always write all the three types of tests, whenever applicable.

All this stuff for what?

• Well because regressions can be much more expensive to your organisation! Writing high quality and thoroughly tested software will save you 💰 and help guarantee a maintainable codebase and a progressive software application.

Happy testing,

Chaiwa

This is a post written on what we call 'Slow Fridays', where we explore and think about stuff we're curious about. The deal is some artifact at the end of the day. So while this post is not complete - I believe part two is warranted.

Phoenix 1.6 started to include esbuild by default. The main reasons cited were the amount of support the team had to continuously give for a toolchain that although necessary is not under the control of the framework teams control.

Back in the day, Phoenix used brunch to do front end builds - despite not being the most common bundler at the time. The objective I assume was to provide something that is easy to get going with and lowers the barrier to entry for Phoenix. We actually switched brunch out for webpack before webpack got into Phoenix in order to more easily use new libraries and newer ES syntax.

It wasn't without its challenges, and still is. Webpack is wonderfully powerful - but it does do a lot, and it's really easy to stumble onto plugins that are either order dependent or mutually exclusive.

So what's this about esbuild​

It's a bundler/compiler for JS/TS projects that is written in Go, along with being faster by the simple fact it's natively compiled it also leverages concurrency. To compare why esbuild is faster than NodeJS on face value is unfair in my opinion. Although having similar objectives, they are wildly different in implementation and in features.

So speed is the big selling point. But thinking back to other bundlers, what made us switch (thinking of gulp, grunt, mixing in babel, browserify etc). Those changes were never about speed, at least a drop in build time was nice - a bit of caching goes a long way. The changes were about being able to use the syntax and libraries we wanted with as little fuss as possible.

Webpack can do almost anything. I'm not convinced esbuild can match that, and as an open-source maintainer I'd argue it shouldn't break its original goals to match Webpack.

Exploring the caveats​

I started breaking down what our current bundle setup actually does, after all if we were just bundling plain JS I'd probably be using rollup and terser. Going through our webpack.config.js file I can see that it:

• Conditionally provide sourcemaps depending on the environment.
• Uses babel to parse js/jsx files
  At the same time cherry picks lodash imports ^*.
• Parse imports for .css files through style-loader and css-loader
  Injecting the styles onto the dom.
• Detect require statements to images, copy them to an assets folder
  and replace the statement with a url to image.
• Can watch files (excluding node_modules) in development.
• Splits files that come from node_modules into a vendor.js bundle.

That's a lot more than just building something, there's some implicit behaviour here.

• The interaction between style-loader and css-loader results in extra functions and behaviour being introduced, it's not producing a .css file.

  I'm personally not a big fan of apps injecting styles, but I think I get why people do it. Maybe I'm old-school and like to have my stylesheets delivered in a few files (or even one).

• The splitting doesn't know about app.js, it puts everything that resolves to node_modules in vendor.js. Subtle but worth pointing out.

^* Could probably ignore this and refactor some files and check that tree-shaking is working properly.

What we need esbuild to provide​

Like I mentioned before, webpack is super versatile and it would be short-sighted to get esbuild to do everything it does.

So I'm approaching this with the idea that I code will need some changes, the less the better - I've always firmly believed the kinds of restrictions smaller tools provide can make things neater and more portable.

• ✅ Provide source maps
  Supported out of the box. docs
• ✅ Watching files
  Yup. docs
• 🆗 Build our css (and vendored css)
  Compiling CSS is supported, however it doesn't (as far as I can tell) doesn't replace style-loader and how it injects CSS into the DOM.
• ✅ Copy image assets referenced in JS
  Indeed it can, using the file loader. docs
• ✅ Can split our files
  I picked a confusing green checkbox to illustrate that while esbuild does support file splitting - it appears to have some caveats, primarily that it only works with esm output files. We don't ship ES Modules to the browser, but seems like a good moment to try given the primary targets for our web front end getting module support in 2019 ².

Giving it a go​

I've tried to go over the process as methodically as possible, after all this isn't a new codebase nor am I alone on this. It's always important to understand who's going to be working with this, and respect the varying skill sets and focuses of our peers.

So we know what we want, but can any of this work? Let's give it a go with the simplest of steps:

./node_modules/.bin/esbuild js/app.js --bundle --outfile=out.js
 > js/app.js:58:2: error: Unexpected "<"
    58 │   <React.StrictMode>
       ╵   ^

1 error

Bang! Right, that one makes sense to me, it doesn't know what JSX is yet. I'm gonna take a wild guess and say it won't know about the CSS or our referenced images.

./node_modules/.bin/esbuild js/app.js \
        --loader:.js=jsx \
        --loader:.png=file \
        --loader:.jpg=file \
        --loader:.jpeg=file \
        --bundle \
        --outdir=out
 > js/marketing/ProductsList.js:53:40: error: Unexpected ":"
    53 │     const displayList = products.filter(::this.includesText);
       ╵                                         ^

 > js/marketing/Banner.js:13:4: warning: Duplicate key "background" in object literal
    13 │     background: `-webkit-linear-gradient(to left, ${theme.palette.banner.right}, ${theme.palette.banner.left})`,
       ╵     ~~~~~~~~~~
...

Nice! Two things worth mentioning here:

1. We're using some non-standard ES syntax here: ::, let's replace that with the equivalent includesText.bind(this).
2. Some duplicate keys in our theme objects, looks like a warning but worth cleaning up.

./node_modules/.bin/esbuild js/app.js \
        --loader:.js=jsx \
        --loader:.png=file \
        --loader:.jpg=file \
        --loader:.jpeg=file \
        --bundle \
        --outdir=out

  out/app.js                               8.0mb ⚠️
  out/commcare-dhis2-JKJLM3MQ.png        471.0kb
  ...
  out/app.css                             85.1kb
  ...and 145 more output files...

⚡ Done in 651ms

Jeepers! 651ms! That's nuts.

We can see it's copied our images and css into the build folder. Note that we're not doing any bundle splitting right now and from the looks of it

... \
>   --minify

  out/app.js                               3.4mb ⚠️
  ...
⚡ Done in 611ms

That's better, wow. It's kinda difficult to not be amazed. For context, a minified and split production build takes about 34s with webpack and that's on my i7 desktop machine, and 197s (3+ mins) on CI/CD.

What's next?​

So our 'can we actually do this' seems to have gone pretty well so far. I'm really excited about what this will give us in the end.

But a shell command doth not a replacement for webpack make. We still need to:

1. Get Phoenix to use esbuild and watch our files as we work.
2. Ensure that our html templates serve the correct files in dev & production.
   Including our CSS that is no longer injected into the DOM.
3. Split at least our vendored modules into their own bundle.
4. Make sure sourcemaps generate correctly for when we upload them to Sentry.
5. Have some kind of cache-busting naming scheme for production builds.

Resources​

• FYI: Phoenix drops webpack and npm for esbuild ¹
• JavaScript modules via script tag ²

As many teams who spend more and more time using docker, it's quite common to get into multi-stage builds; usually resulting in significantly smaller images.

However this comes with a pretty significant dilemma with caching. Even when using the --cache-from flag when building, docker only caches the last image.

One proposed solution¹, is to pull, build and push each individual stage. Coming with tight coupling between the shape of your Dockerfile and your build process/scripts.

The other solution uses Docker Buildx which the document describes as:

Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit. It provides the same user experience as docker build with many new features like creating scoped builder instances and building against multiple nodes concurrently.

While that sounds pretty cool, it doesn't really touch on caching. This actually took me a while to find out that it would in fact do caching very differently. In fact it's a very different experience using it, and has lots of really cool features that further detach you from the local docker state allowing you to build in environments that are stateless - such as Google CloudBuild without having to wire up some kind of persistence or file caching scheme.

Buildx​

We're only going to scratch the surface of Buildx, and with that let's get the absolute minimum working; build our image locally.

Local Cache​

First things first we need to create a builder, and select it for use. This is important as without creating a buildx builder (and setting it as the default), buildx will use the docker driver instead of the docker-container driver which we want in order to take advantage of cache exporting.

docker buildx create --name mybuilder --use

You only need to run this once, except in the case of CloudBuild where each invocation is a new node.

docker buildx build \
  --cache-from=type=local,src=/tmp/buildx-cache \
  --cache-to=type=local,dest=/tmp/buildx-cache \
  --load \
  .

While the --cache-* options aren't specifically required when running build, as buildx does manage its own local cache (distinct from the regular docker cache), it's there to emphasise the options that cache can be provided via the CLI options.

This is about as close as you get to a regular docker build, with the significant difference being that you have to specify where to cache from and to.

The --load flag is to tell buildx to set the output to the local docker daemon. Without that you won't actually get a resulting image to run. However, depending on your use case, this could be seen as a convenience - if you're wanting to run your tests inside your build; a resulting image isn't particularly useful.

Remote Cache​

Now comes to the part I'm most interested in, caching in a stateless/remote environment. Multipart builds for us at OpenFn are essential, since we use Elixir and like other compiled languages there is a lot to be gained by only shipping the stuff you're going to run; and no language is safe from requiring several times more 'stuff' in order to build our apps.

Buildx supports a handful of different types of caching sources and destinations. We're going to be using the registry type, where you point the cache at a repository reference (repo/image:tag style).

One thing to note is that Google Container Registry does not support the metadata/manifest format that buildx uses, so if you're using Google Cloud you will need to start using Artifact Registry.

Inline

Push the image and the cache together:

...
--cache-from=type=registry,ref=$IMAGE_NAME \
--cache-to=type=inline \
...

This comes with the constraint that cache mode is always min, which only exports/caches the resulting layers; which is still better than the plain docker build caching but I think having the intermediary layers is generally a win. We want to avoid a single line change invalidating an entire build step.

Registry

Resulting image and cache are separated:

...
--cache-from=type=registry,ref=$IMAGE_NAME-build-cache \
--cache-to=type=registry,ref=$IMAGE_NAME-build-cache,mode=max \
...

Again coming back to the cache mode, here being max; all intermediary laters are exported to the cache image as well.

I have opted to create two images, one for caching and another for the resulting image used to deploy. This gains us a much more granular cache and the ability to more easily manage the cache image - like deleting the whole thing when wanting to invalidate the cache. Not to mention I'm fairly sure the size of our images that get pulled on kubernetes would get significantly larger with many more layers.

It feels like a safer bet to have lean images for kubernetes to pull, and chunky cache images specifically for speeding up build.

Depending on your setup, pulling large images can get seriously expensive in a reasonably active deployment environment - like on AWS ECS without using PrivateLink.

It appears the moby/buildkit documentation also demonstrates this approach.

IMAGE_NAME=us-east4-docker.pkg.dev/<project-name>/platform/app \
docker buildx build \
  -t $IMAGE_NAME:latest \
  --cache-from=type=registry,ref=$IMAGE_NAME-build-cache \
  --cache-to=type=registry,ref=$IMAGE_NAME-build-cache,mode=max \
  --push \
  --progress=plain \
  .

This implies that the cache image is named with the suffix -build-cache:
us-east4-docker.pkg.dev/<project-name>/platform/app[-build-cache].

The --push argument tells buildx to push the resulting image to the registry.

Tips​

Clearing the local cache

As mentioned before, buildx has its own cache and in order to clear the cache while debugging and readying a Dockerfile for remote building you'll probably need to reach for docker buildx prune.

Closing thoughts​

Using buildx has been a really pleasant experience, having personally attempted using it a few times over the last 3 years; the most recent one being the first time I felt confident getting it into production. As with any sufficiently flexible build tooling, the errors and issues you can run into range from complete gibberish, genuinely concerning inconsistencies to architectural choices that you haven't fully caught up on; requiring an ever growing list of changes you need to make to your own build process.

Our initial observations have been great, reasonable changes on our build have gone from 28 minutes to around 9 minutes.

While I have encountered a few confusing cache invalidations, especially when building locally, exporting the cache to a repository and then having CloudBuild use the image cache. And occasionally locally having what feels like really aggressive caching on intermediate steps, leading me to pruning the local cache.

But overall, these issues aren't necessarily buildx issues and more likely a combination of building docker images in general except with many more steps accounted for by the cache.

It's kinda hard to see now what the exact issues I had with it in the past, but hey!

Buildx has given me what I 'expected' with docker multi-stage builds, and having the cache in a repository completely side-steps having to attach a shared volume or copying from a storage bucket.

Resources​

• Multi-stage builds #3: Speeding up your builds ¹
• Docker Buildx
• buildx build reference
• mody/buildkey Registry cache exporter

First, this is how I think about jobs and what we do at Open Function Group to try to make our job code as readable, future-proof, and concise as possible. There are a million different ways to approach writing jobs. This is one.

It all starts with state​

If a job is a set of instructions for a chef (a recipe?) then the initial state is all of the ingredients they need tied up in a perfect little bundle. It usually looks something like this:

{
  "configuration": {
    "hostUrl": "https://moh.kenya.gov.ke/dhis2",
    "username": "taylor",
    "password": "very-secret"
  },
  "data": {
    "type": "registration",
    "patient": {
      "age": 24,
      "gender": "M",
      "nationalId": "321cs7"
    }
  }
}

This might be the initial state for a real-time, message-triggered job. Some source system generated a new patient payload and sent that payload to OpenFn. The data from our source system will wind up in state.data. Now if my job is meant to take this new patient registration information and use it to create a new record in the national health record system, I'll also need to provide my robot-chef here with a credential so they can access that system. The credential I've specified will get put into state.configuration and now our "raw ingredients" are all ready for our robot chef.

Note that even if this job was initiated by a cron trigger (e.g., "Hey chef, prepare this recipe every Tuesday at 7pm") or by a flow/catch trigger (e.g., "Hey chef, prepare this recipe only when you fail to make banana pancakes") it will have an initial state.

Every job, and every operation inside that job (think "step" in a recipe) is called with state and returns state when it's done.

Initial state for a cron triggered job might look like this:

{
  "configuration": {
    "hostUrl": "https://moh.kenya.gov.ke",
    "apiKey": "abc123"
  },
  "data": {},
  "lastProcessedId": 321
}

And for a fail triggered job like this:

{
  "configuration": {
    "hostUrl": "https://moh.kenya.gov.ke",
    "apiKey": "abc123"
  },
  "data": {},
  "lastProcessedId": 321,
  "error": ["Required field missing", "Patient Surname", "Line 43"]
}

No matter what, jobs start with state. See "Initial and final state for runs" for a detailed breakdown.

It ends with state too​

Now that we've got it in our heads that state is the raw ingredients you hand to your chef when you ask them to prepare a recipe, let's look at the recipe. Boiled down (excuse the pun) a job for loading those patients into the national health record system might look like this:

get('/api/insuranceRegistrations');
post('/api/patients', { ...someData });
post('/api/visits', { ...someData });

We're telling our chef to take those raw ingredients (login info for our national health system and a chunk of information about a newly registered patient) and do the following:

1. Find out whether this person already has a national health insurance number
2. Add this person to the patient registry (making use of some insurance data from step 1)
3. Add a visit record with information about this initial visit (making use of patient registry data from step 2)

When all of this is done, we'll not only have a new patient and visit logged in the national health registry, but we'll also return a final state object with information about what we've done that can be used in subsequent jobs. Imagine that we want to make a cash transfer to this patient so that they can take a cab to the next visit—we might create a job with the Mpesa adaptor that takes the final state of this first job as its initial state. In this way, jobs are composable.

But what about the complexity inside our job—in order to complete step 2, we need some data from the insurance registry and we only get that data in step 1. Crucially, each operation (again, think "step" in a recipe) takes state and returns state. In effect, the OpenFn execution pipeline simply calls all of your action methods with state, passing it along from one operation to the next, waiting for each to finish and using the output from the first as the input for the second.

While you may write your get, post, post job as it's show above, the way it's handled by OpenFn is actually more like:

return get('/api/insurance', { ...useDataFromState })(state)
  .then(state2 => post('/api/patients', { ...useDataFromState2 })(state2))
  .then(state3 => post('/api/visits', { ...useDataFromState3 })(state3));

Each of these operations returns a function which takes state and returns state. This means that within a job, you are essentially modifying state, creating/manipulating records in external systems, and returning state.

It opens up a really interesting world of possibility for data manipulation, cleaning, or transformation. Consider what we might do after we get data from the insurance registry but before we create that patient in the national patient registry:

get('/api/insuranceRegistrations');
fn(state => {
  console.log(state.data); // let's look at the response from the insurance API.
  state.data.people.filter(p => p.HasActiveInsurance); // and modify the payload to only retain those with active insurance
  return state; // before returning state for our create patients operation.
});
post('/api/patients', { ...someData });
post('/api/visits', { ...someData });

We might even need to do some manipulation before we send a get request to the insurance registry. That's no problem:

fn(state => {
  state.data.registrationType = state.data.age > 18 ? 'Adult' : 'Minor';
  return state; // before returning state for our create patients operation.
});
get('/api/insuranceRegistrations', {
  query: { type: dataValue('registrationType') },
});
fn(state => {
  state.data.people.filter(p => p.HasActiveInsurance);
  return state;
});
post('/api/patients', { ...someData });
post('/api/visits', { ...someData });

Here, we've added a step to modify the initial state before we send that first get request to the insurance API. We determine if the new patient is a minor, and then use that newly calculated data to apply a query to the insurance API request.

Using fn(state => state) or alterState(state => state}) is incredibly useful, because it allows us to separate our data manipulation, calculation, and raw Javascript (which will be harder for low-tech users to understand) from our external actions. Let's explore that some more.

Keeping external actions clean​

Inside each operation we could do some data manipulation... all of these operations, across the many different language packages, allow for inline data manipulation like this:

get('/api/insuranceRegistrations', {
  query: state => {
    console.log("I'm doing some fancy stuff here.");
    return { type: state.data.age > 18 ? 'Adult' : 'Minor' };
  },
});
post('/api/patients', {
  body: {
    name: state => {
      return `${state.data.firstName}${state.data.lastName}`;
    },
  },
});

But if you're interacting with both technical and non-technical users, it makes for harder to read jobs. Consider the following instead:

// Perform calculations...
fn(state => {
  // Create several new calculated attributes...
  state.data = {
    ...state.data,
    type: state.data.age > 18 ? 'Adult' : 'Minor',
    fullName: `${state.data.firstName}${state.data.lastName}`,
  };

  return state;
});

// Get insurance data...
get('/api/insuranceRegistrations', { query: { type: dataValue('type') } });

// Create new patient...
post('/api/patients', { body: { name: dataValue('fullName') } });

Since we often have non-developers creating the external operations like get and post above, this pattern makes our handoff easier. The business analyst can say "I need to have a registration type field available for use when querying the insurance registry." A developer might respond, "Great! How do you want to calculate it... I've got all of Javascript at my fingertips." That dev can then make as many API calls as they'd like, perform as many map.reduce(...) calls as their heart desires to complete that calculation... so long as they make sure the hand off state to the business analyst's operation with a valid state.data.type attribute.

A final benefit of this approach is that it becomes much easier to generate job scripts from Google Sheets. Our implementation team frequently works with non-technical clients to generate field maps that look like this:

We can then copy and paste the syntax generated in that final column directly into OpenFn and update the bits that need some sort of custom code, writing an fn(state) block or an alterState(state) block before the external action.

Wrapping up​

Some key takeaways here:

1. Jobs start and end with state — some raw ingredients that will be used in a recipe.

2. Jobs are lists of operations — steps in a recipe that each take state, do some stuff, and then return state.

3. As you move through the steps in a job, you are modifying state. Each subsequent step begins with the final state from the previous step.

4. It may be useful to keep all your custom Javascript data cleaning, manipulation, etc., in a separate operation (e.g., fn(state) or alterState(state)) so that your external actions are clean and easy to follow.

Finally, taking a close look at how developers write those fn(state) steps tells us a lot about what the job execution pipeline is really doing:

// here, "fn" is a function that takes state and returns state
fn(state => {
  console.log("I'm doing some cool stuff.");
  //  I might create some new attribute...
  state.myNewThing = true;

  // And ALWAYS return state for the next operation to use...
  return state;
});

I hope this gives you sense of how I think about structuring jobs and building data pipelines or automation flows on OpenFn. We recognize that this stuff is complex, and are pushing our new documentation regularly, so please do get in touch if you think there are ways we could improve this type of walk-through/helper article.

Happy integrating,

Taylor

So here goes. While this is by no means an exhaustive project planning template, here are a few things to keep in mind if you're planning to implement a CommCare to Salesforce integration on your own.

Most people use "Data Forwarding" in CommCare​

First, most people make use of CommCare's "Data Forwarding" feature to send form submissions and changes in cases (creation, update, closure, etc.) to OpenFn in real-time. You can read about that here but the key consideration at this planning stage is when you'll be performing operations—create(...), update(...), upsert(...), query(...), (bulk(...), etc.—in Salesforce and what data you'll have access to.

Each time a form submission comes into CommCare, we'll get a copy of that submission at OpenFn and can use that data to create or modify some records in Salesforce.

Likewise, each time a case gets updated (or created or closed) we'll get a copy of the case with all the case "properties" and we can use that data to do some stuff in Salesforce.

If you are using "Form Forwarding", the trigger you'd create in OpenFn might look like this {"form":{"@name":"ART Adherence Self-Reporting Tool"}} and it would trigger your job any time an "ART Adherence Self-Reporting Tool" submission arrived from CommCare, giving that job access to all of the data inside that submission.

Working with the data that comes from CommCare​

Assuming you're using making use of case management, the data that arrives from CommCare will look something like this:

{
  "__query_params": {
    "app_id": "some-long-id"
  },
  "app_id": "some-long-id",
  "archived": false,
  "attachments": {
    "1621866020043.jpg": {
      "content_type": "image/jpeg",
      "length": 16423,
      "url": "https://www.commcarehq.org/a/your-project/api/form/attachment/some-uuid/1621866020043.jpg"
    },
    "form.xml": {
      "content_type": "text/xml",
      "length": 2727,
      "url": "https://www.commcarehq.org/a/your-project/api/form/attachment/some-uuid/form.xml"
    }
  },
  "build_id": "0ec83881cd0e420dad5c24ed3a5452fe",
  "domain": "your-project",
  "edited_by_user_id": null,
  "edited_on": null,
  "form": {
    "#type": "data",
    "@name": "ART Adherence Self-Reporting Tool",
    "@uiVersion": "1",
    "@version": "2783",
    "@xmlns": "http://openrosa.org/formdesigner/59E1207B-969F-402D-9EEE-675504036F78",
    "administrative": {
      "coach_verification": "check_here",
      "visit_notes": "",
      "vist_notes_to_save": ""
    },
    "case": {
      "@case_id": "1ec51ee9-5aef-4bd2-b7eb-7599856251bc",
      "@date_modified": "2021-05-24T14:20:28.693000Z",
      "@user_id": "332e893dcd1b413686621bd80aae0cd3",
      "@xmlns": "http://commcarehq.org/case/transaction/v2",
      "update": {
        "consent_received": "yes",
        "home_visit_notes": ""
      }
    },
    "meta": {
      "@xmlns": "http://openrosa.org/jr/xforms",
      "appVersion": "CommCare Android, version \"2.51.2\"(463994). App v2798. CommCare Version 2.51.2. Build 463994, built on: 2021-03-17",
      "app_build_version": 2798,
      "commcare_version": "2.51.2",
      "deviceID": "commcare_a39f55a5-c744-4e33-8e01-d17e7698894f",
      "drift": "0",
      "geo_point": null,
      "instanceID": "130c68c5-7d17-4086-8a85-27d7d7da2216",
      "timeEnd": "2021-05-24T14:20:28.693000Z",
      "timeStart": "2021-05-24T14:18:46.856000Z",
      "userID": "332e893dcd1b413686621bd80aae0cd3",
      "username": "some-chw"
    },
    "participant_information": {
      "participant_id": "007",
      "name": "taylor downs",
      "gender": "male",
      "guardian_information": {
        "guardians_name": "Fake Data",
        "guardians_phone_number": "8675309",
        "guardians_signature": "1621866020043.jpg",
        "relationship_to_participant": "father"
      },
      "current_medications": [
        { "name": "generic-1", "active": true },
        { "name": "fakelyn-notrealiol", "active": false },
        { "name": "sasstra-zenica", "active": false },
        { "name": "ibuprofen", "active": true }
      ]
    },
    "tested_for_hiv_status_tested_for_hiv": "OK",
    "visit_information": {
      "consent_given": "yes",
      "date_consent_given": "2021-05-23",
      "visit_date": "2021-05-23"
    }
  },
  "id": "130c68c5-7d17-4086-8a85-27d7d7da2216",
  "indexed_on": "2021-05-24T14:20:39.045971",
  "initial_processing_complete": true,
  "is_phone_submission": true,
  "metadata": {
    "appVersion": "CommCare Android, version \"2.51.2\"(463994). App v2798. CommCare Version 2.51.2. Build 463994, built on: 2021-03-17",
    "app_build_version": 2798,
    "commcare_version": "2.51.2",
    "deviceID": "commcare_a39f55a5-c744-4e33-8e01-d17e7698894f",
    "drift": "0",
    "geo_point": null,
    "instanceID": "130c68c5-7d17-4086-8a85-27d7d7da2216",
    "location": null,
    "timeEnd": "2021-05-24T14:20:28.693000Z",
    "timeStart": "2021-05-24T14:18:46.856000Z",
    "userID": "332e893dcd1b413686621bd80aae0cd3",
    "username": "some-chw"
  },
  "problem": null,
  "received_on": "2021-05-24T14:20:37.976363Z",
  "resource_uri": "",
  "server_modified_on": "2021-05-24T14:20:38.111789Z",
  "type": "data",
  "uiversion": "1",
  "version": "2783"
}

This is a big blob of JSON—the body of the message that's received at OpenFn when this particular form ("ART Adherence Self-Reporting Tool") is submitted in CommCare—will be handed off to the job to start processing. The question is, what should we do?

When setting up for a self-service implementation on OpenFn, the most important thing you can do at this moment is carefully enumerate the data entry process that you'd like a real human to follow. You can translate it to a job script later.

You'll need to write this up for your own case, but in this fictional example, here's the data entry process.

The instructions for our worker​

1. Every time a messaged is received with {"form":{"@name":"ART Adherence Self-Reporting Tool"}} in the body (this is our trigger)
2. Log into Salesforce and create a new participant with the participant_id you find in the form.participant_information section as their Participant_Code__c. (If one already exists in Salesforce with that code, then update the existing record instead.)
3. Fill out the following fields in Salesforce based on the CommCare data in this message:
   • Name__c with the data from form.participant_information.name
   • Sex__c with the data from form.participant_information.gender
   • CommCare_Case_ID__c with the data from form.case.@case_id
4. After you've created (or updated) this participant in Salesforce, create a record of the visit with the instanceID from the metadata section as the unique identifier Visit_Code__c. (Again, if there's already a visit with that ID please update the existing record.)
5. Fill out the following fields for the visit with data from CommCare"
   • Date__c with form.visit_information.visit_date.
   • Consented__c with form.visit_information.consent_given.
   • Always set Test_Status__c to true, regardless of what's in the message from CommCare.
   • And relate this record with the Community_Health_Worker by their username in form.metadata.username.
6. Finally, add a record for each medication listed in the form.participant_information.current_medications array—matching on a unique ID formed by a combination of the medication name and the participant_id so that we can update existing medication records if they're present.
7. Fill out the following fields for the medication:
   • Generic_Name__c with name
   • Status__c with active
   • And relate this record with the participant you created or updated in step 2 via the participant_id field.

Phew... that's the task. It's just a fictional example and things could be much more straightforward, or much more complicated than this, but it's important to remember that if you can get to this level of precision and granularity in your data entry process, a tool like OpenFn can automate this for you in a flash.

Translating this into an OpenFn project​

If you're streaming data in from CommCare and you've got your Salesforce system all set up so that this data entry person can complete the above steps (are all the objects and fields created? are the right fields marked as "unique" and set to be used as an "external id" in the Salesforce administration section? have you turned on data forwarding in CommCare?) then it's time to turn them into an OpenFn project!

Create a Salesforce credential​

We don't need a CommCare credential, since they'll send data to us. Create a Salesforce credential that will allow the OpenFn worker to log into your Salesforce system.

Read more about credentials here.

Create a message-filter trigger​

• Select Message Filter for the type
• Enter {"form":{"@name":"ART Adherence Self-Reporting Tool"}} for the inclusion criteria

Read more about triggers here.

Create the job​

• Give it a name
• Select the trigger you just created
• Select the salesforce adaptor
• Select the credential you just created

And convert the instructions above to "operations" by using the inline help provided by the Salesforce adaptor:

// Use upsert to create or update a participant based on their participant code.
upsert(
  'Participant__c',
  'Participant_Code__c',
  fields(
    field(
      'Participant_Code__c',
      dataValue('form.participant_information.participant_id')
    ),
    field('Name__c', dataValue('form.participant_information.name')),
    field('Sex__c', dataValue('form.participant_information.gender')),
    field('CommCare_Case_ID__c', dataValue('form.case[@case_id]'))
  )
);

// Then upsert a visit using the visit code.
upsert(
  'Visit__c',
  'Visit_Code__c',
  fields(
    field('Visit_Code__c', dataValue('metadata.instanceID')),
    field('Date__c', dataValue('form.visit_information.visit_date')),
    field('Consented__c', dataValue('form.visit_information.consent_given')),
    // Always set status to true
    field('Test_Status__c', true),
    // And related this visit to the participant we just created by their "code"
    relationship(
      'Participant__r',
      'Participant_Code__c',
      dataValue('form.participant_information.participant_id')
    )
  )
);

// And finally for EACH mediation listed, create a medication record with a status
each(
  merge(
    dataPath('form.participant_information.current_medications[*]'),
    fields(
      field('pID', dataValue('form.participant_information.participant_id'))
    )
  ),
  upsert(
    'Medication_Tx__c',
    'Medication_Tx_ID__c',
    fields(
      field(Medication_Tx_ID__c, state => {
        // Here, inside the medications array we've "scoped" state so that
        // state.data, for each item in the array, looks like this:
        // { pID: 007, name: "sasstra-zenica", active: false }

        // We will concatenate the participant ID with the medication name.
        return state.data.pID + state.data.name;
      }),
      field('Generic_Name__c', dataValue('name')),
      field('Status__c', dataValue('status')),
      relationship('Participant__r', 'Participant_Code__c', dataValue('pID'))
    )
  )
);

Now, every time this job runs (which is every time a CommCare form is submitted) your OpenFn worker will upsert a Participant, upsert a Visit, and upsert a whole list of Medications in Salesforce.

What's next​

Well, in our little example you'd turn the job "on" (setting it to on the inbound messages from CommCare) and let it run. Whenever there was a failure (maybe your Salesforce admin added a new required field on the customMedication object) you'd get an email and you'd have to come back to OpenFn to update your job, including that new field.

If you're in the process of designing your CommCare and Salesforce systems at the moment, this back-and-forth will be pretty common. Keep in mind that you want as much simplicity as possible in those end-user systems because... well because humans have the interact with them every day!

So long as your processes are well defined, OpenFn can handle a bit of complexity (data cleaning, transformation, complex logical flows, etc.) but you should never make sacrifices to the user experience in CommCare and Salesforce—that's a quick way to lose adoption.

So, ideally, you've designed your workflows in CommCare and Salesforce to make your users happy and get them the information they need to do their jobs well and then you come back to OpenFn and spell out our data entry instructions like we've done above.

A final thought​

The two most important resources you've got at your disposal if you're setting this all up on your own are:

1. this site (docs.openfn.org), and
2. the forum (community.openfn.org)

Read through the "What is an integration", "OpenFn Concepts", and "Build" sections if you're a thorough, background-first kind of learner. If you crave snippets and sample job code, head directly to the Job Library to see how other OpenFn users are creating their jobs.

Either way, keep the community posted on your progress in the forum—you'll find lots of helpful folks willing to lend you a hand in your integration journey.

In this post we’ll discuss two different syncing protocols to consider when designing your data integration. These include:

1. Real-time, or event-based, syncs
2. Scheduled syncs

For a recent project in Cambodia, OpenFn is being used by social workers to automate case referrals between the software systems Primero and OSCaR. In the design phase, we evaluated these two syncing options. Below, we'll explain what each one is, the differences between them and which option we chose in the end.

Real Time/Event Based Syncs​

The first option considered for this integration was the real-time/event based sync. This type of sync is triggered whenever a specified event takes place in a system. With this approach, whenever a case is referred in Primero (via the user interface, i.e., when a real case-worker clicks the “refer” button) OpenFn receives a small payload with case data and transmits it to OSCaR and vice versa.

Because of their instantaneous nature, real time/event based syncs are great for integrations that involve mobile payments or sms messages to recipients. Really, anything that needs to be done “now”! Additionally, depending on your data volumes real time syncs might save you money because you’re only using resources when specific events take place. For instance, in the above example, a run is triggered by a referral, so if there are only 10 case referrals/month, you'd only process 10 runs each month.

This type of sync is great because it’s instantaneous, typically quite straightforward to set up, doesn’t require any “state mangagement” on OpenFn, and allows for the reprocessing of individual events. There are, however, drawbacks.

For instance, what happens if the app that’s sending notifications to OpenFn fails to send? What if AWS or GCP goes down, taking half of the internet with it? If Primero “thinks” it sent the referral, OpenFn never receives it, that case might not get referred to Oscar!

Scheduled Syncs​

The second option considered, a bi-directional schedule dependent sync, solves for the issue discussed above. On a scheduled basis (every 5 minutes, for example) OpenFn checks with Primero and Oscar to see if case referrals need to be transmitted between the two systems and then refers the case if required. In the unlikely event that any of the software systems involved crash, the stability provided by the bi-directional sync means that all data is preserved and eventually makes it to its destination safely.

The major drawback here is complexity. We had to use 4 jobs instead of 2, and the job that is responsible for “pulling” data that’s been updated since the time of the last successful sync has to keep “state”—or some sort of working memory of what it’s done in the past. When pulling modified cases from Primero, OpenFn now only pulls cases modified on or after YYYY-MM-DD HH:MM:SS where YYYY-MM-DD HH:MM:SS is the time of the last successful, round-trip synchronization. OpenFn has built-in functionality to handle exactly this requirement, but not all ETL systems do and it’s a design implication that must be considered.

Ultimately, for the project in Cambodia, we decided that this sync option is the right choice because data integrity is more important than the speed of this data flow. That’s a crucial point to understand—the organizations operating in Cambodia decided that for this particular use case, being able to guarantee eventual syncing was more important than having real-time syncing.

Both Sync Options Have Their Pros and Cons​

Both options definitely have their use-cases and OpenFn's platform versatility enables your team to decide which type of sync is right for your project.

As always, we are here to help with any questions as you think through which sync option makes the most sense for your project.

Essentially, Zandile needs to decide if she would like to run the software on someone else’s servers (SaaS), or on her organization’s own servers (deployed locally). Before making a decision she outlines the basic, non-technical considerations for both options.

What is SaaS?​

SaaS is software that is installed and runs on computers maintained by software professionals, rather than on your own computer. While those computers might be anywhere in the world, typically you'll access and use this software via the Internet.

Some benefits of SaaS​

With SaaS, the software vendor is responsible for the expenses of managing and monitoring all of the technical components and issues associated with the software. This means that Zandile’s iNGO will not be responsible for updating the software to ensure compliance with new security regulations, maintaining the servers, backing up the data, purchasing and managing uninterrupted power supplies, and providing a team of physical security guards to protect the computers and data therein against physical theft.

Going the SaaS route is often faster and more secure, because you do not need to develop expertise in "DevOps" or hire IT and physical security specialists. This option also provides the greatest amount of flexibility & scalability– because the SaaS provider is able to deliver more or less computing power, storage, and bandwidth—right when it’s needed.

Having smaller setup costs (you don't have to grow a software delivery company of your own) often makes this a more economical choice for many, though SaaS will always come with some sort of ongoing fee—a price per month or year that goes to the vendor to compensate for the time and money they'll spend to ensure your software works properly.

What is Local Deployment?​

Unlike the SaaS option, local deployment means installing and running software on your own computers—typically on your organization’s servers.

Some benefits of Local Deployment​

If a SaaS provider doesn't offer hosting in your country and your government doesn't allow your data to reside on foreign servers (i.e., you're not allowed to use things like Gmail, WhatsApp or Facebook for communicating sensitive information) then local deployment allows you to use tools like CommCare, DHIS2, and OpenFn while adhering to government data sovereignty regulations.

Local deployment also provides your organization with complete ownership of the end-to-end system. Your IT team will be personally responsible for ensuring that the software works, is maintained, is secure, etc. If your organization does not already have an IT team in place, then this can become a costly headache, but for a large organization with embedded IT experience, local deployment often makes sense.

Ultimately, being able to directly hire and fire the people who are responsible for your software's proper functioning can be very useful. It means you have complete responsibility for whether or not the solution succeeds.

If you've already got the teams in place (security, DevOps, etc.) then this option can be more economical in the long run. With a very good DevOps team, maintaining an extra piece of software might only occupy 20% of a full-time-employee's salary. For your security guards, if the software is installed in the same physical location it's possible that your costs won't increase at all. While there will be very high setup costs, over time you may realize cost savings by running an efficient software delivery unit within your organization that spreads its focus around a number of projects.

Zandile's Decision​

In this fictional case, data residency is a concern—her data is sensitive or contains PII—and CommCare, DHIS2 and OpenFn do not provide hosting in the country she's located. Zandile's organization has a large, experienced IT team that has managed high-availability software projects for many years... they're pros. While they anticipate that the setup costs will be quite high (around $60000 and several months for this set of deployments) they plan on using this software for the next 5 years and have determined that they'll recoup a significant portion of that setup cost by not having to pay license fees for SaaS. They go with local deployment.

Which Deployment Option is Best for your organization?​

The answer is: "it depends", but if your organization has never managed local software deployments, then we recommend going the SaaS approach. SaaS systems, like the one OpenFn and CommCare offer, are simply going to be more secure, more stable, and more scalable for the money.

Crucially, you can always start with SaaS (most tools even offer a free tier) and then decide later to invest in the big startup costs of a local deployment if the license fees for the SaaS feel high enough to make local deployment more economical over the long term. After a few months or years on the SaaS, you'll likely be in a better position to know if you want to continue using the software for 5-10 years.

Should you need any help with your decision though please do not hesitate to contact OpenFn.

A bit more...​

An “UPSERT” is a portmanteau of the database functions UPDATE and INSERT. It’s critical to handle upserts properly when integrating systems. As of version 35 of the API, DHIS2 does not allow for an administrator to upsert tracked entity instances (“TEIs”). OpenFn’s own Chaiwa Berian has come up with a solution that highlights the utility of helper functions in our dhis2 adaptor. By combining various DHIS2 APIs through an upsertTEI function in OpenFn, DHIS2 users can now perform upserts to TEIs.

If you’re curious, check out his implementation here.

Even more!​

A tracked entity instance in DHIS2 is a type of entity that can be tracked through the system. It can be anything from a person to a commodity like a medicine. If I am a database administrator presiding over two different systems that are connected to one another, let’s call them “System A” and “System B,” I would like for any updates made to the TEI of a user named “Jim Smith” in System A to also appear in Jim’s record in System B. Before upserts came about, doing so was difficult because of the possibility of duplicate record creation. Because an upsert simultaneously UPDATES and INSERTS, it prevents duplicates.

Upserts are important and good because they cut down on the risk of duplicate data entry and they also allow for transactions to be retried over and over to ensure data integrity. That last bit is called “idempotency” and you can read about it over here.

Please don’t hesitate to reach out to one of OpenFn’s implementation specialists if you’d like to learn more.

— Taylor

Sign up to set up a project today, absolutely free.

Reach out for more information.

In the past, if you weren't using OpenFn to drive some real-time (or "event-based") automation, you'd need to set up an "interval trigger." Like the photo above, this was essentially a sand timer. Set your trigger to 10 seconds and your job fetches data from DHIS2, some regional public health data set, or whatever, then cleans, transforms, and loads it into some other system.

For the most part, this has got the job done for the last 5 years, but as our NGO and government clients came up with increasingly specific requirements on not only how often but when a crucial job gets executed, we began finding ourselves creating little customizations for them on a once-off basis. We're happy to annouce that as of v1.75 (released today), you can now schedule jobs to run based on cron expressions, giving you incredible control over when your tasks get executed.

Scheduling is better than timing.​

Using cron, you can choose to run a job every minute by typing * * * * *.

Or maybe you've got a batch sync that you want to take place while your users are asleep—why not run it every night at 11pm with 23 * * * *.

What if you've got to submit reuqests for medical inventory only during the onset of flu season? Simply type 0 0 1 2-4 * and your job will run at midnight the 1st of the month, from February through April.

You can still run jobs at the click of a button and create timers with expressions like */10 * * * * for "every 10 minutes", but scheduling with cron gives OpenFn.org users so much more control over how they run their organizations. (And that's a good thing.)

If you're keen on learning by doing but don't have an OpenFn account yet, sign up for free or mess around with cron expressions at crontab.guru, a brilliant site to quickly build complex cron expressions.

That's all from product for today. Speak soon.

Taylor

We just extended our language-mssql adaptor with a custom function that allows upserts (an upsert is when you either insert a new record or update an existing record based on some identifier). Before, you'd need to write something tedious like:

sql({
  query: `MERGE my_table AS [Target]
          USING (SELECT '8675309' AS some_unique_id, 'writing_blog_posts' AS skill) AS [Source]
          ON [Target].some_unique_id = [Source].some_unique_id
          WHEN MATCHED THEN
            UPDATE SET [Target].some_unique_id=8675309, [Target].skill='writing_blog_posts'
          WHEN NOT MATCHED THEN
            INSERT (some_unique_id, skill) VALUES ([Source].some_unique_id, [Source].skill);`,
});

whereas now you can simply write:

upsert('my_table', 'some_unique_id', {
  some_unique_id: 8675309,
  skill: 'writing blog posts',
});

For an operation to be idempotent means that it can be repeated time and time again without producing an unintended result. This is SUPER important for creating S3 (Secure, Stable and Scalable—more on that here) integrations because it provides you with two "get-out-of-jail-free" cards.

1. If a destination application fails, if a connection times out, or if (for whatever reason) you're not sure if the job was completed (say... making a payment to CHW) then an idempotent operation can be RETRIED without fear of making a double-payment.

2. If you make some change to how your job works, make some modification to one of your destination systems, or just because you want to be extra extra sure that all the data in a 9 month survey made it to the national public health reporting system, you can REPROCESS every single message that's come through OpenFn at the click of a button, without having to worry about duplicates.

So... when clients let me mess around with their jobs, I always recommend we design for idempotence. It's common sense when you're passing messages between two different systems that are bound to evolve, go offline, have a bad day, etc.

— Taylor

Sign up to set up a project today, absolutely free.

Reach out for more information.

Our partners use OpenFn automation solutions to drive efficiency and scale their processes, delivering integrated digital systems that work better, faster, and together.​

To date, we have worked with 43 social sector organizations that operate across sectors–from health, education, and agriculture, to livelihoods and emergency response. Over the last 6 years, OpenFn has been implemented worldwide for a wide range of use cases, including building real-time data monitoring systems, streamlining data cleaning pipelines, securely exchanging sensitive information, and automating routine processes like uploading indicator results, sending SMS/email alerts, making mobile payments, and more.

By connecting any app, OpenFn can integrate and automate all apps within a digital ecosystem. However, a question that we frequently ask our partners is:

Just because you can automate—should you?​

While integration and automation have the potential to enable scale and save time and money (we’ve learned this from our partners), solutions require investment to set up and maintain. These costs sometimes outweigh expected efficiency gains and service outcomes. Therefore, when evaluating the cost-benefit of investing in automation and integration solutions, we at Open Function Group typically ask 3 key questions.

1. Security — Will automation limit the exposure of sensitive data?​

Can the exposure of sensitive data be limited by integrating with secure API endpoints (rather than relying on human beings to interact with those data, for example)? Or by automating a data cleaning process?

2. Accuracy — Will automation increase data accuracy and reduce data entry errors?​

Can the process take place more reliably by limiting the opportunity for human error (in automating data manipulation or simple algorithmic work, for example)?

3. Speed — Will automation increase the speed of impact?​

Can the process be done more quickly via automation and is there value in having it done faster? (The answer to the first part is almost always yes, but sometimes there's not actually lots of value generated by doing something faster.)

If you find yourself answering “yes” to these questions, it may be time to consider automating critical processes at your organization.​

Tasks that meet these 3 criteria, take a lot of time to complete or are very repetitive, and/or involve moving data between apps are typically great candidates for automation.

If you find yourself answering “no”, then it may not be worth the investment in automation... at least not yet. This is especially true if these processes are still in flux or require a lot of human involvement to complete. That said, now may be a good time to refine your existing workflows, think about how your processes might change at scale, and consider what new processes, services, or outcomes could be unlocked by automation.

Delegate your busywork to OpenFn, and try it today!​

If you want to try out automation for your organization, sign up for OpenFn, free of charge. Check out our documentation and website to learn how to get started.

Having trouble setting up your first automation "job"? Email us at admin@openfn.org for support. Our team is always happy to assist and help you evaluate the total cost of ownership of automation solutions.

This article was originally posted by Taylor Downs, Head of Product, on The OpenFn Founder's blog as "The power of crow's feet."

It’s Saturday Morning in Cape Town and I’ve just spent an hour talking about how a non-profit is organized. I thought I was getting into a technical discussion—I’ve been doing system architecture discussions for years—but what we ended up talking about was how this NGO thinks.

This engagement is largely about mapping an already existing “people & paper” based system to technology. Vera Solutions will build a system for this client using Open Data Kit for field data collection and Salesforce.com for the management “back-end”. Because we’re not explicitly being asked to help redesign processes at this organization, the client is “telling us how things are”, then expecting us to create a relational database model that facilitates business-as-usual, only in a more efficient, digital way. Seems reasonable.

This organization runs multiple programs focusing on a handful of strategic objectives. They coordinate various activities in their target communities and report on those activities against numerous (sometimes overlapping) indicators. Sound familiar? As we get to the 3rd explanation of these programs, and the 11th iteration of the system schema, it hits me…

Drawing a "relational object model" sounds technical, but it's actually an exercise in clear communication... and everyone can benefit from it.​

When we—human beings—wrestle with complex problems (like managing lots of programs, other humans, community stakeholders, etc.) we have the capacity to trick ourselves into thinking that we have wrapped our heads around a system (for clinic registration, for after-school education, etc.) when, in fact, we’re engaging in mental hand-waving and are simply ‘papering-over’ sections which are secretly not just complicated, but totally incongruous with other parts of the system. We can make ourselves believe that our logic is sound because we want it to be sound, when in reality the organization might be held together by good people, not good, clear, defined processes. By learning a couple of key concepts, it’s possible for non-technical people to articulate their thoughts clearly using “boxes” and “crow’s feet” and see whether or not there is an underlying structure to their organization’s information.

By forcing yourself to reduce complex systems to sketches containing only two elements, you’ll be able to detect important conflicts and confusions in how you think about your organization that you might otherwise miss.​

If you can’t diagram the information structure in your organization using boxes and crow’s feet, it’s a smell that something isn’t quite right (or at least that something isn’t easily scalable… more on this later!). Let me show you the tools in the toolbox and then wrap up by waxing poetic on people, processes, and technology.

Boxes and crow's feet​

The box is my favorite. It represents an entity in your data system. Entities (like teachers) have attributes (like name, phone number, date of birth, gender, etc.) Some people like thinking of entities as simple forms. The “Teacher Registration Form” will ask for the teacher’s name, phone number, gender, etc. These are the fields on your teacher entity. By submitting one of these forms, you’ll add a new teacher to your database. If you’re an Excel person, the attributes are columns in your teachers table.

The crow’s foot is my second favorite. It’s used to show relationships between entities. We know that teachers are related to the sessions that they conduct. (And session might be another entity, with fields like date, subject taught, and venue, to name just a few.) The crow’s foot allows us to specify exactly how they are related. On that session entity, we’ll need to specify the name (or ID) of the teacher who led it. On the teacher entity, however, there won’t be a field to specify the name or ID of the session… because a single teacher can lead MANY sessions. This is a one-to-many relationship. The crow’s foot (that little three-pronged fella) denotes the many. One teacher can have many sessions. One session, however, can only have one teacher. See the diagram below.

If we focus just on teacher and session and think back to MS Excel, we can envision a teachers table and a sessions table. Let’s put them on different sheets in the same workbook. On the teachers table, there is no column for session, but on the sessions table, there is a column for teacher ID. We’ve just established a one-to-many relationship.

Next time, we’ll talk about what’s going with the attendance entity above. It’s sometimes called a “junction object” or a “join table”, and it’s what allows MANY students to be related to MANY sessions. I’ll write more on this next time, but there is no magic going on, no technicalities here. The way that many students are related to many sessions is through this Real World Concept that we call attendance. Attendance is what happens when a student shows up at a session. It’s so important to get the language right in these discussions, and make sure that you’re talking about real-world concepts.

Relational object models with lots of confusing terms are not "technical". They are "bad."​

Remember that as you start to put pen to paper. And allow yourselves time (and multiple drafts) to get the boxes and terminology right. Understanding relational object modelling is an incredibly powerful way to organize a company. As I said before, if you can’t model it with boxes and arrows, it’s a smell that something might not be conceptually sound.

A disclaimer and some thoughts on scaling:​

Some organizations do amazing work without good conceptual systems. They rely on humans, instinct, improvisation, nouse, and other not-so-clearly-defined things. They might do really great work. They might get the job done. But they need to be well aware of their condition and face it head on. If you can’t systematize your program implementation processes, then you need to focus tremendous effort on finding and retaining the right people.

A friend once told me that “people are not scalable.” I couldn’t agree more, and defend my earlier stance that if your organization’s information structure can’t be defined with boxes and crow’s feet, it may be very hard for you to scale responsibly. However, if you can create a ruthlessly efficient, world-class “people operations” system (recruitment, training, management, compensation, HR, etc.) that ensures you’ve always got the right people to figure things out you might be better off than those operating a well defined assembly-line with interchangeable parts. Alas, the middle way is probably the best.

That’s all for now. More soon.

Need help organizing or scaling your organization's information or process flows? Contact our team of ICT4D specialists at support@openfn.org.