![](\"https://user-images.githubusercontent.com/105651463/172341045-62a3eebd-98fb-4189-987f-06bbdb03ac30.png\")
In order to ensure that Global Goods and Digital Public Goods can enable impact\nwithout doing any harm, it is crucial that their authentication and\nauthorization systems are secure from the outset.
\nWe’ve put together a resource to help other Digital Public Goods and Global\nGoods achieve just this. You can view and comment on the resource\nhere,\nor read on to learn why.
\nDigital health technologies have a critical role to play in both improving the\ndelivery of health services and making health systems more robust and\nadaptable. Whilst this has been the consensus for a while, various factors such\nas bureaucracy, lack of capacity, authority/ownership, budget or simply lack of\nprioritization have hindered the adoption and implementation of such\ntechnologies.
\nHowever, the COVID-19 pandemic has brought digital health systems back to the\ntop of the agenda, having forced governments to adapt their health systems and\nprocesses in a short amount of time and confronted them with the fragility and\ninflexibility of some of their existing systems. This has led to an increase in\nfunding for both new and existing health technologies.
\nAt the same time, the recent pandemic also saw an increase in data breaches and\ndata leaks in the healthcare sector. Whilst these are a serious matter in any\nsystem, the nature of the healthcare technologies means that the data they are\nstoring or managing almost always includes Personally Identifiable Information\n(PII) which, if leaked, can have dramatic consequences for those they identify.
\nThe higher the number of technologies and quantity of data stored in these, the\nhigher the risk of vulnerabilities and negative impact of a data breach or data\nleak. If Global Goods are to make the most of this renewed interest and increase\nin funding for digital health technologies, it is crucial for them to keep\nsecurity as a top priority to ensure that they ‘do no harm’, a\ncore Digital Public Good principle.
\nSeveral organizations have already put some thought into documenting security\nbest practices and standards. However, none situate these security decisions\nwithin the wider context of a product development timeline or roadmap. This\nrequires individuals to regularly scan through long lists of security\nrecommendations to pick out and prioritize their implementation, increasing the\nlikelihood of an oversight or early-made product decisions which make it harder\nto implement such security features down the line.
\nAs product manager at Open Function Group, I myself have recently been reviewing\nand analyzing numerous resources to identify and prioritize features required to\nget to fully-secure authentication and authorization for our newest product\nLightning–and make sure that nothing\nfalls through the cracks. I’ve also had the opportunity to gather insights from\nDigital Public Good community forums (OpenMRS, DHIS2, OpenLMIS, etc) and speak\nto other product managers and engineers from the Digital Public Goods community\nto ask them about their learnings, and any tips they might have: Austin and\nMorten from DHIS2, Dev and Biyeun from Dimagi.
\nTo more widely share these learnings, we have drafted this resource\ndocumenting the specific security measures that should be considered at each\nstage of product development _specifically with regards to authentication and\nauthorization. _Our hope is that this article will help other product managers\nin the Global Good (GG) and Digital Public Good (DPG) communities consider these\nfeatures into their backlog/roadmap from the outset and make sure they are\nfollowing a ‘secure by design’ approach. To inform our analysis, we analyzed\nlearnings from the wider DPG community, as well as recommendations from other\nsecurity experts and international standards, including GovStack, OpenHIE,\nOWASP, NIST, and more.
\nThe resource can be found\nhere.
", "url": "https://docs.openfn.org/es/articles/2022/09/19/auth-security", "title": "Secure by design: a roadmap to secure authentication and authorization", "summary": "In order to ensure that Global Goods and Digital Public Goods can enable impact", "date_modified": "2022-09-19T00:00:00.000Z", "author": { "name": "Amber Rignell" }, "tags": [ "security", "authentication", "authorization" ] }, { "id": "https://docs.openfn.org/es/articles/2022/06/07/workflow-automation", "content_html": "Do you think twice when you get an automatic SMS notification because your\nprescription is ready? (Neither do I.) This seamless experience is driven by\n“workflow automation”, a key feature that OpenFn provides. The OpenFn\nIntegration Toolkit is a Digital Public Good (DPG) used by governments and NGOs\nto boost efficiency through workflow automation. The automation that OpenFn\nprovides includes automatically sending SMSs, automating stock updates across\nsupply chain systems, tracking clinical visits, and helping plan vaccine\nrollouts. We support our partners’ work by lifting the burden of manual data\ntransfers between platforms.
\nThis article\noriginally appeared\nas a LinkedIn article from Open Function Group.
Workflow automation is an approach to making the flow of tasks across platforms\nrun independently. A prescription SMS notification feels seamless, but it\nprobably requires complex automation across multiple systems like clinical\nregistries, pharmacy stock databases, and SMS gateways. Tools like OpenFn run\n“behind the scenes” to perform calculations and transfer relevant data to/from\ndifferent systems autonomously, minimising the risk of human error and\nultimately saving time and money that an agency would spend on manual data\ntransfers. The goal is to preserve data integrity, uphold data security, and\nenforce compliance with policies and data standards.
\nSince 2014, OpenFn has been at the forefront of workflow automation in the\nsocial sector, providing a platform to help governments and NGOs focus on the\nhuman aspect of their work rather than wasting time trying to exchange data\nbetween systems or perform calculations manually.
\nMyAgro, an NGO helping West African farmers increase\ntheir yield, uses OpenFn to power its SMS-based savings deposit system for\nsmall-scale farmers. MyAgro distributes and tracks saving vouchers using the\nCommCare app. Smallholder farmers purchase\npre-paid myAgro saving scratch cards and text the one-time code to secure the\nvoucher credit in their account. Once the amount has been added to their\nbalance, a confirmation notification is sent to the farmer. MyAgro works with\nthe farmers to reach their savings goal, which they then use to pay for seeds,\nfertilisers, and training courses. OpenFn:
\n
OpenFn helps myAgro reach more farmers with less administrative overhead by\nautomating these key steps. MyAgro calculated that even at their 2014 levels of\nscale, OpenFn saved them 260+ hours per year and increased customer confidence\nas they worked to serve poor farmers.
\nSinapis, an NGO that empowers entrepreneurs through\ntraining courses, uses OpenFn to automate its payment collection process. Every\ntime a client’s new\nM-Pesa\npayment is registered in Kopo Kopo, OpenFn:
\n![](\"https://user-images.githubusercontent.com/105651463/172341175-5505b647-93dc-44af-9143-00ebdd0b5bae.png\")
With OpenFn handling the routine data processing, Sinapis has more time to\nsupport its entrepreneurs. Precious staff time can be spent delivering training,\nthe essential face-to-face coaching that transforms their clients’ fledgling\nenterprises. Their team does not have to worry about the accuracy of payments\nand account balances; the OpenFn implementation replaces error-prone manual data\nentry with a systematised, intelligible, fully automated process.
\nThe Digital Impact Alliance (DIAL), an\norganisation that works to overcome digital development barriers, wanted to\nprototype a “building-blocks based approach” for their upcoming GovStack.global\nproject. They configured an HR workflow solution that connects mobile data\ncollection apps, databases, and payment apps using OpenFn. Every time an\norganisation registers new workers via\nODK Collect, OpenFn:
\n![](\"https://user-images.githubusercontent.com/105651463/172341214-06a1e74b-c1e3-45e4-83c8-4a60af5a9d2d.png\")
While it’s not in use at scale, this prototype is a powerful illustration of how\nautomation enables scalable, component-based solutions to become more than the\nsum of their parts. The processes that DIAL implement with OpenFn help unlock\nthe true potential of these powerful underlying DPGs.
\nOpenFn’s goal is to increase the efficiency and effectiveness of the social\nsector. Why should health and humanitarian organisations waste precious\nresources manually moving data across systems when workflow automation can do it\nmore safely and quickly? We want to ensure employees focus on using data to do\ntheir work, not transferring data between systems and running rote calculations.
\nSaving organisations time and money is part of the picture, but we also deliver\nbetter program outcomes through efficiency and effectiveness. Suppose OpenFn\nsecurely automates the complex digital processes critical health and\nhumanitarian interventions require. Reducing error rates and security\nvulnerabilities and getting higher quality data into the hands of a doctor\nfaster actually saves lives. Responsible automation leads to better health and\nhumanitarian outcomes, enabling larger-scale intervention delivery via\nefficiency gains and improving service quality by getting accurate data into the\nhands of patients, clinicians, and policymakers when they need it.
\nThink about your own experiences; have you found yourself in a situation where\nmanual data processing—some rote, mindless task—has gotten in the way of doing\nthe more exciting parts of your job? Whether you are in the health and\nhumanitarian space or just want to make your life easier, OpenFn is here for\nyou. Our open-source toolkit is available for everyone to automate tasks such as\nsending SMS and email notifications or automating data cleaning and formatting\nfor faster reporting.
\nHead to docs.openfn.org to learn more about\nenterprise-grade automations with our DPG and check out\ncommunity.openfn.org to connect and collaborate\nwith like-minded social-sector integrators and OpenFn staff. We hope to see you\nsoon!
\nOpen Function Group (OFG) is a global team of integration specialists and\ndevelopers of the OpenFn Integration Toolkit. As a\nDPG, OpenFn’s core technology is free and\nopen-source. If you want more information about the Toolkit or to sign up for a\nfree plan on our associated SaaS application, visit https://www.openfn.org/.
", "url": "https://docs.openfn.org/es/articles/2022/06/07/workflow-automation", "title": "Workflow Automation; Why do it yourself when a program can do it for you?", "summary": "Do you think twice when you get an automatic SMS notification because your", "date_modified": "2022-06-07T00:00:00.000Z", "author": { "name": "Alexa de Vegvar" }, "tags": [ "automation", "solution" ] }, { "id": "https://docs.openfn.org/es/articles/2021/10/29/how-learning-javascript-helped-me-better-understand-jobs", "content_html": "OpenFn automation happens via jobs which\ndefine specific steps (\"operations\") that OpenFn should perform. They're written\nin a basic scripting language that runs on top of (and has full access to)\nJavascript. A basic understanding of Javascript will take your job writing\non OpenFn to the next level. To improve my limited knowledge of JavaScript, I\nhave been taking Codecademy's\nIntroduction to JavaScript Course.
\nIn this post we'll discuss:
\nKeep reading for all the important learnings I have taken from just the first\nfour modules!
\n=> arrow?Javascript allows us to create functions which take arguments and do stuff\nwith them. Most times, function declaration in Javascript looks like this:
\nfunction getAge(dateString) {
if (!dateString) return;
const today = new Date();
const birthDate = new Date(dateString);
var age = today.getFullYear() - birthDate.getFullYear();
var m = today.getMonth() - birthDate.getMonth();
if (m < 0 || (m === 0 && today.getDate() < birthDate.getDate())) {
age--;
}
return age;
}
This was simple enough for me to follow. We are declaring a function,\ngetAge(), which has the steps to calculate a person's age and can be easily\nreused throughout the job.
But other times, function writing looks like this:
\nconst getAge(dateString) => {
if (!dateString) return;
// and so on...
}
In OpenFn, you're often writing unnamed functions that take state as their\nonly argument and you use them right away. See how concise writing and using a\nfunction can be with the arrow:
upsert('tbl_study', 'study_id', {
study_id: state => state.studyIDMap[state.formType],
});
These arrow functions confused me when reading through jobs. I've learned\nthat, for the most part, they can be thought of as short-hand notation which\nremoves the need to type out function for every function declaration.
MDN has a great explanation\nhere.\nTl;dr: \"An arrow function expression is a compact alternative to a traditional\nfunction expression, but is limited and can't be used in all situations.\"
Further, we can make the code even more concise by also removing the\nparenthesis, curly braces, and the return keyword when certain criteria are met.\nIn the code above, the parenthesis are omitted because there are no parameters,\nand the curly braces and return keyword are omitted because the function body is\ncomposed of a single-line block. As you can see, there are a variety of ways to\nwrite functions and this course is helping me better recognize them. Some key\nterminology here is: function expression, arrow function,\nconcise function, anonymous function, and implicit returns.
During the mapping\nphase of integration design, we often discuss how each answer choice for\npicklist values should map from the source system to the destination system.\nSometimes the mapping is simple but other times, there is an extensive list of\npossible choices that can be found in the\nmessage and not all\nare relevant to the destination system. Then the question is, how should the\njob handle values which are not explicitly mapped?
\nSometimes we hear clients say to \"ignore\" those values. But what does it\nreally mean to \"ignore\" a value? Should we set it to 0? An empty string? How\nabout null, undefined or NaN? In Javascript these are all falsy values.
MDN is great for falsy values too! Check their explanation out\nhere.
Determining what is really meant here is important and the value selected has\ndifferent implications depending on the system.
\nTake this sample mapping for different districts in Conakry. The value left of\nthe colon is from the source system and the value to the right is for the\ndestination system.
\nconst districtMapping = {
Ratoma: 'RT',
Kaloum: 'KL',
Dixinn: 'DX',
Matam: 'MA',
Matoto: 'MT',
Other: undefined,
};
If the destination system is Salesforce, this mapping would not upload \"Other\"\nto Salesforce. However if \"Other\" mapped to an empty string instead, this would\nupload the empty string to Salesforce. This distinction is especially\nimportant in cases where we are overwriting existing data. For instance, if a\nstudent previously lived in Ratoma and then moved to an unknown district marked\nas \"Other\", undefined would not update the student's district in Salesforce\nbut the empty string would. Both of these are falsy values but have different\nmeanings in Salesforce.
How about if the message includes a value for a district that is not in the\nmapping? Such as \"New York\". Should the job default to undefined? Or null?
These questions are just a few examples of how understanding falsy values in\nJavascript can make it easier to implement the best mapping for the real-world\nuse case.
Short-circuit evaluation
\nWhen we are syncing forms to a database, we sometimes expect different versions\nof the same form with fields present one version but not the other. One way to\nmitigate this discrepancy is by submitting a dummy value for the field whenever\nit is missing in the message. I just learned the fastest way to add this code to\nmy jobs! For example, the code below will ensure that household_id always has\na value in the destination system: if survey_info/household_id is present in\nthe message this will be the assigned value, otherwise it will assign the dummy\nvalue state.data.body._id.
household_id: state.data.body['survey_info/household_id'] || state.data.body._id,
Template literals
\nLike the arrow discussed above, the \"dollar sign\" was another symbol that often\nconfused me. But template literals are actually very straightforward. They\nincrease the readability of the code and make it easier to see what the\nresulting string will be. Wrap your template in the backtick (`) symbols and\nthen each variable in ${}.
const sign = '$';
console.log(`The ${sign} isn't so confusing!`);
The most challenging module in the course covered higher-order functions.\nThese are defined as functions that accept other functions as arguments and/or\nreturn functions as output. But why are these important and how are they used\nin OpenFn jobs? It turns out we use them quite alot! The code below is an\nexample from an existing integration with the field names replaced.
\nconst participantsToUpdate = state.data.json.filter(data =>
state.idList.includes(data.id)
);
const participantsToCreate = state.data.json.filter(
data => !state.idList.includes(data.id)
);
This code is using a built-in JavaScript method that helps us iterate on\narrays to manipulate elements and return values. The .filter() method is\nbeing used to return a new array after filtering out certain elements from the\noriginal array. We have declared two variables to store the new arrays:\nparticipantsToUpdate and participantsToCreate. Participants whose ids are\nexisting in the destination system (or who are in the idList array) are added\nto participantsToUpdate, and any remaining participants are added to\nparticipantsToCreate. This filtering helped us perform the correct operations\non each participant type. For instance, later in the job, we use the filtered\narrays to only overwrite a participant's reportNumber field if it is a new\nparticipant. .filter() is just one of many higher-order functions that power\nOpenFn jobs.
The final lesson was on objects and key-value pairs--something used in almost\nevery OpenFn job! The result of the design and mapping phase of requirements\ngathering is almost always a mapping document which includes key-value pairs in\nplain English. These specifications are then translated to JavaScript via\nobjects. The code below is a snippet of an object which captures the\nkey-value pairs for states in the US.
\nconst stateMapping = {
AK: 'ALASKA',
AZ: 'ARIZONA',
AR: 'ARKANSAS',
CA: 'CALIFORNIA',
....
The value left of the colon is how the state is represented in the source\nsystem, and the value right of the colon represents how OpenFn will send the\ndata to the destination system. This mapping process is key to integration\ndesign. Learn more about mappings\nhere.
\nMDN has more details on objects\nhere.\nTl;dr: \"The Object class represents one of JavaScript's data types. It is used\nto store various keyed collections and more complex entities.\"
I'm well on my way to becoming a better job reader and writer. Here are some\nnext steps:
\nUnderstand fn(state) and how state can be manipulated in OpenFn jobs.
Explore what's available on the JavaScript docs\nsite.
\nSign up for the next level JavaScript course.
\nHave you ever struggled to layout the strategy for testing your React App? Well,\nyou are not alone! Here a few hints from the lessons I learned during my\nexperience testing a\nReact/Redux app with a\nPhoenix/Elixir\nbackend.
\nBecause a React app is built on\ncomponents, the basic UI units,\nit is natural to think and organise your tests around components! And so unit\ntesting, in this case, would refer to \"component testing\", which may be\nconfusing at times, especially when the concept of unit testing is again applied\nto testing functions such as Redux reducers and action creators or any other\nJavaScript function in your application.
The other challenge that I often faced was whether to write tests for each\ncomponent in isolation or write a test for a feature that encapsulates a set of\nrelated components. The later would be equivalent to writing what I would call\n\"integration tests\".
\nFinally, one would say \"well then you could have just written the tests in a way\nthat resemble the way the application is used\"! This approach is commonly\nrecommended in the React community, but it quickly becomes really complex to\nmaintain the layers of separation between unit tests, integration\ntests and end-to-end tests.
\nGiven a React/Redux application, here is how I would organise my testing\nstrategy:
\nIn a React app, unit tests will largely apply to testing \"helper\nfunctions\" and not to testing components, as justified in the next section.\nHelper functions, in this case, would refer to functions that live outside the\ncomponents and are neither Redux action creators nor reducers. These functions\ncan be used inside components, action creators, reducers or other parts of\nyour application.
\nWriting unit tests for \"helper functions\" would ensure their signatures and\nexpected outputs are protected against regressions. This would also ensure\ntheir use across components or other functions is consistent and as expected.
\nWhere possible, each \"helper function\" must have its own unit test.
An example of a unit test would like:
\nconst sum = require('../../js/sum');
test('adds 1 + 2 to equal 3', () => {
expect(sum(1, 2)).toBe(3);
});
Write a thousand of these.
\nIn the context of a React/Redux app, component tests can be thought of as\nintegration tests. This is because React components are built around\nfeatures such as <Signup />, <Search />, etc. So one React component can\nbe a mix of other components to achieve a UI feature set.
To test a component, write an integration test that covers the use of a\ngiven component for a given UI feature.
\nIf a component being tested dispatches a Redux action, this is the right\nplace to test those actions and their effect on the UI.
Pay attention to the concept of feature isolation vs component isolation\nas it will help you write better integration tests and also easily mock\ncomponent contexts.
\nA classic example of feature isolation is when you have a <UserList />\ncomponent which displays a list of users and has a <button /> to add a new\nuser. Writing a test for <UserList /> would be equivalent to testing a\nfeature.
In this example, one would be tempted to test the action of clicking on the\n<AddUserButton /> and further test the <NewUser /> form... nope! This is\nwhere we draw the line! Only test that the <UserList /> renders the mock\nusers in the list and that the <AddUserButton /> is present/enabled. The\n<UserList /> feature ends there, otherwise you will be sliding into\nEnd-to-End testing :)! The <User /> component, although it is invoked by\n<UserList /> component, it is isolated enough to be tested in its own\nintegration test.
Testing components this way would make \"context mocking\" easier for\ncomponents.
\nAnother important benefit for isolating testing context, as in the example\nabove, is that it will be easier to mock the redux actions and/or api calls\nusing tools such as Jest and\nMock Service Worker (or \"msw\") as explained in the\nChoosing Tools section.
The value of writing integration tests for components, in this way, ensures\nthat a given component renders the UI consistently, given all possible\ncombinations of contexts and interactions. This will also allow you to ensure\nredux actions invoked by the component are called as expected and with the\ncorrect arguments.
\nAn example component integration test would look like:
\n// ....other imports
import { setupServer } from 'msw/node';
// Tell jest to mock the module
jest.mock('../js/actions/UserActions', () => ({
...jest.requireActual('../js/actions/UserActions'),
saveUser: jest.fn(),
}));
import { saveUser as mockSaveUser } from '../js/actions/UserActions';
const server = setupServer(...handlers);
// Enable API mocking before tests
beforeAll(() => server.listen());
// Reset any runtime handlers we may add during the tests
afterEach(() => server.resetHandlers());
// Disable API mocking after the tests are done.
afterAll(() => server.close());
beforeEach(() => {
jest.clearAllMocks();
});
describe('<AddUser/>', () => {
test('create new user', async () => {
const {getByPlaceholderText,getByText} = render(<User {...defaultProps} />);
userEvent.type(getByPlaceholderText('First Name'), 'John');
userEvent.type(getByPlaceholderText('Last Name'), 'Doe');
userEvent.click(getByText('Save'));
expect(mockSaveUser).toHaveBeenCalledTimes(1);
expect(mockSaveUser).toHaveBeenCalledWith({
firstName: 'John',
lastName: 'Doe',
});
}
Write a good couple of these.
\nIn a React/Redux App, this would mean testing a full flow of a given\nfeature. end-to-end tests would require launching the entire application,\nincluding the backend, to run a given test.
\nNote that end-to-end tests are different from integration tests as\nthey require the entire App to run and render the full flow to your component\nunder test.
\nWith this understanding, consider writing e2e tests per workflow.
\nAn example e2e workflow is the \"Viewing and adding users\" workflow.
\nThe e2e test for this workflow would require a test runner to launch the app,\nlog-in, navigate to the users list page, verify existing users are in the\nlist, click on the Add New User button and confirm that the new user has been\nadded to the list.
\nAs you can see, e2e tests have more dependencies and require that you setup\nyour testing environment in way that closely simulates your real application\nusage.
\nAn example e2e test for a React/Redux App with a Phoenix/Elixir backend, using\nHound as a test runner looks like this:
defmodule OpenFn.UsersTest do
setup do
user = insert(:user, confirmed_at: DateTime.utc_now())
{:ok, user: user }
end
@tag :integration
test \"Sign-up.\", %{user: user} do
navigate_to(\"/sign-up\")
form = find_element(:id, \"sign_up_form\")
form
|> find_within_element(:id, \"first-name\")
|> fill_field(\"John\")
form
|> find_within_element(:id, \"last-name\")
|> fill_field(\"Doe\")
form
|> find_within_element(:id, \"email\")
|> fill_field(\"doe@gmail.com\")
form
|> find_within_element(:id, \"save-button\")
|> click
assert page_title() === ~s/Welcome to my page/
end
end
There are many testing tools out there, but for a typical React/Redux app the\nfollowing tools should help you accomplish the above tasks:
\nheadless\nand browser mode.Testing a React App can be really hard, but worth it! By building\nAria-accessible components ahead of time, you save yourself 💰 and good\nhealth! A few more hints would be:
jsdom) rather than a real browser come with their own challenges in\nrendering and traversing complex DOM trees, especially if you are using UI\nlibraries such as MUI.What would I do differently? Here are my few thoughts:
\nAll this stuff for what?
\nGotcha, here you go...
Happy testing,
\nChaiwa
", "url": "https://docs.openfn.org/es/articles/2021/10/22/testing-react-app-with-jest-hound", "title": "Testing a React app, the blurred line between Unit, integration and E2E", "summary": "Have you ever struggled to layout the strategy for testing your React App? Well,", "date_modified": "2021-10-22T00:00:00.000Z", "author": { "name": "Chaiwa Berian" }, "tags": [ "how-to", "tips", "testing", "browser-testing", "react", "elixir" ] }, { "id": "https://docs.openfn.org/es/articles/2021/10/15/webpack-to-esbuild-part1", "content_html": "We're very happy users of Elixir and Phoenix at OpenFn, given that we've been\nusing it continuously for about 6 years - upgrades and all. Our front-end\ntoolchain, albeit far from out of date (Webpack 5.52.1 today) has left some\nroom for improvement.
\n\nThis is a post written on what we call 'Slow Fridays', where we explore and\nthink about stuff we're curious about. The deal is some artifact at the end of\nthe day. So while this post is not complete - I believe part two is warranted.
\n
Phoenix 1.6 started to include esbuild by default. The main reasons\ncited were the amount of support the team had to continuously give for a\ntoolchain that although necessary is not under the control of the framework\nteams control.
\nBack in the day, Phoenix used brunch to do front end builds - despite not being\nthe most common bundler at the time. The objective I assume was to provide\nsomething that is easy to get going with and lowers the barrier to entry for\nPhoenix. We actually switched brunch out for webpack before webpack got into\nPhoenix in order to more easily use new libraries and newer ES syntax.
\nIt wasn't without its challenges, and still is. Webpack is wonderfully\npowerful - but it does do a lot, and it's really easy to stumble onto plugins\nthat are either order dependent or mutually exclusive.
\nIt's a bundler/compiler for JS/TS projects that is written in Go, along with\nbeing faster by the simple fact it's natively compiled it also leverages\nconcurrency. To compare why esbuild is faster than NodeJS on face value is\nunfair in my opinion. Although having similar objectives, they are wildly\ndifferent in implementation and in features.
\nSo speed is the big selling point. But thinking back to other bundlers, what\nmade us switch (thinking of gulp, grunt, mixing in babel, browserify etc). Those\nchanges were never about speed, at least a drop in build time was nice - a bit\nof caching goes a long way. The changes were about being able to use the syntax\nand libraries we wanted with as little fuss as possible.
\nWebpack can do almost anything. I'm not convinced esbuild can match that, and\nas an open-source maintainer I'd argue it shouldn't break its original goals to\nmatch Webpack.
\nI started breaking down what our current bundle setup actually does, after all\nif we were just bundling plain JS I'd probably be using rollup and terser. Going\nthrough our webpack.config.js file I can see that it:
imports for .css files through style-loader and css-loaderrequire statements to images, copy them to an assets foldernode_modules) in development.node_modules into a vendor.js bundle.That's a lot more than just building something, there's some implicit behaviour\nhere.
\nstyle-loader and css-loader results in extra\nfunctions and behaviour being introduced, it's not producing a .css file.\n\n\nI'm personally not a big fan of apps injecting styles, but I think I get why\npeople do it. Maybe I'm old-school and like to have my stylesheets delivered\nin a few files (or even one).
\n
app.js, it puts everything that resolves to\nnode_modules in vendor.js. Subtle but worth pointing out.* Could probably ignore this and refactor some\nfiles and check that tree-shaking is working properly.
\nLike I mentioned before, webpack is super versatile and it would be\nshort-sighted to get esbuild to do everything it does.
\nSo I'm approaching this with the idea that I code will need some changes, the\nless the better - I've always firmly believed the kinds of restrictions smaller\ntools provide can make things neater and more portable.
\nstyle-loader and how it injects CSS into the DOM.file loader.\ndocsesm output files. We don't ship ES Modules to the browser,\nbut seems like a good moment to try given the primary targets for our web\nfront end getting module support in 2019 2.I've tried to go over the process as methodically as possible, after all this\nisn't a new codebase nor am I alone on this. It's always important to understand\nwho's going to be working with this, and respect the varying skill sets and\nfocuses of our peers.
\nSo we know what we want, but can any of this work? Let's give it a go with the\nsimplest of steps:
\n./node_modules/.bin/esbuild js/app.js --bundle --outfile=out.js
> js/app.js:58:2: error: Unexpected \"<\"
58 │ <React.StrictMode>
╵ ^
1 error
Bang! Right, that one makes sense to me, it doesn't know what JSX is yet. I'm\ngonna take a wild guess and say it won't know about the CSS or our referenced\nimages.
\n./node_modules/.bin/esbuild js/app.js \\
--loader:.js=jsx \\
--loader:.png=file \\
--loader:.jpg=file \\
--loader:.jpeg=file \\
--bundle \\
--outdir=out
> js/marketing/ProductsList.js:53:40: error: Unexpected \":\"
53 │ const displayList = products.filter(::this.includesText);
╵ ^
> js/marketing/Banner.js:13:4: warning: Duplicate key \"background\" in object literal
13 │ background: `-webkit-linear-gradient(to left, ${theme.palette.banner.right}, ${theme.palette.banner.left})`,
╵ ~~~~~~~~~~
...
Nice! Two things worth mentioning here:
\n::, let's replace that with\nthe equivalent includesText.bind(this)../node_modules/.bin/esbuild js/app.js \\
--loader:.js=jsx \\
--loader:.png=file \\
--loader:.jpg=file \\
--loader:.jpeg=file \\
--bundle \\
--outdir=out
out/app.js 8.0mb ⚠️
out/commcare-dhis2-JKJLM3MQ.png 471.0kb
...
out/app.css 85.1kb
...and 145 more output files...
⚡ Done in 651ms
Jeepers! 651ms! That's nuts.
\nWe can see it's copied our images and css into the build folder. Note that we're\nnot doing any bundle splitting right now and from the looks of it
\n... \\
> --minify
out/app.js 3.4mb ⚠️
...
⚡ Done in 611ms
That's better, wow. It's kinda difficult to not be amazed. For context, a\nminified and split production build takes about 34s with webpack and that's on\nmy i7 desktop machine, and 197s (3+ mins) on CI/CD.
\nSo our 'can we actually do this' seems to have gone pretty well so far. I'm\nreally excited about what this will give us in the end.
\nBut a shell command doth not a replacement for webpack make. We still need to:
\nSo you're using docker's multi-stage builds and noticed that your build times\naren't nearly as quick as you expected?
\nAs many teams who spend more and more time using docker, it's quite common to\nget into multi-stage builds; usually resulting in significantly smaller images.
\nHowever this comes with a pretty significant dilemma with caching. Even when\nusing the --cache-from flag when building, docker only caches the last image.
One proposed solution1, is to pull, build and push each\nindividual stage. Coming with tight coupling between the shape of your\nDockerfile and your build process/scripts.
\nThe other solution uses Docker Buildx which the document describes as:
\n\n\nDocker Buildx is a CLI plugin that extends the docker command with the full\nsupport of the features provided by Moby BuildKit builder toolkit. It provides\nthe same user experience as docker build with many new features like creating\nscoped builder instances and building against multiple nodes concurrently.
\n
While that sounds pretty cool, it doesn't really touch on caching. This actually\ntook me a while to find out that it would in fact do caching very differently.\nIn fact it's a very different experience using it, and has lots of really cool\nfeatures that further detach you from the local docker state allowing you to\nbuild in environments that are stateless - such as Google CloudBuild without\nhaving to wire up some kind of persistence or file caching scheme.
\nWe're only going to scratch the surface of Buildx, and with that let's get the\nabsolute minimum working; build our image locally.
\nFirst things first we need to create a builder, and select it for use. This is\nimportant as without creating a buildx builder (and setting it as the default),\nbuildx will use the docker driver instead of the docker-container driver\nwhich we want in order to take advantage of cache exporting.
docker buildx create --name mybuilder --use
\n\nYou only need to run this once, except in the case of CloudBuild where each\ninvocation is a new node.
\n
docker buildx build \\
--cache-from=type=local,src=/tmp/buildx-cache \\
--cache-to=type=local,dest=/tmp/buildx-cache \\
--load \\
.
While the --cache-* options aren't specifically required when running build,\nas buildx does manage its own local cache (distinct from the regular docker\ncache), it's there to emphasise the options that cache can be provided via the\nCLI options.
This is about as close as you get to a regular docker build, with the\nsignificant difference being that you have to specify where to cache from and\nto.
\nThe --load flag is to tell buildx to set the output to the local docker\ndaemon. Without that you won't actually get a resulting image to run. However,\ndepending on your use case, this could be seen as a convenience - if you're\nwanting to run your tests inside your build; a resulting image isn't\nparticularly useful.
Now comes to the part I'm most interested in, caching in a stateless/remote\nenvironment. Multipart builds for us at OpenFn are essential, since we use\nElixir and like other compiled languages there is a lot to be gained by only\nshipping the stuff you're going to run; and no language is safe from requiring\nseveral times more 'stuff' in order to build our apps.
\nBuildx supports a\nhandful of different types\nof caching sources and destinations. We're going to be using the registry\ntype, where you point the cache at a repository reference (repo/image:tag\nstyle).
\n\nOne thing to note is that Google Container Registry does not support the\nmetadata/manifest format that buildx uses, so if you're using Google Cloud you\nwill need to start using Artifact Registry.
\n
Inline
\nPush the image and the cache together:
\n...
--cache-from=type=registry,ref=$IMAGE_NAME \\
--cache-to=type=inline \\
...
This comes with the constraint that cache mode is always min, which only\nexports/caches the resulting layers; which is still better than the plain docker\nbuild caching but I think having the intermediary layers is generally a win. We\nwant to avoid a single line change invalidating an entire build step.
Registry
\nResulting image and cache are separated:
\n...
--cache-from=type=registry,ref=$IMAGE_NAME-build-cache \\
--cache-to=type=registry,ref=$IMAGE_NAME-build-cache,mode=max \\
...
Again coming back to the cache mode, here being max; all intermediary laters\nare exported to the cache image as well.
I have opted to create two images, one for caching and another for the\nresulting image used to deploy. This gains us a much more granular cache and the\nability to more easily manage the cache image - like deleting the whole thing\nwhen wanting to invalidate the cache. Not to mention I'm fairly sure the size of\nour images that get pulled on kubernetes would get significantly larger with\nmany more layers.
\nIt feels like a safer bet to have lean images for kubernetes to pull, and chunky\ncache images specifically for speeding up build.
\nDepending on your setup, pulling large images can get seriously expensive in a\nreasonably active deployment environment - like on AWS ECS without using\nPrivateLink.
\n\n\nIt appears the
\nmoby/buildkitdocumentation also demonstrates\nthis\napproach.
IMAGE_NAME=us-east4-docker.pkg.dev/<project-name>/platform/app \\
docker buildx build \\
-t $IMAGE_NAME:latest \\
--cache-from=type=registry,ref=$IMAGE_NAME-build-cache \\
--cache-to=type=registry,ref=$IMAGE_NAME-build-cache,mode=max \\
--push \\
--progress=plain \\
.
This implies that the cache image is named with the suffix -build-cache:
\nus-east4-docker.pkg.dev/<project-name>/platform/app[-build-cache].
The --push argument tells buildx to push the resulting image to the registry.
Clearing the local cache
\nAs mentioned before, buildx has its own cache and in order to clear the cache\nwhile debugging and readying a Dockerfile for remote building you'll probably\nneed to reach for docker buildx prune.
Using buildx has been a really pleasant experience, having personally attempted\nusing it a few times over the last 3 years; the most recent one being the first\ntime I felt confident getting it into production. As with any sufficiently\nflexible build tooling, the errors and issues you can run into range from\ncomplete gibberish, genuinely concerning inconsistencies to architectural\nchoices that you haven't fully caught up on; requiring an ever growing list of\nchanges you need to make to your own build process.
\nOur initial observations have been great, reasonable changes on our build have\ngone from 28 minutes to around 9 minutes.
\nWhile I have encountered a few confusing cache invalidations, especially when\nbuilding locally, exporting the cache to a repository and then having CloudBuild\nuse the image cache. And occasionally locally having what feels like really\naggressive caching on intermediate steps, leading me to pruning the local cache.
\nBut overall, these issues aren't necessarily buildx issues and more likely a\ncombination of building docker images in general except with many more steps\naccounted for by the cache.
\nIt's kinda hard to see now what the exact issues I had with it in the past, but\nhey!
\nBuildx has given me what I 'expected' with docker multi-stage builds, and having\nthe cache in a repository completely side-steps having to attach a shared volume\nor copying from a storage bucket.
\nJobs are business processes turned into functional-style scripts. What does that\nmean, how should you approach writing jobs?
\nFirst, this is how I think about jobs and what we do at Open Function Group to\ntry to make our job code as readable, future-proof, and concise as possible.\nThere are a million different ways to approach writing jobs. This is one.
\nstateIf a job is a set of instructions for a chef (a recipe?) then the initial\nstate is all of the ingredients they need tied up in a perfect little bundle.\nIt usually looks something like this:
{
\"configuration\": {
\"hostUrl\": \"https://moh.kenya.gov.ke/dhis2\",
\"username\": \"taylor\",
\"password\": \"very-secret\"
},
\"data\": {
\"type\": \"registration\",
\"patient\": {
\"age\": 24,
\"gender\": \"M\",
\"nationalId\": \"321cs7\"
}
}
}
This might be the initial state for a real-time, message-triggered job. Some\nsource system generated a new patient payload and sent that payload to OpenFn.\nThe data from our source system will wind up in state.data. Now if my job is\nmeant to take this new patient registration information and use it to create a\nnew record in the national health record system, I'll also need to provide my\nrobot-chef here with a credential so they can access that system. The credential\nI've specified will get put into state.configuration and now our \"raw\ningredients\" are all ready for our robot chef.
Note that even if this job was initiated by a cron trigger (e.g., \"Hey chef,\nprepare this recipe every Tuesday at 7pm\") or by a flow/catch trigger (e.g.,\n\"Hey chef, prepare this recipe only when you fail to make banana pancakes\") it\nwill have an initial state.
\nEvery job, and every operation inside that job (think \"step\" in a recipe) is\ncalled with state and returns state when it's done.
Initial state for a cron triggered job might look like this:
\n{
\"configuration\": {
\"hostUrl\": \"https://moh.kenya.gov.ke\",
\"apiKey\": \"abc123\"
},
\"data\": {},
\"lastProcessedId\": 321
}
And for a fail triggered job like this:
\n{
\"configuration\": {
\"hostUrl\": \"https://moh.kenya.gov.ke\",
\"apiKey\": \"abc123\"
},
\"data\": {},
\"lastProcessedId\": 321,
\"error\": [\"Required field missing\", \"Patient Surname\", \"Line 43\"]
}
No matter what, jobs start with state. See\n\"Initial and final state for runs\" for a detailed\nbreakdown.
\nstate tooNow that we've got it in our heads that state is the raw ingredients you hand\nto your chef when you ask them to prepare a recipe, let's look at the recipe.\nBoiled down (excuse the pun) a job for loading those patients into the national\nhealth record system might look like this:
get('/api/insuranceRegistrations');
post('/api/patients', { ...someData });
post('/api/visits', { ...someData });
We're telling our chef to take those raw ingredients (login info for our\nnational health system and a chunk of information about a newly registered\npatient) and do the following:
\nWhen all of this is done, we'll not only have a new patient and visit logged in\nthe national health registry, but we'll also return a final state object with\ninformation about what we've done that can be used in subsequent jobs. Imagine\nthat we want to make a cash transfer to this patient so that they can take a cab\nto the next visit—we might create a job with the Mpesa adaptor that takes the\nfinal state of this first job as its initial state. In this way, jobs are\ncomposable.
But what about the complexity inside our job—in order to complete step 2, we\nneed some data from the insurance registry and we only get that data in step 1.\nCrucially, each operation (again, think \"step\" in a recipe) takes state and\nreturns state. In effect, the OpenFn execution pipeline simply calls all of your\naction methods with state, passing it along from one operation to the next,\nwaiting for each to finish and using the output from the first as the input for\nthe second.
\nWhile you may write your get, post, post job as it's show above, the way\nit's handled by OpenFn is actually more like:
return get('/api/insurance', { ...useDataFromState })(state)
.then(state2 => post('/api/patients', { ...useDataFromState2 })(state2))
.then(state3 => post('/api/visits', { ...useDataFromState3 })(state3));
Each of these operations returns a function which takes state and returns\nstate. This means that within a job, you are essentially modifying state,\ncreating/manipulating records in external systems, and returning state.
It opens up a really interesting world of possibility for data manipulation,\ncleaning, or transformation. Consider what we might do after we get data from\nthe insurance registry but before we create that patient in the national\npatient registry:
\nget('/api/insuranceRegistrations');
fn(state => {
console.log(state.data); // let's look at the response from the insurance API.
state.data.people.filter(p => p.HasActiveInsurance); // and modify the payload to only retain those with active insurance
return state; // before returning state for our create patients operation.
});
post('/api/patients', { ...someData });
post('/api/visits', { ...someData });
We might even need to do some manipulation before we send a get request to\nthe insurance registry. That's no problem:
fn(state => {
state.data.registrationType = state.data.age > 18 ? 'Adult' : 'Minor';
return state; // before returning state for our create patients operation.
});
get('/api/insuranceRegistrations', {
query: { type: dataValue('registrationType') },
});
fn(state => {
state.data.people.filter(p => p.HasActiveInsurance);
return state;
});
post('/api/patients', { ...someData });
post('/api/visits', { ...someData });
Here, we've added a step to modify the initial state before we send that first\nget request to the insurance API. We determine if the new patient is a minor,\nand then use that newly calculated data to apply a query to the insurance API\nrequest.
Using fn(state => state) or alterState(state => state}) is incredibly\nuseful, because it allows us to separate our data manipulation, calculation, and\nraw Javascript (which will be harder for low-tech users to understand) from our\nexternal actions. Let's explore that some more.
Inside each operation we could do some data manipulation... all of these\noperations, across the many different language packages, allow for inline data\nmanipulation like this:
\nget('/api/insuranceRegistrations', {
query: state => {
console.log(\"I'm doing some fancy stuff here.\");
return { type: state.data.age > 18 ? 'Adult' : 'Minor' };
},
});
post('/api/patients', {
body: {
name: state => {
return `${state.data.firstName}${state.data.lastName}`;
},
},
});
But if you're interacting with both technical and non-technical users, it makes\nfor harder to read jobs. Consider the following instead:
\n// Perform calculations...
fn(state => {
// Create several new calculated attributes...
state.data = {
...state.data,
type: state.data.age > 18 ? 'Adult' : 'Minor',
fullName: `${state.data.firstName}${state.data.lastName}`,
};
return state;
});
// Get insurance data...
get('/api/insuranceRegistrations', { query: { type: dataValue('type') } });
// Create new patient...
post('/api/patients', { body: { name: dataValue('fullName') } });
Since we often have non-developers creating the external operations like get\nand post above, this pattern makes our handoff easier. The business analyst\ncan say \"I need to have a registration type field available for use when\nquerying the insurance registry.\" A developer might respond, \"Great! How do you\nwant to calculate it... I've got all of Javascript at my fingertips.\" That dev\ncan then make as many API calls as they'd like, perform as many\nmap.reduce(...) calls as their heart desires to complete that calculation...\nso long as they make sure the hand off state to the business analyst's\noperation with a valid state.data.type attribute.
A final benefit of this approach is that it becomes much easier to generate job\nscripts from Google Sheets. Our implementation team frequently works with\nnon-technical clients to generate field maps that look like this:
\n| Path to Source Data | Destination Field | Auto-generated syntax (using concat) |
|---|---|---|
| patient.fullName | name | field('name', dataValue('patient.fullName')), |
| patient.age | age | field('age', dataValue('patient.age')), |
| ??? | type | plz help us calculate 'type' based on x, y, z |
| patient.sex | gender | field('gender', dataValue('patient.sex')), |
We can then copy and paste the syntax generated in that final column directly\ninto OpenFn and update the bits that need some sort of custom code, writing an\nfn(state) block or an alterState(state) block before the external action.
Some key takeaways here:
\nJobs start and end with state — some raw ingredients that will be used in a\nrecipe.
Jobs are lists of operations — steps in a recipe that each take state,\ndo some stuff, and then return state.
As you move through the steps in a job, you are modifying state. Each\nsubsequent step begins with the final state from the previous step.
It may be useful to keep all your custom Javascript data cleaning,\nmanipulation, etc., in a separate operation (e.g., fn(state) or\nalterState(state)) so that your external actions are clean and easy to\nfollow.
Finally, taking a close look at how developers write those fn(state) steps\ntells us a lot about what the job execution pipeline is really doing:
// here, \"fn\" is a function that takes state and returns state
fn(state => {
console.log(\"I'm doing some cool stuff.\");
// I might create some new attribute...
state.myNewThing = true;
// And ALWAYS return state for the next operation to use...
return state;
});
I hope this gives you sense of how I think about structuring jobs and building\ndata pipelines or automation flows on OpenFn. We recognize that this stuff is\ncomplex, and are pushing our new documentation regularly, so please do get in\ntouch if you think there are ways we could improve this type of\nwalk-through/helper article.
\nHappy integrating,
\nTaylor
", "url": "https://docs.openfn.org/es/articles/2021/07/05/wrapping-my-head-around-jobs", "title": "Wrapping my head around jobs", "summary": "Jobs are business processes turned into functional-style scripts. What does that", "date_modified": "2021-07-05T00:00:00.000Z", "author": { "name": "Taylor Downs" }, "tags": [ "how-to", "tips", "jobs" ] }, { "id": "https://docs.openfn.org/es/articles/2021/05/24/commcare-events", "content_html": "This is a quick one, but I just got off an exciting call with an organization\nthat's going to set up some jobs to move data into Salesforce from CommCare and\nrealized that despite this being one of our more common integration\nrequirements, we haven't done a 'tips' article for this type of project. Until\nnow.
\nSo here goes. While this is by no means an exhaustive project planning template,\nhere are a few things to keep in mind if you're planning to implement a CommCare\nto Salesforce integration on your own.
\nFirst, most people make use of CommCare's \"Data Forwarding\" feature to send form\nsubmissions and changes in cases (creation, update, closure, etc.) to OpenFn in\nreal-time. You can read about that\nhere\nbut the key consideration at this planning stage is when you'll be performing\noperations—create(...), update(...), upsert(...), query(...),\n(bulk(...), etc.—in Salesforce and what data you'll have access to.
Each time a form submission comes into CommCare, we'll get a copy of that\nsubmission at OpenFn and can use that data to create or modify some records in\nSalesforce.
\nLikewise, each time a case gets updated (or created or closed) we'll get a copy\nof the case with all the case \"properties\" and we can use that data to do some\nstuff in Salesforce.
\nIf you are using \"Form Forwarding\", the trigger you'd create in OpenFn might\nlook like this {\"form\":{\"@name\":\"ART Adherence Self-Reporting Tool\"}} and it\nwould trigger your job any time an \"ART Adherence Self-Reporting Tool\"\nsubmission arrived from CommCare, giving that job access to all of the data\ninside that submission.
Assuming you're using making use of case management, the data that arrives from\nCommCare will look something like this:
\n{
\"__query_params\": {
\"app_id\": \"some-long-id\"
},
\"app_id\": \"some-long-id\",
\"archived\": false,
\"attachments\": {
\"1621866020043.jpg\": {
\"content_type\": \"image/jpeg\",
\"length\": 16423,
\"url\": \"https://www.commcarehq.org/a/your-project/api/form/attachment/some-uuid/1621866020043.jpg\"
},
\"form.xml\": {
\"content_type\": \"text/xml\",
\"length\": 2727,
\"url\": \"https://www.commcarehq.org/a/your-project/api/form/attachment/some-uuid/form.xml\"
}
},
\"build_id\": \"0ec83881cd0e420dad5c24ed3a5452fe\",
\"domain\": \"your-project\",
\"edited_by_user_id\": null,
\"edited_on\": null,
\"form\": {
\"#type\": \"data\",
\"@name\": \"ART Adherence Self-Reporting Tool\",
\"@uiVersion\": \"1\",
\"@version\": \"2783\",
\"@xmlns\": \"http://openrosa.org/formdesigner/59E1207B-969F-402D-9EEE-675504036F78\",
\"administrative\": {
\"coach_verification\": \"check_here\",
\"visit_notes\": \"\",
\"vist_notes_to_save\": \"\"
},
\"case\": {
\"@case_id\": \"1ec51ee9-5aef-4bd2-b7eb-7599856251bc\",
\"@date_modified\": \"2021-05-24T14:20:28.693000Z\",
\"@user_id\": \"332e893dcd1b413686621bd80aae0cd3\",
\"@xmlns\": \"http://commcarehq.org/case/transaction/v2\",
\"update\": {
\"consent_received\": \"yes\",
\"home_visit_notes\": \"\"
}
},
\"meta\": {
\"@xmlns\": \"http://openrosa.org/jr/xforms\",
\"appVersion\": \"CommCare Android, version \\\"2.51.2\\\"(463994). App v2798. CommCare Version 2.51.2. Build 463994, built on: 2021-03-17\",
\"app_build_version\": 2798,
\"commcare_version\": \"2.51.2\",
\"deviceID\": \"commcare_a39f55a5-c744-4e33-8e01-d17e7698894f\",
\"drift\": \"0\",
\"geo_point\": null,
\"instanceID\": \"130c68c5-7d17-4086-8a85-27d7d7da2216\",
\"timeEnd\": \"2021-05-24T14:20:28.693000Z\",
\"timeStart\": \"2021-05-24T14:18:46.856000Z\",
\"userID\": \"332e893dcd1b413686621bd80aae0cd3\",
\"username\": \"some-chw\"
},
\"participant_information\": {
\"participant_id\": \"007\",
\"name\": \"taylor downs\",
\"gender\": \"male\",
\"guardian_information\": {
\"guardians_name\": \"Fake Data\",
\"guardians_phone_number\": \"8675309\",
\"guardians_signature\": \"1621866020043.jpg\",
\"relationship_to_participant\": \"father\"
},
\"current_medications\": [
{ \"name\": \"generic-1\", \"active\": true },
{ \"name\": \"fakelyn-notrealiol\", \"active\": false },
{ \"name\": \"sasstra-zenica\", \"active\": false },
{ \"name\": \"ibuprofen\", \"active\": true }
]
},
\"tested_for_hiv_status_tested_for_hiv\": \"OK\",
\"visit_information\": {
\"consent_given\": \"yes\",
\"date_consent_given\": \"2021-05-23\",
\"visit_date\": \"2021-05-23\"
}
},
\"id\": \"130c68c5-7d17-4086-8a85-27d7d7da2216\",
\"indexed_on\": \"2021-05-24T14:20:39.045971\",
\"initial_processing_complete\": true,
\"is_phone_submission\": true,
\"metadata\": {
\"appVersion\": \"CommCare Android, version \\\"2.51.2\\\"(463994). App v2798. CommCare Version 2.51.2. Build 463994, built on: 2021-03-17\",
\"app_build_version\": 2798,
\"commcare_version\": \"2.51.2\",
\"deviceID\": \"commcare_a39f55a5-c744-4e33-8e01-d17e7698894f\",
\"drift\": \"0\",
\"geo_point\": null,
\"instanceID\": \"130c68c5-7d17-4086-8a85-27d7d7da2216\",
\"location\": null,
\"timeEnd\": \"2021-05-24T14:20:28.693000Z\",
\"timeStart\": \"2021-05-24T14:18:46.856000Z\",
\"userID\": \"332e893dcd1b413686621bd80aae0cd3\",
\"username\": \"some-chw\"
},
\"problem\": null,
\"received_on\": \"2021-05-24T14:20:37.976363Z\",
\"resource_uri\": \"\",
\"server_modified_on\": \"2021-05-24T14:20:38.111789Z\",
\"type\": \"data\",
\"uiversion\": \"1\",
\"version\": \"2783\"
}
This is a big blob of JSON—the body of the message that's received at OpenFn\nwhen this particular form (\"ART Adherence Self-Reporting Tool\") is submitted in\nCommCare—will be handed off to the job to start processing. The question is,\nwhat should we do?
When setting up for a self-service implementation on OpenFn, the most important\nthing you can do at this moment is carefully enumerate the data entry process\nthat you'd like a real human to follow. You can translate it to a job script\nlater.
\nYou'll need to write this up for your own case, but in this fictional example,\nhere's the data entry process.
\nRight from the start, notice that we're being incredibly explicit with these\ninstructions! We're using the \"API Name\" (instead of just the \"label\", which\nmight be ambiguous) of every field we want filled out in Salesforce and we're\nusing the specific \"path\" to the data we want this person to enter from\nCommCare.
Why are we being so specific? Because eventually, a computer will need to\ninterpret this—and they're terrible with ambiguity!
{\"form\":{\"@name\":\"ART Adherence Self-Reporting Tool\"}} in the body (this is\nour trigger)participant_id\nyou find in the form.participant_information section as their\nParticipant_Code__c. (If one already exists in Salesforce with that code,\nthen update the existing record instead.)Name__c with the data from form.participant_information.nameSex__c with the data from form.participant_information.genderCommCare_Case_ID__c with the data from form.case.@case_idinstanceID from the metadata section as the\nunique identifier Visit_Code__c. (Again, if there's already a visit with\nthat ID please update the existing record.)Date__c with form.visit_information.visit_date.Consented__c with form.visit_information.consent_given.Test_Status__c to true, regardless of what's in the message\nfrom CommCare.Community_Health_Worker by their username\nin form.metadata.username.form.participant_information.current_medications array—matching on a unique\nID formed by a combination of the medication name and the participant_id\nso that we can update existing medication records if they're present.Generic_Name__c with nameStatus__c with activeparticipant_id field.Phew... that's the task. It's just a fictional example and things could be much\nmore straightforward, or much more complicated than this, but it's important to\nremember that if you can get to this level of precision and granularity in\nyour data entry process, a tool like OpenFn can automate this for you in a\nflash.
\nIf you're streaming data in from CommCare and you've got your Salesforce system\nall set up so that this data entry person can complete the above steps (are all\nthe objects and fields created? are the right fields marked as \"unique\" and set\nto be used as an \"external id\" in the Salesforce administration section? have\nyou turned on data forwarding in CommCare?) then it's time to turn them into an\nOpenFn project!
\nA quick plug: Did you know that there's an\nOpenFn community forum where you can post\nstuff like the \"steps\" above and get help from other OpenFn users and staff\nconverting these steps into a real, working, OpenFn job?
Well, you do know! Check it out at\ncommunity.openfn.org
We don't need a CommCare credential, since they'll send data to us. Create a\nSalesforce credential that will allow the OpenFn worker to log into your\nSalesforce system.
\nRead more about credentials here.
\nMessage Filter for the type{\"form\":{\"@name\":\"ART Adherence Self-Reporting Tool\"}} for the\ninclusion criteriaRead more about triggers here.
\nsalesforce adaptorAnd convert the instructions above to \"operations\" by using the inline help\nprovided by the Salesforce adaptor:
\n// Use upsert to create or update a participant based on their participant code.
upsert(
'Participant__c',
'Participant_Code__c',
fields(
field(
'Participant_Code__c',
dataValue('form.participant_information.participant_id')
),
field('Name__c', dataValue('form.participant_information.name')),
field('Sex__c', dataValue('form.participant_information.gender')),
field('CommCare_Case_ID__c', dataValue('form.case[@case_id]'))
)
);
// Then upsert a visit using the visit code.
upsert(
'Visit__c',
'Visit_Code__c',
fields(
field('Visit_Code__c', dataValue('metadata.instanceID')),
field('Date__c', dataValue('form.visit_information.visit_date')),
field('Consented__c', dataValue('form.visit_information.consent_given')),
// Always set status to true
field('Test_Status__c', true),
// And related this visit to the participant we just created by their \"code\"
relationship(
'Participant__r',
'Participant_Code__c',
dataValue('form.participant_information.participant_id')
)
)
);
// And finally for EACH mediation listed, create a medication record with a status
each(
merge(
dataPath('form.participant_information.current_medications[*]'),
fields(
field('pID', dataValue('form.participant_information.participant_id'))
)
),
upsert(
'Medication_Tx__c',
'Medication_Tx_ID__c',
fields(
field(Medication_Tx_ID__c, state => {
// Here, inside the medications array we've \"scoped\" state so that
// state.data, for each item in the array, looks like this:
// { pID: 007, name: \"sasstra-zenica\", active: false }
// We will concatenate the participant ID with the medication name.
return state.data.pID + state.data.name;
}),
field('Generic_Name__c', dataValue('name')),
field('Status__c', dataValue('status')),
relationship('Participant__r', 'Participant_Code__c', dataValue('pID'))
)
)
);
Now, every time this job runs (which is every time a CommCare form is submitted)\nyour OpenFn worker will upsert a Participant, upsert a Visit, and upsert a\nwhole list of Medications in Salesforce.
Well, in our little example you'd turn the job \"on\" (setting it to on the\ninbound messages from CommCare) and let it run. Whenever there was a failure\n(maybe your Salesforce admin added a new required field on the\ncustomMedication object) you'd get an email and you'd have to come back to\nOpenFn to update your job, including that new field.
If you're in the process of designing your CommCare and Salesforce systems at\nthe moment, this back-and-forth will be pretty common. Keep in mind that you\nwant as much simplicity as possible in those end-user systems because... well\nbecause humans have the interact with them every day!
\nSo long as your processes are well defined, OpenFn can handle a bit of\ncomplexity (data cleaning, transformation, complex logical flows, etc.) but you\nshould never make sacrifices to the user experience in CommCare and\nSalesforce—that's a quick way to lose adoption.
\nSo, ideally, you've designed your workflows in CommCare and Salesforce to make\nyour users happy and get them the information they need to do their jobs well\nand then you come back to OpenFn and spell out our data entry instructions\nlike we've done above.
\nThe two most important resources you've got at your disposal if you're setting\nthis all up on your own are:
\nRead through the \"What is an integration\",\n\"OpenFn Concepts\", and\n\"Build\" sections if you're a thorough,\nbackground-first kind of learner. If you crave snippets and sample job code,\nhead directly to the Job Library to see how other OpenFn\nusers are creating their jobs.
\nEither way, keep the community posted on your progress in the forum—you'll find\nlots of helpful folks willing to lend you a hand in your integration journey.
", "url": "https://docs.openfn.org/es/articles/2021/05/24/commcare-events", "title": "Forms and Cases: CommCare and event-based integration", "summary": "This is a quick one, but I just got off an exciting call with an organization", "date_modified": "2021-05-24T00:00:00.000Z", "author": { "name": "Taylor Downs" }, "tags": [ "how-to", "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2021/02/17/syncing-options", "content_html": "“Syncing” is getting two systems to a state of harmony. This might mean keeping\na list of patients up to date, though modifications can be made in either\nsystem. It might mean copying transactions from one system to another on a\nnightly basis. It might mean a lot of things, but the key concept is that when\nyou sync systems, you’re asking them to work together while simultaneously\nrespecting both software systems’ independence.
\nIn this post we’ll discuss two different syncing protocols to consider when\ndesigning your data integration. These include:
\nFor a\nrecent project in Cambodia,\nOpenFn is being used by social workers to automate case referrals between the\nsoftware systems Primero and OSCaR. In the design phase, we evaluated these two\nsyncing options. Below, we'll explain what each one is, the differences between\nthem and which option we chose in the end.
\nThe first option considered for this integration was the real-time/event based\nsync. This type of sync is triggered whenever a specified event takes place in a\nsystem. With this approach, whenever a case is referred in Primero (via the user\ninterface, i.e., when a real case-worker clicks the “refer” button) OpenFn\nreceives a small payload with case data and transmits it to OSCaR and vice\nversa.
\n![\"Real_Time_Sync\"](\"https://docs.openfn.org/es/assets/images/syncs1-81446bbef450166c67a52b2ef7a48b10.webp\")
Because of their instantaneous nature, real time/event based syncs are great for\nintegrations that involve mobile payments or sms messages to recipients. Really,\nanything that needs to be done “now”! Additionally, depending on your data\nvolumes real time syncs might save you money because you’re only using resources\nwhen specific events take place. For instance, in the above example, a run is\ntriggered by a referral, so if there are only 10 case referrals/month, you'd\nonly process 10 runs each month.
\nThis type of sync is great because it’s instantaneous, typically quite\nstraightforward to set up, doesn’t require any “state mangagement” on OpenFn,\nand allows for the reprocessing of individual events. There are, however,\ndrawbacks.
\nFor instance, what happens if the app that’s sending notifications to OpenFn\nfails to send? What if AWS or GCP goes down, taking half of the internet with\nit? If Primero “thinks” it sent the referral, OpenFn never receives it, that\ncase might not get referred to Oscar!
\n![\"Schedule_Dependent_Sync\"](\"https://docs.openfn.org/es/assets/images/syncs2-7e3c3eacc4048afa5abfffc4aa1a188a.webp\")
The second option considered, a bi-directional schedule dependent sync, solves\nfor the issue discussed above. On a scheduled basis (every 5 minutes, for\nexample) OpenFn checks with Primero and Oscar to see if case referrals need to\nbe transmitted between the two systems and then refers the case if required. In\nthe unlikely event that any of the software systems involved crash, the\nstability provided by the bi-directional sync means that all data is preserved\nand eventually makes it to its destination safely.
\nThe major drawback here is complexity. We had to use 4 jobs instead of 2, and\nthe job that is responsible for “pulling” data that’s been updated since the\ntime of the last successful sync has to keep “state”—or some sort of working\nmemory of what it’s done in the past. When pulling modified cases from Primero,\nOpenFn now only pulls cases modified on or after YYYY-MM-DD HH:MM:SS where\nYYYY-MM-DD HH:MM:SS is the time of the last successful, round-trip\nsynchronization. OpenFn has built-in functionality to handle exactly this\nrequirement, but not all ETL systems do and it’s a design implication that must\nbe considered.
Ultimately, for the project in Cambodia, we decided that this sync option is the\nright choice because data integrity is more important than the speed of this\ndata flow. That’s a crucial point to understand—the organizations operating in\nCambodia decided that for this particular use case, being able to guarantee\neventual syncing was more important than having real-time syncing.
\nBoth options definitely have their use-cases and OpenFn's platform versatility\nenables your team to decide which type of sync is right for your project.
\nAs always, we are here to help with any questions as you think through which\nsync option makes the most sense for your project.
", "url": "https://docs.openfn.org/es/articles/2021/02/17/syncing-options", "title": "Sync Like You Mean It: Thinking Through System “Syncing” Protocols", "summary": "“Syncing” is getting two systems to a state of harmony. This might mean keeping", "date_modified": "2021-02-17T00:00:00.000Z", "author": { "name": "Jed Goldstein" }, "tags": [ "how-to", "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2021/02/03/hosted-or-local-deployment", "content_html": "Zandile is a program manager at an iNGO and she needs to use CommCare, DHIS2,\nand OpenFn for an upcoming public health project. She understands that all three\npieces of software can be deployed locally, or accessed as SaaS (Software as a\nService).
\nEssentially, Zandile needs to decide if she would like to run the software on\nsomeone else’s servers (SaaS), or on her organization’s own servers (deployed\nlocally). Before making a decision she outlines the basic, non-technical\nconsiderations for both options.
\nSaaS is software that is installed and runs on computers maintained by\nsoftware professionals, rather than on your own computer. While those computers\nmight be anywhere in the world, typically you'll access and use this software\nvia the Internet.
\nWith SaaS, the software vendor is responsible for the expenses of managing and\nmonitoring all of the technical components and issues associated with the\nsoftware. This means that Zandile’s iNGO will not be responsible for updating\nthe software to ensure compliance with new security regulations, maintaining the\nservers, backing up the data, purchasing and managing uninterrupted power\nsupplies, and providing a team of physical security guards to protect the\ncomputers and data therein against physical theft.
\nGoing the SaaS route is often faster and more secure, because you do not need to\ndevelop expertise in \"DevOps\" or hire IT and physical security specialists. This\noption also provides the greatest amount of flexibility & scalability– because\nthe SaaS provider is able to deliver more or less computing power, storage, and\nbandwidth—right when it’s needed.
\nHaving smaller setup costs (you don't have to grow a software delivery company\nof your own) often makes this a more economical choice for many, though SaaS\nwill always come with some sort of ongoing fee—a price per month or year that\ngoes to the vendor to compensate for the time and money they'll spend to ensure\nyour software works properly.
\nUnlike the SaaS option, local deployment means installing and running software\non your own computers—typically on your organization’s servers.
\nIf a SaaS provider doesn't offer hosting in your country and your government\ndoesn't allow your data to reside on foreign servers (i.e., you're not allowed\nto use things like Gmail, WhatsApp or Facebook for communicating sensitive\ninformation) then local deployment allows you to use tools like CommCare, DHIS2,\nand OpenFn while adhering to government data sovereignty regulations.
\nLocal deployment also provides your organization with complete ownership of the\nend-to-end system. Your IT team will be personally responsible for ensuring that\nthe software works, is maintained, is secure, etc. If your organization does not\nalready have an IT team in place, then this can become a costly headache, but\nfor a large organization with embedded IT experience, local deployment often\nmakes sense.
\nUltimately, being able to directly hire and fire the people who are responsible\nfor your software's proper functioning can be very useful. It means you have\ncomplete responsibility for whether or not the solution succeeds.
\nIf you've already got the teams in place (security, DevOps, etc.) then this\noption can be more economical in the long run. With a very good DevOps team,\nmaintaining an extra piece of software might only occupy 20% of a\nfull-time-employee's salary. For your security guards, if the software is\ninstalled in the same physical location it's possible that your costs won't\nincrease at all. While there will be very high setup costs, over time you may\nrealize cost savings by running an efficient software delivery unit within your\norganization that spreads its focus around a number of projects.
\nIn this fictional case, data residency is a concern—her data is sensitive or\ncontains PII—and CommCare, DHIS2 and OpenFn do not provide hosting in the\ncountry she's located. Zandile's organization has a large, experienced IT team\nthat has managed high-availability software projects for many years... they're\npros. While they anticipate that the setup costs will be quite high (around\n$60000 and several months for this set of deployments) they plan on using this\nsoftware for the next 5 years and have determined that they'll recoup a\nsignificant portion of that setup cost by not having to pay license fees for\nSaaS. They go with local deployment.
\nThe answer is: \"it depends\", but if your organization has never managed local\nsoftware deployments, then we recommend going the SaaS approach. SaaS systems,\nlike the one OpenFn and CommCare offer, are simply going to be more secure, more\nstable, and more scalable for the money.
\nCrucially, you can always start with SaaS (most tools even offer a free tier)\nand then decide later to invest in the big startup costs of a local deployment\nif the license fees for the SaaS feel high enough to make local deployment more\neconomical over the long term. After a few months or years on the SaaS, you'll\nlikely be in a better position to know if you want to continue using the\nsoftware for 5-10 years.
\nShould you need any help with your decision though please do not hesitate to\ncontact OpenFn.
", "url": "https://docs.openfn.org/es/articles/2021/02/03/hosted-or-local-deployment", "title": "Our Servers or Yours: Thinking through deployment options", "summary": "Zandile is a program manager at an iNGO and she needs to use CommCare, DHIS2,", "date_modified": "2021-02-03T00:00:00.000Z", "author": { "name": "Jed Goldstein" }, "tags": [ "how-to", "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2020/12/09/upsert-in-dhis2", "content_html": "tl;dr: Lots of our users want to upsert tracked entity instances in dhis2, but\nupserts aren’t supported by a standard DHIS2 API endpoint. We built one in our\ndhis2 adaptor: it’s composed of existing APIs and a bit of logic 🤔. Now you can\nupsert tracked entity instances to DHIS2 👍 ✅.
An “UPSERT” is a portmanteau of the database functions UPDATE and INSERT. It’s\ncritical to handle upserts properly when integrating systems. As of version 35\nof the API, DHIS2 does not allow for an administrator to upsert tracked entity\ninstances (“TEIs”). OpenFn’s own\nChaiwa Berian has come up with a solution\nthat highlights the utility of helper functions in our dhis2 adaptor. By\ncombining various DHIS2 APIs through an upsertTEI function in OpenFn, DHIS2\nusers can now perform upserts to TEIs.
\nIf you’re curious, check out his implementation\nhere.
\nA tracked entity instance in DHIS2 is a type of entity that can be tracked\nthrough the system. It can be anything from a person to a commodity like a\nmedicine. If I am a database administrator presiding over two different systems\nthat are connected to one another, let’s call them “System A” and “System B,” I\nwould like for any updates made to the TEI of a user named “Jim Smith” in System\nA to also appear in Jim’s record in System B. Before upserts came about, doing\nso was difficult because of the possibility of duplicate record creation.\nBecause an upsert simultaneously UPDATES and INSERTS, it prevents duplicates.
\nUpserts are important and good because they cut down on the risk of duplicate\ndata entry and they also allow for transactions to be retried over and over to\nensure data integrity. That last bit is called “idempotency” and you can read\nabout it over here.
\nPlease don’t hesitate to reach out to one of OpenFn’s implementation specialists\nif you’d like to learn more.
\n— Taylor
\nSign up to set up a project today, absolutely\nfree.
\nReach out for more information.
", "url": "https://docs.openfn.org/es/articles/2020/12/09/upsert-in-dhis2", "title": "Tracked entity instances in DHIS2", "summary": "tl;dr: Lots of our users want to upsert tracked entity instances in dhis2, but", "date_modified": "2020-12-09T00:00:00.000Z", "author": { "name": "Taylor Downs" }, "tags": [ "how-to", "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2020/07/14/cron-is-better-than-a-timer", "content_html": "Hi all, this is a quick one from the product team at\nOpenFn — we've made a major upgrade to how timed/period\njobs work.
\nIn the past, if you weren't using OpenFn to drive some real-time (or\n\"event-based\") automation, you'd need to set up an \"interval trigger.\" Like the\nphoto above, this was essentially a sand timer. Set your trigger to 10 seconds\nand your job fetches data from DHIS2, some regional public health data set, or\nwhatever, then cleans, transforms, and loads it into some other system.
For the most part, this has got the job done for the last 5 years, but as our\nNGO and government clients came up with increasingly specific requirements on\nnot only how often but when a crucial job gets executed, we began finding\nourselves creating little customizations for them on a once-off basis. We're\nhappy to annouce that as of v1.75 (released today), you can now schedule jobs\nto run based on cron expressions, giving you incredible control over when your\ntasks get executed.
Using cron, you can choose to run a job every minute by typing * * * * *.
Or maybe you've got a batch sync that you want to take place while your users\nare asleep—why not run it every night at 11pm with 23 * * * *.
What if you've got to submit reuqests for medical inventory only during the\nonset of flu season? Simply type 0 0 1 2-4 * and your job will run at midnight\nthe 1st of the month, from February through April.
You can still run jobs at the click of a button and create timers with\nexpressions like */10 * * * * for \"every 10 minutes\", but scheduling with cron\ngives OpenFn.org users so much more control over how they run their\norganizations. (And that's a good thing.)
If you're keen on learning by doing but don't have an OpenFn account yet,\nsign up for free or mess around with cron\nexpressions at crontab.guru,\na brilliant site to quickly build complex cron expressions.
\nThat's all from product for today. Speak soon.
\nTaylor
", "url": "https://docs.openfn.org/es/articles/2020/07/14/cron-is-better-than-a-timer", "title": "Product News: Enhanced Scheduled/Periodic Job Control", "summary": "Hi all, this is a quick one from the product team at", "date_modified": "2020-07-14T00:00:00.000Z", "author": { "name": "Taylor Downs" }, "tags": [ "annoucement", "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2020/07/02/allow-yourself-to-fail", "content_html": "Hi all, this is a very short post with a simple message: design for failure.\nEven if you've never heard of\nMSSQL (or\nAzure, or Microsoft?), I want to talk for\none moment about the importance of upserts and a funny developer term called\n\"idempotence.\"
\nWe just extended our\nlanguage-mssql adaptor with a custom\nfunction that allows upserts (an upsert is when you either insert a new record\nor update an existing record based on some identifier). Before, you'd need to\nwrite something tedious like:
sql({
query: `MERGE my_table AS [Target]
USING (SELECT '8675309' AS some_unique_id, 'writing_blog_posts' AS skill) AS [Source]
ON [Target].some_unique_id = [Source].some_unique_id
WHEN MATCHED THEN
UPDATE SET [Target].some_unique_id=8675309, [Target].skill='writing_blog_posts'
WHEN NOT MATCHED THEN
INSERT (some_unique_id, skill) VALUES ([Source].some_unique_id, [Source].skill);`,
});
whereas now you can simply write:
\nupsert('my_table', 'some_unique_id', {
some_unique_id: 8675309,
skill: 'writing blog posts',
});
For an operation to be idempotent means that it can be repeated time and time\nagain without producing an unintended result. This is SUPER important for\ncreating S3 (Secure, Stable and Scalable—more on that\nhere) integrations because it provides you with two\n\"get-out-of-jail-free\" cards.
\nIf a destination application fails, if a connection times out, or if (for\nwhatever reason) you're not sure if the job was completed (say... making a\npayment to CHW) then an idempotent operation can be RETRIED without fear of\nmaking a double-payment.
If you make some change to how your job works, make some modification to\none of your destination systems, or just because you want to be extra extra\nsure that all the data in a 9 month survey made it to the national public\nhealth reporting system, you can REPROCESS every single message that's come\nthrough OpenFn at the click of a button, without having to worry about\nduplicates.
So... when clients let me mess around with their jobs, I always recommend we\ndesign for idempotence. It's common sense when you're passing messages between\ntwo different systems that are bound to evolve, go offline, have a bad day, etc.
\n— Taylor
\nSign up to set up a project today, absolutely\nfree.
\nReach out for more information.
", "url": "https://docs.openfn.org/es/articles/2020/07/02/allow-yourself-to-fail", "title": "Allow Yourself to Fail", "summary": "Hi all, this is a very short post with a simple message: design for failure.", "date_modified": "2020-07-02T00:00:00.000Z", "author": { "name": "Taylor Downs" }, "tags": [ "how-to", "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2020/06/24/three-questions-to-ask", "content_html": "Automation can save time, unlock critical resources, and enable scale–but it\ntypically requires investment to set up. Wondering whether you should automate\nyour processes? Ask yourself these 3 questions.
\nTo date, we have worked with 43 social sector organizations that operate across\nsectors–from health, education, and agriculture, to livelihoods and emergency\nresponse. Over the last 6 years, OpenFn has been implemented worldwide for a\nwide range of use cases, including building real-time data monitoring systems,\nstreamlining data cleaning pipelines, securely exchanging sensitive information,\nand automating routine processes like uploading indicator results, sending\nSMS/email alerts, making mobile payments,\nand more.
\nBy connecting any app, OpenFn can integrate and automate all apps within a\ndigital ecosystem. However, a question that we frequently ask our partners is:
\nWhile integration and automation have the potential to enable scale and save\ntime and money (we’ve learned this from our\npartners), solutions require investment to set up\nand maintain. These costs sometimes outweigh expected efficiency gains and\nservice outcomes. Therefore, when evaluating the cost-benefit of investing in\nautomation and integration solutions, we at Open Function Group typically ask 3\nkey questions.
\nCan the exposure of sensitive data be limited by integrating with secure API\nendpoints (rather than relying on human beings to interact with those data, for\nexample)? Or by automating a data cleaning process?
\nCan the process take place more reliably by limiting the opportunity for human\nerror (in automating data manipulation or simple algorithmic work, for example)?
\nCan the process be done more quickly via automation and is there value in having\nit done faster? (The answer to the first part is almost always yes, but\nsometimes there's not actually lots of value generated by doing something\nfaster.)
\nTasks that meet these 3 criteria, take a lot of time to complete or are very\nrepetitive, and/or involve moving data between apps are typically great\ncandidates for automation.
\nIf you find yourself answering “no”, then it may not be worth the investment in\nautomation... at least not yet. This is especially true if these processes are\nstill in flux or require a lot of human involvement to complete. That said, now\nmay be a good time to refine your existing workflows, think about how your\nprocesses might change at scale, and consider what new processes, services, or\noutcomes could be unlocked by automation.
\nIf you want to try out automation for your organization,\nsign up for OpenFn, free of charge. Check out\nour documentation and\nwebsite to learn how to get started.
\nHaving trouble setting up your first automation \"job\"? Email us at\nadmin@openfn.org for support. Our team is always\nhappy to assist and help you evaluate the total cost of ownership of automation\nsolutions.
", "url": "https://docs.openfn.org/es/articles/2020/06/24/three-questions-to-ask", "title": "To Automate or Not to Automate? Ask Yourself These 3 Questions.", "summary": "Automation can save time, unlock critical resources, and enable scale–but it", "date_modified": "2020-06-24T00:00:00.000Z", "author": { "name": "Aleksa Krolls" }, "tags": [ "tips" ] }, { "id": "https://docs.openfn.org/es/articles/2020/06/16/how-information-is-organized", "content_html": "This article was originally posted by Taylor Downs, Head of Product, on\nThe OpenFn Founder's blog as \"The power\nof crow's feet.\"
\nIt’s Saturday Morning in Cape Town and I’ve just spent an hour talking about how\na non-profit is organized. I thought I was getting into a technical\ndiscussion—I’ve been doing system architecture discussions for years—but what we\nended up talking about was how this NGO thinks.
\nThis engagement is largely about mapping an already existing “people & paper”\nbased system to technology. Vera Solutions will build a system for this client\nusing Open Data Kit for field data collection and Salesforce.com for the\nmanagement “back-end”. Because we’re not explicitly being asked to help redesign\nprocesses at this organization, the client is “telling us how things are”, then\nexpecting us to create a relational database model that facilitates\nbusiness-as-usual, only in a more efficient, digital way. Seems reasonable.
\nThis organization runs multiple programs focusing on a handful of strategic\nobjectives. They coordinate various activities in their target communities and\nreport on those activities against numerous (sometimes overlapping) indicators.\nSound familiar? As we get to the 3rd explanation of these programs, and the 11th\niteration of the system schema, it hits me…
\nWhen we—human beings—wrestle with complex problems (like managing lots of\nprograms, other humans, community stakeholders, etc.) we have the capacity to\ntrick ourselves into thinking that we have wrapped our heads around a system\n(for clinic registration, for after-school education, etc.) when, in fact, we’re\nengaging in mental hand-waving and are simply ‘papering-over’ sections which are\nsecretly not just complicated, but totally incongruous with other parts of the\nsystem. We can make ourselves believe that our logic is sound because we want it\nto be sound, when in reality the organization might be held together by good\npeople, not good, clear, defined processes. By learning a couple of key\nconcepts, it’s possible for non-technical people to articulate their thoughts\nclearly using “boxes” and “crow’s feet” and see whether or not there is an\nunderlying structure to their organization’s information.
\nIf you can’t diagram the information structure in your organization using boxes\nand crow’s feet, it’s a smell that something isn’t quite right (or at least that\nsomething isn’t easily scalable… more on this later!). Let me show you the tools\nin the toolbox and then wrap up by waxing poetic on people, processes, and\ntechnology.
\nThe box is my favorite. It represents an entity in your data system. Entities\n(like teachers) have attributes (like name, phone number, date of birth,\ngender, etc.) Some people like thinking of entities as simple forms. The\n“Teacher Registration Form” will ask for the teacher’s name, phone number,\ngender, etc. These are the fields on your teacher entity. By submitting one of\nthese forms, you’ll add a new teacher to your database. If you’re an Excel\nperson, the attributes are columns in your teachers table.
The crow’s foot is my second favorite. It’s used to show relationships between\nentities. We know that teachers are related to the sessions that they conduct.\n(And session might be another entity, with fields like date,\nsubject taught, and venue, to name just a few.) The crow’s foot allows us to\nspecify exactly how they are related. On that session entity, we’ll need to\nspecify the name (or ID) of the teacher who led it. On the teacher entity,\nhowever, there won’t be a field to specify the name or ID of the session…\nbecause a single teacher can lead MANY sessions. This is a one-to-many\nrelationship. The crow’s foot (that little three-pronged fella) denotes the\nmany. One teacher can have many sessions. One session, however, can only have\none teacher. See the diagram below.
![](\"https://docs.openfn.org/es/assets/images/objectmodel3-f8773ce38dd6bda67c896f6546ded810.webp\")
If we focus just on teacher and session and think back to MS Excel, we can\nenvision a teachers table and a sessions table. Let’s put them on different\nsheets in the same workbook. On the teachers table, there is no column for\nsession, but on the sessions table, there is a column for teacher ID.\nWe’ve just established a one-to-many relationship.
Next time, we’ll talk about what’s going with the attendance entity above.\nIt’s sometimes called a “junction object” or a “join table”, and it’s what\nallows MANY students to be related to MANY sessions. I’ll write more on this\nnext time, but there is no magic going on, no technicalities here. The way that\nmany students are related to many sessions is through this Real World Concept\nthat we call attendance. Attendance is what happens when a student shows up at\na session. It’s so important to get the language right in these discussions, and\nmake sure that you’re talking about real-world concepts.
Remember that as you start to put pen to paper. And allow yourselves time (and\nmultiple drafts) to get the boxes and terminology right. Understanding\nrelational object modelling is an incredibly powerful way to organize a company.\nAs I said before, if you can’t model it with boxes and arrows, it’s a smell that\nsomething might not be conceptually sound.
\nSome organizations do amazing work without good conceptual systems. They rely on\nhumans, instinct, improvisation, nouse, and other not-so-clearly-defined things.\nThey might do really great work. They might get the job done. But they need to\nbe well aware of their condition and face it head on. If you can’t systematize\nyour program implementation processes, then you need to focus tremendous effort\non finding and retaining the right people.
\nA friend once told me that “people are not scalable.” I couldn’t agree more, and\ndefend my earlier stance that if your organization’s information structure can’t\nbe defined with boxes and crow’s feet, it may be very hard for you to scale\nresponsibly. However, if you can create a ruthlessly efficient, world-class\n“people operations” system (recruitment, training, management, compensation, HR,\netc.) that ensures you’ve always got the right people to figure things out you\nmight be better off than those operating a well defined assembly-line with\ninterchangeable parts. Alas, the middle way is probably the best.
\nThat’s all for now. More soon.
\nNeed help organizing or scaling your organization's information or process\nflows? Contact our team of ICT4D specialists at\nsupport@openfn.org.
", "url": "https://docs.openfn.org/es/articles/2020/06/16/how-information-is-organized", "title": "How Information Is Organized... In Organizations", "summary": "Does your organization's information have an underlying structure? Try this exercise using boxes and crow's feet.", "date_modified": "2020-06-16T00:00:00.000Z", "author": { "name": "Taylor Downs" }, "tags": [ "how-to", "tips" ] } ] }