Skip to main content

bigquery Adaptor developer README.md

Source: https://github.com/OpenFn/adaptors/tree/main/packages/bigquery

Language BigQuery

Language Pack for building expressions and operations to make HTTP calls.

Documentation​

state.json​

The configuration key must be a valid GCP credential as JSON.

{
  "configuration": {
    "type": "service_account",
    "project_id": "some-project",
    "private_key_id": "670b9e3c8c366e83aa569dd57cbfc5c575b72e42",
    "private_key": "-----BEGIN PRIVATE KEY-----\nblah\nmoreblah=\n-----END PRIVATE KEY-----\n",
    "client_email": "bigquery@some-project.iam.gserviceaccount.com",
    "client_id": "someid",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/bigquery%40some-project.iam.gserviceaccount.com"
  },
  "data": { "a": 1 }
}

operations (expression.js)​

load(...)​

Load data to a table in bigquery from a CSV. See full options here.

load(
  '/home/taylor/Desktop/type-C_r-ALL_ps-201012_freq-M_px-HS_pub-20200731_fmt-csv_ex-20200818.csv',
  'awesome-solutions-project', // project
  'test01', // dataset
  'mytable', // table
  {
    schema:
      'classification:STRING,year:STRING,period:STRING,period_desc:STRING,aggregate_level:STRING,is_leaf_code:STRING,trade_flow_code:STRING,trade_flow:STRING,reporter_code:STRING,reporter:STRING,reporter_iso:STRING,partner_code:STRING,partner:STRING,partner_iso:STRING,commodity_code:STRING,commodity:STRING,qty_unit_code:STRING,qty_unit:STRING,qty:INTEGER,netweight_kg:INTEGER,trade_value:INTEGER,flag:STRING',
    schemaUpdateOptions: ['ALLOW_FIELD_ADDITION'],
    // createDisposition: 'CREATE_IF_NEEDED',
    writeDisposition: 'WRITE_APPEND',
    skipLeadingRows: 1,
  } // loadOptions
);

Development​

Clone the adaptors monorepo. Follow the Getting Started guide inside to get set up.

Run tests using pnpm run test or pnpm run test:watch

Build the project using pnpm build.

To just build the docs run pnpm build docs