Skip to content

A Singer.io tap to extract data from the Codat API

License

Notifications You must be signed in to change notification settings

AutoIDM/tap-codat

 
 

Repository files navigation

tap-codat

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

Quick Start

  1. Install

    pip install tap-codat

  2. Get an API key

    Refer to the Codat documentation here.

  3. Create the config file

    You must create a JSON configuration file that looks like this:

    {
      "start_date": "2010-01-01",
      "api_key": "your-api-key",
      "uat_urls": "true"
    }

    The start_date is the date at which the tap will begin pulling data for streams that support this feature. Note that in the initial version of this tap, this date is unused, as all streams replicate all of the data from Codat during every run.

    Replace your-api-key with the API key you received from Codat. If this token is for the UAT environment, change uat_urls to true.

  4. Run the Tap in Discovery Mode

    tap-codat -c config.json -d

    See the Singer docs on discovery mode here.

  5. Run the Tap in Sync Mode

    tap-codat -c config.json -p catalog-file.json

Data Formatting

For a few endpoints, this tap reformats the structure of "reports" received from Codat. An example is the balance_sheets stream, which returns a structure like this:

"reports": [
    {
        "assets": {
            "name": "Top-level Category",
            "value": 1,
            "items": [
                {"name": "Inner category A", "value": 2},
                {"name": "Inner category B", "value": 3}
            ]
        }
    }
]

Here, assets describes a hierarchical structure. It is recursive in that any of the items can themselves contain an array of items. This is not a structure that easily can fit into a flat, tabular structure. To alleviate this, this tap restructures this data into this format:

"reports": [
    {
        "assets": [
            {
                "name": "Top-level Category",
                "value": 1,
                "name_0": "Top-level Category"
            },
            {
                "name": "Inner category A",
                "value": 2,
                "name_0": "Top-level Category",
                "name_1": "Inner category A"
            },
            {
                "name": "Inner category B",
                "value": 3,
                "name_0": "Top-level Category",
                "name_1": "Inner category B"
            },
        ]
    }
]

The structure is flattened into a single array of objects. The "name" and "value" properties are left as-is for each item, but now each items contains properties "name_X" where X represents the category hierarchy. That is, if your category hierarchy is

A
- B
- - C
- D
E
- F

Then name_0 will always be either A or E, name_1 will always be B, D, or F and name_2 will only ever be C.


Meltano Steps

There's an included meltano.yml file that worked for a test account (Note there are multiple streams being ignored due to errors).

To get Meltano running

  1. pipx install meltano
  2. meltano install
  3. Set your api_key (create a .env file, and add TAP_CODAT_API_KEY="api_key_here")
  4. meltano invoke tap-codat

Copyright © 2017 Stitch

About

A Singer.io tap to extract data from the Codat API

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 100.0%