kiba-extend project sample

A sample base project using kiba-extend.

Includes heavily commented/explained code for:

• configuring your project

• registering your project’s jobs/files

• defining the jobs

• project-specific kiba transforms

  • defining

  • testing

  • using in jobs

Table of Contents

• Prerequisites
• Try it out
  • Set it up
  • Try some commands
  • Run the tests
  • Explore the code and the data
• Start your own project
• More documentation

Prerequisites

You must have a modern Ruby installed. This should work with 2.7.4 and up.

💡	It is highly recommended you use a version manager. The author is using rbenv.

Try it out

Set it up

Clone this repository.

cd into the top level of your repository.

Do bundle install

Try some commands

Now you should be able to run the thor tasks for the project. In your terminal, get a list of the project’s defined files/jobs:

thor reg:list

Run the auth__loc__json job (which has the effect of running all other jobs as its dependencies):

thor run:job auth__loc__json

More about the thor CLI here.

Run the tests

rspec

or

bundle exec rake spec

Explore the code and the data

The code is heavily commented to explain what things are doing.

To understand the relationship between lib/ke_project/registry_data.rb and setting up jobs, I recommend you start by looking at that file and lib/ke_project/target_system/locations.rb, which has more comments explaining this. Then look at lib/ke_project/source_system/locations.rb for a variant approach.

The source and destination files are in the specified locations, so you can see the results of the example jobs.

Start your own project

Create a repository for your project using this repository as a template.

In your new repository, global find/replace KeProject and ke_project with your project name.

ℹ️	Ruby conventions for naming Camel case is for names of modules and classes in your code. When naming files that correspond to said modules or classes, you downcase and separate with underscore instead of camel casing.

Run bundle install in your repo directory.

Configure your project settings in your lib/ke_project.rb equivalent.

Start setting up and running jobs!

ℹ️

Code organization With the exception of registry_data.rb and util.rb, the structure of directories and files in lib/ke_project is completely arbitrary. You could put all your project code in one big .rb file if you wanted to. The author has thus far found it useful to set up the structure has shown here, which follows the basic organization within the top-level module for the project: a module for each data source system a submodule for each spreadsheet or table or other meaningful grouping in source system, named after said table/file. This is used for the initial cleanup/prep of that data---removing fields that won’t be used, merging in values for ID lookups, etc. a module for the target system a submodule for each type of record/data that will migrate into the target system. Reshaping or combination of source data into the format required by target system In reality the distinctions between the kind of logic in the source system and target system submodules will not be clear at all. I’m leaning more and more toward putting almost everything in the source system code, and using the target system submodules to handle any final field renaming, removal of non-importing columns, ordering of columns, and global find/replaces (changing "%CR%" to "\n\n", etc. Each submodule file may contain the methods for a bunch of different jobs.

More documentation

• kiba-extend documentation

  • transforms

    • documentation

    • a lot of the documentation still needs to be written, but all the examples in the docs are tested in the kiba-extend spec. Consult the tests for examples of what each undocumented transform does.

  • data registry entry reference