Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor brigade-project-index loader to use a gitsheets API #33

Open
5 tasks
themightychris opened this issue Sep 22, 2019 · 1 comment
Open
5 tasks
Assignees

Comments

@themightychris
Copy link
Member

themightychris commented Sep 22, 2019

This script makes really limited use of gitsheets as a node module to materialize some tables from evolving online data: https://github.com/codeforamerica/brigade-project-index/tree/loader

This illustrates a major early use case for gitsheets though: as a data backend for civic tools and services.

This is proof-of-concept level code, and it does a lot of stuff directly with git that could be made a lot more pleasant with some of the plumbing wrapped up in an expressive gitsheets module API

  • Get set up to run and debug the loader locally
  • Use gitsheets api to generate more useful commit messages on the index like 🔁 added 1, updated 3 projects from Open Savannah (this example might be accomplished by passing a template for the auto-commit-message generator or making a call to get the generated to use in a sep commit command)
  • Make other revisions to taste to the public gitsheets module API to make for more readable code in the consumer (the loader script)
  • Ship a minor version bump to the gitsheets backend to npm adding module API enhancements
  • Open a PR against the loader inside the cfa repo
@timwis
Copy link
Contributor

timwis commented Jan 10, 2020

So unless I'm underestimating the goal of the brigade-project-index project, I think we only need a couple changes to make it possible with the current high-level API:

  1. Support importing JSON data (right now only CSV is supported; if you convert organizations.json to CSV it works fine)
  2. Allow multiple datasets per repo. In this case, there will be an organizations dataset plus one dataset per organisation, in projects/{organization}/.
  3. Generate more descriptive commit messages, but this is a nice-to-have anyway, I imagine.

At the moment the consumer API looks like this:

const gitSheets = await GitSheets.create()
await gitSheets.setConfigItem('master', 'path', '{{name}}')

const orgsStream = fs.createReadStream('organizations.csv')
await gitSheets.import({
  data: orgsStream,
  dataType: 'csv',
  parentRef: 'master',
  saveToBranch: 'master'
})

It's probably worth sorting out #32 while we're at it but this is the current state.

I think you needed to do all the low-level stuff because (a) json import isn't supported, and (b) multiple datasets per repo isn't supported.

timwis added a commit that referenced this issue Jan 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants