Skip to content

Latest commit

 

History

History
198 lines (135 loc) · 13.7 KB

CONTRIBUTING.md

File metadata and controls

198 lines (135 loc) · 13.7 KB
Raw

How to contribute?

Before you start

If you work with statistics but are not a coder or a developer used to the routine below, or you are just willing to learn, share ideas and catch-up, join us in the Telegram Open Group. This group keeps its language in English in order to be internationally open and we make Telegram the official channel to make it easy for non-developers to reach the technical group.

Also you should read this article, it explains how each part of Serenata de Amor works and how they all come together as whole. After reading it, you'll have a pretty good understanding of all the tools (Jarbas, Rosie and the toolbox) that we use — we’ll refer to them below, so it’s nice to have an idea about what we’re talking about ;-)

The basics

A lot of discussions about ideas take place in the Issues section. There you can catch up with what's going on and also suggest new ideas.

  1. Fork this repository
  2. Clone the repository: git clone http://github.com/[YOUR_GITHUB_USER]/serenata-de-amor.git
  3. Create your branch: $ git checkout -b new-stuff
  4. Commit your changes: $ git commit -am 'My cool contribution'
  5. Push to the branch to your fork: $ git push origin new-stuff
  6. Create a new Pull Request

Environment

Local Installation Environment (without Docker)

The recommended way of setting your environment up is with Anaconda, a Python distribution with useful packages for Data Science.

The project uses Python 3 (specified in the python=3 bit of the commands below). Download Anaconda and create an environment for the project.

You can use either Anaconda 2 and Anaconda 3, since conda environments manage Python versions. But if you're going to do a fresh install, we recommend Anaconda 3 — it's newer and it makes no sense to have Python 2 as your default Anaconda Python.

$ cd serenata-de-amor
$ conda update conda
$ conda create --name serenata_de_amor python=3
$ source activate serenata_de_amor
$ ./setup

The activate serenata_de_amor command must be run every time you enter the project folder to start working.

Common (virtual) environment issues

In some environments Jupyter might not be able to access packages from the Conda environment (basically… for some reason). If this is the case we recommend you to ensure you are using Anaconda 4.1.0 or higher, delete the old environment and follow these steps:

$ rm -rf ~/<path to your anaconda>/envs/serenata_de_amor
$ conda update conda
$ conda create --name serenata_de_amor python=3 ipykernel
$ source activate serenata_de_amor
$ conda install notebook ipykernel
$ ipython kernel install --user
$ conda install jupyter
$ ./setup

Pyenv users

If you installed Anaconda via pyenv, source activate serenata_de_amor will probably fail unless you explicitly use the path to the Anaconda activate script. For example:

$ source /usr/local/var/pyenv/versions/anaconda3-4.1.1/bin/activate serenata_de_amor

Docker Installation Environment

You can user Docker and Docker Compose to have a working environment:

  1. Start the environment (it might take a while, the base image has 5.8GB and we also pull in lots of dependencies): $ docker-compose up -d
  2. Create your config.ini file from the example: $ cp config.ini.example config.ini
  3. Run the script to download data and other useful files: $ docker-compose run --rm jupyter python src/fetch_datasets.py
  4. You can start Jupyter Notebooks and access them at localhost:8888: $ docker-compose run --rm jupyter jupyter notebook

If you want to access the console:

$ docker-compose run --rm jupyter bash

Best practices

In order to avoid tons of conflicts when trying to merge Jupyter Notebooks, there are some guidelines we follow.

That said one of our best practices is creating a .html and a .py version of each notebook. This can be done automatically and painlessly by editing the config file ~/.jupyter/jupyter_notebook_config.py and adding the folowing code:

### If you want to auto-save .html and .py versions of your notebook:
# modified from: https://github.com/ipython/ipython/issues/8009
import os
from subprocess import check_call

def post_save(model, os_path, contents_manager):
    """post-save hook for converting notebooks to .py scripts"""
    if model['type'] != 'notebook':
        return # only do this for notebooks
    d, fname = os.path.split(os_path)
    check_call(['jupyter', 'nbconvert', '--to', 'script', fname], cwd=d)
    check_call(['jupyter', 'nbconvert', '--to', 'html', fname], cwd=d)

c.FileContentsManager.post_save_hook = post_save

Beyond that we have five big directories with different purposes:

Directory Purpose File naming
develop/ This is where we explore data, feel free to create your own notebook for your exploration. [ISO 8601 date]-[author-username]-[2-4 word description].ipynb (e.g. 2016-05-13-anaschwendler-air-tickets.ipynb)
report/ This is where we write up the findings and results, here is where we put together different data, analysis and strategies to make a point, feel free to jump in. Meaningful title for the report (e.g. Transport-allowances.ipynb
src/ This is where our auxiliary scripts lie: code to scrap data, to convert stuff, etc. Small caps, no special character, - instead of spaces.
data/ This is not supposed to be committed, but it is where saved databases will be stored locally (scripts from src/ should be able to get this data for you); a copy of this data will be available elsewhere (just in case). Date prefix, small caps, no special character, - instead of spaces, preference for .xz compressed CSV (YYYY-MM-DD-my-dataset.xz).
docs/ Once a new subject, theme or datset is added to project, would be nice to have some documentation describing these items and how others can use them. Small caps whenever possible, no special character, - instead of spaces, preference for .md Markdown files.

The toolbox and our the source files (src/)

Here we explain what each script from src/ and the serenata_toolbox do for you:

One toolbox to rule them all

With the toolbox you can download, translate and convert the dataset from XML to CSV. You can chec the toolbox docs too.

When you run our setup, the toolbox is installed and all our datasets are downloaded to your data/ directory. This is handled by these two single lines:

from serenata_toolbox.datasets import fetch_latest_backup
fetch_latest_backup('data/')
Quota for Exercising Parliamentary Activity (CEAP)
  1. src/group_receipts.py creates a data/YYYY-MM-DD-reimbursements.xz file with grouped data from all of the available datasets (data/YYYY-MM-DD-current-year.xz, data/YYYY-MM-DD-last-year.xz and data/YYYY-MM-DD-previous-years.xz)
  2. src/translation_table.py creates a data/YYYY-MM-DD-ceap-datasets.md file with details of the meaning and of the translation of each variable from the Quota for Exercising Parliamentary Activity datasets.
Suppliers information (CNPJ)
  1. src/fetch_cnpj_info.py iterates over the CEAP datasets looking for supplier unique documents (CNPJ) and creates a local dataset with each supplier info.
  2. src/clean_cnpj_info_dataset.py clean up and translate the supplier info dataset.
  3. src/geocode_addresses.py iterates over the supplier info dataset and add geolocation data to it (it uses the Google Maps API set in config.ini).
Miscellaneous
  1. src/backup_data.py uploads files from data/ to an Amazon S3 bucket set on config.ini .
Politician's relatives
  1. src/get_family_names.py gets the names of the parents of congresspeople from the congress website and saves them to data/YYYY-MM-DD-congressperson_relatives.xz (and it may save some data to data/YYYY-MM-DD-congressperson_relatives_raw.xz in case it fails to parse the names)
Federal Budget
  1. src/fetch_federal_budget_datasets.py downloads datasets files of agreements made with Federal Budget and their related amendments. The script gets the lastest version available for each dataset, unpacks, translates columns to english and saves them into data/. The files are named as follows:
  • Agreements: YYYY-MM-DD-agreements.xz
  • Amendments: YYYY-MM-DD-amendments.xz
Companies and Non-Profit Entities with sanctions (CEIS, CEPIM and CNEP).
  1. src/fetch_federal_sanctions.py downloads all three datasets files (CEIS, CEPIM and CNEP) from official source. The script gets the lastest version available for each dataset, unpacks, translates columns to english and saves them into data/. The files are named as follows:
  • CEIS: YYYY-MM-DD-inident-and-suspended-companies.xz
  • CEPIM: YYYY-MM-DD-impeded-non-profit-entities.xz
  • CNEP: YYYY-MM-DD-national-register-punished-companies.xz

Datasets (data/)

Here we explain what are the datasets inside data/. They are not part of this repository, but can be downloaded with the toolbox. Most files are .xz compressed CSV. All files are named with a ISO 8601 date suffix.

  1. data/YYYY-MM-DD-current-year.xz, data/YYYY-MM-DD-last-year.xz and data/YYYY-MM-DD-previous-years.xz: Datasets from the Quota for Exercising Parliamentary Activity; for details on its variables and meaning, check data/YYYY-MM-DD-ceap-datasets.md.
  2. data/datasets-format.html: Original HTML in Portuguese from the Chamber of Deputies explaining CEAP dataset variables.
  3. data/YYYY-MM-DD-ceap-datasets.md: Table comparing contents from data/YYYY-MM-DD-datasets_format.html and our translation of variable names and descriptions.
  4. data/YYYY-MM-DD-companies.xz: Dataset with suppliers info containing all the fields offered in the Federal Revenue alternative API and complemented with geolocation (latitude and longitude) gathered from Google Maps.
  5. data/YYYY-MM-DD-congressperson_relatives.xz contains data on the relatives of congresspeople and the nature of their relationship.
  6. data/YYYY-MM-DD-congressperson_relatives_raw.xz also contains data on the relatives of congresspeople, but is only created if src/get_family_names.py fails to handle some names.

Four moments

The project basically happens in four moments, and contributions are welcomed in all of them:

Moment Description Focus Target
Possibilities To structure hypotheses and strategies taking into account (a) the source of the data, (b) how feasible it is to get this data, and (c) what is the purpose of bringing this data into the project. Contributions here require more sagacity than technical skills. GitHub Issues
Data collection Once one agrees that a certain possibility is worth it, one might want to start writing code to get the data (these scripts go into src/). Technical skills in scrapping data and using APIs. src/, data/ and docs/
Exploring Once data is ready to be used, one might want to start exploring and analyzing it. Here what matters is mostly data science skills. develop/
Reporting Once a relevant finding emerges from the previous stages, this finding might be gathered with other similar findings (e.g. put together explorations on airline tickets, car rentals and geolocation under a report on transportation) on a report. Contributions here require good communication skills and very basic understanding of quantitative methods. report/

More about the Quota for Exercising Parliamentary Activity (CEAP)

If you read Portuguese there is the official page with the legal pieces defining the quota and also a human version of the main text we made.

Also you can find more about the dataset variables in Jarbas or in data/YYYY-MM-DD-ceap-datasets.md that was downloaded when you ran the setup.

More about Federal Budget

As a secondary goal, some datasets related to Federal Budget and its uses were analyzed crossing them with datasets of inident and suspect companies that have suffered some sanction by Federal Government and are suspended from entering into any type of contract with Federal Government during sactions.

It is a work in progress as other datasets can be downloaded from SICONV and documentation can also be improved.

You can read more about these datasets at:

The notebook with the analysis are:

  • 2016-12-12-marcusrehm-federal-budget-companies-with-sanctions.ipynb
  • 2017-01-15-marcusrehm-congressperson-reimbursements-from-companies-with-sanctions.ipynb