Ferry v2 - UNDER DEVELOPMENT

Installation

1. Ferry uses VSCode's Dev Container for the dev environment. All environment setup is handled through the standard Dev Container pipeline. For a quick start, use the Dev Containers: Reopen in Container command in VSCode.

2. MacOS: If you want to develop outside of the Dev Container, please install pygraphviz first before running pip install -e .

   pip install -U --no-cache-dir  \
               --config-settings="--global-option=build_ext" \
               --config-settings="--global-option=-I$(brew --prefix graphviz)/include/" \
               --config-settings="--global-option=-L$(brew --prefix graphviz)/lib/" \
               pygraphviz

Installing submodules

Ferry uses a submodule for the data repository. To clone the submodule, run:

git submodule update --init --depth 1

Any time in the future, you can update the submodule with:

git submodule foreach git reset --hard
git submodule update --rebase --remote

Then, you can commit the new submodule link to the main repo. This should be rarely done.

Running Ferry

Make sure you have all dependencies installed or are using the Dev Container.

python main.py -f config/dev_fetch.yml

Options

Options can be passed either through the command line, through a YAML config file, or through environment variables, in decreasing order of precedence.

Note: because we use argparse, you can provide just the prefix of each argument, such as --config instead of --config-file.

CLI flag	Config option	Env key	Default	Description
--cas-cookie	cas_cookie	CAS_COOKIE	None; prompt if crawl_evals	Only used for fetching evals; see below
-f, --config-file	N/A	N/A	None	Path to YAML config file, relative to PWD; if unspecified, all options are read from command
--data-dir	data_dir	N/A	data	Directory to load/store parsed data. This is usually where the ferry-data is cloned.
--database-connect-string	database_connect_string	POSTGRES_URI	None; prompt if sync_db	Postgres connection string; for dev, see dev_sync_db.yml
-d, --debug	debug	N/A	False	Enable debug logging
-r, --release	release	N/A	False	Run in release mode; see below
-s, --seasons	seasons	N/A	None	A list of seasons to fetch; see below
--sentry-url	sentry_url	SENTRY_URL	None; prompt if release	Sentry URL for error reporting; required in release mode, ignored in dev mode
--use-cache	use_cache	N/A	False; always False if release	Use cached data instead of fetching fresh data. Even if not using cache, cache will still be updated.

Stage switches

You can selectively run certain stages of Ferry. Options below are in the order of the stages. All stages are False by default, in which case Ferry basically does nothing.

CLI flag	Config option	Description
--save-config	save_config	Save the parsed config options to config_file; does nothing if config_file is unspecified.
--crawl-seasons	crawl_seasons	Run the seasons crawler. If False and no cached data is available, then --seasons must be provided and all be valid seasons.
--crawl-classes	crawl_classes	Run the class crawler.
--crawl-evals	crawl_evals	Run the eval crawler.
--transform	transform	Run the transformer. Always True if sync_db or snapshot_tables.
--snapshot-tables	snapshot_tables	Generate CSV files capturing data that would be written to DB.
--sync-db	sync_db	Sync the transformed data to the database.
--generate-diagram	generate_diagram	Generate a DB visualization diagram to docs/db_diagram.pdf

Release mode

In release mode:

• use_cache is always False; we always fetch fresh data.
• sentry_url is required; we will send error reports to Sentry.

In non-release mode:

• We never initialize Sentry and sentry_url is ignored.

CAS cookie

To get a valid CAS cookie to connect to OCE, first log into https://oce.app.yale.edu/ocedashboard/studentViewer. Refresh with the network inspector tab open, and find the first HTTP request. Copy the Cookie header from the request and paste into the prompt or the cas_cookie option. It should look like:

JSESSIONID=...; ...

Specifying seasons

• If the seasons option is unspecified, we default to fetching ALL viable seasons.
• If the seasons option contains a single value called LATEST_n, we fetch the latest n seasons.
• Otherwise, the seasons option should be a list of seasons to fetch, using the standard season code format: e.g. --seasons 202301 202303.

In almost all cases, it is sufficient to only fetch the last 3 seasons in dev. In fact, ferry will not work when fetching and syncing to DB all seasons in dev due to professor_id mapping requiring legacy seasons. Please clone ferry-data if working with all seasons is necessary.

Linting & formatting

black .
pyright

TODO

• Ensure that parallel evals is accurate
• Rearchitect GitHub action to create Postgres dump, upload with VC (enable lifecycle management) to Azure Blob, and restart prod postgres container to trigger new pull.
• Modernize dependencies
• SIGINT handler
• Write documentation, standardize variable names, cleanup, etc.
• Cleanup those branches
• Lint and format CI
• Parallelize db sync scripts (stage, transform, deploy)