FS-Tox: A Small Molecule Toxicity Benchmark

🔎 Overview

We are building FS-Tox: a toxicity benchmark for small molecule toxicology assays. Toxicity prediction tasks differ from traditional machine learning tasks in that there are usually only a small number of training examples per toxicity assay. Here, we provide a few-shot learning dataset built using several publicly available toxicity datasets (e.g. EPA's ToxRefDB), and an associated benchmarking pipeline. We will incorporate the different assays from these datsets consisting of the molecular representation of a small molecule, with an associated binary marker of whether the drug was toxic or not for the given assay.

🗺️ Roadmap

Mid-May 2023 - benchmark SOTA models

Test the performance of the following state-of-the-art few-shot prediction methods on existing toxicity benchmark:

• Gradient Boosted Random Forest (XGBoost)
• Text-embedding-ada-002 on SMILES (OpenAI)
• [] Galactica 125M (Hugging Face)
• [] Galactica 1.3B (Hugging Face)
• ChemGPT 19M (Hugging Face)
• [] ChemGPT 1.2B (Hugging Face)
• [] Uni-Mol (docker)
• [] Uni-Mol+ (docker)
• [] MoLeR (Microsoft)

Late-May 2023 - create FS-Tox benchmarking tool

Incorporate the following datsets containing results from in vivo toxicity assays:

• [] ToxRefDB (subacute and chronic toxicity)
• [] TDCommon, Zhu 2009 (acute toxicity)
• [] MEIC (small, curated clinical toxicity)

Early-June 2023 - benchmark SOTA small molecule language models on FS-Tox

Test the following language models on the FS-Tox benchmark:

• [] Text-embedding-ada-002 on SMILES (OpenAI)
• [] Galactica 125M (Hugging Face)
• [] Galactica 1.3B (Hugging Face)
• [] ChemGPT 19M (Hugging Face)
• [] ChemGPT 1.2B (Hugging Face)
• [] Uni-Mol (docker)
• [] Uni-Mol+ (docker)
• [] MoLeR (Microsoft)

Mid-June 2023 - extend FS-Tox with in vitro data

Incorporate in vitro assays into the FS-Tox benchmark:

• [] ToxCast
• [] Extended Tox21
• [] NCI60 data

📂 Project Organization

├── LICENSE
├── Makefile           <- Makefile with commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── docs               <- A default Sphinx project; see sphinx-doc.org for details
│
├── models             <- Trained and serialized models, model predictions, or model summaries
│
├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
│                         and a short `-` delimited description, e.g.
│                         `1.0-initial-data-exploration`.
│
├── references         <- Data dictionaries, manuals, and all other explanatory materials.
│
├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures        <- Generated graphics and figures to be used in reporting
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
│
├── setup.py           <- makes project pip installable (pip install -e .) so src can be imported
├── src                <- Source code for use in this project.
│   ├── __init__.py    <- Makes src a Python module
│   │
│   ├── data           <- Scripts to download or generate data
│   │   └── make_dataset.py
│   │
│   ├── features       <- Scripts to turn raw data into features for modeling
│   │   └── build_features.py
│   │
│   ├── models         <- Scripts to train models and then use trained models to make
│   │   │                 predictions
│   │   ├── predict_model.py
│   │   └── train_model.py
│   │
│   └── visualization  <- Scripts to create exploratory and results oriented visualizations
│       └── visualize.py
│
└── tox.ini            <- tox file with settings for running tox; see tox.readthedocs.io

📚 Resources

1. ToxRefDB version 2.0: Improved utility for predictive and retrospective toxicology analyses
2. ChemGPT: a transformer model for generative molecular modeling