DS_TechWriting_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Technical exercise - Data scientist intern (Technical Writing) @ Giskard\n",
    "\n",
    "Hi! As part of our recruitment process, we’d like you to complete the following test in 10 days. Once you finish the exercise, you can send your submission by email to luca@giskard.ai and jmjm@giskard.ai. If you want to share a private GitHub repository, make sure you give read access to `luca-martial` and `jmsquare`.\n",
    "\n",
    "If you have problems running the notebook, get in touch with Luca at luca@giskard.ai."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install numpy pandas scikit-learn datasets transformers torch \"giskard>=2.0.0b\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise TW1: Not bad reviews\n",
    "\n",
    "\n",
    "We trained a random forest model to predict if a film review is positive or negative. Here is the training code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "\n",
    "# Load training data\n",
    "train_data = datasets.load_dataset(\"sst2\", split=\"train[:20000]\").to_pandas()\n",
    "valid_data = datasets.load_dataset(\"sst2\", split=\"validation\").to_pandas()\n",
    "\n",
    "# Prepare model\n",
    "with open(\"stopwords.txt\", \"r\") as f:\n",
    "    stopwords = [w.strip() for w in f.readlines()]\n",
    "\n",
    "preprocessor = TfidfVectorizer(stop_words=stopwords, max_features=5000, lowercase=False)\n",
    "classifier = RandomForestClassifier(n_estimators=400, n_jobs=-1)\n",
    "\n",
    "model = Pipeline([(\"preprocessor\", preprocessor), (\"classifier\", classifier)])\n",
    "\n",
    "# Train\n",
    "X = train_data.sentence\n",
    "y = train_data.label\n",
    "\n",
    "model.fit(X, y)\n",
    "\n",
    "print(\n",
    "    \"Training complete.\",\n",
    "    \"Accuracy:\",\n",
    "    model.score(valid_data.sentence, valid_data.label),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, it works quite well, but we noticed it has some problems with reviews containing negations, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class labels are:\n",
    "# 1 = Positive, 0 = Negative\n",
    "\n",
    "# this returns positive, that’s right!\n",
    "assert model.predict([\"This movie is good\"]) == [1]\n",
    "\n",
    "# negative! bingo!\n",
    "assert model.predict([\"This movie is bad\"]) == [0]\n",
    "\n",
    "# WHOOPS! this ↓ is predicted as negative?! uhm…\n",
    "assert model.predict([\"This movie is not bad at all!\"]) == [1]\n",
    "\n",
    "# WHOOPS! this ↓ is predicted as negative?! why?\n",
    "assert model.predict([\"This movie is not perfect, but very good!\"]) == [1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you help us understand what is going on? Do you have any idea on how to fix it?\n",
    "You can edit the code above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise TW2: Write about Giskard\n",
    "\n",
    "The Giskard python library provides an automatic scanner to find weaknesses and vulnerabilities in ML models. \n",
    "\n",
    "You can find a quickstart here: https://docs.giskard.ai/en/latest/getting-started/quickstart.html\n",
    "\n",
    "Using this tool do the following:\n",
    "- identify some issues in the movie classification model mentioned in the previous exercise (TW1)\n",
    "- generate hypotheses about what is causing these issues\n",
    "- choose one of the issues you found and try to improve the model to mitigate or resolve it — just one, no need to spend the whole weekend over it!\n",
    "\n",
    "Write about this end-to-end testing and mitigation process and share your findings in blogpost format (aim for around 1000 words excluding code snippets). The blogpost submission should be shared as a Google Docs document. We then ask you to record a short demo video (1-5mns) of the blogpost you submit, to show your skills in oral communication."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}