qasper_generate_explanations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate explanations to qasper train examples to use it in the dynamic few-shot prompt\n",
    "\n",
    "This notebook uses GPT-3 for generating explanations to the examples of the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "qasper_train = json.load(open('data/qasper/qasper-train-v0.3.json','r'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "\n",
    "openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "\n",
    "def generate(prompt,max_tokens=1000, temperature=0):\n",
    "    response = openai.Completion.create(\n",
    "        engine=\"text-davinci-002\",\n",
    "        prompt=prompt,\n",
    "        temperature=temperature,\n",
    "        max_tokens=256,\n",
    "        top_p=1,\n",
    "        frequency_penalty=0,\n",
    "        presence_penalty=0\n",
    "    )\n",
    "\n",
    "    return response[\"choices\"][0]['text']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random pertube\n",
    "Insert unrelevant documents to the question list of context documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def random_pertube(full_text, evidence,n=1,attempt=1):\n",
    "    pertubations = []\n",
    "    for i in range(n):\n",
    "        section = random.sample(full_text,k=1)\n",
    "        # print(section)\n",
    "        paragraph = random.sample(section[0]['paragraphs'],k=1)\n",
    "        if paragraph[0] not in evidence and paragraph[0] != \"\":\n",
    "            pertubations.append(paragraph[0])\n",
    "        else:\n",
    "            pertubations = pertubations + random_pertube(full_text, evidence,n=1)\n",
    "\n",
    "    if len(pertubations) == 0 and attempt<5:\n",
    "        return random_pertube(full_text, evidence,n,attempt+1)\n",
    "        \n",
    "    return pertubations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qas = 0\n",
    "for k in list(qasper_train.keys()):\n",
    "    qas += len(qasper_train[k]['qas'])\n",
    "qas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "explained_questions = []\n",
    "for k in tqdm(qasper_train):\n",
    "    for qa in qasper_train[k]['qas']:\n",
    "        prompt=\"For each example, explain how each document is used to answer the question:\\n\\nExample 1:\\n\\n[Document 1]: In this section we describe a number of experiments targeted to compare the performance of popular named entity recognition algorithms on our data. We trained and evaluated Stanford NER, spaCy 2.0, and a recurrent model similar to BIBREF13 , BIBREF14 that uses bidirectional LSTM cells for character-based feature extraction and CRF, described in Guillaume Genthial's Sequence Tagging with Tensorflow blog post BIBREF15 .\\n\\n[Document 2]: Stanford NER is conditional random fields (CRF) classifier based on lexical and contextual features such as the current word, character-level n-grams of up to length 6 at its beginning and the end, previous and next words, word shape and sequence features BIBREF16 .\\n\\n[Document 3]: spaCy 2.0 uses a CNN-based transition system for named entity recognition. For each token, a Bloom embedding is calculated based on its lowercase form, prefix, suffix and shape, then using residual CNNs, a contextual representation of that token is extracted that potentially draws information from up to 4 tokens from each side BIBREF17 . Each update of the transition system's configuration is a classification task that uses the contextual representation of the top token on the stack, preceding and succeeding tokens, first two tokens of the buffer, and their leftmost, second leftmost, rightmost, second rightmost children. The valid transition with the highest score is applied to the system. This approach reportedly performs within 1% of the current state-of-the-art for English . In our experiments, we tried out 50-, 100-, 200- and 300-dimensional pre-trained GloVe embeddings. Due to time constraints, we did not tune the rest of hyperparameters and used their default values.\\n\\n[Document 4]: In order to evaluate the models trained on generated data, we manually annotated a named entities dataset comprising 53453 tokens and 2566 sentences selected from over 250 news texts from ilur.am. This dataset is comparable in size with the test sets of other languages (Table TABREF10 ). Included sentences are from political, sports, local and world news (Figures FIGREF8 , FIGREF9 ), covering the period between August 2012 and July 2018. The dataset provides annotations for 3 popular named entity classes: people (PER), organizations (ORG), and locations (LOC), and is released in CoNLL03 format with IOB tagging scheme. Tokens and sentences were segmented according to the UD standards for the Armenian language BIBREF11 .\\n\\n[Document 5]: The main model that we focused on was the recurrent model with a CRF top layer, and the above-mentioned methods served mostly as baselines. The distinctive feature of this approach is the way contextual word embeddings are formed. For each token separately, to capture its word shape features, character-based representation is extracted using a bidirectional LSTM BIBREF18 . This representation gets concatenated with a distributional word vector such as GloVe, forming an intermediate word embedding. Using another bidirectional LSTM cell on these intermediate word embeddings, the contextual representation of tokens is obtained (Figure FIGREF17 ). Finally, a CRF layer labels the sequence of these contextual representations. In our experiments, we used Guillaume Genthial's implementation of the algorithm. We set the size of character-based biLSTM to 100 and the size of second biLSTM network to 300\\n\\nQuestion: what ner models were evaluated?\\n\\nAnswer: Stanford NER algorithm, the spaCy 2.0 algorithm, recurrent model with a CRF top layer.\\n\\nExplanation: According to [Document 1], the Stanford NER algorithm, the spaCy 2.0 algorithm, and a recurrent model with a CRF top layer were evaluated. This information is further supported by [Document 2], [Document 3], and [Document 5].\\n\\nExample 2:\\n\\n\"\n",
    "\n",
    "        answer_info = qa['answers'][0]['answer']\n",
    "        if answer_info[\"unanswerable\"]:\n",
    "            answer = \"Unanswerable\"\n",
    "        else:\n",
    "            if answer_info[\"extractive_spans\"]:\n",
    "                answer = \", \".join(answer_info[\"extractive_spans\"])\n",
    "                answer_type = \"extractive\"\n",
    "            elif answer_info[\"free_form_answer\"]:\n",
    "                answer = answer_info[\"free_form_answer\"]\n",
    "                answer_type = \"abstractive\"\n",
    "            elif answer_info[\"yes_no\"]:\n",
    "                answer = \"Yes\"\n",
    "                answer_type = \"boolean\"\n",
    "            elif answer_info[\"yes_no\"] is not None:\n",
    "                answer = \"No\"\n",
    "                answer_type = \"boolean\"\n",
    "        evidence = [text for text in answer_info[\"evidence\"] if \"FLOAT SELECTED\" not in text]\n",
    "\n",
    "        pertubed_evidence = evidence + random_pertube(qasper_train[k][\"full_text\"],evidence)\n",
    "\n",
    "        random.shuffle(pertubed_evidence)\n",
    "        \n",
    "        for i, d in enumerate(pertubed_evidence):\n",
    "            prompt += \"[Document {0}]: {1}\\n\\n\".format(i+1, d)\n",
    "        prompt += \"Question: {0}\\n\\nAnswer: {1}\\n\\nExplanation:\".format(qa['question'], answer)\n",
    "\n",
    "        res = generate(prompt)\n",
    "\n",
    "        explained_questions.append({\n",
    "            \"question\": qa['question'],\n",
    "            \"evidence\": pertubed_evidence,\n",
    "            \"answer\": answer,\n",
    "            \"explanation\": res\n",
    "        })\n",
    "json.dump(explained_questions, open(\"./data/qasper/explained_train.json\",'w'))"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}