keyword_precision_eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1d95355b-76b2-4abf-8e8b-95710b35db70",
   "metadata": {
    "tags": []
   },
   "source": [
    "# KW eval - precision\n",
    "* Inspired by [RAGAS](https://github.com/explodinggradients/ragas)\n",
    "* Utilizing GPT-4 prompt eval for keyword extraction precision.\n",
    "* Key point is this is all done without any golden set or human annotation..\n",
    "* KeyBert scores 0.5 precision - however this is whithout and cut off set.\n",
    "* GPT4 using a very simple prompt scores 0.786 (cut off is not applicable).\n",
    "* Next steps include implementing a recall and F1 score then perhaps some other new custom score of some sort.\n",
    "* Once there are in place it will be interesting to test various KW extraction prompts and see how they perform.\n",
    "* The biggest drawback of this type of evaluation is it's slow and perhaps can be expensive. Perhaps finetuning an opensource model may resolve this in the near future.\n",
    "* Also, while I was able to get the precision prompt to work reasonably well it is difficult to instruct it to do exactly what I want in the evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01d01885-faef-4c04-92d9-b434e379b8fd",
   "metadata": {},
   "source": [
    "## Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1f52abf9-5525-46f1-ab2f-6f46a4b342f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install keybert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6031e509-f908-4c9b-962a-7d55e1ed3b42",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eadcca06-30f7-4a76-bcd5-2739ded31045",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bf496262-f415-4538-afcd-82b8fafc9dbd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d35c75a7-3707-4530-96a8-0276b6bb233f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install python-dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "127fb0be-5a8c-43bb-91de-05073d0b95ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cc7207e5-5f09-412d-a56a-55f493b050d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !conda install -y fastparquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c3cb41aa-c606-476d-818b-54aee33be5d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import random\n",
    "from tqdm import tqdm\n",
    "import ast\n",
    "from random import sample\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b89fedbf-d2c5-4d27-ae91-c576de5d2d2d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import the LangChain libraries\n",
    "from langchain.llms import OpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.chat_models import ChatOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1d222d30-8971-48dd-98a4-fe54a66a64a0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "load_dotenv(override=True)\n",
    "\n",
    "import openai\n",
    "openaikey = os.getenv('OPENAI_API_KEY')\n",
    "openai.api_key = openaikey"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f6b254-dd6c-49ac-a9e8-d916b5cf15b6",
   "metadata": {},
   "source": [
    "## Precision metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056dd0ae-c4a5-426a-8ba2-811a71d3b3be",
   "metadata": {},
   "source": [
    "### Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bf47f7d0-6ff5-42ed-b7a2-76fe264f8ac2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "kw_precision_prompt = \"\"\"Given a document and proposed keywords, identify the good and bad keywords following these steps. \n",
    "\n",
    "Steps\n",
    "* start by generating the main keyword phrases following the below guidlines.\n",
    "* Compare the proposed keywords with the generated keywords.\n",
    "* Mark as good any keywords that are very close matches and for these good:\"yes\"\n",
    "* Read the guidlines <guidlines>.\n",
    "* Use the guildines to determine which proposed keywords are good vs bad.\n",
    "* Take a breadth and reevaluate your decisions to make sure they are correct.\n",
    "* Return the results of your consternation.\n",
    "\n",
    "<guidlines>\n",
    "General:\n",
    "In some cases, all keywords will be bad. If there aren't any good keywords that fully conform to all guidelines then reject all of them. Make sure all proposed keywords are concise, clear, and unambiguous keywords rather than phrases or fragments. You must compare each keyword to the ideal keyword that should be assigned to this document using the guidelines.\n",
    "\n",
    "Features of Good Keywords for Document Classification:\n",
    "- Relevance: They should directly relate to the core content or theme of the document. Relevant keywords ensure that the document is correctly categorized and easily retrievable. When assigning keywords, ensure they reflect the primary focus and recurring themes of the document, rather than isolated incidents or secondary details.\n",
    "- Contextual Relevance: Ensure the keywords capture the main theme or unique aspects of the document. They should reflect the core message or the most significant information presented. For instance, if the document discusses the impact of vermin on grain stocks, a keyword like \"vermin impact on agriculture\" would be contextually relevant.\n",
    "- Specificity: Keywords should be specific enough to distinguish the document's unique aspects. Generic terms may lead to misclassification or difficulty in locating the document among numerous others. Choose keywords that are specific to the content of the document. Avoid broad or generic terms that could apply to many different contexts.\n",
    "- Consistency: Using standardized terms within a given classification system or subject area promotes uniformity. This aids in linking similar documents and facilitates systematic retrieval.\n",
    "- Clarity: Keywords should be clear and unambiguous. This minimizes confusion and ensures that users with different levels of expertise or background knowledge can understand and use them effectively.\n",
    "- Comprehensiveness: They should cover all major themes or topics in the document. This ensures that the document can be found under all relevant searches.\n",
    "- Conciseness: While being comprehensive, keywords should also be concise, avoiding overly long or complex phrases that may not be commonly used or searched for.\n",
    "- Predictive Power: Good keywords anticipate the terms users might use when searching for that type of document, aligning with potential search queries.\n",
    "- Balance Specificity with Breadth: While being specific, also ensure that the keywords are not so narrow that they miss other important aspects of the document. They should collectively provide a balanced overview of the document’s content.\n",
    "\n",
    "\n",
    "Features of Bad Keywords for Document Classification:\n",
    "- Too Broad or Vague: Keywords that are overly general can lead to a document being lost among too many irrelevant search results, making it hard to locate.\n",
    "- Too Obscure or Technical: Using highly specialized or niche terms that are not widely recognized can make a document inaccessible to a broader audience.\n",
    "- Irrelevant: Keywords that don't accurately reflect the document's content can mislead users and result in poor retrieval accuracy.\n",
    "- Inconsistent Usage: Using different terms for the same concept across documents can lead to fragmentation and difficulty in retrieving all relevant documents on a topic.\n",
    "- Redundancy: Overlapping or repetitive keywords add no value and can clutter the classification system. Avoid Redundancy. Don't use multiple keywords that convey the same idea. Each keyword should add new information or a different perspective on the document's content.\n",
    "- Ambiguity: Ambiguous terms can lead to misinterpretation and misclassification, especially in documents covering multiple topics.\n",
    "- Too Lengthy: Long phrases or sentences used as keywords can complicate the search process and are often overlooked in favor of shorter, more direct terms. Choose keywords that unambiguously relate to the document’s content. Avoid words that have multiple meanings or could be easily misinterpreted out of context.\n",
    "</guidlines>\n",
    "\n",
    "Document: Supervised learning is the machine learning task of learning a function that\n",
    "maps an input to an output based on example input-output pairs. It infers a\n",
    "function from labeled training data consisting of a set of training examples.\n",
    "In supervised learning, each example is a pair consisting of an input object\n",
    "(typically a vector) and a desired output value (also called the supervisory signal).\n",
    "A supervised learning algorithm analyzes the training data and produces an inferred function,\n",
    "which can be used for mapping new examples. An optimal scenario will allow for the\n",
    "algorithm to correctly determine the class labels for unseen instances. This requires\n",
    "the learning algorithm to generalize from the training data to unseen situations in a\n",
    "'reasonable' way (see inductive bias)\n",
    "Proposed keywords: supervised learning, supervised, signal supervised, labeled training\n",
    "\n",
    "Accuracy:\n",
    "[\n",
    "    {{  \"keyword\":\"supervised learning\"\n",
    "        \"reason\": \"This is the main topic of this document. The document is a definition of this keyword\",\n",
    "        \"good\": \"Yes\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"supervised\"\n",
    "        \"reason\": \"This single word is ambiguous, lacks precision, clarity, and specificity. It could be easily misinterpreted.\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"signal supervised\"\n",
    "        \"reason\": \"This is not a normally used phrase. It's a sentence fragment.\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"Training data\"\n",
    "        \"reason\": \"This is described in the document within the context of supervised learning. It is clear, specific and non-ambiguous, and relevant to the document.\",\n",
    "        \"good\": \"Yes\"\n",
    "    }},\n",
    "]\n",
    "\n",
    "Document: The 2022 ICC Men's T20 World Cup, held from October 16 to November 13, 2022, in Australia, was the eighth edition of the tournament. Originally scheduled for 2020, it was postponed due to the COVID-19 pandemic. England emerged victorious, defeating Pakistan by five wickets in the final to clinch their second ICC Men's T20 World Cup title.\n",
    "Proposed keywords: World Cup, 2020\n",
    "Accuracy:\n",
    "[\n",
    "\t{{  \"keyword\":\"2020\"\n",
    "        \"reason\": \"2020 is a year and is not the subject of this document and as such while it is mentioned in it is not a the topic of this document..\",\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"World cup\"\n",
    "        \"reason\": \"This is the main topic of the document.\",\n",
    "        \"good\": \"Yes\"\n",
    "    }},\n",
    "]\n",
    "\n",
    "Document:\"HIGHER 1986 PROFIT FOR DUTCH CHEMICAL FIRM DSM\n",
    "The fully state-owned Dutch\n",
    "chemical firm NV DSM &lt;DSMN.AS> said its 1986 net profit rose to\n",
    "412 mln guilders from 402 mln in 1985, while turnover fell to\n",
    "17.7 billion guilders in 1986 from 24.1 billion in 1985.\n",
    "  The company said 1986 dividend, which will be paid to the\n",
    "Dutch state in its capacity of the firm's sole shareholder,\n",
    "would be raised to 98 mln guilders from 70 mln guilders in\n",
    "1985.\n",
    "  In an initial comment on its 1986 results, DSM said the\n",
    "drop in 1986 turnover had been caused mainly by losses in the\n",
    "company's fertilizer division.\"\n",
    "Proposed keywords: 1986 profit, net profit, 1986 dividend, nv dsm, firm dsm'\n",
    "Accuracy: [\n",
    "    {{  \"keyword\":\"1986 profit\",\n",
    "        \"reason\": \"This is a main topic of the document. The document discusses the profit of the company in 1986.\",\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"net profit\",\n",
    "        \"reason\": \"This is a main topic of the document. The document discusses the net profit of the company.\",\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"1986 dividend\",\n",
    "        \"reason\": \"This is a main topic of the document. The document discusses the dividend of the company in 1986.\",\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"nv dsm\",\n",
    "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "    {{  \"keyword\":\"firm dsm\",\n",
    "        \"reason\": \"This is a repetition of the company name and does not add any new information. It is redundant.\",\n",
    "        \"good\": \"No\"\n",
    "    }},\n",
    "]\n",
    "\n",
    "Document:\"{doc_text}\"\n",
    "Proposed keywords:{extracted_keywords}\n",
    "Accuracy:\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f070a4-bf14-4bf8-96eb-746b42bbe28a",
   "metadata": {},
   "source": [
    "### Helpers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "29243a6a-2e3a-4114-a530-943966891077",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_kws_precision(completion):\n",
    "    \"\"\" Processes completion to json. \n",
    "    For a document text with keyword, returns judgements per each keyword.\n",
    "    Convert resonse to a list and count the number of 'yes' then devide thsi by total kw count.\"\"\"\n",
    "    judges = eval(completion)\n",
    "    if len(judges) == 0:\n",
    "        return -1\n",
    "    goods = []\n",
    "    for judge in judges:\n",
    "        goods.append(judge['good'])\n",
    "    goods_count = goods.count('Yes')\n",
    "    precision = goods_count / len(goods)\n",
    "    return precision\n",
    "\n",
    "# get_kw_precision(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "daaa1a63-4190-4271-85db-5fee0d63c744",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_precisions_judgements(df):\n",
    "    \"\"\" Runs presision prompt, processes to \"goog\" judgemnts into a precision column.\n",
    "    Requires 'doc_text' and 'keywords' columns.\"\"\"\n",
    "    \n",
    "    # Run the precision evaluation prompt.\n",
    "    completions = []\n",
    "    for row in tqdm(df.itertuples()):\n",
    "        keywords = ', '.join([kw for kw, rank in row.keywords])\n",
    "        try:\n",
    "            response = precision_llm_chain.run(doc_text=row.doc_text, extracted_keywords=keywords)\n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "            response = ''\n",
    "        completions.append(response)\n",
    "\n",
    "    df['completions'] = completions\n",
    "    \n",
    "    # Process repsonses into precision column.\n",
    "    precisions = [get_kws_precision(c) for c in completions]\n",
    "    df['precision'] = precisions\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3aa59076-7fe0-4904-99e7-cba065cb71e2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def gen_run_precision(df):\n",
    "    # Calculate the average precision.\n",
    "    precisions = [p for p in df.precision.tolist() if p != -1]\n",
    "    precision_avg = sum(precisions) / len(precisions)\n",
    "    precision_avg = round(precision_avg, 3)\n",
    "    \n",
    "    ## Harmonic mean of precision.\n",
    "    from scipy.stats import hmean\n",
    "    # precisions = [get_kws_precision(c) for c in completions if c!=-1]\n",
    "    precisions_harmonic_mean = round(hmean(precisions), 3)\n",
    "    \n",
    "    return precision_avg, precisions_harmonic_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "4b1135b1-819f-4fbc-8620-d21c49061349",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Set up the llm prompt chat chain.\n",
    "#\n",
    "\n",
    "model_name = 'gpt-4'\n",
    "precision_llm = ChatOpenAI(model=model_name, temperature=0.25)\n",
    "precision_llm_chain = LLMChain(\n",
    "    llm=precision_llm,\n",
    "    prompt=PromptTemplate.from_template(kw_precision_prompt)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea93130c-07c9-45fd-a81c-471f11f18f6c",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Datasets\n",
    "* load a few news datasets for usee in manual evaluation prompt creation.\n",
    "* And eventual testing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0978e25-e755-400a-ab86-9d14741d9b39",
   "metadata": {},
   "source": [
    "### Reuters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "20f55ae7-ad7b-479c-986b-7309f2665087",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# from sklearn.datasets import fetch_20newsgroups\n",
    "# newsgroups = fetch_20newsgroups(subset='all')\n",
    "# print(newsgroups.target_names[newsgroups.target[0]])\n",
    "# print(newsgroups.data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "647e2ada-c4f1-4c62-b354-340b01aa62be",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package reuters to /home/jupyter/nltk_data...\n",
      "[nltk_data]   Package reuters is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# Reuters - has some longer docs.\n",
    "import nltk\n",
    "from nltk.corpus import reuters\n",
    "nltk.download('reuters')\n",
    "from nltk.corpus import reuters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "86eec46c-a92f-4563-927a-e2866af60d52",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Sample reuters.\n",
    "reuters_docs = []\n",
    "for doc_id in reuters.fileids():\n",
    "    reuters_docs.append(reuters.raw(doc_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "0e6ab314-e74a-498c-b624-e045a1ded6c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Sample reuters.\n",
    "random.seed(42)\n",
    "doc_ids = random.sample(reuters.fileids(), 20)\n",
    "reuters_samp = [reuters.raw(doc_id) for doc_id in doc_ids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bb7e5f35-c2a3-42d8-a723-193684aa8387",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(reuters_samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b696f90e-e435-4201-ae55-1d6fc640f411",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "METRO FUNDING SHAREHOLDERS APPROVE MERGER\n",
      "  &lt;Metro Funding Corp> said its\n",
      "  shareholders approved its merger into &lt;Maxcom Corp> and its\n",
      "  change of incorporation from Nevada to Delaware.\n",
      "      Metro Funding also said its subsidiary, Comet Corp, will be\n",
      "  renamed Maxcom USA.\n",
      "      The company also reported shareholders approved the\n",
      "  authorization of 500,000 shares of common stock to be set aside\n",
      "  for an incentive stock option plan.\n",
      "  \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(reuters_samp[8])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d18f1f1-9333-4c39-954c-4ee260820883",
   "metadata": {},
   "source": [
    "## KW evals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe3afea6-5845-4df6-803e-fdb9d7b02455",
   "metadata": {},
   "source": [
    "### KeyBert - eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7c694acd-75ff-4d34-958a-65cfb09d84b8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/nlp_learn/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# Prep for kw extraction.\n",
    "from keybert import KeyBERT\n",
    "kw_model = KeyBERT()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "1bbc5e58-de60-4fa2-aedb-fb94eceb931c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Extract KWs.\n",
    "docs_with_kws = []\n",
    "for doc in reuters_samp:\n",
    "    kws = kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 2), stop_words=None)\n",
    "    docs_with_kws.append((doc, kws))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "887e11c3-5d47-4215-931c-c5f0b36705fe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_text</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PAN ATLANTIC RE INC &amp;lt;PNRE&gt; 4TH QTR NET\\n  O...</td>\n",
       "      <td>[(pan atlantic, 0.4595), (year net, 0.4489), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OECD SEES GERMAN GROWTH HIT BY LOW DOMESTIC DE...</td>\n",
       "      <td>[(german economic, 0.5334), (german economy, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>YUGOSLAVIA TO TENDER FOR 100,000 TONNES WHEAT\\...</td>\n",
       "      <td>[(wheat yugoslavia, 0.7061), (soft wheat, 0.53...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TRANSDUCER SYSTEMS INC YEAR\\n  Shr profit 12 c...</td>\n",
       "      <td>[(shr profit, 0.4864), (transducer, 0.4716), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SWISS MONEY MARKET PAPER YIELDS 3.286 PCT\\n  T...</td>\n",
       "      <td>[(swiss francs, 0.5559), (swiss money, 0.5137)...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            doc_text  \\\n",
       "0  PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\\n  O...   \n",
       "1  OECD SEES GERMAN GROWTH HIT BY LOW DOMESTIC DE...   \n",
       "2  YUGOSLAVIA TO TENDER FOR 100,000 TONNES WHEAT\\...   \n",
       "3  TRANSDUCER SYSTEMS INC YEAR\\n  Shr profit 12 c...   \n",
       "4  SWISS MONEY MARKET PAPER YIELDS 3.286 PCT\\n  T...   \n",
       "\n",
       "                                            keywords  \n",
       "0  [(pan atlantic, 0.4595), (year net, 0.4489), (...  \n",
       "1  [(german economic, 0.5334), (german economy, 0...  \n",
       "2  [(wheat yugoslavia, 0.7061), (soft wheat, 0.53...  \n",
       "3  [(shr profit, 0.4864), (transducer, 0.4716), (...  \n",
       "4  [(swiss francs, 0.5559), (swiss money, 0.5137)...  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert output to df for convenience.\n",
    "df_keybert = pd.DataFrame(docs_with_kws, columns=['doc_text', 'keywords'])\n",
    "df_keybert.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "c9ab603f-081f-420d-8423-d7a0c27a2c2b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------- Document ----------\n",
      "KEYWORDS: pan atlantic, year net, net oper, oper net, investment\n",
      "\n",
      "PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\n",
      "  Oper shr 15 cts vs 1.07 dlrs\n",
      "      Oper net 372,000 vs 2,601,000\n",
      "      Year\n",
      "      Oper shr 80 cts vs 61 cts\n",
      "      Oper net 1,952,000 vs 1,491,000\n",
      "      NOTE: Net excludes realized investment loss 13,000 dlrs vs\n",
      "  gain 986,000 dlrs in quarter and gains 1,047,000 dlrs vs\n",
      "  1,152,000 dlrs in year.\n",
      "      1986 year net excludes tax credit 919,000 dlrs.\n",
      "  \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print out a few docs for manual prompt iteration.\n",
    "for doc, kws in docs_with_kws[:1]:\n",
    "    print('---------- Document ----------')\n",
    "    kws_str = ', '.join([k[0] for k in kws])\n",
    "    print(f\"KEYWORDS: {kws_str}\\n\")\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "554292e9-d668-4364-b593-787e27c39215",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "20it [05:00, 15.02s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.5, 0.397)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run the precision evaluation.\n",
    "get_precisions_judgements(df_keybert)\n",
    "gen_run_precision(df_keybert)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "8da3d743-217e-4562-afb1-cee5aa32d359",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_text</th>\n",
       "      <th>keywords</th>\n",
       "      <th>completions</th>\n",
       "      <th>precision</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PAN ATLANTIC RE INC &amp;lt;PNRE&gt; 4TH QTR NET\\n  O...</td>\n",
       "      <td>[(pan atlantic, 0.4595), (year net, 0.4489), (...</td>\n",
       "      <td>[\\n    {  \"keyword\":\"pan atlantic\",\\n        \"...</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            doc_text  \\\n",
       "0  PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\\n  O...   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [(pan atlantic, 0.4595), (year net, 0.4489), (...   \n",
       "\n",
       "                                         completions  precision  \n",
       "0  [\\n    {  \"keyword\":\"pan atlantic\",\\n        \"...        0.4  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_keybert.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "05f17d83-bdc1-4f1c-b65d-cf7a5130054f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Persist results.\n",
    "kw_method = 'keybert'\n",
    "timestamp = datetime.now().strftime(\"%Y-%m-%d-%H%M%S\")\n",
    "fpath = f'data_output/kw_precision_{kw_method}_{timestamp}.parq'\n",
    "df_keybert.to_parquet(fpath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ed9b33d2-24a3-4525-a5aa-e96672e0e654",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Print out a few docs for manual prompt iteration.\n",
    "for row in df_keybert.itertuples():\n",
    "    print('---------- Document ----------')\n",
    "    print(f\"PRECISION: {row.precision}\")\n",
    "    kws_str = ', '.join([k[0] for k in row.keywords])\n",
    "    print(f\"KEYWORDS: {kws_str}\")\n",
    "    print(f\"DOCUMENT: {row.doc_text}\")\n",
    "    print(f\"COMPLETION: {row.completions}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695a71f0-c749-4c86-a23b-dc6522d89640",
   "metadata": {},
   "source": [
    "### LLM kws extraction - eval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de338cd-b1ed-46fd-87b1-4575ea0ecbe7",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Extract KWs using GPT-4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7767608a-de63-4ebb-92b1-ae3ae68da3d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "kw_extract_prompt = \"\"\"Identify the main categories of this Reuters news story. These should be unambiguous phrases. Output the results in a Python list.\n",
    "\n",
    "{doc_text}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "74a04dab-efdb-48a2-b874-65d43209e618",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Set up the llm prompt chat chain.\n",
    "#\n",
    "\n",
    "model_name = 'gpt-4'\n",
    "kw_extract_llm = ChatOpenAI(model=model_name, temperature=0.00)\n",
    "kw_extract_chain = LLMChain(\n",
    "    llm=kw_extract_llm,\n",
    "    prompt=PromptTemplate.from_template(kw_extract_prompt)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "53ef47d8-ebb4-41a2-b009-c196298faca2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def llm_extract_kws(df):\n",
    "    \"\"\" Run kw extraction using the llm prompt. \"\"\"\n",
    "    # Run the precision evaluation prompt.\n",
    "    completions = []\n",
    "    for row in tqdm(df.itertuples()):\n",
    "        try:\n",
    "            response = kw_extract_chain.run(doc_text=row.doc_text)\n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "            response = []\n",
    "        completions.append(response)\n",
    "\n",
    "    df['kw_completions'] = completions    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "e88d3a6c-6f21-4230-92b0-3c2301782b56",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PAN ATLANTIC RE INC &amp;lt;PNRE&gt; 4TH QTR NET\\n  O...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            doc_text\n",
       "0  PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\\n  O..."
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert data to df.\n",
    "df_gpt = pd.DataFrame(reuters_samp, columns=['doc_text'])\n",
    "df_gpt.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "da396910-982a-48f9-b960-e5430d81c267",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "20it [00:52,  2.63s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 264 ms, sys: 21.3 ms, total: 285 ms\n",
      "Wall time: 52.6 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Run the kw extraction.\n",
    "llm_extract_kws(df_gpt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "669225a2-db7e-4d24-9cc2-981841445569",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_text</th>\n",
       "      <th>kw_completions</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PAN ATLANTIC RE INC &amp;lt;PNRE&gt; 4TH QTR NET\\n  O...</td>\n",
       "      <td>[\"Business\", \"Finance\", \"Company Earnings\", \"I...</td>\n",
       "      <td>[Business, Finance, Company Earnings, Insuranc...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            doc_text  \\\n",
       "0  PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\\n  O...   \n",
       "\n",
       "                                      kw_completions  \\\n",
       "0  [\"Business\", \"Finance\", \"Company Earnings\", \"I...   \n",
       "\n",
       "                                            keywords  \n",
       "0  [Business, Finance, Company Earnings, Insuranc...  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# process completions to keywords.\n",
    "df_gpt['keywords'] = df_gpt['kw_completions'].apply(lambda x: ast.literal_eval(x))\n",
    "df_gpt.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "18a3af5a-ce09-4b31-bdf8-86be2e64a721",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_text</th>\n",
       "      <th>kw_completions</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PAN ATLANTIC RE INC &amp;lt;PNRE&gt; 4TH QTR NET\\n  O...</td>\n",
       "      <td>[\"Business\", \"Finance\", \"Company Earnings\", \"I...</td>\n",
       "      <td>[(Business, 1.0), (Finance, 1.0), (Company Ear...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            doc_text  \\\n",
       "0  PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\\n  O...   \n",
       "\n",
       "                                      kw_completions  \\\n",
       "0  [\"Business\", \"Finance\", \"Company Earnings\", \"I...   \n",
       "\n",
       "                                            keywords  \n",
       "0  [(Business, 1.0), (Finance, 1.0), (Company Ear...  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add a facke probability)\n",
    "df_gpt['keywords'] = df_gpt.keywords.apply(lambda x: [(k, 1.0) for k in x])\n",
    "df_gpt.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "e6e68dad-5ade-43e6-9df7-c18e15ba9cac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# df_gpt.keywords.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "164b5737-9059-41c1-8cc3-f62c2186aa52",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Run precision eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "c3401c6e-b4c3-4fb6-a957-3b885ba4b96c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# df_samp = df_gpt.iloc[0:1].copy()\n",
    "# df_samp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "09a0779f-12c3-4eb2-bb23-22788803ac5b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "20it [06:10, 18.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 301 ms, sys: 30.7 ms, total: 332 ms\n",
      "Wall time: 6min 10s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "get_precisions_judgements(df_gpt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "a879165c-41f9-4cc4-97f7-6a1226e2b062",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.786, 0.716)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_run_precision(df_gpt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "75cd017a-3410-428d-a764-aa36a6a00d72",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Persist results.\n",
    "kw_method = 'llm-gpt4-iter1'\n",
    "timestamp = datetime.now().strftime(\"%Y-%m-%d-%H%M%S\")\n",
    "fpath = f'data_output/kw_precision_{kw_method}_{timestamp}.parq'\n",
    "df_gpt.to_parquet(fpath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "c37f090e-0a8c-43fa-b10f-e34018aa810e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_text</th>\n",
       "      <th>kw_completions</th>\n",
       "      <th>keywords</th>\n",
       "      <th>completions</th>\n",
       "      <th>precision</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PAN ATLANTIC RE INC &amp;lt;PNRE&gt; 4TH QTR NET\\n  O...</td>\n",
       "      <td>[\"Business\", \"Finance\", \"Company Earnings\", \"I...</td>\n",
       "      <td>[(Business, 1.0), (Finance, 1.0), (Company Ear...</td>\n",
       "      <td>[\\n    {  \"keyword\":\"Business\",\\n        \"reas...</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            doc_text  \\\n",
       "0  PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\\n  O...   \n",
       "\n",
       "                                      kw_completions  \\\n",
       "0  [\"Business\", \"Finance\", \"Company Earnings\", \"I...   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [(Business, 1.0), (Finance, 1.0), (Company Ear...   \n",
       "\n",
       "                                         completions  precision  \n",
       "0  [\\n    {  \"keyword\":\"Business\",\\n        \"reas...        0.4  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gpt.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "ceefdf7f-d876-4949-a477-65c0e7a2b298",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------- Document ----------\n",
      "PRECISION: 0.4\n",
      "KEYWORDS: Business, Finance, Company Earnings, Insurance Industry, Investment\n",
      "DOCUMENT: PAN ATLANTIC RE INC &lt;PNRE> 4TH QTR NET\n",
      "  Oper shr 15 cts vs 1.07 dlrs\n",
      "      Oper net 372,000 vs 2,601,000\n",
      "      Year\n",
      "      Oper shr 80 cts vs 61 cts\n",
      "      Oper net 1,952,000 vs 1,491,000\n",
      "      NOTE: Net excludes realized investment loss 13,000 dlrs vs\n",
      "  gain 986,000 dlrs in quarter and gains 1,047,000 dlrs vs\n",
      "  1,152,000 dlrs in year.\n",
      "      1986 year net excludes tax credit 919,000 dlrs.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business\",\n",
      "        \"reason\": \"This is a broad term that could apply to many documents. It lacks specificity and does not directly relate to the core content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Finance\",\n",
      "        \"reason\": \"This is a broad term that could apply to many documents. It lacks specificity and does not directly relate to the core content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Company Earnings\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the earnings of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Insurance Industry\",\n",
      "        \"reason\": \"This is not mentioned in the document. It is irrelevant and does not accurately reflect the document's content.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Investment\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the investment gains and losses of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.45454545454545453\n",
      "KEYWORDS: German Economic Growth, OECD Economic Review, Domestic Demand in Germany, International Economic Competition, Gross National Product Forecast, Business Confidence in Germany, Government Tax Reform in Germany, Reduction of Subsidies, West German Federal Statistics Office Report, Group of Seven Industrial Nations, West Germany's Current Account Surplus\n",
      "DOCUMENT: OECD SEES GERMAN GROWTH HIT BY LOW DOMESTIC DEMAND\n",
      "  West German economic growth will slow to\n",
      "  1.5 pct this year from 2.4 pct in 1986 due to weak domestic\n",
      "  demand and tougher competition from abroad, the Organisation\n",
      "  for Economic Cooperation and Development (OECD) said in its\n",
      "  semi-annual review of the world economy.\n",
      "      This view is less favourable than the West German\n",
      "  government's forecast of a growth rate of under two pct this\n",
      "  year, but is in line with forecasts by independent economic\n",
      "  institutes of growth ranging from 1.5 to two pct.\n",
      "      The OECD said that the economy should pick up next year,\n",
      "  with the gross national product rising by two pct in real\n",
      "  terms.\n",
      "      The OECD said it assumed the German economy was passing\n",
      "  through a period of temporary weakness and there would be some\n",
      "  recovery in business confidence in the near future.\n",
      "      But it warned that the key to an improvement in the economy\n",
      "  was higher domestic demand, which is only forecast to rise by\n",
      "  2.5 pct this year and 2.75 pct in 1988, below 1986's 3.7 pct.\n",
      "      While noting that the government is bringing forward a five\n",
      "  billion mark tax reform to January 1988, the OECD said that \"the\n",
      "  medium to longer-term performance of the West German economy\n",
      "  could be improved by reduction of subsidies - which would allow\n",
      "  relatively lower tax rates.\"\n",
      "      Since the OECD report was compiled, the West German Federal\n",
      "  Statistics Office has released figures showing that the GNP\n",
      "  actually fell 0.5 pct in real terms in the first quarter of\n",
      "  this year compared with the final three months of 1986.\n",
      "      Diplomatic sources here said that West Germany appeared\n",
      "  likely to finish the year with the lowest growth rate of any of\n",
      "  the Group of Seven leading industrial nations.\n",
      "      West Germany's current account surplus, the target of\n",
      "  considerable criticism by the Reagan administration, is\n",
      "  expected to rise slightly to 37 billion dlrs this year from\n",
      "  35.8 billion in 1986, before declining to 29 billion dlrs in\n",
      "  1988.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"German Economic Growth\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the economic growth of Germany.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"OECD Economic Review\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the review of the OECD on the German economy.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Domestic Demand in Germany\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the domestic demand in Germany.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"International Economic Competition\",\n",
      "        \"reason\": \"This is not a main topic of the document. The document does not discuss international economic competition in detail.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Gross National Product Forecast\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the forecast of the gross national product.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Business Confidence in Germany\",\n",
      "        \"reason\": \"This is not a main topic of the document. The document does not discuss business confidence in Germany in detail.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Government Tax Reform in Germany\",\n",
      "        \"reason\": \"This is not a main topic of the document. The document does not discuss government tax reform in Germany in detail.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Reduction of Subsidies\",\n",
      "        \"reason\": \"This is not a main topic of the document. The document does not discuss the reduction of subsidies in detail.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"West German Federal Statistics Office Report\",\n",
      "        \"reason\": \"This is not a main topic of the document. The document does not discuss the report of the West German Federal Statistics Office in detail.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Group of Seven Industrial Nations\",\n",
      "        \"reason\": \"This is not a main topic of the document. The document does not discuss the Group of Seven Industrial Nations in detail.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"West Germany's Current Account Surplus\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the current account surplus of West Germany.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: Yugoslavia Wheat Tender, U.S. Agriculture Department's Counselor Report, 1986 and 1987 Wheat Harvest, Exclusion of European and Other Suppliers, Rebuilding Federal Reserves, Exemption from Import Surcharges\n",
      "DOCUMENT: YUGOSLAVIA TO TENDER FOR 100,000 TONNES WHEAT\n",
      "  Yugoslavia will tender April 14 for\n",
      "  100,000 tonnes of wheat, the U.S. Agriculture Department's\n",
      "  Counselor in Belgrade said in a field report.\n",
      "      The report, dated April 7, said the wheat must be from 1986\n",
      "  and 1987 harvest, and imports of soft wheat from Europe and\n",
      "  from other suppliers will not be considered.\n",
      "      It said the imports will be used to rebuild the federal\n",
      "  reserves and as a result will not be subject to import\n",
      "  surcharges.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Yugoslavia Wheat Tender\",\n",
      "        \"reason\": \"This is the main topic of the document. The document discusses the tender for wheat in Yugoslavia.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"U.S. Agriculture Department's Counselor Report\",\n",
      "        \"reason\": \"This is relevant as it is the source of the information in the document.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"1986 and 1987 Wheat Harvest\",\n",
      "        \"reason\": \"This is relevant as it specifies the harvest years for the wheat being tendered.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Exclusion of European and Other Suppliers\",\n",
      "        \"reason\": \"This is relevant as it specifies who is not considered for the tender.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Rebuilding Federal Reserves\",\n",
      "        \"reason\": \"This is relevant as it specifies the purpose of the wheat import.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Exemption from Import Surcharges\",\n",
      "        \"reason\": \"This is relevant as it specifies a condition of the wheat import.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: Company Financials, Annual Profit/Loss, Revenue, Tax Benefits\n",
      "DOCUMENT: TRANSDUCER SYSTEMS INC YEAR\n",
      "  Shr profit 12 cts vs loss 49 cts\n",
      "      Net profit 117,000 vs loss 506,000\n",
      "      Revs 1.1 mln vs 1.2 mln\n",
      "      Year\n",
      "      Shr profit seven cts vs loss 89 cts\n",
      "      Net profit 66,000 vs loss 921,000\n",
      "      Revs 4.4 mln vs 3.9 mln\n",
      "      NOTE:1986 reflects tax benefit of 24,000. 1985 reflects tax\n",
      "  benefit of 186,000 for quarter and 573,000 for year.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Company Financials\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the financials of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Annual Profit/Loss\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the annual profit and loss of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Revenue\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the revenue of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Tax Benefits\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the tax benefits of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: Swiss Federal Government, money market certificates, Swiss francs, issue price, average annual yield, National Bank, payment date, six-month paper\n",
      "DOCUMENT: SWISS MONEY MARKET PAPER YIELDS 3.286 PCT\n",
      "  The Swiss Federal Government's new series\n",
      "  of six-month money market certificates raised 177.5 mln Swiss\n",
      "  francs at an issue price of 98.401 pct to give an average\n",
      "  annual yield of 3.286 pct, the National Bank said.\n",
      "      Payment date is March 5.\n",
      "      The last series of six-month paper issued in January raised\n",
      "  159.6 mln francs at 98.392 pct to give an average yield of\n",
      "  3.251 pct.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Swiss Federal Government\",\n",
      "        \"reason\": \"This is relevant as it is the issuer of the money market certificates being discussed in the document.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"money market certificates\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the issue of money market certificates.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Swiss francs\",\n",
      "        \"reason\": \"This is relevant as it is the currency in which the money market certificates are issued.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"issue price\",\n",
      "        \"reason\": \"This is relevant as it is a key detail in the issuance of the money market certificates.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"average annual yield\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the yield of the money market certificates.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"National Bank\",\n",
      "        \"reason\": \"This is relevant as it is the institution that announced the details of the money market certificates.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"payment date\",\n",
      "        \"reason\": \"This is relevant as it is a key detail in the issuance of the money market certificates.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"six-month paper\",\n",
      "        \"reason\": \"This is relevant as it is the type of money market certificates being issued.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: International Finance, Currency Stabilization, UK Economy, Interest Rates, Inflation, Government Fiscal Policy\n",
      "DOCUMENT: PARIS MEET AGREED NOT TO PUBLISH BANDS - LAWSON\n",
      "  U.K. Chancellor of the Exchequer Nigel\n",
      "  Lawson said the meeting of six major industrial nations in\n",
      "  Paris last month agreed not to publish any bands in connection\n",
      "  with their pact to stabilise currencies.\n",
      "      Questioned by a parliamentary select committee about why\n",
      "  the participating countries had not announced any bands, Lawson\n",
      "  replied, \"we all agreed it would be much more sensible not to.\"\n",
      "  When asked whether that meant an informal joint currency float\n",
      "  with set ranges was arranged in Paris, Lawson said \"I do not\n",
      "  want to reveal the precise nature of the agreement, so as not\n",
      "  to make it easy for\" speculation against the accord.\n",
      "      Lawson said the Paris accord presumed that individual\n",
      "  countries would take corrective action if their currency began\n",
      "  reacting significantly to domestic macroeconomic factors.\n",
      "      But if such movements were due to extraneous factors,\n",
      "  Lawson said the other pact countries would come to its aid\n",
      "  through concerted intervention on the foreign exchanges.\n",
      "      \"It is clear that both Germany and Japan are having\n",
      "  difficulty adjusting to their very large exchange rate\n",
      "  appreciations and making their economies more domestically\n",
      "  oriented, just as it is taking time for the United States to\n",
      "  make its own economy more export oriented,\" Lawson said.\n",
      "      In his oral evidence to Parliament's Select Treasury\n",
      "  Committee, Lawson repeated that he was happy with the pound's\n",
      "  current level, adding that \"It is an objective ... To try to\n",
      "  keep it around that level.\"\n",
      "      He said the perception of sterling on foreign exchanges had\n",
      "  changed since the steep drop in oil prices, largely because the\n",
      "  pound had weathered that period so successfully. \"There has been\n",
      "  a reassessment of sterling's fundamentals,\" Lawson said.  He\n",
      "  disagreed with what he termed \"the grossly exaggerated claim\"\n",
      "  that real U.K. Interest rates were much higher than those of\n",
      "  other major industrialsed countries.\n",
      "      Using as a reference the key three-month sterling interbank\n",
      "  rate as quoted in London, Lawson said Britain now had a real\n",
      "  interest rate level of 5.75 pct - the same as Japan did and\n",
      "  only a 0.75 percentaage point above the Group of Five average.\n",
      "      Lawson confirmed that \"over the medium and longer term, the\n",
      "  government's objective is zero inflation.\"\n",
      "      He said the government's intention of its PSBR constant at\n",
      "  1.0 pct of GDP \"is the modern equivalent of the balanced budget\n",
      "  doctrine.\"  He added that \"to allow the debt/GDP ratio to remain\n",
      "  constant on anything other than zero inflation basis is simply\n",
      "  a recipe for accelerating inflation.\"\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"International Finance\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses international finance issues such as currency stabilization.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Currency Stabilization\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses efforts to stabilize currencies among major industrial nations.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"UK Economy\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses aspects of the UK economy such as interest rates and inflation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Interest Rates\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses interest rates in the UK and other major industrialized countries.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Inflation\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses inflation and the government's objective of zero inflation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Government Fiscal Policy\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the government's fiscal policy, including its objective of zero inflation and keeping the PSBR constant at 1.0 pct of GDP.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.4\n",
      "KEYWORDS: Business, Finance, Earnings Reports, Retail Industry, Sears, Roebuck and Co.\n",
      "DOCUMENT: SEARS, ROEBUCK AND CO &lt;S> 3RD QTR NET\n",
      "  Shr 1.08 dlrs vs 88 cts\n",
      "      Net 409,000,000 vs 328,000,000\n",
      "      Revs 12.19 billion vs 11.17 billion\n",
      "      Avg shrs 378.2 mln vs 368.4 mln\n",
      "      Nine mths\n",
      "      Shr 2.86 dlrs vs 2.17 dlrs\n",
      "      Net 1.09 billion vs 808.3 mln\n",
      "      Revs 34.39 billion vs 31.31 billion\n",
      "      Avg shrs 377.7 mln vs 366.2 mln\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business\",\n",
      "        \"reason\": \"This is a broad category that the document falls under, but it is not specific enough to be a good keyword for this document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Finance\",\n",
      "        \"reason\": \"This is a broad category that the document falls under, but it is not specific enough to be a good keyword for this document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Earnings Reports\",\n",
      "        \"reason\": \"This is a main topic of the document. The document is an earnings report for Sears, Roebuck and Co.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Retail Industry\",\n",
      "        \"reason\": \"This is relevant to the document as Sears, Roebuck and Co. is a retail company. However, it is not a main topic of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Sears, Roebuck and Co.\",\n",
      "        \"reason\": \"This is the company that the document is about. It is a main topic of the document.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.75\n",
      "KEYWORDS: Company Financials, Quarterly Earnings, PENWEST INC, Stock Market\n",
      "DOCUMENT: PENWEST INC &lt;PENW.O> 3RD QTR MAY 31 NET\n",
      "  Shr 1.08 dlrs vs 37 cts\n",
      "      Net 3,025,000 vs 1,188,000\n",
      "      Sales 35.2 mln vs 35.4 mln\n",
      "      Avg shrs 2,794,298 vs 3,187,051\n",
      "      Nine mths\n",
      "      Shr 2.43 dlrs vs 83 cts\n",
      "      Net 6,781,000 vs 2,576,000\n",
      "      Sales 101.0 mln vs 99.7 mln\n",
      "      Avg shrs 2,794,298 vs 3,115,499\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Company Financials\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the financials of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Quarterly Earnings\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the quarterly earnings of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"PENWEST INC\",\n",
      "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Stock Market\",\n",
      "        \"reason\": \"While the document discusses financials of a company, it does not specifically discuss the stock market. This keyword is too broad.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.8\n",
      "KEYWORDS: Business, Mergers and Acquisitions, Corporate Actions, Shareholder Voting, Stock Options\n",
      "DOCUMENT: METRO FUNDING SHAREHOLDERS APPROVE MERGER\n",
      "  &lt;Metro Funding Corp> said its\n",
      "  shareholders approved its merger into &lt;Maxcom Corp> and its\n",
      "  change of incorporation from Nevada to Delaware.\n",
      "      Metro Funding also said its subsidiary, Comet Corp, will be\n",
      "  renamed Maxcom USA.\n",
      "      The company also reported shareholders approved the\n",
      "  authorization of 500,000 shares of common stock to be set aside\n",
      "  for an incentive stock option plan.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business\",\n",
      "        \"reason\": \"This is too broad and vague. It does not specifically describe the content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Mergers and Acquisitions\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the merger of two companies.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Corporate Actions\",\n",
      "        \"reason\": \"This is relevant to the document as it discusses corporate actions such as mergers and changes of incorporation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Shareholder Voting\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the approval of shareholders for various actions.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Stock Options\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the authorization of shares for a stock option plan.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: Japan's Oil Industry, Deregulation, Potential Mergers, Petroleum Council Advisory, Ministry of International Trade and Industry (MITI), End of Industry Protection, Promotion of Competition and Efficiency, Survival of Smaller Firms, Streamlining and Efficiency Programs, Refining Capacity Reduction, Workforce Reduction, Oil Company Integration, Potential Further Mergers\n",
      "DOCUMENT: JAPAN OIL INDUSTRY DECONTROL MAY LEAD TO MERGERS\n",
      "  Deregulation of Japan's oil industry could\n",
      "  mean hardship for smaller firms and lead to their merging into\n",
      "  bigger refining and marketing groups, industry sources said.\n",
      "      They said the relaxation of controls was now under review\n",
      "  by the Petroleum Council, an advisory panel to the Ministry of\n",
      "  International Trade and Industry (MITI).\n",
      "      A spokesman for a major firm said, \"Deregulation would bring\n",
      "  about a reorganization. If it's a by-product of freer\n",
      "  competition, we have no choice but to accept it.\"\n",
      "      The Council is due to close its discussions on June 12.\n",
      "      The sources said the Council was likely to tell MITI it\n",
      "  should end its 50 year-old protection of the industry. It\n",
      "  should cut capacity to 3.8 mln barrels per day, about 75 pct of\n",
      "  current capacity. Quotas should end for crude throughput and\n",
      "  gasoline output, and oil tariffs should be abolished.\n",
      "      They said deregulation was vital to promote more\n",
      "  competition and efficiency, and most saw it as inevitable.\n",
      "      \"Deregulation is taking place everywhere. Now it's our turn\n",
      "  to see if we can survive cut-throat competition,\" said a source\n",
      "  at one major Japanese oil company.\n",
      "      A spokesman for a smaller refiner said, \"We'll have a hard\n",
      "  time surviving, but that's something we must go through.\"\n",
      "      \"In addition to our streamlining and efficiency programs for\n",
      "  the oil division, we will exert efforts towards branching out\n",
      "  further into other lines such as real estate and travel\n",
      "  agencies,\" he said.\n",
      "      Larger companies are also streamlining. Nippon Oil Co Ltd\n",
      "  which had the largest share of refined products sales in the\n",
      "  Japanese market in fiscal 1985, cut nine pct of its refining\n",
      "  capacity in fiscal 1986.\n",
      "      Cosmo Oil Co Ltd, the third largest seller of oil products\n",
      "  in 1985, cut its workforce by some 20 pct last year, a\n",
      "  spokesman for the company said.\n",
      "      Between 1984 and 1986, on the recommendation of the\n",
      "  Council, 13 oil companies were integrated into 11 companies\n",
      "  within seven refining and marketing groups to improve the\n",
      "  efficiency of the industry.\n",
      "      Oil industry sources said this structure was now likely to\n",
      "  be streamlined further into five refining groups.\n",
      "      \"MITI means business. It will urge the major seven groups to\n",
      "  merge into five to build up their muscles,\" a source said.\n",
      "      A MITI official told Reuters he did not rule out the\n",
      "  possibility of further mergers within the Japanese oil industry\n",
      "  in the event of the relaxation of oil controls.\n",
      "      He declined to officially confirm or deny that the\n",
      "  Petroleum Council had recommended deregulatory measures but\n",
      "  said that in principal MITI would encourage a move towards\n",
      "  deregulation.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Japan's Oil Industry\",\n",
      "        \"reason\": \"This is the main topic of the document. The document discusses the potential impact of deregulation on Japan's oil industry.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Deregulation\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential impact of deregulation on Japan's oil industry.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Potential Mergers\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for mergers as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Petroleum Council Advisory\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the role of the Petroleum Council in advising on deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Ministry of International Trade and Industry (MITI)\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the role of the MITI in the potential deregulation of Japan's oil industry.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"End of Industry Protection\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential end of industry protection as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Promotion of Competition and Efficiency\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for increased competition and efficiency as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Survival of Smaller Firms\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential impact of deregulation on smaller firms in Japan's oil industry.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Streamlining and Efficiency Programs\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for streamlining and efficiency programs as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Refining Capacity Reduction\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for refining capacity reduction as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Workforce Reduction\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for workforce reduction as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Oil Company Integration\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for oil company integration as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Potential Further Mergers\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the potential for further mergers as a result of deregulation.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.8\n",
      "KEYWORDS: Business, Acquisitions, Energy Industry, Cogeneration Projects, Petroleum-Coke Fueled Power Plants\n",
      "DOCUMENT: ENERGY FACTORS &lt;EFAC> TO BUY ALLIED &lt;ALD> UNITS\n",
      "  Energy Factors Inc said it has signed\n",
      "  an agreement to acquire GWF Power Systems Co and Combustion\n",
      "  Power Co Inc from Allied-Signal Inc for undisclosed terms.\n",
      "      The two Allied units operate and are developing\n",
      "  cogeneration projects and small petroleum-coke fueled power\n",
      "  plants.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business\",\n",
      "        \"reason\": \"This keyword is too broad and vague. It does not provide specific information about the content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Acquisitions\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses an acquisition.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Energy Industry\",\n",
      "        \"reason\": \"This is relevant to the document as it discusses an acquisition within the energy industry.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Cogeneration Projects\",\n",
      "        \"reason\": \"This is a specific aspect of the document, as the companies being acquired are involved in cogeneration projects.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Petroleum-Coke Fueled Power Plants\",\n",
      "        \"reason\": \"This is a specific aspect of the document, as the companies being acquired are involved in petroleum-coke fueled power plants.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: USX Subsidiary, Marathon Petroleum Co, Crude Oil Prices, West Texas Intermediate, West Texas Sour, Light Louisiana grade\n",
      "DOCUMENT: USX &lt;X> UNIT TO RAISE MOST CRUDE POSTINGS\n",
      "  USX's subsidiary, Marathon Petroleum\n",
      "  Co, said it will raise its contract prices for eleven grades of\n",
      "  crude oil 50 cts a barrel, effective tomorrow.\n",
      "      The increase brings Marathon's posted price for West Texas\n",
      "  Intermediate and West Texas Sour grades to 17.50 dlrs a barrel.\n",
      "      The Light Louisiana grade was also raised 50 cts to 17.85\n",
      "  dlrs a barrel.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"USX Subsidiary\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the actions of the USX subsidiary, Marathon Petroleum Co.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Marathon Petroleum Co\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the actions of Marathon Petroleum Co.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Crude Oil Prices\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the increase in crude oil prices.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"West Texas Intermediate\",\n",
      "        \"reason\": \"This is a specific grade of crude oil mentioned in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"West Texas Sour\",\n",
      "        \"reason\": \"This is a specific grade of crude oil mentioned in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Light Louisiana grade\",\n",
      "        \"reason\": \"This is a specific grade of crude oil mentioned in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.6\n",
      "KEYWORDS: Company Financials, Silicon Systems Inc, Quarterly Results, Profit and Loss Statement, Revenue\n",
      "DOCUMENT: SILICON SYSTEMS INC &lt;SLCN> 2ND QTR MARCH 28\n",
      "  Shr profit five cts vs profit two cts\n",
      "      Net profit 325,000 vs profit 105,000\n",
      "      Revs 19.5 mln vs 16.1 mln\n",
      "      Six Mths\n",
      "      Shr profit nine cts vs loss 35 cts\n",
      "      Net profit 627,000 vs loss 2,280,000\n",
      "      Revs 36.9 mln vs 27.4 mln\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Company Financials\",\n",
      "        \"reason\": \"This is a broad term that could apply to any company's financial report. It lacks specificity and relevance to the specific document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Silicon Systems Inc\",\n",
      "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Quarterly Results\",\n",
      "        \"reason\": \"This is a relevant keyword as the document discusses the quarterly results of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Profit and Loss Statement\",\n",
      "        \"reason\": \"This is a relevant keyword as the document discusses the profit and loss of the company. However, it is a bit broad and could be more specific.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Revenue\",\n",
      "        \"reason\": \"This is a relevant keyword as the document discusses the company's revenue.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.75\n",
      "KEYWORDS: Business, Financial Results, FAB Industries Inc, First Quarter Results\n",
      "DOCUMENT: FAB INDUSTRIES INC &lt;FIT> 1ST QTR FEB 28 NET\n",
      "  Shr 69 cts vs 67 cts\n",
      "      Net 2,488,000 vs 2,435,000\n",
      "      Revs 27.6 mln vs 26.5 mln\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business\",\n",
      "        \"reason\": \"This keyword is too broad and does not directly relate to the core content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Financial Results\",\n",
      "        \"reason\": \"This keyword is relevant as the document discusses the financial results of FAB Industries Inc.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"FAB Industries Inc\",\n",
      "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"First Quarter Results\",\n",
      "        \"reason\": \"This keyword is relevant as the document discusses the first quarter results of FAB Industries Inc.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 1.0\n",
      "KEYWORDS: Company Financial Results, Nodaway Valley Co Performance, First Quarter Net Income, Share Value, Revenue\n",
      "DOCUMENT: NODAWAY VALLEY CO &lt;NVCO.O> 1ST QTR NET\n",
      "  qtr ended April 30\n",
      "      Shr seven cts vs 10 cts\n",
      "      Net 158,294 vs 234,829\n",
      "      Revs 8,727,242 vs 8,152,478\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Company Financial Results\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the financial results of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Nodaway Valley Co Performance\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the performance of Nodaway Valley Co.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"First Quarter Net Income\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the net income of the company in the first quarter.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Share Value\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the share value of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Revenue\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the revenue of the company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.8571428571428571\n",
      "KEYWORDS: U.S. Government, International Coffee Organization, Coffee Quotas Negotiations, U.S. Delegation to ICO, Brazil Coffee Production, Establishing Quotas Criteria, ICO Executive Board Meeting\n",
      "DOCUMENT: U.S. SEES NO NEW COFFEE AGREEMENT TALKS SOON\n",
      "  The United States does not expect\n",
      "  the executive board meeting of the International Coffee\n",
      "  Organization, ICO, to call for a new round of negotiations on\n",
      "  reinstating coffee quotas, a U.S. government official said.\n",
      "      The official, a member of the U.S. delegation to ICO talks\n",
      "  earlier this year, said no new coffee agreement talks are\n",
      "  expected because there is no indication the negotiating\n",
      "  positions of major producers and consumers have changed.\n",
      "      The U.S. still demands, as a condition of reimposition of\n",
      "  coffee quotas, that \"objective criteria\" be set for\n",
      "  establishing quotas, said the U.S. official, who asked not to\n",
      "  be identified. Brazil, the major producer, insists on quotas\n",
      "  based on a traditional formula.\n",
      "      The U.S. remains open to a negotiating meeting but only if\n",
      "  some new flexibility is apparent from major countries, the\n",
      "  official said.\n",
      "      The ICO executive board meets tomorrow in London.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"U.S. Government\",\n",
      "        \"reason\": \"This is relevant as the document discusses the stance of the U.S. government on the coffee agreement talks.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"International Coffee Organization\",\n",
      "        \"reason\": \"This is relevant as the document discusses the role of the International Coffee Organization in the coffee agreement talks.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Coffee Quotas Negotiations\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the negotiations on reinstating coffee quotas.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"U.S. Delegation to ICO\",\n",
      "        \"reason\": \"This is relevant as the document discusses the role of the U.S. delegation in the ICO talks.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Brazil Coffee Production\",\n",
      "        \"reason\": \"This is relevant as the document discusses Brazil's stance on coffee quotas. However, it does not specifically discuss Brazil's coffee production.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Establishing Quotas Criteria\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the criteria for establishing quotas.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"ICO Executive Board Meeting\",\n",
      "        \"reason\": \"This is relevant as the document discusses the upcoming ICO executive board meeting.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.7777777777777778\n",
      "KEYWORDS: Business Acquisition, Share Trading, Sugar Industry, CSR Ltd, Pioneer Sugar Mills Ltd, Stock Market, Business Management, Delhi Petroleum Pty Ltd, Exxon Corp\n",
      "DOCUMENT: CSR BIDS 2.20 DLRS A SHARE FOR PIONEER SUGAR MILLS\n",
      "  CSR Ltd &lt;CSRA.S> said it will offer 2.20\n",
      "  dlrs cash each for the shares it does not already hold in\n",
      "  &lt;Pioneer Sugar Mills Ltd>.\n",
      "      CSR already holds 31.74 pct of Pioneer's 99.80 mln issued\n",
      "  shares, it said in a statement.\n",
      "      The offer price values the entire Pioneer Sugar share\n",
      "  capital at 219.6 mln dlrs and compares with today's closing\n",
      "  market level of 1.85 dlrs a share.\n",
      "      CSR said it will announce further details of the offer\n",
      "  soon, including an alternative offer of CSR shares for Pioneer\n",
      "  Sugar stock.\n",
      "      It said the offer is generous since it will give Pioneer\n",
      "  Sugar shareholders a price equivalent to 29 times Pioneer's net\n",
      "  earnings last financial year and a premium of 22 pct over\n",
      "  yesterday's market price which CSR said it believed already\n",
      "  contained an element of takeover speculation.\n",
      "      It also gives a premium of 91 pct over Pioneer's last\n",
      "  reported net tangible assets per share, CSR said.\n",
      "      CSR said the generous offer price reflects the cost savings\n",
      "  which will flow from integrated management of CSR's and\n",
      "  Pioneer's raw sugar mills and building materials businesses.\n",
      "      These economies can only be achieved through CSR control\n",
      "  and management of Pioneer Sugar, it added.\n",
      "      The takeover announcement came soon after CSR's earlier\n",
      "  reported statement that it will sell its &lt;Delhi Petroleum Pty\n",
      "  Ltd> unit to an Exxon Corp &lt;XON> unit for 985 mln dlrs and not\n",
      "  proceed with the previously announced float of part of Delhi.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business Acquisition\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the acquisition of Pioneer Sugar Mills by CSR Ltd.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Share Trading\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the trading of shares in the context of the acquisition.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Sugar Industry\",\n",
      "        \"reason\": \"This is relevant to the document. The companies involved are in the sugar industry.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"CSR Ltd\",\n",
      "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Pioneer Sugar Mills Ltd\",\n",
      "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Stock Market\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the trading of shares in the context of the acquisition.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Business Management\",\n",
      "        \"reason\": \"This is relevant to the document. The document discusses the management of the companies involved.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Delhi Petroleum Pty Ltd\",\n",
      "        \"reason\": \"This is mentioned in the document but it is not a main topic. The main focus of the document is the acquisition of Pioneer Sugar Mills by CSR Ltd.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Exxon Corp\",\n",
      "        \"reason\": \"This is mentioned in the document but it is not a main topic. The main focus of the document is the acquisition of Pioneer Sugar Mills by CSR Ltd.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.8333333333333334\n",
      "KEYWORDS: Investment, Stock Acquisition, Securities and Exchange Commission (SEC), CooperVision Inc, Siebel Capital Management Inc, Corporate Structure Changes\n",
      "DOCUMENT: FIRM HAS COOPERVISION &lt;EYE> STOCK, WILL BUY MORE\n",
      "  Siebel Capital Management Inc, a\n",
      "  California investment adviser, told the Securities and Exchange\n",
      "  Commission (SEC) it bought 1.3 mln CooperVision Inc common\n",
      "  shares or 5.9 pct of the total outstanding.\n",
      "      In its SEC filing, the firm said it \"intends to acquire more\n",
      "  stock and may attempt to influence management of the company to\n",
      "  make major changes in the company's business or corporate\n",
      "  structure.\"\n",
      "      Siebel said it made net purchases of 163,200 CooperVision\n",
      "  shares since Jan. 1.\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Investment\",\n",
      "        \"reason\": \"This keyword is too broad and does not specifically relate to the content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Stock Acquisition\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses Siebel Capital Management Inc's acquisition of CooperVision Inc's stock.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Securities and Exchange Commission (SEC)\",\n",
      "        \"reason\": \"This is relevant to the document as the SEC is mentioned in relation to the stock acquisition.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"CooperVision Inc\",\n",
      "        \"reason\": \"This is the company whose stock is being acquired. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Siebel Capital Management Inc\",\n",
      "        \"reason\": \"This is the company that is acquiring the stock. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Corporate Structure Changes\",\n",
      "        \"reason\": \"This is a potential outcome mentioned in the document, hence it is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.5\n",
      "KEYWORDS: Business, Finance, Earnings Reports, Communications Corp of America\n",
      "DOCUMENT: COMMUNICATIONS CORP OF AMERICA 2ND QTR DEC 31\n",
      "  Shr loss 33 cts vs loss 48 cts\n",
      "      Net loss 1.7 mln vs loss 2.5 mln\n",
      "      Revs 6.3 mln vs 10.2 mln\n",
      "      Six months\n",
      "      Shr loss 54 cts vs loss 75 cts\n",
      "      Net loss 2.8 mln vs loss 3.9 mln\n",
      "      Revs 15.2 mln vs 23.4 mln\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Business\",\n",
      "        \"reason\": \"This keyword is too broad and does not accurately represent the specific content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Finance\",\n",
      "        \"reason\": \"This keyword is too broad and does not accurately represent the specific content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "    {  \"keyword\":\"Earnings Reports\",\n",
      "        \"reason\": \"This keyword is relevant as the document discusses the earnings report of a company.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Communications Corp of America\",\n",
      "        \"reason\": \"This is the name of the company being discussed in the document. It is relevant and specific.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "]\n",
      "---------- Document ----------\n",
      "PRECISION: 0.8\n",
      "KEYWORDS: Group of 10, G-7 Agreement, Paris Accord on Currency Stability, Exchange Rates Stability, International Monetary Fund Interim Committee Meeting\n",
      "DOCUMENT: GROUP OF 10 WELCOMES LATEST G-7 AGREEMENT\n",
      "  The Group of 10 developed nations\n",
      "  issued a communique welcoming the reaffirmation of the Paris\n",
      "  accord on currency stability by the Group of Seven leading\n",
      "  industrial democracies yesterday.\n",
      "      In the communique issued this morning, the G-10 said\n",
      "  \"prospects of member countries' economies would be improved by\n",
      "  stability in the exchange rates of their currencies.\"\n",
      "      The G-10 includes the group of seven - the United States,\n",
      "  Canada, Britain, Japan, Italy, West Germany and France - plus\n",
      "  Belgium, the Netherlands, Sweden and also Switzerland.\n",
      "      Sources who attended the G-10 conference this morning said\n",
      "  the reference to the latest G-7 agreement was especially added\n",
      "  to the brief communique because currency stability will benefit\n",
      "  all the G-10 members.\n",
      "      The G-10 met briefly before the International Monetary Fund\n",
      "  Interim Committee meeting scheduled for today.\n",
      "  .\n",
      "  \n",
      "\n",
      "\n",
      "COMPLETION: [\n",
      "    {  \"keyword\":\"Group of 10\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the Group of 10's response to the G-7 agreement.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"G-7 Agreement\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the Group of 10's response to the G-7 agreement.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Paris Accord on Currency Stability\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the Group of 10's response to the reaffirmation of the Paris accord on currency stability.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"Exchange Rates Stability\",\n",
      "        \"reason\": \"This is a main topic of the document. The document discusses the Group of 10's response to the reaffirmation of the Paris accord on currency stability.\",\n",
      "        \"good\": \"Yes\"\n",
      "    },\n",
      "    {  \"keyword\":\"International Monetary Fund Interim Committee Meeting\",\n",
      "        \"reason\": \"This is mentioned in the document but it is not the main focus of the document. It is too specific and does not reflect the overall content of the document.\",\n",
      "        \"good\": \"No\"\n",
      "    },\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# Print out a few docs for manual prompt iteration.\n",
    "for row in df_gpt.itertuples():\n",
    "    print('---------- Document ----------')\n",
    "    print(f\"PRECISION: {row.precision}\")\n",
    "    kws_str = ', '.join([k[0] for k in row.keywords])\n",
    "    print(f\"KEYWORDS: {kws_str}\")\n",
    "    print(f\"DOCUMENT: {row.doc_text}\")\n",
    "    print(f\"COMPLETION: {row.completions}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c463c563-6753-43d4-b0c9-04c70e88c345",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-env-ai_explorations-py",
   "name": "workbench-notebooks.m113",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/workbench-notebooks:m113"
  },
  "kernelspec": {
   "display_name": "Python 3 (ai_explorations) (Local)",
   "language": "python",
   "name": "conda-env-ai_explorations-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}