Chapter_03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing Pipelines\n",
    "\n",
    "https://course.spacy.io/chapter3\n",
    "\n",
    "This chapter will show you to everything you need to know about spaCy's processing pipeline. You'll learn what goes on under the hood when you process a text, how to write your own components and add them to the pipeline, and how to use custom attributes to add your own meta data to the documents, spans and tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Processing pipelines\n",
    "\n",
    "In this lesson, you'll learn about the pipeline components provided by spaCy, and what happens behind the scenes when you call nlp on a string of text.\n",
    "\n",
    "You've already written this plenty of times by now: pass a string of text to the nlp object, and receive a Doc object.\n",
    "\n",
    "But what does the nlp object actually do?\n",
    "\n",
    "First, the tokenizer is applied to turn the string of text into a Doc object. Next, a series of pipeline components is applied to the Doc in order. In this case, the tagger, then the parser, then the entity recognizer. Finally, the processed Doc is returned, so you can work with it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/whathappens.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy ships with the following built-in pipeline components:\n",
    "\n",
    "* The part-of-speech tagger sets the token dot tag attribute.\n",
    "\n",
    "* The dependency parser adds the token dot dep and token dot head attributes and is also responsible for detecting sentences and base noun phrases, also known as noun chunks.\n",
    "\n",
    "* The named entity recognizer adds the detected entities to the doc dot ents property. It also sets entity type attributes on the tokens that indicate if a token is part of an entity or not.\n",
    "\n",
    "* Finally, the text classifier sets category labels that apply to the whole text, and adds them to the doc dot cats property. Because text categories are always very specific, the text classifier is not included in any of the pre-trained models by default. But you can use it to train your own system.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/builtin.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All models you can load into spaCy include several files and a meta JSON.\n",
    "\n",
    "The meta defines things like the language and pipeline. This tells spaCy which components to instantiate.\n",
    "\n",
    "The built-in components that make predictions also need binary data. The data is included in the model package and loaded into the component when you load the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/hood.png\" /> \n",
    "\n",
    "### Pipelines Attributes\n",
    "\n",
    "To see the names of the pipeline components present in the current nlp object, you can use the nlp dot pipe names attribute.\n",
    "\n",
    "For a list of component name and component function tuples, you can use the nlp dot pipeline attribute.\n",
    "\n",
    "The component functions are the functions applied to the Doc to process it and set attributes – for example, part-of-speech tags or named entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tagger', 'parser', 'ner']\n"
     ]
    }
   ],
   "source": [
    "# Import the English language class and create the nlp object\n",
    "from spacy.lang.en import English\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Process a text\n",
    "doc = nlp(\"This is a table.\")\n",
    "\n",
    "print(nlp.pipe_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('tagger', <spacy.pipeline.pipes.Tagger object at 0x000001A481A9D630>), ('parser', <spacy.pipeline.pipes.DependencyParser object at 0x000001A4869A7A08>), ('ner', <spacy.pipeline.pipes.EntityRecognizer object at 0x000001A4869A7A68>)]\n"
     ]
    }
   ],
   "source": [
    "print(nlp.pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. What happens when you call NLP?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does spaCy do when you call nlp on a string of text?\n",
    "\n",
    "Tokenize the text and apply each pipeline component in order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Inspecting the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s inspect the small English model’s pipeline!\n",
    "\n",
    "* Load the en_core_web_sm model and create the nlp object.\n",
    "\n",
    "* Print the names of the pipeline components using nlp.pipe_names.\n",
    "\n",
    "* Print the full pipeline of (name, component) tuples using nlp.pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tagger', 'parser', 'ner']\n",
      "[('tagger', <spacy.pipeline.pipes.Tagger object at 0x000001A484FDE5F8>), ('parser', <spacy.pipeline.pipes.DependencyParser object at 0x000001A485178CA8>), ('ner', <spacy.pipeline.pipes.EntityRecognizer object at 0x000001A485178D08>)]\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Load the en_core_web_sm model\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Print the names of the pipeline components\n",
    "print(nlp.pipe_names)\n",
    "\n",
    "# Print the full pipeline of (name, component) tuples\n",
    "print(nlp.pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Custom Pipeline components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you know how spaCy's pipeline works, let's take a look at another very powerful feature: custom pipeline components.\n",
    "\n",
    "Custom pipeline components let you add your own function to the spaCy pipeline that is executed when you call the nlp object on a text – for example, to modify the Doc and add more data to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the text is tokenized and a Doc object has been created, pipeline components are applied in order. spaCy supports a range of built-in components, but also lets you define your own.\n",
    "\n",
    "Custom components are executed automatically when you call the nlp object on a text.\n",
    "\n",
    "They're especially useful for adding your own custom metadata to documents and tokens.\n",
    "\n",
    "You can also use them to update built-in attributes, like the named entity spans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Fundamentally, a **pipeline component** is a function or callable that takes a doc, modifies it and returns it, so it can be processed by the next component in the pipeline.\n",
    "\n",
    "Components can be added to the pipeline using the nlp dot add pipe method. The method takes at least one argument: the component function.\n",
    "\n",
    "<img src=\"img/components.png\" /> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To specify where to add the component in the pipeline, you can use the following keyword arguments:\n",
    "\n",
    "Setting **last** to True will add the component last in the pipeline. This is the default behavior.\n",
    "\n",
    "Setting **first** to True will add the component first in the pipeline, right after the tokenizer.\n",
    "\n",
    "The \"before\" and \"after\" arguments let you define the name of an existing component to add the new component before or after. For example, before equals \"ner\" will add it before the named entity recognizer.\n",
    "\n",
    "The other component to add the new component before or after needs to exist, though – otherwise, spaCy will raise an error.\n",
    "\n",
    "<img src=\"img/components2.png\" /> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: a simple component\n",
    "\n",
    "Here's an example of a simple pipeline component.\n",
    "\n",
    "We start off with the small English model.\n",
    "\n",
    "We then define the component – a function that takes a Doc object and returns it.\n",
    "\n",
    "Let's do something simple and print the length of the Doc that passes through the pipeline.\n",
    "\n",
    "Don't forget to return the Doc so it can be processed by the next component in the pipeline! The Doc created by the tokenizer is passed through all components, so it's important that they all return the modified doc.\n",
    "\n",
    "We can now add the component to the pipeline. Let's add it to the very beginning right after the tokenizer by setting first equals True.\n",
    "\n",
    "When we print the pipeline component names, the custom component now shows up at the start. This means it will be applied first when we process a Doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline: ['custom_component', 'tagger', 'parser', 'ner']\n"
     ]
    }
   ],
   "source": [
    "# Create the nlp object\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Define a custom component\n",
    "def custom_component(doc):\n",
    "    # Print the doc's length\n",
    "    print('Doc length:', len(doc))\n",
    "    # Return the doc object\n",
    "    return doc\n",
    "\n",
    "# Add the component first in the pipeline\n",
    "nlp.add_pipe(custom_component, first=True)\n",
    "\n",
    "# Print the pipeline component names\n",
    "print('Pipeline:', nlp.pipe_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when we process a text using the nlp object, the custom component will be applied to the Doc and the length of the document will be printed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc length: 3\n"
     ]
    }
   ],
   "source": [
    "# Create the nlp object\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Define a custom component\n",
    "def custom_component(doc):\n",
    "\n",
    "    # Print the doc's length\n",
    "    print('Doc length:', len(doc))\n",
    "\n",
    "    # Return the doc object\n",
    "    return doc\n",
    "\n",
    "# Add the component first in the pipeline\n",
    "nlp.add_pipe(custom_component, first=True)\n",
    "\n",
    "# Process a text\n",
    "doc = nlp(\"Hello world!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Use Cases for Custom Components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which of these problems can be solved by custom pipeline components? Choose all that apply:\n",
    "\n",
    "1. Updating the pre-trained models and improving their predictions\n",
    "\n",
    "2. Computing your own values based on tokens and their attributes\n",
    "\n",
    "3. Adding named entities, for example based on a dictionary\n",
    "\n",
    "4. Implementing support for an additional language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: 2 and 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Simple Components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example shows a custom component that prints the character length of a document. Can you complete it?\n",
    "\n",
    "Complete the component function with the doc’s length.\n",
    "Add the length_component to the existing pipeline as the first component.\n",
    "Try out the new pipeline and process any text with the nlp object – for example “This is a sentence.”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['length_component', 'tagger', 'parser', 'ner']\n",
      "This document is 5 tokens long.\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Define the custom component\n",
    "def length_component(doc):\n",
    "    # Get the doc's length\n",
    "    doc_length = len(doc)\n",
    "    print(\"This document is {} tokens long.\".format(doc_length))\n",
    "    # Return the doc\n",
    "    return doc\n",
    "\n",
    "\n",
    "# Load the small English model\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Add the component first in the pipeline and print the pipe names\n",
    "nlp.add_pipe(length_component, first=True)\n",
    "print(nlp.pipe_names)\n",
    "\n",
    "# Process a text\n",
    "doc = nlp(\"This is a sentence.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Complex Components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, you’ll be writing a custom component that uses the PhraseMatcher to find animal names in the document and adds the matched spans to the doc.ents. A PhraseMatcher with the animal patterns has already been created as the variable matcher.\n",
    "\n",
    "* Define the custom component and apply the matcher to the doc.\n",
    "* Create a Span for each match, assign the label ID for 'ANIMAL' and overwrite the doc.ents with the new spans.\n",
    "* Add the new component to the pipeline after the 'ner' component.\n",
    "* Process the text and print the entity text and entity label for the entities in doc.ents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "animal_patterns: [Golden Retriever, cat, turtle, Rattus norvegicus]\n",
      "['tagger', 'parser', 'ner', 'animal_component']\n",
      "[('cat', 'ANIMAL'), ('Golden Retriever', 'ANIMAL')]\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy.matcher import PhraseMatcher\n",
    "from spacy.tokens import Span\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "animals = [\"Golden Retriever\", \"cat\", \"turtle\", \"Rattus norvegicus\"]\n",
    "animal_patterns = list(nlp.pipe(animals))\n",
    "print(\"animal_patterns:\", animal_patterns)\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "matcher.add(\"ANIMAL\", None, *animal_patterns)\n",
    "\n",
    "# Define the custom component\n",
    "def animal_component(doc):\n",
    "    # Apply the matcher to the doc\n",
    "    matches = matcher(doc)\n",
    "    # Create a Span for each match and assign the label 'ANIMAL'\n",
    "    spans = [Span(doc, start, end, label=\"ANIMAL\") for match_id, start, end in matches]\n",
    "    # Overwrite the doc.ents with the matched spans\n",
    "    doc.ents = spans\n",
    "    return doc\n",
    "\n",
    "\n",
    "# Add the component to the pipeline after the 'ner' component\n",
    "nlp.add_pipe(animal_component, after=\"ner\")\n",
    "print(nlp.pipe_names)\n",
    "\n",
    "# Process the text and print the text and label for the doc.ents\n",
    "doc = nlp(\"I have a cat and a Golden Retriever\")\n",
    "print([(ent.text, ent.label_) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Extension Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson, you'll learn how to add custom attributes to the Doc, Token and Span objects to store custom data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Custom attributes let you add any meta data to Docs, Tokens and Spans. The data can be added once, or it can be computed dynamically.\n",
    "\n",
    "Custom attributes are available via the dot-underscore property. This makes it clear that they were added by the user, and not built into spaCy, like token dot text.\n",
    "\n",
    "Attributes need to be registered on the global Doc, Token and Span classes you can import from spacy dot tokens. You've already worked with those in the previous chapters. To register a custom attribute on the Doc, Token or Span, you can use the set extension method.\n",
    "\n",
    "The first argument is the attribute name. Keyword arguments let you define how the value should be computed. In this case, it has a default value and can be overwritten."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting custom attributes\n",
    "\n",
    "* Add custom metadata to documents, tokens and spans\n",
    "* Accessible via the ._ property"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc._.title = 'My document'\n",
    "\n",
    "token._.is_color = True\n",
    "\n",
    "span._.has_color = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Registered on the global Doc, Token or Span using the set_extension method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import global classes\n",
    "from spacy.tokens import Doc, Token, Span\n",
    "\n",
    "# Set extensions on the Doc, Token and Span\n",
    "Doc.set_extension('title', default=None)\n",
    "Token.set_extension('is_color', default=False)\n",
    "Span.set_extension('has_color', default=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three types of extensions: \n",
    "    * attribute extensions, \n",
    "    * property extensions and \n",
    "    * method extensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attribute extensions\n",
    "\n",
    "Attribute extensions set a default value that can be overwritten.\n",
    "\n",
    "For example, a custom \"is color\" attribute on the token that defaults to False.\n",
    "\n",
    "On individual tokens, its value can be changed by overwriting it – in this case, True for the token \"blue\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.tokens import Token\n",
    "\n",
    "# Set extension on the Token with default value\n",
    "Token.set_extension('is_color', default=False, force=True)\n",
    "\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "\n",
    "# Overwrite extension attribute value\n",
    "doc[3]._.is_color = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Property extensions\n",
    "\n",
    "* Define a getter and an optional setter function\n",
    "* Getter only called when you retrieve the attribute value\n",
    "\n",
    "Property extensions work like properties in Python: they can define a getter function and an optional setter.\n",
    "\n",
    "The getter function is only called when you retrieve the attribute. This lets you compute the value dynamically, and even take other custom attributes into account.\n",
    "\n",
    "Getter functions take one argument: the object, in this case, the token. In this example, the function returns whether the token text is in our list of colors.\n",
    "\n",
    "We can then provide the function via the getter keyword argument when we register the extension.\n",
    "\n",
    "The token \"blue\" now returns True for \"is color\".\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True - blue\n"
     ]
    }
   ],
   "source": [
    "from spacy.tokens import Token\n",
    "\n",
    "# Define getter function\n",
    "def get_is_color(token):\n",
    "    colors = ['red', 'yellow', 'blue']\n",
    "    return token.text in colors\n",
    "\n",
    "# Set extension on the Token with getter\n",
    "Token.set_extension('is_color', getter=get_is_color, force=True)\n",
    "\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "print(doc[3]._.is_color, '-', doc[3].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Span extensions should almost always use a getter\n",
    "\n",
    "If you want to set extension attributes on a Span, you almost always want to use a property extension with a getter. Otherwise, you'd have to update every possible span ever by hand to set all the values.\n",
    "\n",
    "In this example, the \"get has color\" function takes the span and returns whether the text of any of the tokens is in the list of colors.\n",
    "\n",
    "After we've processed the doc, we can check different slices of the doc and the custom \"has color\" property returns whether the span contains a color token or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True - sky is blue\n",
      "False - The sky\n"
     ]
    }
   ],
   "source": [
    "from spacy.tokens import Span\n",
    "\n",
    "# Define getter function\n",
    "def get_has_color(span):\n",
    "    colors = ['red', 'yellow', 'blue']\n",
    "    return any(token.text in colors for token in span)\n",
    "\n",
    "# Set extension on the Span with getter\n",
    "Span.set_extension('has_color', getter=get_has_color, force=True)\n",
    "\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "print(doc[1:4]._.has_color, '-', doc[1:4].text)\n",
    "print(doc[0:2]._.has_color, '-', doc[0:2].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method extensions\n",
    "\n",
    "* Assign a function that becomes available as an object method\n",
    "* Lets you pass arguments to the extension function\n",
    "\n",
    "Method extensions make the extension attribute a callable method.\n",
    "\n",
    "You can then pass one or more arguments to it, and compute attribute values dynamically – for example, based on a certain argument or setting.\n",
    "\n",
    "In this example, the method function checks whether the doc contains a token with a given text. The first argument of the method is always the object itself – in this case, the Doc. It's passed in automatically when the method is called. All other function arguments will be arguments on the method extension. In this case, \"token text\".\n",
    "\n",
    "Here, the custom \"has token\" method returns True for the word \"blue\" and False for the word \"cloud\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True - blue\n",
      "False - cloud\n"
     ]
    }
   ],
   "source": [
    "from spacy.tokens import Doc\n",
    "\n",
    "# Define method with arguments\n",
    "def has_token(doc, token_text):\n",
    "    in_doc = token_text in [token.text for token in doc]\n",
    "    return in_doc\n",
    "\n",
    "# Set extension on the Doc with method\n",
    "Doc.set_extension('has_token', method=has_token)\n",
    "\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "print(doc._.has_token('blue'), '- blue')\n",
    "print(doc._.has_token('cloud'), '- cloud')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Setting extension attributes\n",
    "\n",
    "Let’s practice setting some extension attributes.\n",
    "\n",
    "Step 1\n",
    "\n",
    "* Use Token.set_extension to register is_country (default False).\n",
    "* Update it for \"Spain\" and print it for all tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('I', False), ('live', False), ('in', False), ('Spain', True), ('.', False)]\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "from spacy.tokens import Token\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "# Register the Token extension attribute 'is_country' with the default value False\n",
    "Token.set_extension(\"is_country\", default=False)\n",
    "\n",
    "# Process the text and set the is_country attribute to True for the token \"Spain\"\n",
    "doc = nlp(\"I live in Spain.\")\n",
    "doc[3]._.is_country = True\n",
    "\n",
    "# Print the token text and the is_country attribute for all tokens\n",
    "print([(token.text, token._.is_country) for token in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 2\n",
    "* Use Token.set_extension to register 'reversed' (getter function get_reversed).\n",
    "* Print its value for each token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reversed: llA\n",
      "reversed: snoitazilareneg\n",
      "reversed: era\n",
      "reversed: eslaf\n",
      "reversed: ,\n",
      "reversed: gnidulcni\n",
      "reversed: siht\n",
      "reversed: eno\n",
      "reversed: .\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "from spacy.tokens import Token\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "# Define the getter function that takes a token and returns its reversed text\n",
    "def get_reversed(token):\n",
    "    return token.text[::-1]\n",
    "\n",
    "\n",
    "# Register the Token property extension 'reversed' with the getter get_reversed\n",
    "Token.set_extension(\"reversed\", getter=get_reversed)\n",
    "\n",
    "# Process the text and print the reversed attribute for each token\n",
    "doc = nlp(\"All generalizations are false, including this one.\")\n",
    "for token in doc:\n",
    "    print(\"reversed:\", token._.reversed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Setting extension attributes\n",
    "\n",
    "Let’s try setting some more complex attributes using getters and method extensions.\n",
    "\n",
    "Part 1\n",
    "* Complete the has_number function .\n",
    "* Use Doc.set_extension to register has_number (getter get_has_number) and print its value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Let’s try setting some more complex attributes using getters and method extensions.\n",
    "\n",
    "Part 1\n",
    "Complete the has_number function .\n",
    "Use Doc.set_extension to register has_number (getter get_has_number) and print its value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "has_number: True\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "# Define the getter function\n",
    "def get_has_number(doc):\n",
    "    # Return if any of the tokens in the doc return True for token.like_num\n",
    "    return any(token.like_num for token in doc)\n",
    "\n",
    "# Register the Doc property extension 'has_number' with the getter get_has_number\n",
    "Doc.set_extension(\"has_number\", getter=get_has_number)\n",
    "\n",
    "# Process the text and check the custom has_number attribute\n",
    "doc = nlp(\"The museum closed for five years in 2012.\")\n",
    "print(\"has_number:\", doc._.has_number)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 2\n",
    "* Use Span.set_extension to register 'to_html' (method to_html).\n",
    "* Call it on doc[0:2] with the tag 'strong'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<strong>Hello world</strong>\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "from spacy.tokens import Span\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "# Define the method\n",
    "def to_html(span, tag):\n",
    "    # Wrap the span text in a HTML tag and return it\n",
    "    return \"<{tag}>{text}</{tag}>\".format(tag=tag, text=span.text)\n",
    "\n",
    "\n",
    "# Register the Span property extension 'to_html' with the method to_html\n",
    "Span.set_extension(\"to_html\", method=to_html)\n",
    "\n",
    "# Process the text and call the to_html method on the span with the tag name 'strong'\n",
    "doc = nlp(\"Hello world, this is a sentence.\")\n",
    "span = doc[0:2]\n",
    "print(span._.to_html(\"strong\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Entities and extensions\n",
    "\n",
    "In this exercise, you’ll combine custom extension attributes with the model’s predictions and create an attribute getter that returns a Wikipedia search URL if the span is a person, organization, or location.\n",
    "\n",
    "* Complete the get_wikipedia_url getter so it only returns the URL if the span’s label is in the list of labels.\n",
    "* Set the Span extension 'wikipedia_url' using the getter get_wikipedia_url.\n",
    "* Iterate over the entities in the doc and output their Wikipedia URL.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "over fifty years None\n",
      "first None\n",
      "David Bowie https://en.wikipedia.org/w/index.php?search=David_Bowie\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy.tokens import Span\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "\n",
    "def get_wikipedia_url(span):\n",
    "    # Get a Wikipedia URL if the span has one of the labels\n",
    "    if span.label_ in (\"PERSON\", \"ORG\", \"GPE\", \"LOCATION\"):\n",
    "        entity_text = span.text.replace(\" \", \"_\")\n",
    "        return \"https://en.wikipedia.org/w/index.php?search=\" + entity_text\n",
    "\n",
    "\n",
    "# Set the Span extension wikipedia_url using get getter get_wikipedia_url\n",
    "Span.set_extension(\"wikipedia_url\", getter=get_wikipedia_url)\n",
    "\n",
    "doc = nlp(\n",
    "    \"In over fifty years from his very first recordings right through to his \"\n",
    "    \"last album, David Bowie was at the vanguard of contemporary culture.\"\n",
    ")\n",
    "for ent in doc.ents:\n",
    "    # Print the text and Wikipedia URL of the entity\n",
    "    print(ent.text, ent._.wikipedia_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Components with extensions\n",
    "\n",
    "Extension attributes are especially powerful if they’re combined with custom pipeline components. In this exercise, you’ll write a pipeline component that finds country names and a custom extension attribute that returns a country’s capital, if available.\n",
    "\n",
    "A phrase matcher with all countries is available as the variable matcher. A dictionary of countries mapped to their capital cities is available as the variable CAPITALS.\n",
    "\n",
    "* Complete the countries_component and create a Span with the label 'GPE' (geopolitical entity) for all matches.\n",
    "* Add the component to the pipeline.\n",
    "* Register the Span extension attribute 'capital' with the getter get_capital.\n",
    "* Process the text and print the entity text, entity label and entity capital for each entity span in doc.ents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['countries_component']\n",
      "[('Czech Republic', 'GPE', 'Prague'), ('Slovakia', 'GPE', 'Bratislava')]\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from spacy.lang.en import English\n",
    "from spacy.tokens import Span\n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "with open(\"exercises/countries.json\") as f:\n",
    "    COUNTRIES = json.loads(f.read())\n",
    "\n",
    "with open(\"exercises/capitals.json\") as f:\n",
    "    CAPITALS = json.loads(f.read())\n",
    "\n",
    "nlp = English()\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "matcher.add(\"COUNTRY\", None, *list(nlp.pipe(COUNTRIES)))\n",
    "\n",
    "\n",
    "def countries_component(doc):\n",
    "    # Create an entity Span with the label 'GPE' for all matches\n",
    "    matches = matcher(doc)\n",
    "    doc.ents = [Span(doc, start, end, label=\"GPE\") for match_id, start, end in matches]\n",
    "    return doc\n",
    "\n",
    "\n",
    "# Add the component to the pipeline\n",
    "nlp.add_pipe(countries_component)\n",
    "print(nlp.pipe_names)\n",
    "\n",
    "# Getter that looks up the span text in the dictionary of country capitals\n",
    "get_capital = lambda span: CAPITALS.get(span.text)\n",
    "\n",
    "# Register the Span extension attribute 'capital' with the getter get_capital\n",
    "Span.set_extension(\"capital\", getter=get_capital)\n",
    "\n",
    "# Process the text and print the entity text, label and capital attributes\n",
    "doc = nlp(\"Czech Republic may help Slovakia protect its airspace\")\n",
    "print([(ent.text, ent.label_, ent._.capital) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Scaling and performance\n",
    "\n",
    "In this lesson, I'll show you a few tips and tricks to make your spaCy pipelines run as fast as possible, and process large volumes of text efficiently.\n",
    "\n",
    "### Processing large volumes of text\n",
    "\n",
    "* Use nlp.pipe method\n",
    "* Processes texts as a stream, yields Doc objects\n",
    "* Much faster than calling nlp on each text\n",
    "\n",
    "BAD:\n",
    "\n",
    "docs = [nlp(text) for text in LOTS_OF_TEXTS]\n",
    "\n",
    "GOOD:\n",
    "\n",
    "docs = list(nlp.pipe(LOTS_OF_TEXTS))\n",
    "\n",
    "If you need to process a lot of texts and create a lot of Doc objects in a row, the nlp dot pipe method can speed this up significantly.\n",
    "\n",
    "It processes the texts as a stream and yields Doc objects.\n",
    "\n",
    "It is much faster than just calling nlp on each text, because it batches up the texts.\n",
    "\n",
    "nlp dot pipe is a generator that yields Doc objects, so in order to get a list of Docs, remember to call the list method around it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passing in context\n",
    "\n",
    "* Setting as_tuples=True on nlp.pipe lets you pass in (text, context) tuples\n",
    "* Yields (doc, context) tuples\n",
    "* Useful for associating metadata with the doc\n",
    "\n",
    "nlp dot pipe also supports passing in tuples of text / context if you set \"as tuples\" to True.\n",
    "\n",
    "The method will then yield doc / context tuples.\n",
    "\n",
    "This is useful for passing in additional metadata, like an ID associated with the text, or a page number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a text 15\n",
      "And another text 16\n"
     ]
    }
   ],
   "source": [
    "data = [\n",
    "    ('This is a text', {'id': 1, 'page_number': 15}),\n",
    "    ('And another text', {'id': 2, 'page_number': 16}),\n",
    "]\n",
    "\n",
    "for doc, context in nlp.pipe(data, as_tuples=True):\n",
    "    print(doc.text, context['page_number'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even add the context meta data to custom attributes.\n",
    "\n",
    "In this example, we're registering two extensions, \"id\" and \"page number\", which default to None.\n",
    "\n",
    "After processing the text and passing through the context, we can overwrite the doc extensions with our context metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.tokens import Doc\n",
    "\n",
    "Doc.set_extension('id', default=None)\n",
    "Doc.set_extension('page_number', default=None)\n",
    "\n",
    "data = [\n",
    "    ('This is a text', {'id': 1, 'page_number': 15}),\n",
    "    ('And another text', {'id': 2, 'page_number': 16}),\n",
    "]\n",
    "\n",
    "for doc, context in nlp.pipe(data, as_tuples=True):\n",
    "    doc._.id = context['id']\n",
    "    doc._.page_number = context['page_number']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using only the tokenizer\n",
    "\n",
    "Another common scenario: Sometimes you already have a model loaded to do other processing, but you only need the tokenizer for one particular text.\n",
    "\n",
    "Running the whole pipeline is unnecessarily slow, because you'll be getting a bunch of predictions from the model that you don't need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/tokenizer.png\" /> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you only need a tokenized Doc object, you can use the nlp dot make doc method instead, which takes a text and returns a Doc.\n",
    "\n",
    "This is also how spaCy does it behind the scenes: nlp dot make doc turns the text into a Doc before the pipeline components are called.\n",
    "\n",
    "Use nlp.make_doc to turn a text in to a Doc object\n",
    "\n",
    "BAD:\n",
    "\n",
    "doc = nlp(\"Hello world\")\n",
    "\n",
    "GOOD:\n",
    "\n",
    "doc = nlp.make_doc(\"Hello world!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Disabling pipeline components\n",
    "\n",
    "spaCy also allows you to temporarily disable pipeline components using the nlp dot disable pipes context manager.\n",
    "\n",
    "It takes a variable number of arguments, the string names of the pipeline components to disable. For example, if you only want to use the entity recognizer to process a document, you can temporarily disable the tagger and parser.\n",
    "\n",
    "After the with block, the disabled pipeline components are automatically restored.\n",
    "\n",
    "In the with block, spaCy will only run the remaining components.\n",
    "\n",
    "* Use nlp.disable_pipes to temporarily disable one or more pipes\n",
    "* Restores them after the with block\n",
    "* Only runs the remaining components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(American, College Park, Georgia)\n"
     ]
    }
   ],
   "source": [
    "# Disable tagger and parser\n",
    "with nlp.disable_pipes('tagger', 'parser'):\n",
    "    # Process the text and print the entities\n",
    "    doc = nlp(text)\n",
    "    print(doc.ents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14. Processing streams\n",
    "\n",
    "In this exercise, you’ll be using nlp.pipe for more efficient text processing. The nlp object has already been created for you. A list of tweets about a popular American fast food chain are available as the variable TEXTS.\n",
    "\n",
    "Part 1\n",
    "\n",
    "* Rewrite the example to use nlp.pipe. Instead of iterating over the texts and processing them, iterate over the doc objects yielded by nlp.pipe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['favorite']\n",
      "['sick']\n",
      "[]\n",
      "['happy']\n",
      "['delicious', 'fast']\n",
      "[]\n",
      "['terrible', 'gettin', 'payin']\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "with open(\"exercises/tweets.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "# Process the texts and print the adjectives\n",
    "for text in TEXTS:\n",
    "    doc = nlp(text)\n",
    "    print([token.text for token in doc if token.pos_ == \"ADJ\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['favorite']\n",
      "['sick']\n",
      "[]\n",
      "['happy']\n",
      "['delicious', 'fast']\n",
      "[]\n",
      "['terrible', 'gettin', 'payin']\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "with open(\"exercises/tweets.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "# Process the texts and print the adjectives\n",
    "for doc in nlp.pipe(TEXTS):\n",
    "    print([token.text for token in doc if token.pos_ == \"ADJ\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rewrite the example to use nlp.pipe. Don’t forget to call list() around the result to turn it into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(McDonalds,) (@McDonalds,) (McDonalds,) (McDonalds, Spain) (The Arch Deluxe,) (WANT, McRib) (This morning,)\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "with open(\"exercises/tweets.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "# Process the texts and print the entities\n",
    "docs = [nlp(text) for text in TEXTS]\n",
    "entities = [doc.ents for doc in docs]\n",
    "print(*entities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(McDonalds,) (@McDonalds,) (McDonalds,) (McDonalds, Spain) (The Arch Deluxe,) (WANT, McRib) (This morning,)\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "with open(\"exercises/tweets.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "# Process the texts and print the entities\n",
    "docs = list(nlp.pipe(TEXTS))\n",
    "entities = [doc.ents for doc in docs]\n",
    "print(*entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rewrite the example to use nlp.pipe. Don’t forget to call list() around the result to turn it into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[David Bowie, Angela Merkel, Lady Gaga]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "people = [\"David Bowie\", \"Angela Merkel\", \"Lady Gaga\"]\n",
    "\n",
    "# Create a list of patterns for the PhraseMatcher\n",
    "patterns = [nlp(person) for person in people]\n",
    "patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[David Bowie, Angela Merkel, Lady Gaga]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from spacy.lang.en import English\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "people = [\"David Bowie\", \"Angela Merkel\", \"Lady Gaga\"]\n",
    "\n",
    "# Create a list of patterns for the PhraseMatcher\n",
    "patterns = list(nlp.pipe(people))\n",
    "patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15 Processing data with context\n",
    "\n",
    "In this exercise, you’ll be using custom attributes to add author and book meta information to quotes.\n",
    "\n",
    "A list of [text, context] examples is available as the variable DATA. The texts are quotes from famous books, and the contexts dictionaries with the keys 'author' and 'book'.\n",
    "\n",
    "Use the set_extension method to register the custom attributes 'author' and 'book' on the Doc, which default to None.\n",
    "Process the [text, context] pairs in DATA using nlp.pipe with as_tuples=True.\n",
    "Overwrite the doc._.book and doc._.author with the respective info passed in as the context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin. \n",
      " — 'Metamorphosis' by Franz Kafka \n",
      "\n",
      "I know not all that may be coming, but be it what it will, I'll go to it laughing. \n",
      " — 'Moby-Dick or, The Whale' by Herman Melville \n",
      "\n",
      "It was the best of times, it was the worst of times. \n",
      " — 'A Tale of Two Cities' by Charles Dickens \n",
      "\n",
      "The only people for me are the mad ones, the ones who are mad to live, mad to talk, mad to be saved, desirous of everything at the same time, the ones who never yawn or say a commonplace thing, but burn, burn, burn like fabulous yellow roman candles exploding like spiders across the stars. \n",
      " — 'On the Road' by Jack Kerouac \n",
      "\n",
      "It was a bright cold day in April, and the clocks were striking thirteen. \n",
      " — '1984' by George Orwell \n",
      "\n",
      "Nowadays people know the price of everything and the value of nothing. \n",
      " — 'The Picture Of Dorian Gray' by Oscar Wilde \n",
      "\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from spacy.lang.en import English\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "with open(\"exercises/bookquotes.json\") as f:\n",
    "    DATA = json.loads(f.read())\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "# Register the Doc extension 'author' (default None)\n",
    "Doc.set_extension(\"author\", default=None)\n",
    "\n",
    "# Register the Doc extension 'book' (default None)\n",
    "Doc.set_extension(\"book\", default=None)\n",
    "\n",
    "for doc, context in nlp.pipe(DATA, as_tuples=True):\n",
    "    # Set the doc._.book and doc._.author attributes from the context\n",
    "    doc._.book = context[\"book\"]\n",
    "    doc._.author = context[\"author\"]\n",
    "\n",
    "    # Print the text and custom attribute data\n",
    "    print(doc.text, \"\\n\", \"— '{}' by {}\".format(doc._.book, doc._.author), \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16 Selective processing\n",
    "\n",
    "In this exercise, you’ll use the nlp.make_doc and nlp.disable_pipes methods to only run selected components when processing a text.\n",
    "\n",
    "Part 1\n",
    "\n",
    "Rewrite the code to only tokenize the text using nlp.make_doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Chick', '-', 'fil', '-', 'A', 'is', 'an', 'American', 'fast', 'food', 'restaurant', 'chain', 'headquartered', 'in', 'the', 'city', 'of', 'College', 'Park', ',', 'Georgia', ',', 'specializing', 'in', 'chicken', 'sandwiches', '.']\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = (\n",
    "    \"Chick-fil-A is an American fast food restaurant chain headquartered in \"\n",
    "    \"the city of College Park, Georgia, specializing in chicken sandwiches.\"\n",
    ")\n",
    "\n",
    "# Only tokenize the text\n",
    "doc = nlp(text)\n",
    "print([token.text for token in doc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Chick', '-', 'fil', '-', 'A', 'is', 'an', 'American', 'fast', 'food', 'restaurant', 'chain', 'headquartered', 'in', 'the', 'city', 'of', 'College', 'Park', ',', 'Georgia', ',', 'specializing', 'in', 'chicken', 'sandwiches', '.']\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = (\n",
    "    \"Chick-fil-A is an American fast food restaurant chain headquartered in \"\n",
    "    \"the city of College Park, Georgia, specializing in chicken sandwiches.\"\n",
    ")\n",
    "\n",
    "# Only tokenize the text\n",
    "doc = nlp.make_doc(text)\n",
    "print([token.text for token in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 2\n",
    "\n",
    "Disable the tagger and parser using the nlp.disable_pipes method.\n",
    "\n",
    "Process the text and print all entities in the doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(American, College Park, Georgia)\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = (\n",
    "    \"Chick-fil-A is an American fast food restaurant chain headquartered in \"\n",
    "    \"the city of College Park, Georgia, specializing in chicken sandwiches.\"\n",
    ")\n",
    "\n",
    "# Disable the tagger and parser\n",
    "with nlp.disable_pipes(\"tagger\", \"parser\"):\n",
    "    # Process the text\n",
    "    doc = nlp(text)\n",
    "    # Print the entities in the doc\n",
    "    print(doc.ents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contens",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "332.807px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}