03e - Optical Character Recognition.ipynb

{"cells":[{"cell_type":"markdown","source":["# Optical Character Recognition\n","\n","![A robot reading a newspaper](./images/ocr.jpg)\n","\n","A common computer vision challenge is to detect and interpret text in an image. This kind of processing is often referred to as *optical character recognition* (OCR).\n","\n","## Use the Computer Vision Service to Read Text in an Image\n","\n","The **Computer Vision** cognitive service provides support for OCR tasks, including:\n","\n","- An **OCR** API that you can use to read text in multiple languages. This API can be used synchronously, and works well when you need to detect and read a small amount of text in an image.\n","- A **Read** API that is optimized for larger documents. This API is used asynchronously, and can be used for both printed and handwritten text.\n","\n","You can use this service by creating either a **Computer Vision** resource or a **Cognitive Services** resource.\n","\n","If you haven't already done so, create a **Cognitive Services** resource in your Azure subscription.\n","\n","1. In another browser tab, open the Azure portal at https://portal.azure.com, and sign in with your Microsoft account.\n","2. Click the **&#65291;Create a resource** button, search for *Cognitive Services*, and create a **Cognitive Services** resource with the following settings:\n","    - **Name**: *Enter a unique name*.\n","    - **Subscription**: *Your Azure subscription*.\n","    - **Location**: *Any available location*.\n","    - **Pricing tier**: S0\n","    - **Resource group**: *Create a resource group with a unique name*.\n","3. Wait for deployment to complete. Then go to your cognitive services resource, and on the **Overview** page, click the link to manage the keys for the service. You will need the endpoint and keys to connect to your cognitive services resource from client applications.\n","\n","### Get the Key and Endpoint for your Cognitive Services resource\n","\n","To use your cognitive services resource, client applications need its  endpoint and authentication key:\n","\n","1. In the Azure portal, on the **Keys and Endpoint** page for your cognitive service resource, copy the **Key1** for your resource and paste it in the code below, replacing **YOUR_COG_KEY**.\n","2. Copy the **endpoint** for your resource and and paste it in the code below, replacing **YOUR_COG_ENDPOINT**.\n","3. Run the code in the cell below by clicking the **Run cell** (&#9655;) button (to the left of the cell)."],"metadata":{}},{"cell_type":"code","source":["cog_key = 'YOUR_COG_KEY'\n","cog_endpoint = 'YOUR_COG_ENDPOINT'\n","\n","print('Ready to use cognitive services at {} using key {}'.format(cog_endpoint, cog_key))"],"outputs":[],"execution_count":null,"metadata":{"gather":{"logged":1599694246277}}},{"cell_type":"markdown","source":["Now that you've set up the key and endpoint, you can use your computer vision service resource to extract text from an image.\n","\n","Let's start with the **OCR** API, which enables you to synchronously analyze an image and read any text it contains. In this case, you have an adventising image for the fictional Northwind Traders retail company that includes some text. Run the cell below to read it.. "],"metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":["from azure.cognitiveservices.vision.computervision import ComputerVisionClient\n","from msrest.authentication import CognitiveServicesCredentials\n","import matplotlib.pyplot as plt\n","from PIL import Image, ImageDraw\n","import os\n","%matplotlib inline\n","\n","# Get a client for the computer vision service\n","computervision_client = ComputerVisionClient(cog_endpoint, CognitiveServicesCredentials(cog_key))\n","\n","# Read the image file\n","image_path = os.path.join('data', 'ocr', 'advert.jpg')\n","image_stream = open(image_path, \"rb\")\n","\n","# Use the Computer Vision service to find text in the image\n","read_results = computervision_client.recognize_printed_text_in_stream(image_stream)\n","\n","# Process the text line by line\n","for region in read_results.regions:\n","    for line in region.lines:\n","\n","        # Read the words in the line of text\n","        line_text = ''\n","        for word in line.words:\n","            line_text += word.text + ' '\n","        print(line_text.rstrip())\n","\n","# Open image to display it.\n","fig = plt.figure(figsize=(7, 7))\n","img = Image.open(image_path)\n","draw = ImageDraw.Draw(img)\n","plt.axis('off')\n","plt.imshow(img)"],"outputs":[],"execution_count":null,"metadata":{"gather":{"logged":1599694257280}}},{"cell_type":"markdown","source":["The text found in the image is organized into a hierarchical structure of regions, lines, and words, and the code reads these to retrieve the results.\n","\n","In the results, view the text that was read above the image. \n","\n","## Display bounding boxes\n","\n","The results also include *bounding box* coordinates for the lines of text and individual words found in the image. Run the cell below to see the bounding boxes for the lines of text in the advertising image you retrieved above."],"metadata":{}},{"cell_type":"code","source":["# Open image to display it.\n","fig = plt.figure(figsize=(7, 7))\n","img = Image.open(image_path)\n","draw = ImageDraw.Draw(img)\n","\n","# Process the text line by line\n","for region in read_results.regions:\n","    for line in region.lines:\n","\n","        # Show the position of the line of text\n","        l,t,w,h = list(map(int, line.bounding_box.split(',')))\n","        draw.rectangle(((l,t), (l+w, t+h)), outline='magenta', width=5)\n","\n","        # Read the words in the line of text\n","        line_text = ''\n","        for word in line.words:\n","            line_text += word.text + ' '\n","        print(line_text.rstrip())\n","\n","# Show the image with the text locations highlighted\n","plt.axis('off')\n","plt.imshow(img)"],"outputs":[],"execution_count":null,"metadata":{"gather":{"logged":1599694266106}}},{"cell_type":"markdown","source":["In the result, the bounding box for each line of text is shown as a rectangle on the image.\n","\n","## Use the Read API\n","\n","The OCR API you used previously works well for images with a small amount of text. When you need to read larger bodies of text, such as scanned documents, you can use the **Read** API. This requires a multi-step process:\n","\n","1. Submit an image to the Computer Vision service to be read and analyzed asynchronously.\n","2. Wait for the analysis operation to complete.\n","3. Retrieve the results of the analysis.\n","\n","Run the following cell to use this process to read the text in a scanned letter to the manager of a Northwind Traders store."],"metadata":{}},{"cell_type":"code","source":["from azure.cognitiveservices.vision.computervision import ComputerVisionClient\n","from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes\n","from msrest.authentication import CognitiveServicesCredentials\n","import matplotlib.pyplot as plt\n","from PIL import Image\n","import time\n","import os\n","%matplotlib inline\n","\n","# Read the image file\n","image_path = os.path.join('data', 'ocr', 'letter.jpg')\n","image_stream = open(image_path, \"rb\")\n","\n","# Get a client for the computer vision service\n","computervision_client = ComputerVisionClient(cog_endpoint, CognitiveServicesCredentials(cog_key))\n","\n","# Submit a request to read printed text in the image and get the operation ID\n","read_operation = computervision_client.read_in_stream(image_stream,\n","                                                      raw=True)\n","operation_location = read_operation.headers[\"Operation-Location\"]\n","operation_id = operation_location.split(\"/\")[-1]\n","\n","# Wait for the asynchronous operation to complete\n","while True:\n","    read_results = computervision_client.get_read_result(operation_id)\n","    if read_results.status not in [OperationStatusCodes.running]:\n","        break\n","    time.sleep(1)\n","\n","# If the operation was successfuly, process the text line by line\n","if read_results.status == OperationStatusCodes.succeeded:\n","    for result in read_results.analyze_result.read_results:\n","        for line in result.lines:\n","            print(line.text)\n","\n","# Open image and display it.\n","print('\\n')\n","fig = plt.figure(figsize=(12,12))\n","img = Image.open(image_path)\n","plt.axis('off')\n","plt.imshow(img)"],"outputs":[],"execution_count":null,"metadata":{"gather":{"logged":1599694312346}}},{"cell_type":"markdown","source":["Review the results. There's a full transcription of the letter, which consists mostly of printed text with a handwritten signature. The original image of the letter is shown beneath the OCR results (you may need to scroll to see it).\n","\n","## Read handwritten text\n","\n","In the previous example, the request to analyze the image specified a text recognition mode that optimized the operation for *printed* text. Note that despite this, the handwritten signature was read.\n","\n","This ability to read handwritten text is extremely useful. For example, suppose you've written a note containing a shopping list, and want to use an app on your phone to read the note and transcribe the text it contains.\n","\n","Run the cell below to see an example of a read operation for a handwritten shopping list."],"metadata":{}},{"cell_type":"code","source":["from azure.cognitiveservices.vision.computervision import ComputerVisionClient\n","from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes\n","from msrest.authentication import CognitiveServicesCredentials\n","import matplotlib.pyplot as plt\n","from PIL import Image\n","import time\n","import os\n","%matplotlib inline\n","\n","# Read the image file\n","image_path = os.path.join('data', 'ocr', 'note.jpg')\n","image_stream = open(image_path, \"rb\")\n","\n","# Get a client for the computer vision service\n","computervision_client = ComputerVisionClient(cog_endpoint, CognitiveServicesCredentials(cog_key))\n","\n","# Submit a request to read printed text in the image and get the operation ID\n","read_operation = computervision_client.read_in_stream(image_stream,\n","                                                      raw=True)\n","operation_location = read_operation.headers[\"Operation-Location\"]\n","operation_id = operation_location.split(\"/\")[-1]\n","\n","# Wait for the asynchronous operation to complete\n","while True:\n","    read_results = computervision_client.get_read_result(operation_id)\n","    if read_results.status not in [OperationStatusCodes.running]:\n","        break\n","    time.sleep(1)\n","\n","# If the operation was successfuly, process the text line by line\n","if read_results.status == OperationStatusCodes.succeeded:\n","    for result in read_results.analyze_result.read_results:\n","        for line in result.lines:\n","            print(line.text)\n","\n","# Open image and display it.\n","print('\\n')\n","fig = plt.figure(figsize=(12,12))\n","img = Image.open(image_path)\n","plt.axis('off')\n","plt.imshow(img)"],"outputs":[],"execution_count":null,"metadata":{"gather":{"logged":1599694340593}}},{"cell_type":"markdown","source":["## More Information\n","\n","For more information about using the Computer Vision service for OCR, see [the Computer Vision documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-recognizing-text)"],"metadata":{}}],"metadata":{"language_info":{"name":"python","version":"3.6.9","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"name":"python3-azureml","language":"python","display_name":"Python 3.6 - AzureML"},"kernel_info":{"name":"python3-azureml"},"nteract":{"version":"nteract-front-end@1.0.0"}},"nbformat":4,"nbformat_minor":2}