Lesson 6 - File access in Python3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading and Writing files\n",
    "\n",
    "So far, we have typed all of our \"data\" into the code of our software (e.g. the names of the students, and their ages.\n",
    "\n",
    "Most of the time, this kind of data is stored in files.  We need to read (and write) files so that we can create and use permanent copies of these data (and exchange these data with other people or software)\n",
    "\n",
    "The function we will use to open a file is called (surprise!) \"open\"\n",
    "\n",
    "open takes two arguments\":\n",
    "1. the path to the file\n",
    "2. \"how\" to open the file (for read?  for write?  for \"append\"? for read and write?)\n",
    "\n",
    "it looks like this:\n",
    "\n",
    "    myfile = open(\"/path/to/file.csv\", \"r\")  # opens file.csv for \"read\"\n",
    "    \n",
    "I have already created a file called students.csv that we can open now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_io.TextIOWrapper name='students.csv' mode='r' encoding='UTF-8'>\n"
     ]
    }
   ],
   "source": [
    "studentfile = open(\"students.csv\", \"r\")\n",
    "print(studentfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Does the output of that print statement surprise you?  What it tells you is that 'studentfile' is a Python \"object\" (again, we will discuss objects in more detail later, but you will start to see how they work now...)\n",
    "\n",
    "studentfile is an object of type \"TextIOWrapper\" (from the \"io\" Python library, which is automatically installed in all Python distributions).  It knows what its filename is, it knows that it is open for \"read\", and it also has guessed the \"encoding\" of the file (UTF-8 is a kind of text encoding that allows extended text characters like the German umlaut's, and greek alpha, beta, etc.   This is a good default for us!)\n",
    "\n",
    "# Reading information from a file\n",
    "\n",
    "surprise!  The most basic method used to read information is.... 'read'!  This reads **the entire file**\n",
    "\n",
    "    print(studentfile.read())\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mark,53\n",
      "Alejandro,25\n",
      "Julia,26\n",
      "Denise,23\n",
      "Josef,21\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(studentfile.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now we need to talk about a feature of file input/output, called a \"pointer\".  The pointer is the position where the code \"is\" in the file - is it at the beginning?  is it at the end\"?  Is it at line 5?\n",
    "\n",
    "Where is the pointer now?  Let's try the same command again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print(studentfile.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Nada de nada!  That's because the pointer is at the end of the file - when we say file.read it tries to read starting from the end of the file...and of course, there is nothing there.  \n",
    "\n",
    "To reset back to the beginning, we will use the \"seek\" function, and set it to position '0':\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mark,53\n",
      "Alejandro,25\n",
      "Julia,26\n",
      "Denise,23\n",
      "Josef,21\n",
      "\n"
     ]
    }
   ],
   "source": [
    "studentfile.seek(0)\n",
    "print(studentfile.read())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## More refined file access - line-by-line\n",
    "\n",
    "Most of the time, you do not want to read the enire file into memory (tell me why this can be very very bad!.... please)\n",
    "\n",
    "MOST of the time, a file will have one \"record\" per line.  e.g. our CSV file has the \"name,age\" for one student per line.  We want to read those lines one-at-a-time and do something useful with each record.\n",
    "\n",
    "The method we want to use is called \"readlines()\"\n",
    "\n",
    "    print(studentfile.readlines())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Mark,53\\n', 'Alejandro,25\\n', 'Julia,26\\n', 'Denise,23\\n', 'Josef,21\\n']\n"
     ]
    }
   ],
   "source": [
    "studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "\n",
    "print(studentfile.readlines())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "You will see that this returns a list, which means we can use it in a FOR loop...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the current record is Mark,53\n",
      "\n",
      "the current record is Alejandro,25\n",
      "\n",
      "the current record is Julia,26\n",
      "\n",
      "the current record is Denise,23\n",
      "\n",
      "the current record is Josef,21\n",
      "\n"
     ]
    }
   ],
   "source": [
    "studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "\n",
    "for line in studentfile.readlines():\n",
    "    print(\"the current record is\", line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We're getting closer to what we want!  We have each record as a string in the format \"Mark,50\".  What we want is to separate the \"Mark\" and the \"50\" so that we could put them into separate variables (e.g. *name* and *age*)\n",
    "\n",
    "There is a ***correct*** way to this, but you already know one way to solve this problem!  \n",
    "\n",
    "In the box below, use regular expressions to capture the name and the age into the variables *name* and *age*\n",
    "\n",
    "<p style=\"visibility: hidden;\">\n",
    "#!/usr/bin/python3\n",
    "import re  # this brings the python regular expression object into your program\n",
    "\n",
    "studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "\n",
    "for line in studentfile.readlines():\n",
    "    #print(\"the current record is\", line)\n",
    "    matchObj = re.search( r'(\\w+),(\\d+)', line)  # match the index letter, then CAPTURE the rest of the sentence\n",
    "    if matchObj:\n",
    "        name = matchObj.group(1)\n",
    "        age = matchObj.group(2)\n",
    "        print(\"Name: \", name, \"   Age: \", age)\n",
    "    else:\n",
    "        print (\"No match!!\")\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: Mark Age: 53\n",
      "Name: Alejandro Age: 25\n",
      "Name: Julia Age: 26\n",
      "Name: Denise Age: 23\n",
      "Name: Josef Age: 21\n"
     ]
    }
   ],
   "source": [
    "# put your amazing solution here!\n",
    "import re\n",
    "\n",
    "studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "\n",
    "for line in studentfile.readlines():\n",
    "    matchObj = re.search(r'(\\w+),(\\d+)', line)\n",
    "    if matchObj:\n",
    "        print(f'Name: {matchObj.group(1)} Age: { matchObj.group(2)}')\n",
    "    else:\n",
    "        print(\"No valid record!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "OK, so now you have solved the problem using regular expressions, however... the solution isn't very \"abstract\".  In another case, you might have a more complex record:\n",
    "\n",
    "    Mark,50,190cm,95kg,163483,113mmhg,29mg/ml\n",
    "\n",
    "Your regular expression would start to get ugly!  What is the one thing that is constant in this CSV file?  (in fact,the name of the file-type tells you!)\n",
    "\n",
    "In cases like this, there is a method called \"split\", which will take a string and split it based on whatever separator you give it.  In this case, the comma.\n",
    "\n",
    "  \n",
    "    for line in studentfile.readlines():\n",
    "        print(\"the current record is\", line)\n",
    "        name, age = line.split(',')\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the current record is Mark,53\n",
      "\n",
      "Name: Mark     Age: 53\n",
      "\n",
      "the current record is Alejandro,25\n",
      "\n",
      "Name: Alejandro     Age: 25\n",
      "\n",
      "the current record is Julia,26\n",
      "\n",
      "Name: Julia     Age: 26\n",
      "\n",
      "the current record is Denise,23\n",
      "\n",
      "Name: Denise     Age: 23\n",
      "\n",
      "the current record is Josef,21\n",
      "\n",
      "Name: Josef     Age: 21\n",
      "\n"
     ]
    }
   ],
   "source": [
    "studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "\n",
    "for line in studentfile.readlines():\n",
    "    print(\"the current record is\", line)\n",
    "    name, age = line.split(',')\n",
    "    print(\"Name:\", name, \"    Age:\", age)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Much better!  But... *Still not quite right!!*  What are all of those blank lines?  We didn't ask for blank lines...\n",
    "\n",
    "Remember just a few minutes ago we looked at the output from readlines():\n",
    "\n",
    "    studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "    print(studentfile.readlines())\n",
    "    \n",
    "    ==>  ['Mark,50\\n', 'Alejandro,25\\n', 'Julia,26\\n', 'Denise,23\\n', 'Josef,21\\n']\n",
    "    \n",
    "Those blank lines are because of the \\n (newline) character at the end of every line.  What is happening is that the print statements above ACTUALLY look like this:\n",
    "\n",
    "    the current record is Alejandro,25\\n\n",
    "    Name: Alejandro     Age: 25\\n      <----- the value of the age variable after the spit is '25\\n'\n",
    "    \n",
    "Can we discard this newline?  Sure!   The method *rstrip()* will strip all blank space (including newlines) from the end (right-hand end --> **r**strip() ) of the line:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the current record is Mark,53\n",
      "Name: Mark     Age: 53\n",
      "the current record is Alejandro,25\n",
      "Name: Alejandro     Age: 25\n",
      "the current record is Julia,26\n",
      "Name: Julia     Age: 26\n",
      "the current record is Denise,23\n",
      "Name: Denise     Age: 23\n",
      "the current record is Josef,21\n",
      "Name: Josef     Age: 21\n"
     ]
    }
   ],
   "source": [
    "studentfile.seek(0)   # set it back to the beginning again for this lesson...\n",
    "\n",
    "for line in studentfile.readlines():\n",
    "    line = line.rstrip()\n",
    "    print(\"the current record is\", line)\n",
    "    name, age = line.split(',')\n",
    "    print(\"Name:\", name, \"    Age:\", age)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "When you have finished with an open file, it is a very good idea to close it!\n",
    "\n",
    "    studentfile.close()   # it's a good idea to close a file once you are finished with it!  We are...\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "studentfile.close()   # it's a good idea to close a file once you are finished with it!  We are..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "\n",
    "## Writing to a file\n",
    "\n",
    "Writing to a file is very straightforward.  Use the same \"open\" command that you have already learned, but using the \"w\" flag (\"open for **w**rite), then write information to that open file using the *write()* method.\n",
    "\n",
    "Python will help you by creating the file if it doesn't exist.  For example, the box below will create a file named \"OLDERstudents.csv\" if that file doesn't exist.  ***IF IT DOES EXIST, IT WILL BE DESTROYED!!!!!  YOU CANNOT GET THE CONTENT BACK!!!!  BE CAREFUL!!!***\n",
    "\n",
    "The file pointer is set to the beginning of the file.\n",
    "\n",
    "Here is how easy it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello, I am writing stuff to a file!\n",
      "This is very cool!\n"
     ]
    }
   ],
   "source": [
    "olderstudents = open(\"OLDERstudents.csv\", \"w\")\n",
    "olderstudents.write(\"hello, I am writing stuff to a file!\\nThis is very cool!\")  # the write function, using \\n (newline)\n",
    "olderstudents.close()\n",
    "\n",
    "checkcontent = open(\"OLDERstudents.csv\", \"r\")\n",
    "print(checkcontent.read())  # print the content of the file\n",
    "checkcontent.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "## Now you!\n",
    "\n",
    "* create the file OLDERstudents.csv\n",
    "* using the data from the original students.csv, make everyone 5 years older\n",
    "* write the new older student data to the OLDERstudents.csv file, in an identical format (Mark,55....)\n",
    "* do this again, but this time, create a \"header line\" (Student Name, Student Age)\n",
    "        Student Name, Student Age\n",
    "        Mark,55\n",
    "        Alejandro,35\n",
    "        ...\n",
    "        ...\n",
    "       \n",
    "* do this again, but instead of creating a CSV (comma-separated value) file, create a TSV (tab-separated value)\n",
    " * call it ***OLDERstudents.tsv***\n",
    " * You need to know: the symbol for TAB is \\t\n",
    " * these are the two most common structured text-file formats\n",
    " * both of these can be imported into software like MS Excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "students = open(\"students.csv\", 'r')\n",
    "\n",
    "old_students_csv = open(\"OLDstudents.csv\", 'w')\n",
    "old_students_csv.write(\"Student Name,Student Age\\n\")\n",
    "\n",
    "old_students_tsv = open(\"OLDstudents.tsv\", 'w')\n",
    "old_students_tsv.write(\"Student Name\\tStudent Age\\n\")\n",
    "\n",
    "\n",
    "for line in students.readlines():\n",
    "    \n",
    "    name, age = line.rstrip().split(',')\n",
    "    age = int(age) + 5\n",
    "\n",
    "    old_students_csv.write(f'{name},{str(age)}\\n')\n",
    "    old_students_tsv.write(f'{name}\\t{str(age)}\\n')\n",
    "\n",
    "old_students_csv.close()\n",
    "old_students_tsv.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Append\n",
    "\n",
    "The final mode for writing to a file is \"append\", which means, \"open the file, and put the pointer at the END of the file\".  This allows you to open a file and add new information to it, without destroying the existing information.\n",
    "\n",
    "The append flag is 'a'\n",
    "\n",
    "    olderstudents = open(\"OLDERstudents.csv\", \"a\")   # open for append\n",
    "    \n",
    "We wont go through an example here, but... you might have a question about this on your exam, so try it yourself! :-)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}