Part1_Web_Scraping-and-data-collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a95380de",
   "metadata": {},
   "source": [
    "# Part I : Data collection using web scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "598c0b25",
   "metadata": {},
   "source": [
    "Data scraping is the process of extracting information from a website and putting it into a spreadsheet. For a data collection activity, this method is an effective way to obtain a large amount of information for analysis, processing or inference. \n",
    "\n",
    "In our case, we want to create a recent dataset of phishing sites and legitimate sites. Therefore, we chose the phishtank.org site as a reference for phishing sites and we chose the most 1000 visited sites in 2021 as a reference for legitimate sites assuming that these sites are not phishing sites. Moreover, we extract the other part of ligitimate sites ( another 2000 links ) from an existing kaggle dataset that we merge into the other links to have finally a multisource data. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbc24d6c",
   "metadata": {},
   "source": [
    "## I.1 Phishing websites extraction using selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f93fc336",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.support.ui import Select\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "import time\n",
    "import unittest\n",
    "import csv\n",
    "import re\n",
    "import requests\n",
    "from urllib.parse import urlparse,urlencode\n",
    "import ipaddress\n",
    "import socket\n",
    "from bs4 import BeautifulSoup as bs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "db0eca40",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\souhm\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:3: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PhishTank | Join the fight against phishing\n"
     ]
    }
   ],
   "source": [
    "PATH = \"./chromedriver.exe\"\n",
    "# chromedriver has to be installled in the same folder as the notebook in this case or you have to add the path \n",
    "driver = webdriver.Chrome(PATH)   \n",
    "url_to_fetch=\"https://phishtank.org/\"\n",
    "driver.get(url_to_fetch)\n",
    "#Naviger vers le site phishtank.org\n",
    "print(driver.title) \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dfc2797e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Go to search page\n",
    "link=driver.find_element(By.LINK_TEXT,\"Phish Search\")\n",
    "link.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0470c441",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Search in website for urls that are confirmed to be phish\n",
    "select = Select(driver.find_element(By.NAME,\"valid\"))\n",
    "select.select_by_value('y')\n",
    "button = driver.find_element(By.NAME,\"Search\")\n",
    "button.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b77c64a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows 22\n",
      "Number of columns 5\n"
     ]
    }
   ],
   "source": [
    "#extract XML data using XPATH : \n",
    "\n",
    "#nombre de lignes \n",
    "nbre_rows=len(driver.find_elements(By.XPATH,\"//table/tbody/tr\"))\n",
    "print (\"Number of rows\", nbre_rows)\n",
    "#nombre de colonnes\n",
    "nbre_cols = len (driver.find_elements(By.XPATH,\"//table/tbody/tr[1]/th\"))\n",
    "print(\"Number of columns\"  ,nbre_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6c5030e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Here, I extract 20 websites per iteration and store them in a csv file that will be saved in the same folder \n",
    "K is the total number of iterations that we can fix according to the data size\n",
    "Finally, we extract a total number of 20*K links\n",
    "This process may take a while for large numbers of K\n",
    "\n",
    "\"\"\"\n",
    "def get_links_of_current_page(K):\n",
    "    \n",
    "    for i in range(K):\n",
    "        for r in range(2, (nbre_rows)) :\n",
    "            phish_website = []\n",
    "            row=[]\n",
    "            for c in [1,2,4,5]:\n",
    "                x = driver.find_element(By.XPATH,\"//tr[\"+str(r)+\"]/td[\"+str(c)+\"]\").text\n",
    "                row.append(x)\n",
    "            phish_website.append(row[1:3])\n",
    "            with open('phish_website.csv','a') as file: \n",
    "                file.writer=csv.writer(file)\n",
    "                file.writer.writerow((phish_website[0][0],phish_website[0][1]))\n",
    "            next = driver.find_element(By.XPATH,\"//*[@id='widecol']/div/table/tbody/tr[22]/td[5]/b/a\") # click older in phishtank\n",
    "            next.click()\n",
    "\n",
    "        file.close()\n",
    "    \n",
    "        #print(i/K) # to track the execution state\n",
    "     \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6c220d6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell is used to run the code and extract 20000 phishing URL\n",
    "get_links_of_current_page(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e446c09",
   "metadata": {},
   "source": [
    "PS: this code was inspired by the tutoriels of [**Tech with Tim youtube channel** ](https://www.youtube.com/watch?v=Xjv1sY630Uc&list=PLzMcBGfZo4-n40rB1XaJ0ak1bemvlqumQ&ab_channel=TechWithTim) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb8beb03",
   "metadata": {},
   "source": [
    "## I.2 Kaggle dataset extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89fee5da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('./Datasets/dataset_phishing.csv')\n",
    "data = data[ data[\"status\"] == \"legitimate\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd447537",
   "metadata": {},
   "source": [
    "### Merging the partial datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f79d7aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "websites = list(data[\"url\"]) + websites\n",
    "with open('legitimate_websites', 'w') as f:\n",
    "      \n",
    "    # using csv.writer method from CSV package\n",
    "    write = csv.writer(f)\n",
    "    for i in websites:\n",
    "      write.writerow([i])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}