Pro-Scraper

Pro-Scraper Frontend (Repo) | Pro-Scraper Hosted/API (Repo)

• Clone repository to local machine
• install python, pip & mysql
• install python requirements.txt
• install mysql server ("winget install -e Oracle.MySQL" in terminal | or manual install from mysql website)
• setup mysql server, create user (make sure to grant all privileges | or use root user instead)
• sometimes you are required to set current use db (Set current database: "use <database_name>;")
• add user & mysql credentials to config.py
• start mysql server
• run python3 app.py
• open browser and go to http://localhost:5000
• Try scraping something. The more you scrape the more optimized it gets over time.
• you dont have to export data as csv, you can use the API to get data in JSON format or even better use the Database directly to fetch data.locally.
• fiddle around and have fun !

Search API

• URL

  /api/search /api/quick_search /api/deep_search

  
  

• Method:

• GET

• POST
  

• URL Params
  

  • Required: searchQuery=[string]
    

• Data Params

• searchQuery=[string]
  

• Success Response:

  • Code: 200 Content: { searchQuery : "iphone 13" }
    

• Error Response:

• Code: 404 NOT FOUND Content: { error : "No results found" }
  

• Sample Call:

  $.ajax({
    url: "http://localhost:5000/api/search?searchQuery=iphone+13",
    dataType: "json",
    type: "GET",
    success: function (r) {
      console.log(r);
    },
  });

  $.ajax({
    url: "http://localhost:5000/api/search",
    dataType: "json",
    type: "POST",
    data: { searchQuery: "iphone 13" },
    success: function (r) {
      console.log(r);
    },
  });

## Other Endpoints :

• api/search/product
• api/search/product_quick
• api/search/product_deep
  
• api/search/reviews
• api/search/reviews_quick
• api/search/reviews_deep
  
• api/search/search
• api/search/search_quick
• api/search/search_deep

Related Papers :

Comparative Web Product Analysis: A Data-Driven Approach

If you use this project, plz cite or add reference to it :

@INPROCEEDINGS{10452652,
  author={Jonathan, Jesvi and Muthusundar, S. K.},
  booktitle={2023 International Conference on Data Science, Agents & Artificial Intelligence (ICDSAAI)}, 
  title={Comparative Web Product Analysis: A Data-Driven Approach}, 
  year={2023},
  volume={},
  number={},
  pages={1-4},
  keywords={Sentiment analysis;Decision making;Data visualization;Pricing;Aerodynamics;Real-time systems;Electronic commerce;E-commerce;Data Scraping;Machine Learning;Data Optimization;Decision Making;Data Analysis},
  doi={10.1109/ICDSAAI59313.2023.10452652}}