Skip to content

Scrapes all of the current issue pages for AJM's various journals to search for new articles. If new articles are found then a summary email is created and sent to anyone who is on the recipients list.

Notifications You must be signed in to change notification settings

oldo/webscraper-emailer

Repository files navigation

American Journal of Meteorology New Article Notifier

The following setup will scrape all of the current issue pages for AJM's various journals to search for new articles. If new articles are found then a summary email is created and sent to anyone who is on the recipients list.

With some modification I'm sure it could be adapted to scrape any other page and automatically email a list of recipients.

How It Works

All programmed in Python

Scrapy > MongoDB > Jinja2 > Email delivery with SMTPLib via gmail

Installation

  • MongoDB must be installed and mongod running.
  • $ pip install -r requirements.txt
  • The script depends on a file that I've excluded from git which has the gmail login details and a list of the recipients. It is located in the root directory of the project and is of the form:
    recipients = [
        {
            'name': 'Oliver',
            'email': '[email protected]'
        },
        {
            'name': 'Nollie',
            'email': '[email protected]'
        }
    ]
    
    GMAIL_CREDENTIALS = {
        'email': 'senders email address',
        'username': 'gmail username',
        'pass': 'gmail password'
    }
  • Run $ python main.py
  • Setup a cron job to run the script at any interval that you desire.

About

Scrapes all of the current issue pages for AJM's various journals to search for new articles. If new articles are found then a summary email is created and sent to anyone who is on the recipients list.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published