Astra scraper #31
Astra scraper #31
Conversation
|
The current scraper logs in to Astra successfully but then throws the below error before exiting the sign in function: After that I think I'll just have to get the cookies sorted and it'll be able to make its first successful scraping request. |
Thanks for looking into this! As far as that error goes, it pops up on the other scrapers occasionally as well -- I'm not actually 100% certain what causes it. It seems to just be an error that chromedp throws internally when it can't handle a |
|
Ah ok, good to know. It was never exiting the |
@TyHil There seems to be a suggestion at this GitHub issue comment to add the I tried it with |
TODOs: sorting scrape each day look into login inputting user/pass in wrong sometimes
After this and next semester seems there's only ever 2 events, one in FO 3.616 with no time (?) and one with no location that always shows up after the current semester and says either the holiday and "Events for Future Terms" as well as "No Events Allowed". This just scrapes 90 days into that, stops at about a year and 2 months out.
Closes chromedp when not necessary
Resolves #9
This Astra scraper dumps the data in JSON format with a key for each day scraped followed by a list of each room reservation in the format mentioned in issue #9 .
It stops after 90 days of fewer than 10 events. There is one persistent all day event that shows up on holidays and any semester after this one that marks "No Events Allowed". There is also a perpetual event with no time in FO 3.616, idk why. So once you get past this semester and the next you regularly have fewer than 10 events per day. I choose 90 to be safe about holidays. This works out to scraping about a year and 2 months out which takes a couple minutes on my machine.
The events are sorted by start time but this could easily be switched to room with
LocationName.