-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run scraper on local machine to gather regulations.gov comments for the NILC #57
Comments
I started to take a look at this issue as a first contribution and have a couple of questions. I was able to get the docker container running but was unable to run I also ran into an error running |
Ahh I see where it is creating the volume in the docker run command It seems that the issue is I don't have CURDIR set. I am not too familiar but I think it might be related to make. The other issue I had is also related to the DISPLAY env var which also isn't getting set properly. |
Continuation of #48:
We have not yet been able to get a VM up to run the scraper, so we need your help running the scraper locally in order to gather an initial dataset that the NILC can look at.
The documentation for the scraper can be found here: https://github.com/Data4Democracy/immigration-connect/tree/master/public-charge/scraper
Ping me (@dotj) or @alejandrox1 here or post in the
#immigration-connect
slack page if you need help,We've seen each page (50 comments) take about 4 minutes to scrape, and there are currently almost 10k comments, so it will take about 13 hours total. Of course, this is dependent on your internet speed and various other factors.
Tasks
The text was updated successfully, but these errors were encountered: