You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running ./generate.sh, which scrapes all Scheme implementations from scratch, I now understand the point someone made about the scrapers being too slow.
The easiest solution would be to only run the scrapers that have changed since the last run. This could be done by:
Storing a copy of the last working version of each scraper .sh, and skipping it if the current .sh is equal to the copy.
Storing a checksum of the last working version and comparing checksums.
Using make.
Scrapers could also be run in parallel.
Any of the above would work, but we're going to hit the same problem when scraping other data (not just SRFI listings) so a more generic solution would be in order. I'll dust of the prototype Scheme framework I have for generic scraping.
The text was updated successfully, but these errors were encountered:
Running
./generate.sh
, which scrapes all Scheme implementations from scratch, I now understand the point someone made about the scrapers being too slow.The easiest solution would be to only run the scrapers that have changed since the last run. This could be done by:
.sh
, and skipping it if the current.sh
is equal to the copy.make
.Scrapers could also be run in parallel.
Any of the above would work, but we're going to hit the same problem when scraping other data (not just SRFI listings) so a more generic solution would be in order. I'll dust of the prototype Scheme framework I have for generic scraping.
The text was updated successfully, but these errors were encountered: