Faster scraping #20
Description
Running ./generate.sh, which scrapes all Scheme implementations from scratch, I now understand the point someone made about the scrapers being too slow.
The easiest solution would be to only run the scrapers that have changed since the last run. This could be done by:
- Storing a copy of the last working version of each scraper
.sh, and skipping it if the current.shis equal to the copy. - Storing a checksum of the last working version and comparing checksums.
- Using
make.
Scrapers could also be run in parallel.
Any of the above would work, but we're going to hit the same problem when scraping other data (not just SRFI listings) so a more generic solution would be in order. I'll dust of the prototype Scheme framework I have for generic scraping.