9 Commits (808a6fb43a48806eb930186fb2e3929b1ddf0f0f)

Author	SHA1	Message	Date
Thibaut	3eb5ccb7ea	Raise error and stop scraping on 4xx/5xx status code	9 years ago
ShaneQful	3465933543	Added dojo to devdocs & ability to define headers in scraper requests	9 years ago
Thibaut	018628ea7d	Add two-pass redirection rewriter ... ... to avoid having to maintain huge lists of redirects. This works by doing a first pass to detect which internal URL is redirected where, before doing a second (normal) pass that rewrites all these URLs (links) with their final destination. There's a bit of monkey-patching I'm not proud of, but this works(tm).	10 years ago
Thibaut	bcd4a5b522	Use String#remove	11 years ago
Thibaut	5e69d6df5e	Make Docs::FileScraper#request_all accept an array of URLs	11 years ago
Thibaut	cd6057e392	Make Docs::UrlScraper#request_all accept an array of URLs	11 years ago
Thibaut	24347f312b	Add CleanLocalUrls filter for removing localhost URLs ... Enabled by default in FileScraper	11 years ago
Thibaut	9089d03211	Set the default FileScraper::base_url to localhost	11 years ago
Thibaut	18986c1814	Going open source	11 years ago