28 Commits (0b2cde3fa884a6b895cf6d36427ca419bae496dc)

Author	SHA1	Message	Date
Simon Legner	961bbeb92b	fix: replace calls to `Dir.exists?` with `Dir.exist?`	2 years ago
Jasper van Merle	8325ed78ef	Fix OpenJDK scraper	6 years ago
Thibaut Courouble	bf003669ba	Simplify file scraper setup; scrape files in the "docs/[slug]" directory	6 years ago
Thibaut Courouble	0095779b3f	Instrument FileScraper read errors rather than use puts	7 years ago
Thibaut Courouble	472f03d5b0	Update C/C++ documentation ... Fixes #688. Fixes #736.	7 years ago
Thibaut Courouble	ad9276e901	Add rate limiter	7 years ago
Thibaut Courouble	914100a657	Improve error message on failed requests	7 years ago
Thibaut Courouble	65e8236ab5	Switch to headless Chrome for capybara/selenium scraping	8 years ago
Thibaut Courouble	6fc48db8af	Improve error logging	8 years ago
Thibaut Courouble	476c69e419	Improve MDN scrapers ... Closes #488. Closes #572.	8 years ago
Thibaut Courouble	4e41ed9f25	Add <base> support	8 years ago
Thibaut Courouble	0d4c6b51bb	Update TensorFlow documentation (0.11)	8 years ago
Thibaut Courouble	6fdf81e649	Update Elixir documentation (1.3.3)	8 years ago
Thibaut Courouble	46a9ed16f6	Refactor scrapers with multiple base URLs	8 years ago
Thibaut Courouble	b279f84a4c	Update and version Meteor documentation	9 years ago
Thibaut Courouble	f6530eddcf	Update Angular.js documentation (1.5.6) ... Fixes #405.	9 years ago
Thibaut Courouble	1a4bea8510	Change User-Agent to 'DevDocs'	9 years ago
Thibaut Courouble	5f2d9cde50	Finish CakePHP documentation	9 years ago
Thibaut Courouble	b8658d9708	Refactor FixRedirectionsBehavior	9 years ago
Thibaut	3eb5ccb7ea	Raise error and stop scraping on 4xx/5xx status code	9 years ago
ShaneQful	3465933543	Added dojo to devdocs & ability to define headers in scraper requests	9 years ago
Thibaut	018628ea7d	Add two-pass redirection rewriter ... ... to avoid having to maintain huge lists of redirects. This works by doing a first pass to detect which internal URL is redirected where, before doing a second (normal) pass that rewrites all these URLs (links) with their final destination. There's a bit of monkey-patching I'm not proud of, but this works(tm).	10 years ago
Thibaut	bcd4a5b522	Use String#remove	11 years ago
Thibaut	5e69d6df5e	Make Docs::FileScraper#request_all accept an array of URLs	11 years ago
Thibaut	cd6057e392	Make Docs::UrlScraper#request_all accept an array of URLs	11 years ago
Thibaut	24347f312b	Add CleanLocalUrls filter for removing localhost URLs ... Enabled by default in FileScraper	11 years ago
Thibaut	9089d03211	Set the default FileScraper::base_url to localhost	11 years ago
Thibaut	18986c1814	Going open source	11 years ago