Commit Graph

56 Commits (e9b36319cbb882140329428775c0867bd0fedc5a)

Author SHA1 Message Date
Thibaut Courouble 9e1b9ca2a9 Improve MDN/JavaScript scraper
9 years ago
Thibaut Courouble 70b19c238a Sort types/categories by number when they start with a number
9 years ago
Thibaut Courouble d366e14ea7 Fix Docs::Parse#document? when document has no doctype
9 years ago
Thibaut Courouble 6c9fc464c2 Add :fix_urls_before_parse option for Angular doc
9 years ago
Thibaut Courouble 63c77322d3 Handle unencoded spaces in link hrefs
9 years ago
Thibaut Courouble c3b9502657 Set version attributes before evaluating block
9 years ago
Thibaut Courouble 3df9cfff98 Add support for blank and non-number version names
9 years ago
Thibaut Courouble 16ddcb100c Simplify version path separator
9 years ago
Thibaut Courouble b67a02ed35 Add version to doc manifest
9 years ago
Thibaut Courouble b2d2066d96 Multi-version support
9 years ago
Thibaut Courouble bd6e27eca2 Optionally include 'release' and 'links' in docs manifest
9 years ago
Thibaut Courouble a639aedcd9 Remove index_path and db_path from docs manifest
9 years ago
Thibaut Courouble e1c0218230 Rename version -> release
9 years ago
Thibaut 3eb5ccb7ea Raise error and stop scraping on 4xx/5xx status code
9 years ago
Thibaut 6939865137 Finish Dojo scraper
9 years ago
ShaneQful 3465933543 Added dojo to devdocs & ability to define headers in scraper requests
9 years ago
Thibaut 7de19cf800 Make EntryIndex a unique index (don't add the same entry twice)
10 years ago
Thibaut 018628ea7d Add two-pass redirection rewriter
10 years ago
Thibaut b29d6ca002 Move doc links to manifest
10 years ago
Thibaut cf7f446738 Change home_url to a list of links
10 years ago
Thu Trang Pham 642c1cff7d Make sure that home_url can be nil
10 years ago
Thibaut a59ef1cdb6 Add db_size attribute in doc manifest
10 years ago
Thibaut 456c4cb811 Add Store#size
10 years ago
Thibaut bc5488faa2 Make docs mtime the greatest of the index and db files' mtime
10 years ago
Thibaut ca7ff6086e Exclude docs without a db file from the manifest
10 years ago
Thibaut ca61a2b746 Add Doc#db_path
10 years ago
Thibaut 5c46eabc67 Output a JSON file containing all the pages' content
10 years ago
Thibaut e9125c6ec2 Refactor Doc.store_pages
10 years ago
Thibaut ecf774e22c Add EntryIndex#blank?
10 years ago
Thibaut 1655a00fb6 Refactor Doc.store_page
10 years ago
Thibaut a9c8c228bc Fix default argument assignment for Ruby 2.2
10 years ago
Thibaut c68133eb8c Revert "Replace colons with dashes in internal URL paths"
11 years ago
Thibaut c122caa7c4 Add :skip_link option for ignoring certain links in scrapers
11 years ago
Thibaut ab1de67032 Update Nokogiri
11 years ago
Thibaut bcd4a5b522 Use String#remove
11 years ago
Thibaut d51a6f9042 Replace colons with dashes in internal URL paths
11 years ago
Thibaut c0be178556 Auto-require gems in the "docs" bundle
11 years ago
Thibaut 864188e24c Use String#sub instead of String#gsub when possible
11 years ago
Thibaut b92db88506 Refactor Docs::Scraper
11 years ago
Thibaut ca06cc7ad9 Implement Docs::Filter#initial_page?
11 years ago
Thibaut f8298181e9 Implement initial_paths scraper option
11 years ago
Thibaut 5e69d6df5e Make Docs::FileScraper#request_all accept an array of URLs
11 years ago
Thibaut cd6057e392 Make Docs::UrlScraper#request_all accept an array of URLs
11 years ago
Thibaut c6be9b6ae4 Make Docs:Requester#request accept an array of URLs
11 years ago
Thibaut 562a4c74fe Fix failing Docs::Requester test
11 years ago
Thibaut 95ec3a5dc3 Refactor :trailing_slash option tests
11 years ago
Thibaut 4a2a393ed4 Refactor :follow_links option
11 years ago
Thibaut bf6c900ca7 Make :skip_links option work with true/false
11 years ago
Thibaut 47eb7eec7f Make :skip_links option apply to pages instead of individual links
11 years ago
Thibaut 6c8eea1adb Add :follow_links option to not follow links on select pages
11 years ago