Commit Graph

143 Commits (main)

Author SHA1 Message Date
Thibaut Courouble a6855329e8 Add image scraping and optimization filter
8 years ago
Thibaut Courouble 6fc48db8af Improve error logging
8 years ago
Thibaut Courouble 8ef3502cb3 Throw an error when response code == 0
8 years ago
Thibaut Courouble ee21c0bf29 Add frozen_string_literal to subscriber files"
8 years ago
Thibaut Courouble 2efce74521 Finish OpenJDK scraper
8 years ago
Thibaut Courouble 057a3a9ce9 Ensure root page is processed first for docs with multiple base urls
8 years ago
Thibaut Courouble 509fb443e6 Remove leftover debug log
8 years ago
Thibaut Courouble c1ebb7a0b9 Improve Doc#name and Doc#slug
8 years ago
Andreas Stenius b36f3f8095 core/doc: make sure name is usable as slug.
8 years ago
Thibaut Courouble 476c69e419 Improve MDN scrapers
8 years ago
Thibaut Courouble 8ddb812035 Pass response object to Scraper#parse method
8 years ago
Thibaut Courouble 4e41ed9f25 Add <base> support
8 years ago
Thibaut Courouble 6f0214eaf3 Make Docs::Parser return the entire document instead of <body>
8 years ago
Thibaut Courouble 8147f9f90a Update TensorFlow documentation (0.12)
8 years ago
Thibaut Courouble 0c8ca4e5fa Add SQLite documentation
8 years ago
Thibaut Courouble 0d4c6b51bb Update TensorFlow documentation (0.11)
8 years ago
Thibaut Courouble 721adf8e21 Don't rewrite data URIs
8 years ago
Thibaut Courouble 6fdf81e649 Update Elixir documentation (1.3.3)
8 years ago
Thibaut Courouble 46a9ed16f6 Refactor scrapers with multiple base URLs
8 years ago
Thibaut Courouble 64eb1c86fc Log errors instead of exiting when scraping docs
9 years ago
Thibaut Courouble 82d0725747 Improve ordering of entries and types
9 years ago
Thibaut Courouble b279f84a4c Update and version Meteor documentation
9 years ago
Thibaut Courouble d7726f98b1 Fix exception when trying to find a version of a doc that has no versions
9 years ago
Thibaut Courouble f6530eddcf Update Angular.js documentation (1.5.6)
9 years ago
Thibaut Courouble 5bb96f804a Require all entries to have a name, path and type
9 years ago
Thibaut Courouble 1a4bea8510 Change User-Agent to 'DevDocs'
9 years ago
nucular 034ecfae72 Replace File.basename in URL#relative_path_to because it doesn't handle special characters in URLs well
9 years ago
Thibaut Courouble 2813cf9be3 Fix internal paths with ':' characters not getting clean up properly
9 years ago
Thibaut Courouble e2b7004078 Update Laravel documentation (5.2.31, 5.1.33)
9 years ago
Thibaut Courouble 70b19c238a Sort types/categories by number when they start with a number
9 years ago
Thibaut Courouble d366e14ea7 Fix Docs::Parse#document? when document has no doctype
9 years ago
Thibaut Courouble adba5c1cac Update URL stubs
9 years ago
Thibaut Courouble 5f7005729c Update Meteor documentation (1.3.0) and add new URL stubbing mechanism
9 years ago
Thibaut Courouble fbbf9d5d09 Finish Haxe documentation
9 years ago
Thibaut Courouble 2bec61a1a7 Finish TensorFlow scraper
9 years ago
Thibaut Courouble 1fa82430b0 Finish Chef documentation
9 years ago
Thibaut Courouble 5f2d9cde50 Finish CakePHP documentation
9 years ago
Thibaut Courouble 44e1f216c4 Update and improve PHP documentation (7.0.3)
9 years ago
Thibaut Courouble b8658d9708 Refactor FixRedirectionsBehavior
9 years ago
Thibaut Courouble c3b9502657 Set version attributes before evaluating block
9 years ago
Thibaut Courouble 3df9cfff98 Add support for blank and non-number version names
9 years ago
Thibaut Courouble 16ddcb100c Simplify version path separator
9 years ago
Thibaut Courouble b67a02ed35 Add version to doc manifest
9 years ago
Thibaut Courouble 4f3ea92bea Improve 'docs:list' command
9 years ago
Thibaut Courouble b2d2066d96 Multi-version support
9 years ago
Thibaut Courouble bd6e27eca2 Optionally include 'release' and 'links' in docs manifest
9 years ago
Thibaut Courouble a639aedcd9 Remove index_path and db_path from docs manifest
9 years ago
Thibaut Courouble e1c0218230 Rename version -> release
9 years ago
Thibaut 3eb5ccb7ea Raise error and stop scraping on 4xx/5xx status code
9 years ago
ShaneQful 3465933543 Added dojo to devdocs & ability to define headers in scraper requests
9 years ago
Thibaut c69136056e Log current URL on scraper error
10 years ago
Thibaut 7de19cf800 Make EntryIndex a unique index (don't add the same entry twice)
10 years ago
Thibaut 018628ea7d Add two-pass redirection rewriter
10 years ago
Thibaut 9ecb1f9498 Refactor StubRootPage
10 years ago
Thibaut b29d6ca002 Move doc links to manifest
10 years ago
Thibaut 5c4c1ce2b6 Log entries/types/files diff in docs:generate command
10 years ago
Thibaut cf7f446738 Change home_url to a list of links
10 years ago
Thu Trang Pham ab85da334d home_url must be applied after inner_html
10 years ago
Thu Trang Pham 642c1cff7d Make sure that home_url can be nil
10 years ago
Thu Trang Pham 53bba10212 Adding home_url attr to doc.rb
10 years ago
Thu Trang Pham 42910c62ce Adding home_url to scraper
10 years ago
Thu Trang Pham 5d7ad301ea Give scrapper access to home_url
10 years ago
Thu Trang Pham 610e49119f Renamed official_url to home_url
10 years ago
Thu Trang Pham 0dc8cfd5d3 Adding Official Url for url scrapped documentations
10 years ago
Thibaut a59ef1cdb6 Add db_size attribute in doc manifest
10 years ago
Thibaut bc5488faa2 Make docs mtime the greatest of the index and db files' mtime
10 years ago
Thibaut ca7ff6086e Exclude docs without a db file from the manifest
10 years ago
Thibaut ca61a2b746 Add Doc#db_path
10 years ago
Thibaut 5c46eabc67 Output a JSON file containing all the pages' content
10 years ago
Thibaut e9125c6ec2 Refactor Doc.store_pages
10 years ago
Thibaut ecf774e22c Add EntryIndex#blank?
10 years ago
Thibaut 1655a00fb6 Refactor Doc.store_page
10 years ago
Thibaut e0556365a8 Remove deprecated Kernel#quietly calls
10 years ago
Thibaut 1fa81b8461 Silence Nokogiri warnings
10 years ago
Thibaut 6769c90c8a Silent nokorigi/libxml2 warnings
10 years ago
Thibaut bcd4a5b522 Use String#remove
11 years ago
Thibaut c0be178556 Auto-require gems in the "docs" bundle
11 years ago
Thibaut 864188e24c Use String#sub instead of String#gsub when possible
11 years ago
Thibaut b92db88506 Refactor Docs::Scraper
11 years ago
Thibaut ca06cc7ad9 Implement Docs::Filter#initial_page?
11 years ago
Thibaut f8298181e9 Implement initial_paths scraper option
11 years ago
Thibaut 5e69d6df5e Make Docs::FileScraper#request_all accept an array of URLs
11 years ago
Thibaut cd6057e392 Make Docs::UrlScraper#request_all accept an array of URLs
11 years ago
Thibaut c6be9b6ae4 Make Docs:Requester#request accept an array of URLs
11 years ago
Thibaut 6e9c16a1fc Increase default :max_concurrency
11 years ago
Thibaut 706270d89c Don't store pages with no entries
11 years ago
Thibaut 24347f312b Add CleanLocalUrls filter for removing localhost URLs
11 years ago
Thibaut 9089d03211 Set the default FileScraper::base_url to localhost
11 years ago
Thibaut 5c4775201b Move code in scraper.rb
11 years ago
Thibaut d964fd4ca9 Make Docs::URL#relative_path_to work with unqualified URLs
11 years ago
Thibaut 4df387d475 Make Docs::URL#subpath_to work with unqualified URLs
11 years ago
Thibaut 1204390f81 Make Docs::Scraper.root_path inheritable
11 years ago
Thibaut 18986c1814 Going open source
11 years ago