Commit Graph

73 Commits (5dbb8536069b2956f9c733e627989e6806eb94e3)

Author SHA1 Message Date
Jasper van Merle b31dc9d0c2 Fix tests
6 years ago
Thibaut Courouble bf003669ba Simplify file scraper setup; scrape files in the "docs/[slug]" directory
6 years ago
Thibaut Courouble 68b80bce36 Generate pretty JSON in docs.json manifest
7 years ago
Thibaut Courouble 0725a69af5 Store docs' metadata in meta.json files
7 years ago
Thibaut Courouble dd8c80060a Fix :follow_links option not doing anything when set to false
8 years ago
Thibaut Courouble 6fc48db8af Improve error logging
8 years ago
Thibaut Courouble ed5a5cadd9 Fix handling of invalid iframe URLs
8 years ago
Thibaut Courouble c1ebb7a0b9 Improve Doc#name and Doc#slug
8 years ago
Andreas Stenius b36f3f8095 core/doc: make sure name is usable as slug.
8 years ago
Thibaut Courouble 94470251fe Bump Ruby 2.4.0
8 years ago
Thibaut Courouble 4e41ed9f25 Add <base> support
8 years ago
Thibaut Courouble 6f0214eaf3 Make Docs::Parser return the entire document instead of <body>
8 years ago
Thibaut Courouble 0c8ca4e5fa Add SQLite documentation
8 years ago
Thibaut Courouble 721adf8e21 Don't rewrite data URIs
8 years ago
Thibaut Courouble 82d0725747 Improve ordering of entries and types
9 years ago
Thibaut Courouble 5bb96f804a Require all entries to have a name, path and type
9 years ago
nucular 034ecfae72 Replace File.basename in URL#relative_path_to because it doesn't handle special characters in URLs well
9 years ago
Thibaut Courouble 9e1b9ca2a9 Improve MDN/JavaScript scraper
9 years ago
Thibaut Courouble 70b19c238a Sort types/categories by number when they start with a number
9 years ago
Thibaut Courouble d366e14ea7 Fix Docs::Parse#document? when document has no doctype
9 years ago
Thibaut Courouble 6c9fc464c2 Add :fix_urls_before_parse option for Angular doc
9 years ago
Thibaut Courouble 63c77322d3 Handle unencoded spaces in link hrefs
9 years ago
Thibaut Courouble c3b9502657 Set version attributes before evaluating block
9 years ago
Thibaut Courouble 3df9cfff98 Add support for blank and non-number version names
9 years ago
Thibaut Courouble 16ddcb100c Simplify version path separator
9 years ago
Thibaut Courouble b67a02ed35 Add version to doc manifest
9 years ago
Thibaut Courouble b2d2066d96 Multi-version support
9 years ago
Thibaut Courouble bd6e27eca2 Optionally include 'release' and 'links' in docs manifest
9 years ago
Thibaut Courouble a639aedcd9 Remove index_path and db_path from docs manifest
9 years ago
Thibaut Courouble e1c0218230 Rename version -> release
9 years ago
Thibaut 3eb5ccb7ea Raise error and stop scraping on 4xx/5xx status code
9 years ago
Thibaut 6939865137 Finish Dojo scraper
9 years ago
ShaneQful 3465933543 Added dojo to devdocs & ability to define headers in scraper requests
9 years ago
Thibaut 7de19cf800 Make EntryIndex a unique index (don't add the same entry twice)
10 years ago
Thibaut 018628ea7d Add two-pass redirection rewriter
10 years ago
Thibaut b29d6ca002 Move doc links to manifest
10 years ago
Thibaut cf7f446738 Change home_url to a list of links
10 years ago
Thu Trang Pham 642c1cff7d Make sure that home_url can be nil
10 years ago
Thibaut a59ef1cdb6 Add db_size attribute in doc manifest
10 years ago
Thibaut 456c4cb811 Add Store#size
10 years ago
Thibaut bc5488faa2 Make docs mtime the greatest of the index and db files' mtime
10 years ago
Thibaut ca7ff6086e Exclude docs without a db file from the manifest
10 years ago
Thibaut ca61a2b746 Add Doc#db_path
10 years ago
Thibaut 5c46eabc67 Output a JSON file containing all the pages' content
10 years ago
Thibaut e9125c6ec2 Refactor Doc.store_pages
10 years ago
Thibaut ecf774e22c Add EntryIndex#blank?
10 years ago
Thibaut 1655a00fb6 Refactor Doc.store_page
10 years ago
Thibaut a9c8c228bc Fix default argument assignment for Ruby 2.2
10 years ago
Thibaut c68133eb8c Revert "Replace colons with dashes in internal URL paths"
11 years ago
Thibaut c122caa7c4 Add :skip_link option for ignoring certain links in scrapers
11 years ago