Jasper van Merle
b31dc9d0c2
Fix tests
6 years ago
Thibaut Courouble
bf003669ba
Simplify file scraper setup; scrape files in the "docs/[slug]" directory
6 years ago
Thibaut Courouble
68b80bce36
Generate pretty JSON in docs.json manifest
...
To make it easier to track changes in Git.
7 years ago
Thibaut Courouble
0725a69af5
Store docs' metadata in meta.json files
...
To avoid relying on the filesystem for modified times.
7 years ago
Thibaut Courouble
dd8c80060a
Fix :follow_links option not doing anything when set to false
8 years ago
Thibaut Courouble
6fc48db8af
Improve error logging
8 years ago
Thibaut Courouble
ed5a5cadd9
Fix handling of invalid iframe URLs
...
Fixes #590 .
8 years ago
Thibaut Courouble
c1ebb7a0b9
Improve Doc#name and Doc#slug
8 years ago
Andreas Stenius
b36f3f8095
core/doc: make sure name is usable as slug.
8 years ago
Thibaut Courouble
94470251fe
Bump Ruby 2.4.0
8 years ago
Thibaut Courouble
4e41ed9f25
Add <base> support
8 years ago
Thibaut Courouble
6f0214eaf3
Make Docs::Parser return the entire document instead of <body>
8 years ago
Thibaut Courouble
0c8ca4e5fa
Add SQLite documentation
8 years ago
Thibaut Courouble
721adf8e21
Don't rewrite data URIs
8 years ago
Thibaut Courouble
82d0725747
Improve ordering of entries and types
9 years ago
Thibaut Courouble
5bb96f804a
Require all entries to have a name, path and type
9 years ago
nucular
034ecfae72
Replace File.basename in URL#relative_path_to because it doesn't handle special characters in URLs well
9 years ago
Thibaut Courouble
9e1b9ca2a9
Improve MDN/JavaScript scraper
9 years ago
Thibaut Courouble
70b19c238a
Sort types/categories by number when they start with a number
9 years ago
Thibaut Courouble
d366e14ea7
Fix Docs::Parse#document? when document has no doctype
9 years ago
Thibaut Courouble
6c9fc464c2
Add :fix_urls_before_parse option for Angular doc
9 years ago
Thibaut Courouble
63c77322d3
Handle unencoded spaces in link hrefs
9 years ago
Thibaut Courouble
c3b9502657
Set version attributes before evaluating block
...
Ref #25 .
9 years ago
Thibaut Courouble
3df9cfff98
Add support for blank and non-number version names
...
Ref #25 .
9 years ago
Thibaut Courouble
16ddcb100c
Simplify version path separator
...
Ref #25 .
9 years ago
Thibaut Courouble
b67a02ed35
Add version to doc manifest
...
Ref #25 .
9 years ago
Thibaut Courouble
b2d2066d96
Multi-version support
...
Ref #25 .
9 years ago
Thibaut Courouble
bd6e27eca2
Optionally include 'release' and 'links' in docs manifest
9 years ago
Thibaut Courouble
a639aedcd9
Remove index_path and db_path from docs manifest
9 years ago
Thibaut Courouble
e1c0218230
Rename version -> release
9 years ago
Thibaut
3eb5ccb7ea
Raise error and stop scraping on 4xx/5xx status code
9 years ago
Thibaut
6939865137
Finish Dojo scraper
9 years ago
ShaneQful
3465933543
Added dojo to devdocs & ability to define headers in scraper requests
9 years ago
Thibaut
7de19cf800
Make EntryIndex a unique index (don't add the same entry twice)
10 years ago
Thibaut
018628ea7d
Add two-pass redirection rewriter
...
... to avoid having to maintain huge lists of redirects. This works by doing a first pass to detect which internal URL is redirected where, before doing a second (normal) pass that rewrites all these URLs (links) with their final destination. There's a bit of monkey-patching I'm not proud of, but this works(tm).
10 years ago
Thibaut
b29d6ca002
Move doc links to manifest
10 years ago
Thibaut
cf7f446738
Change home_url to a list of links
10 years ago
Thu Trang Pham
642c1cff7d
Make sure that home_url can be nil
10 years ago
Thibaut
a59ef1cdb6
Add db_size attribute in doc manifest
10 years ago
Thibaut
456c4cb811
Add Store#size
10 years ago
Thibaut
bc5488faa2
Make docs mtime the greatest of the index and db files' mtime
10 years ago
Thibaut
ca7ff6086e
Exclude docs without a db file from the manifest
10 years ago
Thibaut
ca61a2b746
Add Doc#db_path
10 years ago
Thibaut
5c46eabc67
Output a JSON file containing all the pages' content
10 years ago
Thibaut
e9125c6ec2
Refactor Doc.store_pages
10 years ago
Thibaut
ecf774e22c
Add EntryIndex#blank?
10 years ago
Thibaut
1655a00fb6
Refactor Doc.store_page
10 years ago
Thibaut
a9c8c228bc
Fix default argument assignment for Ruby 2.2
10 years ago
Thibaut
c68133eb8c
Revert "Replace colons with dashes in internal URL paths"
...
This reverts commit d51a6f9042
.
11 years ago
Thibaut
c122caa7c4
Add :skip_link option for ignoring certain links in scrapers
11 years ago