diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index d2317ab5..d0fccff5 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -61,7 +61,7 @@ In addition to the [guidelines for contributing code](#contributing-code-and-fea Please don't submit a pull request updating the version number of a documentation, unless a change is required in the scraper and you've verified that it works. -To ask that an existing documentation be updated, please use the [Trello board](https://trello.com/c/2B0hmW7M/52-request-updates-here). +To ask that an existing documentation be updated, first check the last two [Documentation versions reports](https://github.com/freeCodeCamp/devdocs/issues?utf8=%E2%9C%93&q=Documentation+versions+report+is%3Aissue+author%3Adevdocs-bot+sort%3Aupdated-desc). Only create an issue if the documentation has been wrongly marked as up-to-date. ## Coding conventions diff --git a/.travis.yml b/.travis.yml index 0e64a76a..f3839425 100644 --- a/.travis.yml +++ b/.travis.yml @@ -6,6 +6,10 @@ before_script: - gem update --system - gem install bundler +script: + - if [ "$TRAVIS_EVENT_TYPE" != "cron" ]; then bundle exec rake; fi + - if [ "$TRAVIS_EVENT_TYPE" = "cron" ]; then bundle exec thor updates:check --github-token $GH_TOKEN --upload; fi + deploy: provider: heroku app: devdocs diff --git a/Gemfile b/Gemfile index a06f57d7..e3f75fce 100644 --- a/Gemfile +++ b/Gemfile @@ -42,6 +42,7 @@ group :docs do gem 'unix_utils', require: false gem 'tty-pager', require: false gem 'net-sftp', '>= 2.1.3.rc2', require: false + gem 'terminal-table', require: false end group :test do diff --git a/Gemfile.lock b/Gemfile.lock index 4ae8c270..b7023a31 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -104,6 +104,8 @@ GEM unicode-display_width (~> 1.4.0) unicode_utils (~> 1.4.0) strings-ansi (0.1.0) + terminal-table (1.8.0) + unicode-display_width (~> 1.1, >= 1.1.1) thin (1.7.2) daemons (~> 1.0, >= 1.0.9) eventmachine (~> 1.0, >= 1.0.4) @@ -157,6 +159,7 @@ DEPENDENCIES sinatra-contrib sprockets sprockets-helpers + terminal-table sprockets-sass thin thor diff --git a/docs/adding-docs.md b/docs/adding-docs.md index dfc96cb1..a89db7bb 100644 --- a/docs/adding-docs.md +++ b/docs/adding-docs.md @@ -17,6 +17,7 @@ Adding a documentation may look like a daunting task but once you get the hang o 10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works). 11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. It'll be added to the icon spritesheet after your pull request is merged. 12. Add the documentation's copyright details to the list in `assets/javascripts/templates/pages/about_tmpl.coffee`. This is the data shown in the table on the [about](https://devdocs.io/about) page, and is ordered alphabetically. Simply copying an existing item, placing it in the right slot and updating the values to match the new scraper will do the job. +13. Ensure `thor updates:check [my_doc]` shows the correct latest version. If the documentation includes more than a few hundreds pages and is available for download, try to scrape it locally (e.g. using `FileScraper`). It'll make the development process much faster and avoids putting too much load on the source site. (It's not a problem if your scraper is coupled to your local setup, just explain how it works in your pull request.) diff --git a/docs/scraper-reference.md b/docs/scraper-reference.md index d9fe8b4a..c2872388 100644 --- a/docs/scraper-reference.md +++ b/docs/scraper-reference.md @@ -184,3 +184,44 @@ More information about how filters work is available on the [Filter Reference](. Overrides the `:title` option for the root page only. _Note: this filter is disabled by default._ + +## Keeping scrapers up-to-date + +In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations. + +To make life easier, there are a few utility methods that you can use in `get_latest_version`: +* `fetch(url, opts)` + + Makes a GET request to the url and returns the response body. + + Example: [lib/docs/scrapers/bash.rb](../lib/docs/scrapers/bash.rb) +* `fetch_doc(url, opts)` + + Makes a GET request to the url and returns the HTML body converted to a Nokogiri document. + + Example: [lib/docs/scrapers/git.rb](../lib/docs/scrapers/git.rb) +* `fetch_json(url, opts)` + + Makes a GET request to the url and returns the JSON body converted to a dictionary. + + Example: [lib/docs/scrapers/mdn/mdn.rb](../lib/docs/scrapers/mdn/mdn.rb) +* `get_npm_version(package, opts)` + + Returns the latest version of the given npm package. + + Example: [lib/docs/scrapers/bower.rb](../lib/docs/scrapers/bower.rb) +* `get_latest_github_release(owner, repo, opts)` + + Returns the tag name of the latest GitHub release of the given repository. If the tag name is preceded by a "v", the "v" will be removed. + + Example: [lib/docs/scrapers/jsdoc.rb](../lib/docs/scrapers/jsdoc.rb) +* `get_github_tags(owner, repo, opts)` + + Returns the list of tags on the given repository ([format](https://developer.github.com/v3/repos/#list-tags)). + + Example: [lib/docs/scrapers/liquid.rb](../lib/docs/scrapers/liquid.rb) +* `get_github_file_contents(owner, repo, path, opts)` + + Returns the contents of the requested file in the default branch of the given repository. + + Example: [lib/docs/scrapers/minitest.rb](../lib/docs/scrapers/minitest.rb) diff --git a/lib/docs/core/doc.rb b/lib/docs/core/doc.rb index cb1cd209..da21daf8 100644 --- a/lib/docs/core/doc.rb +++ b/lib/docs/core/doc.rb @@ -152,7 +152,6 @@ module Docs end end - def initialize raise NotImplementedError, "#{self.class} is an abstract class and cannot be instantiated." if self.class.abstract end @@ -164,5 +163,104 @@ module Docs def build_pages(&block) raise NotImplementedError end + + def get_scraper_version(opts) + if self.class.method_defined?(:options) and !options[:release].nil? + options[:release] + else + # If options[:release] does not exist, we return the Epoch timestamp of when the doc was last modified in DevDocs production + json = fetch_json('https://devdocs.io/docs.json', opts) + items = json.select {|item| item['name'] == self.class.name} + items = items.map {|item| item['mtime']} + items.max + end + end + + # Should return the latest version of this documentation + # If options[:release] is defined, it should be in the same format + # If options[:release] is not defined, it should return the Epoch timestamp of when the documentation was last updated + # If the docs will never change, simply return '1.0.0' + def get_latest_version(opts) + raise NotImplementedError + end + + # Returns whether or not this scraper is outdated. + # + # The default implementation assumes the documentation uses a semver(-like) approach when it comes to versions. + # Patch updates are ignored because there are usually little to no documentation changes in bug-fix-only releases. + # + # Scrapers of documentations that do not use this versioning approach should override this method. + # + # Examples of the default implementation: + # 1 -> 2 = outdated + # 1.1 -> 1.2 = outdated + # 1.1.1 -> 1.1.2 = not outdated + def is_outdated(scraper_version, latest_version) + scraper_parts = scraper_version.to_s.split(/\./).map(&:to_i) + latest_parts = latest_version.to_s.split(/\./).map(&:to_i) + + # Only check the first two parts, the third part is for patch updates + [0, 1].each do |i| + break if i >= scraper_parts.length or i >= latest_parts.length + return true if latest_parts[i] > scraper_parts[i] + return false if latest_parts[i] < scraper_parts[i] + end + + false + end + + private + + # + # Utility methods for get_latest_version + # + + def fetch(url, opts) + headers = {} + + if opts.key?(:github_token) and url.start_with?('https://api.github.com/') + headers['Authorization'] = "token #{opts[:github_token]}" + end + + opts[:logger].debug("Fetching #{url}") + response = Request.run(url, { connecttimeout: 15, headers: headers }) + + if response.success? + response.body + else + reason = response.timed_out? ? "Timed out while connecting to #{url}" : "Couldn't fetch #{url} (response code #{response.code})" + opts[:logger].error(reason) + raise reason + end + end + + def fetch_doc(url, opts) + body = fetch(url, opts) + Nokogiri::HTML.parse(body, nil, 'UTF-8') + end + + def fetch_json(url, opts) + JSON.parse fetch(url, opts) + end + + def get_npm_version(package, opts) + json = fetch_json("https://registry.npmjs.com/#{package}", opts) + json['dist-tags']['latest'] + end + + def get_latest_github_release(owner, repo, opts) + release = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", opts) + tag_name = release['tag_name'] + tag_name.start_with?('v') ? tag_name[1..-1] : tag_name + end + + def get_github_tags(owner, repo, opts) + fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", opts) + end + + def get_github_file_contents(owner, repo, path, opts) + json = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", opts) + Base64.decode64(json['content']) + end end end diff --git a/lib/docs/scrapers/angular.rb b/lib/docs/scrapers/angular.rb index c318ce25..3365ec67 100644 --- a/lib/docs/scrapers/angular.rb +++ b/lib/docs/scrapers/angular.rb @@ -155,6 +155,10 @@ module Docs end end + def get_latest_version(opts) + get_npm_version('@angular/core', opts) + end + private def parse(response) diff --git a/lib/docs/scrapers/angularjs.rb b/lib/docs/scrapers/angularjs.rb index b8ff08b9..9d663e35 100644 --- a/lib/docs/scrapers/angularjs.rb +++ b/lib/docs/scrapers/angularjs.rb @@ -69,5 +69,9 @@ module Docs self.release = '1.2.32' self.base_url = "https://code.angularjs.org/#{release}/docs/partials/" end + + def get_latest_version(opts) + get_npm_version('angular', opts) + end end end diff --git a/lib/docs/scrapers/ansible.rb b/lib/docs/scrapers/ansible.rb index 2d62909a..b2363d4d 100644 --- a/lib/docs/scrapers/ansible.rb +++ b/lib/docs/scrapers/ansible.rb @@ -87,5 +87,10 @@ module Docs quickstart.html list_of_all_modules.html) end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.ansible.com/ansible/latest/index.html', opts) + doc.at_css('.DocSiteProduct-CurrentVersion').content.strip + end end end diff --git a/lib/docs/scrapers/apache.rb b/lib/docs/scrapers/apache.rb index 9ee82f12..1301b574 100644 --- a/lib/docs/scrapers/apache.rb +++ b/lib/docs/scrapers/apache.rb @@ -33,5 +33,10 @@ module Docs © 2018 The Apache Software Foundation
Licensed under the Apache License, Version 2.0. HTML + + def get_latest_version(opts) + doc = fetch_doc('http://httpd.apache.org/docs/', opts) + doc.at_css('#apcontents > ul a')['href'][0...-1] + end end end diff --git a/lib/docs/scrapers/apache_pig.rb b/lib/docs/scrapers/apache_pig.rb index 65897a78..f35085e6 100644 --- a/lib/docs/scrapers/apache_pig.rb +++ b/lib/docs/scrapers/apache_pig.rb @@ -43,5 +43,10 @@ module Docs self.base_url = "https://pig.apache.org/docs/r#{release}/" end + def get_latest_version(opts) + doc = fetch_doc('https://pig.apache.org/', opts) + item = doc.at_css('div[id="menu_1.2"] > .menuitem:last-child') + item.content.strip.sub(/Release /, '') + end end end diff --git a/lib/docs/scrapers/async.rb b/lib/docs/scrapers/async.rb index 40022f19..61615b54 100644 --- a/lib/docs/scrapers/async.rb +++ b/lib/docs/scrapers/async.rb @@ -17,5 +17,10 @@ module Docs © 2010–2018 Caolan McMahon
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://caolan.github.io/async/v3/', opts) + doc.at_css('#version-dropdown > a').content.strip[1..-1] + end end end diff --git a/lib/docs/scrapers/babel.rb b/lib/docs/scrapers/babel.rb index c9e40212..c8d716f1 100644 --- a/lib/docs/scrapers/babel.rb +++ b/lib/docs/scrapers/babel.rb @@ -22,5 +22,10 @@ module Docs stub '' do '
' end + + def get_latest_version(opts) + doc = fetch_doc('https://babeljs.io/docs/en/', opts) + doc.at_css('a[href="/versions"] > h3').content + end end end diff --git a/lib/docs/scrapers/backbone.rb b/lib/docs/scrapers/backbone.rb index b72b1084..2b33505e 100644 --- a/lib/docs/scrapers/backbone.rb +++ b/lib/docs/scrapers/backbone.rb @@ -20,5 +20,10 @@ module Docs © 2010–2016 Jeremy Ashkenas, DocumentCloud
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://backbonejs.org/', opts) + doc.at_css('.version').content[1...-1] + end end end diff --git a/lib/docs/scrapers/bash.rb b/lib/docs/scrapers/bash.rb index feb0ddce..92ef894a 100644 --- a/lib/docs/scrapers/bash.rb +++ b/lib/docs/scrapers/bash.rb @@ -17,5 +17,10 @@ module Docs Copyright © 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.
Licensed under the GNU Free Documentation License. HTML + + def get_latest_version(opts) + body = fetch('https://www.gnu.org/software/bash/manual/html_node/index.html', opts) + body.scan(/, Version ([0-9.]+)/)[0][0][0...-1] + end end end diff --git a/lib/docs/scrapers/bluebird.rb b/lib/docs/scrapers/bluebird.rb index e5cd6b59..8f38120a 100644 --- a/lib/docs/scrapers/bluebird.rb +++ b/lib/docs/scrapers/bluebird.rb @@ -18,5 +18,9 @@ module Docs © 2013–2017 Petka Antonov
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('bluebird', opts) + end end end diff --git a/lib/docs/scrapers/bootstrap.rb b/lib/docs/scrapers/bootstrap.rb index 7b2406b8..8571462e 100644 --- a/lib/docs/scrapers/bootstrap.rb +++ b/lib/docs/scrapers/bootstrap.rb @@ -34,5 +34,10 @@ module Docs options[:only] = %w(getting-started/ css/ components/ javascript/) end + + def get_latest_version(opts) + doc = fetch_doc('https://getbootstrap.com/', opts) + doc.at_css('#bd-versions').content.strip[1..-1] + end end end diff --git a/lib/docs/scrapers/bottle.rb b/lib/docs/scrapers/bottle.rb index 25ad7f6e..d0397ec7 100644 --- a/lib/docs/scrapers/bottle.rb +++ b/lib/docs/scrapers/bottle.rb @@ -27,5 +27,11 @@ module Docs self.release = '0.11.7' self.base_url = "https://bottlepy.org/docs/#{self.version}/" end + + def get_latest_version(opts) + doc = fetch_doc('https://bottlepy.org/docs/stable/', opts) + label = doc.at_css('.sphinxsidebarwrapper > ul > li > b') + label.content.sub(/Bottle /, '') + end end end diff --git a/lib/docs/scrapers/bower.rb b/lib/docs/scrapers/bower.rb index b032f1d3..aab2a1e9 100644 --- a/lib/docs/scrapers/bower.rb +++ b/lib/docs/scrapers/bower.rb @@ -19,5 +19,9 @@ module Docs © 2018 Bower contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('bower', opts) + end end end diff --git a/lib/docs/scrapers/c.rb b/lib/docs/scrapers/c.rb index f9289617..ec99f704 100644 --- a/lib/docs/scrapers/c.rb +++ b/lib/docs/scrapers/c.rb @@ -26,6 +26,13 @@ module Docs Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0. HTML + def get_latest_version(opts) + doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts) + link = doc.at_css('a[title^="File:"]') + date = link.content.scan(/(\d+)\./)[0][0] + DateTime.strptime(date, '%Y%m%d').to_time.to_i + end + private def file_path_for(*) diff --git a/lib/docs/scrapers/cakephp.rb b/lib/docs/scrapers/cakephp.rb index 08dbead0..6291b4ab 100644 --- a/lib/docs/scrapers/cakephp.rb +++ b/lib/docs/scrapers/cakephp.rb @@ -71,6 +71,11 @@ module Docs self.base_url = 'https://api.cakephp.org/2.7/' end + def get_latest_version(opts) + doc = fetch_doc('https://api.cakephp.org/3.7/', opts) + doc.at_css('.version-picker .dropdown-toggle').content.strip + end + private def parse(response) diff --git a/lib/docs/scrapers/chai.rb b/lib/docs/scrapers/chai.rb index 9d8aa4d2..759f7540 100644 --- a/lib/docs/scrapers/chai.rb +++ b/lib/docs/scrapers/chai.rb @@ -23,5 +23,9 @@ module Docs © 2016 Chai.js Assertion Library
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('chai', opts) + end end end diff --git a/lib/docs/scrapers/chef.rb b/lib/docs/scrapers/chef.rb index 2fd32a83..f0b7d6b0 100644 --- a/lib/docs/scrapers/chef.rb +++ b/lib/docs/scrapers/chef.rb @@ -47,5 +47,10 @@ module Docs options[:only_patterns] = [/\A#{client_path}\//, /\A#{server_path}\//] end + + def get_latest_version(opts) + doc = fetch_doc('https://downloads.chef.io/chef', opts) + doc.at_css('h1.product-heading > span').content.strip + end end end diff --git a/lib/docs/scrapers/clojure.rb b/lib/docs/scrapers/clojure.rb index c6bdcdea..b5785bd3 100644 --- a/lib/docs/scrapers/clojure.rb +++ b/lib/docs/scrapers/clojure.rb @@ -27,5 +27,10 @@ module Docs self.release = '1.7' self.base_url = 'https://clojure.github.io/clojure/branch-clojure-1.7.0/' end + + def get_latest_version(opts) + doc = fetch_doc('http://clojure.github.io/clojure/index.html', opts) + doc.at_css('#header-version').content[1..-1] + end end end diff --git a/lib/docs/scrapers/cmake.rb b/lib/docs/scrapers/cmake.rb index c455e4fd..7548a4a9 100644 --- a/lib/docs/scrapers/cmake.rb +++ b/lib/docs/scrapers/cmake.rb @@ -59,5 +59,11 @@ module Docs self.release = '3.5.2' self.base_url = 'https://cmake.org/cmake/help/v3.5/' end + + def get_latest_version(opts) + doc = fetch_doc('https://cmake.org/documentation/', opts) + link = doc.at_css('.entry-content ul > li > strong > a > big') + link.content.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/codeception.rb b/lib/docs/scrapers/codeception.rb index 919f146d..caafc9cf 100644 --- a/lib/docs/scrapers/codeception.rb +++ b/lib/docs/scrapers/codeception.rb @@ -18,5 +18,10 @@ module Docs © 2011 Michael Bodnarchuk and contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://codeception.com/changelog', opts) + doc.at_css('#page > h4').content + end end end diff --git a/lib/docs/scrapers/codeceptjs.rb b/lib/docs/scrapers/codeceptjs.rb index 13189340..34d9b855 100644 --- a/lib/docs/scrapers/codeceptjs.rb +++ b/lib/docs/scrapers/codeceptjs.rb @@ -21,5 +21,9 @@ module Docs © 2015 DavertMik <davert@codegyre.com> (http://codegyre.com)
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('codeceptjs', opts) + end end end diff --git a/lib/docs/scrapers/codeigniter.rb b/lib/docs/scrapers/codeigniter.rb index 573f9b8c..05258d9a 100644 --- a/lib/docs/scrapers/codeigniter.rb +++ b/lib/docs/scrapers/codeigniter.rb @@ -38,5 +38,11 @@ module Docs version '3' do self.release = '3.1.8' end + + def get_latest_version(opts) + doc = fetch_doc('https://codeigniter.com/user_guide/changelog.html', opts) + header = doc.at_css('#change-log h2') + header.content.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/coffeescript.rb b/lib/docs/scrapers/coffeescript.rb index 23e9557f..695f3697 100644 --- a/lib/docs/scrapers/coffeescript.rb +++ b/lib/docs/scrapers/coffeescript.rb @@ -30,5 +30,9 @@ module Docs options[:container] = '.container' end + + def get_latest_version(opts) + get_npm_version('coffeescript', opts) + end end end diff --git a/lib/docs/scrapers/cordova.rb b/lib/docs/scrapers/cordova.rb index f74c72ff..65cf7f60 100644 --- a/lib/docs/scrapers/cordova.rb +++ b/lib/docs/scrapers/cordova.rb @@ -42,5 +42,15 @@ module Docs self.release = '6.5.0' self.base_url = 'https://cordova.apache.org/docs/en/6.x/' end + + def get_latest_version(opts) + doc = fetch_doc('https://cordova.apache.org/docs/en/latest/', opts) + + label = doc.at_css('#versionDropdown').content.strip + version = label.scan(/([0-9.]+)/)[0][0] + version = version[0...-1] if version.end_with?('.') + + version + end end end diff --git a/lib/docs/scrapers/cpp.rb b/lib/docs/scrapers/cpp.rb index 374f6883..f96ee8f1 100644 --- a/lib/docs/scrapers/cpp.rb +++ b/lib/docs/scrapers/cpp.rb @@ -34,6 +34,14 @@ module Docs Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0. HTML + # Same as get_latest_version in lib/docs/scrapers/c.rb + def get_latest_version(opts) + doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts) + link = doc.at_css('a[title^="File:"]') + date = link.content.scan(/(\d+)\./)[0][0] + DateTime.strptime(date, '%Y%m%d').to_time.to_i + end + private def file_path_for(*) diff --git a/lib/docs/scrapers/crystal.rb b/lib/docs/scrapers/crystal.rb index 29061a1d..14537c7f 100644 --- a/lib/docs/scrapers/crystal.rb +++ b/lib/docs/scrapers/crystal.rb @@ -34,5 +34,10 @@ module Docs HTML end } + + def get_latest_version(opts) + body = fetch('https://crystal-lang.org/api', opts) + body.scan(/Crystal Docs ([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/d.rb b/lib/docs/scrapers/d.rb index 6126380e..e1475b45 100644 --- a/lib/docs/scrapers/d.rb +++ b/lib/docs/scrapers/d.rb @@ -26,5 +26,10 @@ module Docs def initial_urls %w(https://dlang.org/phobos/index.html https://dlang.org/spec/intro.html) end + + def get_latest_version(opts) + doc = fetch_doc('https://dlang.org/changelog/', opts) + doc.at_css('#content > ul > li:nth-child(2) > a')['id'] + end end end diff --git a/lib/docs/scrapers/d3.rb b/lib/docs/scrapers/d3.rb index 26b27ca5..e26c1f3d 100644 --- a/lib/docs/scrapers/d3.rb +++ b/lib/docs/scrapers/d3.rb @@ -58,5 +58,9 @@ module Docs options[:root_title] = 'D3.js' options[:only_patterns] = [/\.md\z/] end + + def get_latest_version(opts) + get_npm_version('d3', opts) + end end end diff --git a/lib/docs/scrapers/dart.rb b/lib/docs/scrapers/dart.rb index c345c22f..322bfe2a 100644 --- a/lib/docs/scrapers/dart.rb +++ b/lib/docs/scrapers/dart.rb @@ -31,5 +31,11 @@ module Docs self.release = '1.24.3' self.base_url = "https://api.dartlang.org/stable/#{release}/" end + + def get_latest_version(opts) + doc = fetch_doc('https://api.dartlang.org/', opts) + label = doc.at_css('footer > span').content.strip + label.sub(/Dart /, '') + end end end diff --git a/lib/docs/scrapers/django.rb b/lib/docs/scrapers/django.rb index 45273540..6d48c6d7 100644 --- a/lib/docs/scrapers/django.rb +++ b/lib/docs/scrapers/django.rb @@ -63,5 +63,10 @@ module Docs self.release = '1.8.18' self.base_url = 'https://docs.djangoproject.com/en/1.8/' end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.djangoproject.com/', opts) + doc.at_css('#doc-versions > li.current > span > strong').content + end end end diff --git a/lib/docs/scrapers/docker.rb b/lib/docs/scrapers/docker.rb index 92494f8a..3ef60aab 100644 --- a/lib/docs/scrapers/docker.rb +++ b/lib/docs/scrapers/docker.rb @@ -137,5 +137,11 @@ module Docs options[:container] = '#docs' options[:only_patterns] << /\Aswarm\// end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.docker.com/', opts) + label = doc.at_css('.nav-container button.dropdown-toggle').content.strip + label.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/dojo.rb b/lib/docs/scrapers/dojo.rb index 937ed21a..79898916 100644 --- a/lib/docs/scrapers/dojo.rb +++ b/lib/docs/scrapers/dojo.rb @@ -36,6 +36,11 @@ module Docs urls.map { |url| "#{url}" }.join end + def get_latest_version(opts) + doc = fetch_doc('https://dojotoolkit.org/api/', opts) + doc.at_css('#versionSelector > option[selected]').content + end + private def get_url_list(json, set = Set.new) diff --git a/lib/docs/scrapers/drupal.rb b/lib/docs/scrapers/drupal.rb index 5710eb36..3798caec 100644 --- a/lib/docs/scrapers/drupal.rb +++ b/lib/docs/scrapers/drupal.rb @@ -98,5 +98,10 @@ module Docs /\A[\w\-\.]+\.php\/7\.x\z/ ] end + + def get_latest_version(opts) + json = fetch_json('https://packagist.org/packages/drupal/drupal.json', opts) + json['package']['versions'].keys.find {|version| !version.end_with?('-dev')} + end end end diff --git a/lib/docs/scrapers/electron.rb b/lib/docs/scrapers/electron.rb index 3cb399f0..8e635f49 100644 --- a/lib/docs/scrapers/electron.rb +++ b/lib/docs/scrapers/electron.rb @@ -22,5 +22,10 @@ module Docs © 2013–2018 GitHub Inc.
Licensed under the MIT license. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://electronjs.org/docs', opts) + doc.at_css('.docs-version').content + end end end diff --git a/lib/docs/scrapers/elixir.rb b/lib/docs/scrapers/elixir.rb index 10d5aac1..a25cca41 100644 --- a/lib/docs/scrapers/elixir.rb +++ b/lib/docs/scrapers/elixir.rb @@ -97,5 +97,10 @@ module Docs 'https://elixir-lang.org/getting-started/' ] end + + def get_latest_version(opts) + doc = fetch_doc('https://hexdocs.pm/elixir/api-reference.html', opts) + doc.at_css('h2.sidebar-projectVersion').content.strip[1..-1] + end end end diff --git a/lib/docs/scrapers/ember.rb b/lib/docs/scrapers/ember.rb index 3db20c94..6f853bb9 100644 --- a/lib/docs/scrapers/ember.rb +++ b/lib/docs/scrapers/ember.rb @@ -56,5 +56,10 @@ module Docs https://emberjs.com/api/ember-data/2.14/classes/DS ) end + + def get_latest_version(opts) + doc = fetch_doc('https://emberjs.com/api/ember/release', opts) + doc.at_css('.sidebar > .select-container .ember-power-select-selected-item').content.strip + end end end diff --git a/lib/docs/scrapers/erlang.rb b/lib/docs/scrapers/erlang.rb index d6aa2a0b..14a87cf5 100644 --- a/lib/docs/scrapers/erlang.rb +++ b/lib/docs/scrapers/erlang.rb @@ -55,5 +55,10 @@ module Docs version '18' do self.release = '18.3' end + + def get_latest_version(opts) + doc = fetch_doc('https://www.erlang.org/downloads', opts) + doc.at_css('.col-lg-3 > ul > li').content.strip.sub(/OTP /, '') + end end end diff --git a/lib/docs/scrapers/eslint.rb b/lib/docs/scrapers/eslint.rb index 8b4c9a2e..c213eacc 100644 --- a/lib/docs/scrapers/eslint.rb +++ b/lib/docs/scrapers/eslint.rb @@ -20,5 +20,9 @@ module Docs © JS Foundation and other contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('eslint', opts) + end end end diff --git a/lib/docs/scrapers/express.rb b/lib/docs/scrapers/express.rb index 0fb4ed14..990019fb 100644 --- a/lib/docs/scrapers/express.rb +++ b/lib/docs/scrapers/express.rb @@ -28,5 +28,9 @@ module Docs © 2017 StrongLoop, IBM, and other expressjs.com contributors.
Licensed under the Creative Commons Attribution-ShareAlike License v3.0. HTML + + def get_latest_version(opts) + get_npm_version('express', opts) + end end end diff --git a/lib/docs/scrapers/falcon.rb b/lib/docs/scrapers/falcon.rb index 5bfd8efc..8ba69150 100644 --- a/lib/docs/scrapers/falcon.rb +++ b/lib/docs/scrapers/falcon.rb @@ -33,5 +33,10 @@ module Docs self.release = '1.2.0' self.base_url = "https://falcon.readthedocs.io/en/#{self.release}/" end + + def get_latest_version(opts) + doc = fetch_doc('https://falcon.readthedocs.io/en/stable/changes/index.html', opts) + doc.at_css('#changelogs ul > li > a').content + end end end diff --git a/lib/docs/scrapers/fish.rb b/lib/docs/scrapers/fish.rb index 5ccfa71c..c9a98802 100644 --- a/lib/docs/scrapers/fish.rb +++ b/lib/docs/scrapers/fish.rb @@ -46,5 +46,10 @@ module Docs self.release = '2.2.0' self.base_url = "https://fishshell.com/docs/#{version}/" end + + def get_latest_version(opts) + doc = fetch_doc('http://fishshell.com/docs/current/index.html', opts) + doc.at_css('#toc-index').content.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/flow.rb b/lib/docs/scrapers/flow.rb index 16ea70dd..b3b5a02f 100644 --- a/lib/docs/scrapers/flow.rb +++ b/lib/docs/scrapers/flow.rb @@ -18,5 +18,9 @@ module Docs © 2013–present Facebook Inc.
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('flow-bin', opts) + end end end diff --git a/lib/docs/scrapers/git.rb b/lib/docs/scrapers/git.rb index 26b2da95..9de5cb0d 100644 --- a/lib/docs/scrapers/git.rb +++ b/lib/docs/scrapers/git.rb @@ -19,5 +19,10 @@ module Docs © 2005–2018 Linus Torvalds and others
Licensed under the GNU General Public License version 2. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://git-scm.com/', opts) + doc.at_css('.version').content.strip + end end end diff --git a/lib/docs/scrapers/gnu/gcc.rb b/lib/docs/scrapers/gnu/gcc.rb index be3bb54e..565706d9 100644 --- a/lib/docs/scrapers/gnu/gcc.rb +++ b/lib/docs/scrapers/gnu/gcc.rb @@ -99,5 +99,11 @@ module Docs options[:replace_paths] = CPP_PATHS end + + def get_latest_version(opts) + doc = fetch_doc('https://gcc.gnu.org/onlinedocs/', opts) + label = doc.at_css('ul > li > ul > li > a').content.strip + label.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/gnu/gnu_fortran.rb b/lib/docs/scrapers/gnu/gnu_fortran.rb index 2610178e..dd18827c 100644 --- a/lib/docs/scrapers/gnu/gnu_fortran.rb +++ b/lib/docs/scrapers/gnu/gnu_fortran.rb @@ -25,5 +25,11 @@ module Docs self.release = '4.9.3' self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gfortran/" end + + def get_latest_version(opts) + doc = fetch_doc('https://gcc.gnu.org/onlinedocs/', opts) + label = doc.at_css('ul > li > ul > li > a').content.strip + label.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/go.rb b/lib/docs/scrapers/go.rb index 7b233317..27b46652 100644 --- a/lib/docs/scrapers/go.rb +++ b/lib/docs/scrapers/go.rb @@ -24,6 +24,11 @@ module Docs Licensed under the Creative Commons Attribution License 3.0. HTML + def get_latest_version(opts) + doc = fetch_doc('https://golang.org/project/', opts) + doc.at_css('#page ul > li > a').text[3..-1] + end + private def parse(response) # Hook here because Nokogori removes whitespace from textareas diff --git a/lib/docs/scrapers/godot.rb b/lib/docs/scrapers/godot.rb index 7e7da9a6..06c330b2 100644 --- a/lib/docs/scrapers/godot.rb +++ b/lib/docs/scrapers/godot.rb @@ -37,5 +37,10 @@ module Docs self.release = '2.1' self.base_url = "http://docs.godotengine.org/en/#{self.version}/" end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.godotengine.org/', opts) + doc.at_css('.version').content.strip + end end end diff --git a/lib/docs/scrapers/graphite.rb b/lib/docs/scrapers/graphite.rb index 49ade898..83e9314a 100644 --- a/lib/docs/scrapers/graphite.rb +++ b/lib/docs/scrapers/graphite.rb @@ -17,5 +17,10 @@ module Docs © 2011–2016 The Graphite Project
Licensed under the Apache License, Version 2.0. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://graphite.readthedocs.io/en/latest/releases.html', opts) + doc.at_css('#release-notes li > a').content + end end end diff --git a/lib/docs/scrapers/grunt.rb b/lib/docs/scrapers/grunt.rb index 2201c043..469d10a0 100644 --- a/lib/docs/scrapers/grunt.rb +++ b/lib/docs/scrapers/grunt.rb @@ -26,5 +26,9 @@ module Docs © GruntJS Team
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('grunt-cli', opts) + end end end diff --git a/lib/docs/scrapers/handlebars.rb b/lib/docs/scrapers/handlebars.rb index 22935d21..046cdf0f 100644 --- a/lib/docs/scrapers/handlebars.rb +++ b/lib/docs/scrapers/handlebars.rb @@ -19,5 +19,9 @@ module Docs © 2011–2017 by Yehuda Katz
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('handlebars', opts) + end end end diff --git a/lib/docs/scrapers/haskell.rb b/lib/docs/scrapers/haskell.rb index 442339b3..fb118851 100755 --- a/lib/docs/scrapers/haskell.rb +++ b/lib/docs/scrapers/haskell.rb @@ -10,7 +10,7 @@ module Docs html_filters.push 'haskell/entries', 'haskell/clean_html' - options[:container] = ->(filter) { filter.subpath.start_with?('users_guide') ? '.body' : '#content' } + options[:container] = ->(filter) {filter.subpath.start_with?('users_guide') ? '.body' : '#content'} options[:only_patterns] = [/\Alibraries\//, /\Ausers_guide\//] options[:skip_patterns] = [ @@ -68,5 +68,12 @@ module Docs options[:only_patterns] = [/\Alibraries\//] end + + def get_latest_version(opts) + doc = fetch_doc('https://downloads.haskell.org/~ghc/latest/docs/html/', opts) + links = doc.css('a').to_a + versions = links.map {|link| link['href'].scan(/ghc-([0-9.]+)/)} + versions.find {|version| !version.empty?}[0][0] + end end end diff --git a/lib/docs/scrapers/haxe.rb b/lib/docs/scrapers/haxe.rb index 33f20b93..2dbab01a 100644 --- a/lib/docs/scrapers/haxe.rb +++ b/lib/docs/scrapers/haxe.rb @@ -66,5 +66,11 @@ module Docs version 'Python' do self.base_url = 'https://api.haxe.org/python/' end + + def get_latest_version(opts) + doc = fetch_doc('https://api.haxe.org/', opts) + label = doc.at_css('.container.main-content h1 > small').content + label.sub(/version /, '') + end end end diff --git a/lib/docs/scrapers/homebrew.rb b/lib/docs/scrapers/homebrew.rb index fba79ec0..9dd1581a 100644 --- a/lib/docs/scrapers/homebrew.rb +++ b/lib/docs/scrapers/homebrew.rb @@ -19,5 +19,9 @@ module Docs © 2009–present Homebrew contributors
Licensed under the BSD 2-Clause License. HTML + + def get_latest_version(opts) + get_latest_github_release('Homebrew', 'brew', opts) + end end end diff --git a/lib/docs/scrapers/http.rb b/lib/docs/scrapers/http.rb index 60f15f75..90e6f5c1 100644 --- a/lib/docs/scrapers/http.rb +++ b/lib/docs/scrapers/http.rb @@ -7,6 +7,8 @@ module Docs html_filters.push 'http/clean_html', 'http/entries', 'title' + options[:mdn_tag] = 'HTTP' + options[:root_title] = 'HTTP' options[:title] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? false : filter.default_title } options[:container] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? '.content' : nil } diff --git a/lib/docs/scrapers/immutable.rb b/lib/docs/scrapers/immutable.rb index fa7fb81b..8b1b47a2 100644 --- a/lib/docs/scrapers/immutable.rb +++ b/lib/docs/scrapers/immutable.rb @@ -54,5 +54,9 @@ module Docs JS capybara.html end + + def get_latest_version(opts) + get_npm_version('immutable', opts) + end end end diff --git a/lib/docs/scrapers/influxdata.rb b/lib/docs/scrapers/influxdata.rb index 6c83b66b..db160f9c 100644 --- a/lib/docs/scrapers/influxdata.rb +++ b/lib/docs/scrapers/influxdata.rb @@ -46,5 +46,11 @@ module Docs © 2015 InfluxData, Inc.
Licensed under the MIT license. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://docs.influxdata.com/influxdb/', opts) + label = doc.at_css('.navbar--current-product').content.strip + label.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/jasmine.rb b/lib/docs/scrapers/jasmine.rb index 82f3c9cf..b1971ecd 100644 --- a/lib/docs/scrapers/jasmine.rb +++ b/lib/docs/scrapers/jasmine.rb @@ -17,5 +17,9 @@ module Docs © 2008–2017 Pivotal Labs
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_latest_github_release('jasmine', 'jasmine', opts) + end end end diff --git a/lib/docs/scrapers/jekyll.rb b/lib/docs/scrapers/jekyll.rb index 1faaa9de..500eee10 100644 --- a/lib/docs/scrapers/jekyll.rb +++ b/lib/docs/scrapers/jekyll.rb @@ -28,5 +28,10 @@ module Docs © 2008–2018 Tom Preston-Werner and Jekyll contributors
Licensed under the MIT license. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://jekyllrb.com/docs/', opts) + doc.at_css('.meta a').content[1..-1] + end end end diff --git a/lib/docs/scrapers/jest.rb b/lib/docs/scrapers/jest.rb index f4ce944f..a495d939 100644 --- a/lib/docs/scrapers/jest.rb +++ b/lib/docs/scrapers/jest.rb @@ -17,5 +17,10 @@ module Docs © 2014–present Facebook Inc.
Licensed under the BSD License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://jestjs.io/docs/en/getting-started', opts) + doc.at_css('header > a > h3').content + end end end diff --git a/lib/docs/scrapers/jquery/jquery_core.rb b/lib/docs/scrapers/jquery/jquery_core.rb index 20aca0dc..a0c8b97a 100644 --- a/lib/docs/scrapers/jquery/jquery_core.rb +++ b/lib/docs/scrapers/jquery/jquery_core.rb @@ -22,5 +22,9 @@ module Docs /Selectors\/odd/i, /index/i ] + + def get_latest_version(opts) + get_npm_version('jquery', opts) + end end end diff --git a/lib/docs/scrapers/jquery/jquery_mobile.rb b/lib/docs/scrapers/jquery/jquery_mobile.rb index 8e5abf1c..5b856a95 100644 --- a/lib/docs/scrapers/jquery/jquery_mobile.rb +++ b/lib/docs/scrapers/jquery/jquery_mobile.rb @@ -16,5 +16,10 @@ module Docs options[:fix_urls] = ->(url) do url.sub! 'http://api.jquerymobile.com/', 'https://api.jquerymobile.com/' end + + def get_latest_version(opts) + doc = fetch_doc('https://jquerymobile.com/', opts) + doc.at_css('.download-box > .download-option:last-child > span').content.sub(/Version /, '') + end end end diff --git a/lib/docs/scrapers/jquery/jquery_ui.rb b/lib/docs/scrapers/jquery/jquery_ui.rb index 0c90fc1a..021d1d22 100644 --- a/lib/docs/scrapers/jquery/jquery_ui.rb +++ b/lib/docs/scrapers/jquery/jquery_ui.rb @@ -15,5 +15,9 @@ module Docs options[:fix_urls] = ->(url) do url.sub! 'http://api.jqueryui.com/', 'https://api.jqueryui.com/' end + + def get_latest_version(opts) + get_npm_version('jquery-ui', opts) + end end end diff --git a/lib/docs/scrapers/jsdoc.rb b/lib/docs/scrapers/jsdoc.rb index bb3781ca..df27e578 100644 --- a/lib/docs/scrapers/jsdoc.rb +++ b/lib/docs/scrapers/jsdoc.rb @@ -21,5 +21,9 @@ module Docs © 2011–2017 the contributors to the JSDoc 3 documentation project
Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0. HTML + + def get_latest_version(opts) + get_latest_github_release('jsdoc3', 'jsdoc', opts) + end end end diff --git a/lib/docs/scrapers/julia.rb b/lib/docs/scrapers/julia.rb index 5bc16b77..c9c96da6 100644 --- a/lib/docs/scrapers/julia.rb +++ b/lib/docs/scrapers/julia.rb @@ -49,5 +49,9 @@ module Docs html_filters.push 'julia/entries_sphinx', 'julia/clean_html_sphinx', 'sphinx/clean_html' end + + def get_latest_version(opts) + get_latest_github_release('JuliaLang', 'julia', opts) + end end end diff --git a/lib/docs/scrapers/knockout.rb b/lib/docs/scrapers/knockout.rb index 663f7847..efad86f0 100644 --- a/lib/docs/scrapers/knockout.rb +++ b/lib/docs/scrapers/knockout.rb @@ -33,5 +33,9 @@ module Docs © Steven Sanderson, the Knockout.js team, and other contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_latest_github_release('knockout', 'knockout', opts) + end end end diff --git a/lib/docs/scrapers/koa.rb b/lib/docs/scrapers/koa.rb index 3ce79cac..cac14920 100644 --- a/lib/docs/scrapers/koa.rb +++ b/lib/docs/scrapers/koa.rb @@ -34,5 +34,9 @@ module Docs © 2018 Koa contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('koa', opts) + end end end diff --git a/lib/docs/scrapers/kotlin.rb b/lib/docs/scrapers/kotlin.rb index 415393d1..5055b65e 100644 --- a/lib/docs/scrapers/kotlin.rb +++ b/lib/docs/scrapers/kotlin.rb @@ -28,5 +28,9 @@ module Docs © 2010–2018 JetBrains s.r.o.
Licensed under the Apache License, Version 2.0. HTML + + def get_latest_version(opts) + get_latest_github_release('JetBrains', 'kotlin', opts) + end end end diff --git a/lib/docs/scrapers/laravel.rb b/lib/docs/scrapers/laravel.rb index 5c88ae0f..e45b0bed 100644 --- a/lib/docs/scrapers/laravel.rb +++ b/lib/docs/scrapers/laravel.rb @@ -133,5 +133,9 @@ module Docs url end end + + def get_latest_version(opts) + get_latest_github_release('laravel', 'laravel', opts) + end end end diff --git a/lib/docs/scrapers/leaflet.rb b/lib/docs/scrapers/leaflet.rb index c8e2071c..38e497e7 100644 --- a/lib/docs/scrapers/leaflet.rb +++ b/lib/docs/scrapers/leaflet.rb @@ -39,5 +39,10 @@ module Docs self.base_url = "https://leafletjs.com/reference-#{release}.html" end + def get_latest_version(opts) + doc = fetch_doc('https://leafletjs.com/index.html', opts) + link = doc.css('ul > li > a').to_a.select {|node| node.content == 'Docs'}.first + link['href'].scan(/reference-([0-9.]+)\.html/)[0][0] + end end end diff --git a/lib/docs/scrapers/less.rb b/lib/docs/scrapers/less.rb index a0947e1a..b19bbe17 100644 --- a/lib/docs/scrapers/less.rb +++ b/lib/docs/scrapers/less.rb @@ -21,5 +21,11 @@ module Docs © 2009–2016 The Core Less Team
Licensed under the Creative Commons Attribution License 3.0. HTML + + def get_latest_version(opts) + doc = fetch_doc('http://lesscss.org/features/', opts) + label = doc.at_css('.footer-links > li').content + label.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/liquid.rb b/lib/docs/scrapers/liquid.rb index 9ebc4041..b8e40d59 100644 --- a/lib/docs/scrapers/liquid.rb +++ b/lib/docs/scrapers/liquid.rb @@ -19,5 +19,10 @@ module Docs © 2005, 2006 Tobias Luetke
Licensed under the MIT License. HTML + + def get_latest_version(opts) + tags = get_github_tags('Shopify', 'liquid', opts) + tags[0]['name'][1..-1] + end end end diff --git a/lib/docs/scrapers/lodash.rb b/lib/docs/scrapers/lodash.rb index 0461f7b7..bce625e6 100644 --- a/lib/docs/scrapers/lodash.rb +++ b/lib/docs/scrapers/lodash.rb @@ -32,5 +32,10 @@ module Docs self.release = '2.4.2' self.base_url = "https://lodash.com/docs/#{release}" end + + def get_latest_version(opts) + doc = fetch_doc('https://lodash.com/docs/', opts) + doc.at_css('#version > option[selected]').content + end end end diff --git a/lib/docs/scrapers/love.rb b/lib/docs/scrapers/love.rb index 7f23bded..887b796f 100644 --- a/lib/docs/scrapers/love.rb +++ b/lib/docs/scrapers/love.rb @@ -39,5 +39,10 @@ module Docs © 2006–2016 LÖVE Development Team
Licensed under the GNU Free Documentation License, Version 1.3. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://love2d.org/wiki/Version_History', opts) + doc.at_css('#mw-content-text table a').content + end end end diff --git a/lib/docs/scrapers/lua.rb b/lib/docs/scrapers/lua.rb index 40a5c007..e3608918 100644 --- a/lib/docs/scrapers/lua.rb +++ b/lib/docs/scrapers/lua.rb @@ -26,5 +26,10 @@ module Docs self.release = '5.1.5' self.base_url = 'https://www.lua.org/manual/5.1/' end + + def get_latest_version(opts) + doc = fetch_doc('https://www.lua.org/manual/', opts) + doc.at_css('p.menubar > a').content + end end end diff --git a/lib/docs/scrapers/marionette.rb b/lib/docs/scrapers/marionette.rb index fea6617f..fd1eab8e 100644 --- a/lib/docs/scrapers/marionette.rb +++ b/lib/docs/scrapers/marionette.rb @@ -38,5 +38,9 @@ module Docs html_filters.push 'marionette/entries_v2' end + + def get_latest_version(opts) + get_npm_version('backbone.marionette', opts) + end end end diff --git a/lib/docs/scrapers/markdown.rb b/lib/docs/scrapers/markdown.rb index 87e9c957..b837c692 100644 --- a/lib/docs/scrapers/markdown.rb +++ b/lib/docs/scrapers/markdown.rb @@ -13,5 +13,9 @@ module Docs © 2004 John Gruber
Licensed under the BSD License. HTML + + def get_latest_version(opts) + '1.0.0' + end end end diff --git a/lib/docs/scrapers/matplotlib.rb b/lib/docs/scrapers/matplotlib.rb index ddd1f9de..eeecea71 100644 --- a/lib/docs/scrapers/matplotlib.rb +++ b/lib/docs/scrapers/matplotlib.rb @@ -64,5 +64,9 @@ module Docs "https://matplotlib.org/#{release}/mpl_toolkits/axes_grid/api/" ] end + + def get_latest_version(opts) + get_latest_github_release('matplotlib', 'matplotlib', opts) + end end end diff --git a/lib/docs/scrapers/mdn/css.rb b/lib/docs/scrapers/mdn/css.rb index 4c44f1f1..abb69b3a 100644 --- a/lib/docs/scrapers/mdn/css.rb +++ b/lib/docs/scrapers/mdn/css.rb @@ -6,6 +6,8 @@ module Docs html_filters.push 'css/clean_html', 'css/entries', 'title' + options[:mdn_tag] = 'CSS' + options[:root_title] = 'CSS' options[:skip] = %w(/CSS3 /Media/Visual /paged_media /Media/TV /Media/Tactile) diff --git a/lib/docs/scrapers/mdn/dom.rb b/lib/docs/scrapers/mdn/dom.rb index bbf95b20..a2202929 100644 --- a/lib/docs/scrapers/mdn/dom.rb +++ b/lib/docs/scrapers/mdn/dom.rb @@ -8,6 +8,8 @@ module Docs html_filters.push 'dom/clean_html', 'dom/entries', 'title' + options[:mdn_tag] = 'XSLT_Reference' + options[:root_title] = 'DOM' options[:skip] = %w( diff --git a/lib/docs/scrapers/mdn/dom_events.rb b/lib/docs/scrapers/mdn/dom_events.rb index fcbdc08f..258fbcd4 100644 --- a/lib/docs/scrapers/mdn/dom_events.rb +++ b/lib/docs/scrapers/mdn/dom_events.rb @@ -9,6 +9,8 @@ module Docs html_filters.insert_after 'clean_html', 'dom_events/clean_html' html_filters.push 'dom_events/entries', 'title' + options[:mdn_tag] = 'events' + options[:root_title] = 'DOM Events' options[:skip] = %w(/MozOrientation) diff --git a/lib/docs/scrapers/mdn/html.rb b/lib/docs/scrapers/mdn/html.rb index 4b28cefd..f38432f1 100644 --- a/lib/docs/scrapers/mdn/html.rb +++ b/lib/docs/scrapers/mdn/html.rb @@ -7,6 +7,8 @@ module Docs html_filters.push 'html/clean_html', 'html/entries', 'title' + options[:mdn_tag] = 'HTML' + options[:root_title] = 'HTML' options[:title] = ->(filter) do diff --git a/lib/docs/scrapers/mdn/javascript.rb b/lib/docs/scrapers/mdn/javascript.rb index 935df61c..cea55fc8 100644 --- a/lib/docs/scrapers/mdn/javascript.rb +++ b/lib/docs/scrapers/mdn/javascript.rb @@ -8,6 +8,8 @@ module Docs html_filters.push 'javascript/clean_html', 'javascript/entries', 'title' + options[:mdn_tag] = 'JavaScript' + options[:root_title] = 'JavaScript' # Don't want diff --git a/lib/docs/scrapers/mdn/mdn.rb b/lib/docs/scrapers/mdn/mdn.rb index 2ebd38aa..defb4533 100644 --- a/lib/docs/scrapers/mdn/mdn.rb +++ b/lib/docs/scrapers/mdn/mdn.rb @@ -21,6 +21,11 @@ module Docs Licensed under the Creative Commons Attribution-ShareAlike License v2.5 or later. HTML + def get_latest_version(opts) + json = fetch_json("https://developer.mozilla.org/en-US/docs/feeds/json/tag/#{options[:mdn_tag]}", opts) + DateTime.parse(json[0]['pubdate']).to_time.to_i + end + private def process_response?(response) diff --git a/lib/docs/scrapers/mdn/svg.rb b/lib/docs/scrapers/mdn/svg.rb index db9de7a1..66baf60d 100644 --- a/lib/docs/scrapers/mdn/svg.rb +++ b/lib/docs/scrapers/mdn/svg.rb @@ -8,6 +8,8 @@ module Docs html_filters.push 'svg/clean_html', 'svg/entries', 'title' + options[:mdn_tag] = 'XSLT_Reference' + options[:root_title] = 'SVG' options[:title] = ->(filter) do diff --git a/lib/docs/scrapers/mdn/xslt_xpath.rb b/lib/docs/scrapers/mdn/xslt_xpath.rb index 5d812dd4..9bf01c01 100644 --- a/lib/docs/scrapers/mdn/xslt_xpath.rb +++ b/lib/docs/scrapers/mdn/xslt_xpath.rb @@ -8,6 +8,8 @@ module Docs html_filters.push 'xslt_xpath/clean_html', 'xslt_xpath/entries', 'title' + options[:mdn_tag] = 'XSLT_Reference' + options[:root_title] = 'XSLT' options[:only_patterns] = [/\A\/XSLT/, /\A\/XPath/] diff --git a/lib/docs/scrapers/meteor.rb b/lib/docs/scrapers/meteor.rb index b38d5dc2..a758d154 100644 --- a/lib/docs/scrapers/meteor.rb +++ b/lib/docs/scrapers/meteor.rb @@ -45,5 +45,10 @@ module Docs self.base_urls = ['https://guide.meteor.com/v1.3/', "https://docs.meteor.com/v#{self.release}/"] options[:fix_urls] = nil end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.meteor.com/#/full/', opts) + doc.at_css('select.version-select > option').content + end end end diff --git a/lib/docs/scrapers/mocha.rb b/lib/docs/scrapers/mocha.rb index 8ab9bdc8..04945425 100644 --- a/lib/docs/scrapers/mocha.rb +++ b/lib/docs/scrapers/mocha.rb @@ -18,5 +18,9 @@ module Docs © 2011–2018 JS Foundation and contributors
Licensed under the Creative Commons Attribution 4.0 International License. HTML + + def get_latest_version(opts) + get_npm_version('mocha', opts) + end end end diff --git a/lib/docs/scrapers/modernizr.rb b/lib/docs/scrapers/modernizr.rb index 96c82153..01ad49a7 100644 --- a/lib/docs/scrapers/modernizr.rb +++ b/lib/docs/scrapers/modernizr.rb @@ -15,5 +15,9 @@ module Docs © 2009–2017 The Modernizr team
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('modernizr', opts) + end end end diff --git a/lib/docs/scrapers/moment.rb b/lib/docs/scrapers/moment.rb index 88df0d14..5b7491ea 100644 --- a/lib/docs/scrapers/moment.rb +++ b/lib/docs/scrapers/moment.rb @@ -22,5 +22,10 @@ module Docs © JS Foundation and other contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('http://momentjs.com/', opts) + doc.at_css('.hero-title > h1 > span').content + end end end diff --git a/lib/docs/scrapers/mongoose.rb b/lib/docs/scrapers/mongoose.rb index 71ee04d2..fbd4ca92 100644 --- a/lib/docs/scrapers/mongoose.rb +++ b/lib/docs/scrapers/mongoose.rb @@ -26,5 +26,11 @@ module Docs © 2010 LearnBoost
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://mongoosejs.com/docs/', opts) + label = doc.at_css('.pure-menu-link').content.strip + label.sub(/Version /, '') + end end end diff --git a/lib/docs/scrapers/nginx.rb b/lib/docs/scrapers/nginx.rb index 5a3dbf1e..5354b8bd 100644 --- a/lib/docs/scrapers/nginx.rb +++ b/lib/docs/scrapers/nginx.rb @@ -25,5 +25,11 @@ module Docs © 2011-2018 Nginx, Inc.
Licensed under the BSD License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://nginx.org/en/download.html', opts) + table = doc.at_css('#content > table').inner_html + table.scan(/nginx-([0-9.]+) Licensed under the BSD License. HTML + + def get_latest_version(opts) + tags = get_github_tags('openresty', 'lua-nginx-module', opts) + tag = tags.find {|tag| !tag['name'].include?('rc')} + tag['name'][1..-1] + end end end diff --git a/lib/docs/scrapers/nim.rb b/lib/docs/scrapers/nim.rb index cce16f1c..a927605d 100644 --- a/lib/docs/scrapers/nim.rb +++ b/lib/docs/scrapers/nim.rb @@ -17,5 +17,10 @@ module Docs © 2006–2018 Andreas Rumpf
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://nim-lang.org/docs/overview.html', opts) + doc.at_css('.container > .docinfo > tbody > tr:last-child > td').content.strip + end end end diff --git a/lib/docs/scrapers/node.rb b/lib/docs/scrapers/node.rb index 66c6a456..0e5ee8fa 100644 --- a/lib/docs/scrapers/node.rb +++ b/lib/docs/scrapers/node.rb @@ -46,5 +46,10 @@ module Docs self.release = '4.9.1' self.base_url = 'https://nodejs.org/dist/latest-v4.x/docs/api/' end + + def get_latest_version(opts) + doc = fetch_doc('https://nodejs.org/en/', opts) + doc.at_css('#home-intro > .home-downloadblock:last-of-type > a')['data-version'][1..-1] + end end end diff --git a/lib/docs/scrapers/nokogiri2.rb b/lib/docs/scrapers/nokogiri2.rb index 084ad2fb..7c28ca92 100644 --- a/lib/docs/scrapers/nokogiri2.rb +++ b/lib/docs/scrapers/nokogiri2.rb @@ -19,5 +19,9 @@ module Docs Patrick Mahoney, Yoko Harada, Akinori Musha, John Shahid, Lars Kanis
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_latest_github_release('sparklemotion', 'nokogiri', opts) + end end end diff --git a/lib/docs/scrapers/npm.rb b/lib/docs/scrapers/npm.rb index 2c854e6e..3f868a3c 100644 --- a/lib/docs/scrapers/npm.rb +++ b/lib/docs/scrapers/npm.rb @@ -29,5 +29,9 @@ module Docs Licensed under the npm License.
npm is a trademark of npm, Inc. HTML + + def get_latest_version(opts) + get_latest_github_release('npm', 'cli', opts) + end end end diff --git a/lib/docs/scrapers/numpy.rb b/lib/docs/scrapers/numpy.rb index 1327fc02..84de6cab 100644 --- a/lib/docs/scrapers/numpy.rb +++ b/lib/docs/scrapers/numpy.rb @@ -49,5 +49,9 @@ module Docs self.release = '1.10.4' self.base_url = "https://docs.scipy.org/doc/numpy-#{self.release}/reference/" end + + def get_latest_version(opts) + get_latest_github_release('numpy', 'numpy', opts) + end end end diff --git a/lib/docs/scrapers/openjdk.rb b/lib/docs/scrapers/openjdk.rb index 944ac416..c26bce4c 100644 --- a/lib/docs/scrapers/openjdk.rb +++ b/lib/docs/scrapers/openjdk.rb @@ -86,5 +86,25 @@ module Docs def read_file(path) File.read(path).force_encoding('iso-8859-1').encode('utf-8') rescue nil end + + def get_latest_version(opts) + latest_version = 8 + current_attempt = latest_version + attempts = 0 + + while attempts < 3 + current_attempt += 1 + + doc = fetch_doc("https://packages.debian.org/sid/openjdk-#{current_attempt}-doc", opts) + if doc.at_css('.perror').nil? + latest_version = current_attempt + attempts = 0 + else + attempts += 1 + end + end + + latest_version + end end end diff --git a/lib/docs/scrapers/opentsdb.rb b/lib/docs/scrapers/opentsdb.rb index 0372e331..6eec407c 100644 --- a/lib/docs/scrapers/opentsdb.rb +++ b/lib/docs/scrapers/opentsdb.rb @@ -18,5 +18,9 @@ module Docs © 2010–2016 The OpenTSDB Authors
Licensed under the GNU LGPLv2.1+ and GPLv3+ licenses. HTML + + def get_latest_version(opts) + get_latest_github_release('OpenTSDB', 'opentsdb', opts) + end end end diff --git a/lib/docs/scrapers/padrino.rb b/lib/docs/scrapers/padrino.rb index 218f1731..d34b7db5 100644 --- a/lib/docs/scrapers/padrino.rb +++ b/lib/docs/scrapers/padrino.rb @@ -23,5 +23,9 @@ module Docs stub 'index2' do request_one(url_for('index')).body end + + def get_latest_version(opts) + get_github_tags('padrino', 'padrino-framework', opts)[0]['name'] + end end end diff --git a/lib/docs/scrapers/pandas.rb b/lib/docs/scrapers/pandas.rb index 1355a012..a8c2ea51 100644 --- a/lib/docs/scrapers/pandas.rb +++ b/lib/docs/scrapers/pandas.rb @@ -49,5 +49,11 @@ module Docs self.release = '0.18.1' self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/" end + + def get_latest_version(opts) + doc = fetch_doc('http://pandas.pydata.org/pandas-docs/stable/', opts) + label = doc.at_css('.body > .section > p').content + label.scan(/Version: ([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/perl.rb b/lib/docs/scrapers/perl.rb index 142ceaa5..ebf0a653 100644 --- a/lib/docs/scrapers/perl.rb +++ b/lib/docs/scrapers/perl.rb @@ -43,5 +43,11 @@ module Docs self.release = '5.20.2' self.base_url = "https://perldoc.perl.org/#{self.release}/" end + + def get_latest_version(opts) + doc = fetch_doc('https://perldoc.perl.org/', opts) + header = doc.at_css('h2.h1').content + header.scan(/Perl ([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/phalcon.rb b/lib/docs/scrapers/phalcon.rb index c2cb9242..c6ca63f2 100644 --- a/lib/docs/scrapers/phalcon.rb +++ b/lib/docs/scrapers/phalcon.rb @@ -29,5 +29,10 @@ module Docs self.release = '2.0.13' self.base_url = 'https://docs.phalconphp.com/en/2.0.0/' end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.phalconphp.com/', opts) + doc.at_css('.header__lang.expand > div > ul > li > a').content + end end end diff --git a/lib/docs/scrapers/phaser.rb b/lib/docs/scrapers/phaser.rb index 5ae371c1..bd5f411c 100644 --- a/lib/docs/scrapers/phaser.rb +++ b/lib/docs/scrapers/phaser.rb @@ -25,5 +25,9 @@ module Docs © 2016 Richard Davey, Photon Storm Ltd.
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_latest_github_release('photonstorm', 'phaser', opts) + end end end diff --git a/lib/docs/scrapers/phoenix.rb b/lib/docs/scrapers/phoenix.rb index 26b0144f..2ad053e2 100644 --- a/lib/docs/scrapers/phoenix.rb +++ b/lib/docs/scrapers/phoenix.rb @@ -46,5 +46,10 @@ module Docs HTML end } + + def get_latest_version(opts) + doc = fetch_doc('https://hexdocs.pm/phoenix/Phoenix.html', opts) + doc.at_css('.sidebar-projectVersion').content.strip[1..-1] + end end end diff --git a/lib/docs/scrapers/php.rb b/lib/docs/scrapers/php.rb index d4a66b5b..181d8b67 100644 --- a/lib/docs/scrapers/php.rb +++ b/lib/docs/scrapers/php.rb @@ -66,5 +66,11 @@ module Docs © 1997–2018 The PHP Documentation Group
Licensed under the Creative Commons Attribution License v3.0 or later. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://secure.php.net/manual/en/doc.changelog.php', opts) + label = doc.at_css('tbody.gen-changelog > tr > td').content + label.split(',').last.strip + end end end diff --git a/lib/docs/scrapers/phpunit.rb b/lib/docs/scrapers/phpunit.rb index 665ca4c7..5f2cbfea 100644 --- a/lib/docs/scrapers/phpunit.rb +++ b/lib/docs/scrapers/phpunit.rb @@ -37,5 +37,11 @@ module Docs self.release = '4.8' self.base_url = "https://phpunit.de/manual/#{release}/en/" end + + def get_latest_version(opts) + doc = fetch_doc('https://phpunit.readthedocs.io/', opts) + label = doc.at_css('.rst-current-version').content.strip + label.scan(/v: ([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/postgresql.rb b/lib/docs/scrapers/postgresql.rb index 109685ba..cc7a85c8 100644 --- a/lib/docs/scrapers/postgresql.rb +++ b/lib/docs/scrapers/postgresql.rb @@ -80,5 +80,11 @@ module Docs html_filters.insert_before 'postgresql/extract_metadata', 'postgresql/normalize_class_names' end + + def get_latest_version(opts) + doc = fetch_doc('https://www.postgresql.org/docs/current/index.html', opts) + label = doc.at_css('#pgContentWrap h1.title').content + label.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/pug.rb b/lib/docs/scrapers/pug.rb index 76949529..79a1b0a8 100644 --- a/lib/docs/scrapers/pug.rb +++ b/lib/docs/scrapers/pug.rb @@ -18,6 +18,10 @@ module Docs Licensed under the MIT license. HTML + def get_latest_version(opts) + get_npm_version('pug', opts) + end + private def parse(response) # Hook here because Nokogori removes whitespace from textareas diff --git a/lib/docs/scrapers/puppeteer.rb b/lib/docs/scrapers/puppeteer.rb index 2e26c180..043380d5 100644 --- a/lib/docs/scrapers/puppeteer.rb +++ b/lib/docs/scrapers/puppeteer.rb @@ -14,5 +14,10 @@ module Docs © 2017 Google Inc
Licensed under the Apache License 2.0. HTML + + def get_latest_version(opts) + contents = get_github_file_contents('GoogleChrome', 'puppeteer', 'README.md', opts) + contents.scan(/\/v([0-9.]+)\/docs\/api\.md/)[0][0] + end end end diff --git a/lib/docs/scrapers/pygame.rb b/lib/docs/scrapers/pygame.rb index 9da3148d..94c3d508 100644 --- a/lib/docs/scrapers/pygame.rb +++ b/lib/docs/scrapers/pygame.rb @@ -2,6 +2,7 @@ module Docs class Pygame < UrlScraper self.type = 'simple' self.release = '1.9.4' + self.base_url = 'https://www.pygame.org/docs/' self.root_path = 'py-modindex.html' self.links = { home: 'https://www.pygame.org/', @@ -16,5 +17,9 @@ module Docs © Pygame Developpers.
Licensed under the GNU LGPL License version 2.1. HTML + + def get_latest_version(opts) + get_latest_github_release('pygame', 'pygame', opts) + end end end diff --git a/lib/docs/scrapers/python.rb b/lib/docs/scrapers/python.rb index aa4336d9..c7905591 100644 --- a/lib/docs/scrapers/python.rb +++ b/lib/docs/scrapers/python.rb @@ -50,5 +50,10 @@ module Docs html_filters.push 'python/entries_v2', 'sphinx/clean_html', 'python/clean_html' end + + def get_latest_version(opts) + doc = fetch_doc('https://docs.python.org/', opts) + doc.at_css('.version_switcher_placeholder').content + end end end diff --git a/lib/docs/scrapers/q.rb b/lib/docs/scrapers/q.rb index bb66c156..a4c449c0 100644 --- a/lib/docs/scrapers/q.rb +++ b/lib/docs/scrapers/q.rb @@ -19,5 +19,9 @@ module Docs © 2009–2017 Kristopher Michael Kowal
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('q', opts) + end end end diff --git a/lib/docs/scrapers/qt.rb b/lib/docs/scrapers/qt.rb index 412eca6d..a1098b20 100644 --- a/lib/docs/scrapers/qt.rb +++ b/lib/docs/scrapers/qt.rb @@ -117,5 +117,10 @@ module Docs self.release = '5.6' self.base_url = 'https://doc.qt.io/qt-5.6/' end + + def get_latest_version(opts) + doc = fetch_doc('https://doc.qt.io/qt-5/index.html', opts) + doc.at_css('.mainContent h1.title').content.sub(/Qt /, '') + end end end diff --git a/lib/docs/scrapers/ramda.rb b/lib/docs/scrapers/ramda.rb index 09ff46c8..0b86e365 100644 --- a/lib/docs/scrapers/ramda.rb +++ b/lib/docs/scrapers/ramda.rb @@ -15,6 +15,11 @@ module Docs © 2013–2016 Scott Sauyet and Michael Hurley
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://ramdajs.com/docs/', opts) + doc.at_css('.navbar-brand > .version').content[1..-1] + end end end diff --git a/lib/docs/scrapers/rdoc/minitest.rb b/lib/docs/scrapers/rdoc/minitest.rb index 761da1de..2a4249fc 100644 --- a/lib/docs/scrapers/rdoc/minitest.rb +++ b/lib/docs/scrapers/rdoc/minitest.rb @@ -21,5 +21,10 @@ module Docs © Ryan Davis, seattle.rb
Licensed under the MIT License. HTML + + def get_latest_version(opts) + contents = get_github_file_contents('seattlerb', 'minitest', 'History.rdoc', opts) + contents.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/rdoc/rails.rb b/lib/docs/scrapers/rdoc/rails.rb index 6bdce34a..0dec42a9 100644 --- a/lib/docs/scrapers/rdoc/rails.rb +++ b/lib/docs/scrapers/rdoc/rails.rb @@ -93,5 +93,9 @@ module Docs version '4.1' do self.release = '4.1.16' end + + def get_latest_version(opts) + get_latest_github_release('rails', 'rails', opts) + end end end diff --git a/lib/docs/scrapers/rdoc/ruby.rb b/lib/docs/scrapers/rdoc/ruby.rb index dd296765..bc064660 100644 --- a/lib/docs/scrapers/rdoc/ruby.rb +++ b/lib/docs/scrapers/rdoc/ruby.rb @@ -84,5 +84,16 @@ module Docs version '2.2' do self.release = '2.2.10' end + + def get_latest_version(opts) + tags = get_github_tags('ruby', 'ruby', opts) + tags.each do |tag| + version = tag['name'].gsub(/_/, '.')[1..-1] + + if !/^([0-9.]+)$/.match(version).nil? && version.count('.') == 2 + return version + end + end + end end end diff --git a/lib/docs/scrapers/react.rb b/lib/docs/scrapers/react.rb index 704f3189..3c41ea5a 100644 --- a/lib/docs/scrapers/react.rb +++ b/lib/docs/scrapers/react.rb @@ -30,5 +30,10 @@ module Docs © 2013–present Facebook Inc.
Licensed under the Creative Commons Attribution 4.0 International Public License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://reactjs.org/docs/getting-started.html', opts) + doc.at_css('a[href="/versions"]').content.strip[1..-1] + end end end diff --git a/lib/docs/scrapers/react_native.rb b/lib/docs/scrapers/react_native.rb index d1f94001..fe87e492 100644 --- a/lib/docs/scrapers/react_native.rb +++ b/lib/docs/scrapers/react_native.rb @@ -30,5 +30,10 @@ module Docs © 2015–2018 Facebook Inc.
Licensed under the Creative Commons Attribution 4.0 International Public License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://facebook.github.io/react-native/docs/getting-started.html', opts) + doc.at_css('header > a > h3').content + end end end diff --git a/lib/docs/scrapers/redis.rb b/lib/docs/scrapers/redis.rb index 201dbb47..81440a82 100644 --- a/lib/docs/scrapers/redis.rb +++ b/lib/docs/scrapers/redis.rb @@ -19,5 +19,11 @@ module Docs © 2009–2018 Salvatore Sanfilippo
Licensed under the Creative Commons Attribution-ShareAlike License 4.0. HTML + + def get_latest_version(opts) + body = fetch('http://download.redis.io/redis-stable/00-RELEASENOTES', opts) + body = body.lines[1..-1].join + body.scan(/Redis ([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/redux.rb b/lib/docs/scrapers/redux.rb index cd7369ba..14f8e8b4 100644 --- a/lib/docs/scrapers/redux.rb +++ b/lib/docs/scrapers/redux.rb @@ -20,5 +20,9 @@ module Docs stub '' do request_one('http://redux.js.org/index.html').body end + + def get_latest_version(opts) + get_npm_version('redux', opts) + end end end diff --git a/lib/docs/scrapers/relay.rb b/lib/docs/scrapers/relay.rb index a020e9d7..807d6e1d 100644 --- a/lib/docs/scrapers/relay.rb +++ b/lib/docs/scrapers/relay.rb @@ -18,5 +18,10 @@ module Docs © 2013–present Facebook Inc.
Licensed under the BSD License. HTML + + def get_latest_version(opts) + doc = fetch_doc('http://facebook.github.io/relay/en/', opts) + doc.at_css('header > a > h3').content[1..-1] + end end end diff --git a/lib/docs/scrapers/requirejs.rb b/lib/docs/scrapers/requirejs.rb index 200f1f2d..80ecc722 100644 --- a/lib/docs/scrapers/requirejs.rb +++ b/lib/docs/scrapers/requirejs.rb @@ -30,5 +30,9 @@ module Docs © jQuery Foundation and other contributors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('requirejs', opts) + end end end diff --git a/lib/docs/scrapers/rethinkdb.rb b/lib/docs/scrapers/rethinkdb.rb index 023bbe77..35d8d334 100644 --- a/lib/docs/scrapers/rethinkdb.rb +++ b/lib/docs/scrapers/rethinkdb.rb @@ -58,6 +58,10 @@ module Docs CODE end + def get_latest_version(opts) + get_latest_github_release('rethinkdb', 'rethinkdb', opts) + end + private def process_response?(response) diff --git a/lib/docs/scrapers/rust.rb b/lib/docs/scrapers/rust.rb index 1677e138..635afc1c 100644 --- a/lib/docs/scrapers/rust.rb +++ b/lib/docs/scrapers/rust.rb @@ -39,6 +39,12 @@ module Docs Licensed under the Apache License, Version 2.0 or the MIT license, at your option. HTML + def get_latest_version(opts) + doc = fetch_doc('https://www.rust-lang.org/', opts) + label = doc.at_css('.button-download + p > a').content + label.sub(/Version /, '') + end + private REDIRECT_RGX = /http-equiv="refresh"/i diff --git a/lib/docs/scrapers/sass.rb b/lib/docs/scrapers/sass.rb index 244a88e7..c9774cdf 100644 --- a/lib/docs/scrapers/sass.rb +++ b/lib/docs/scrapers/sass.rb @@ -23,5 +23,9 @@ module Docs © 2006–2016 Hampton Catlin, Nathan Weizenbaum, and Chris Eppstein
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_latest_github_release('sass', 'libsass', opts) + end end end diff --git a/lib/docs/scrapers/scikit_image.rb b/lib/docs/scrapers/scikit_image.rb index 2a60aaec..a5959581 100644 --- a/lib/docs/scrapers/scikit_image.rb +++ b/lib/docs/scrapers/scikit_image.rb @@ -20,5 +20,10 @@ module Docs © 2011 the scikit-image team
Licensed under the BSD 3-clause License. HTML + + def get_latest_version(opts) + tags = get_github_tags('scikit-image', 'scikit-image', opts) + tags[0]['name'][1..-1] + end end end diff --git a/lib/docs/scrapers/scikit_learn.rb b/lib/docs/scrapers/scikit_learn.rb index fe6a8665..973c9d7e 100644 --- a/lib/docs/scrapers/scikit_learn.rb +++ b/lib/docs/scrapers/scikit_learn.rb @@ -25,5 +25,9 @@ module Docs Licensed under the 3-clause BSD License. HTML + def get_latest_version(opts) + doc = fetch_doc('https://scikit-learn.org/stable/documentation.html', opts) + doc.at_css('.body h1').content.scan(/([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/sinon.rb b/lib/docs/scrapers/sinon.rb index 9e81cba9..632348dc 100644 --- a/lib/docs/scrapers/sinon.rb +++ b/lib/docs/scrapers/sinon.rb @@ -52,5 +52,10 @@ module Docs self.release = '1.17.7' self.base_url = "https://sinonjs.org/releases/v#{release}/" end + + def get_latest_version(opts) + body = fetch('https://sinonjs.org/', opts) + body.scan(/\/releases\/v([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/socketio.rb b/lib/docs/scrapers/socketio.rb index 427098c4..e9bc264a 100644 --- a/lib/docs/scrapers/socketio.rb +++ b/lib/docs/scrapers/socketio.rb @@ -20,5 +20,9 @@ module Docs © 2014–2015 Automattic
Licensed under the MIT License. HTML + + def get_latest_version(opts) + get_npm_version('socket.io', opts) + end end end diff --git a/lib/docs/scrapers/sqlite.rb b/lib/docs/scrapers/sqlite.rb index 790acf83..d3a5fa6b 100644 --- a/lib/docs/scrapers/sqlite.rb +++ b/lib/docs/scrapers/sqlite.rb @@ -41,6 +41,11 @@ module Docs options[:attribution] = 'SQLite is in the Public Domain.' + def get_latest_version(opts) + doc = fetch_doc('https://sqlite.org/chronology.html', opts) + doc.at_css('#chrontab > tbody > tr > td:last-child > a').content + end + private def parse(response) diff --git a/lib/docs/scrapers/statsmodels.rb b/lib/docs/scrapers/statsmodels.rb index 255bde6c..3e8a357c 100644 --- a/lib/docs/scrapers/statsmodels.rb +++ b/lib/docs/scrapers/statsmodels.rb @@ -21,5 +21,9 @@ module Docs Licensed under the 3-clause BSD License. HTML + def get_latest_version(opts) + doc = fetch_doc('http://www.statsmodels.org/stable/', opts) + doc.at_css('.sphinxsidebarwrapper h3 + p > b').content[1..-1] + end end end diff --git a/lib/docs/scrapers/support_tables.rb b/lib/docs/scrapers/support_tables.rb index 9a550c1e..d04072da 100644 --- a/lib/docs/scrapers/support_tables.rb +++ b/lib/docs/scrapers/support_tables.rb @@ -178,5 +178,11 @@ module Docs

HTML + + def get_latest_version(opts) + body = fetch('https://feeds.feedburner.com/WhenCanIUse?format=xml', opts) + timestamp = body.scan(/([^<]+)<\/updated>/)[0][0] + DateTime.parse(timestamp).to_time.to_i + end end end diff --git a/lib/docs/scrapers/symfony.rb b/lib/docs/scrapers/symfony.rb index f5e4285a..4bdde054 100644 --- a/lib/docs/scrapers/symfony.rb +++ b/lib/docs/scrapers/symfony.rb @@ -70,5 +70,9 @@ module Docs self.release = '2.7.35' self.base_url = "https://api.symfony.com/#{version}/" end + + def get_latest_version(opts) + get_latest_github_release('symfony', 'symfony', opts) + end end end diff --git a/lib/docs/scrapers/tcl_tk.rb b/lib/docs/scrapers/tcl_tk.rb index 1e4b2567..bca840c6 100644 --- a/lib/docs/scrapers/tcl_tk.rb +++ b/lib/docs/scrapers/tcl_tk.rb @@ -25,5 +25,10 @@ module Docs options[:attribution] = <<-HTML Licensed under Tcl/Tk terms HTML + + def get_latest_version(opts) + doc = fetch_doc('https://www.tcl.tk/man/tcl/contents.htm', opts) + doc.at_css('h2').content.scan(/Tk([0-9.]+)/)[0][0] + end end end diff --git a/lib/docs/scrapers/tensorflow.rb b/lib/docs/scrapers/tensorflow.rb index 0296299f..9638b516 100644 --- a/lib/docs/scrapers/tensorflow.rb +++ b/lib/docs/scrapers/tensorflow.rb @@ -56,6 +56,10 @@ module Docs /\Aextend/] end + def get_latest_version(opts) + get_latest_github_release('tensorflow', 'tensorflow', opts) + end + private def parse(response) diff --git a/lib/docs/scrapers/terraform.rb b/lib/docs/scrapers/terraform.rb index ab12775d..0965ad06 100644 --- a/lib/docs/scrapers/terraform.rb +++ b/lib/docs/scrapers/terraform.rb @@ -18,5 +18,10 @@ module Docs © 2018 HashiCorp
Licensed under the MPL 2.0 License. HTML + + def get_latest_version(opts) + contents = get_github_file_contents('hashicorp', 'terraform-website', 'content/config.rb', opts) + contents.scan(/version\s+=\s+"([0-9.]+)"/)[0][0] + end end end diff --git a/lib/docs/scrapers/twig.rb b/lib/docs/scrapers/twig.rb index 4781dcb2..e52ff6bd 100755 --- a/lib/docs/scrapers/twig.rb +++ b/lib/docs/scrapers/twig.rb @@ -28,5 +28,10 @@ module Docs self.release = '1.34.3' self.base_url = 'https://twig.symfony.com/doc/1.x/' end + + def get_latest_version(opts) + tags = get_github_tags('twigphp', 'Twig', opts) + tags[0]['name'][1..-1] + end end end diff --git a/lib/docs/scrapers/typescript.rb b/lib/docs/scrapers/typescript.rb index 1117314a..777f5dd6 100644 --- a/lib/docs/scrapers/typescript.rb +++ b/lib/docs/scrapers/typescript.rb @@ -24,6 +24,10 @@ module Docs © Microsoft and other contributors
Licensed under the Apache License, Version 2.0. HTML + + def get_latest_version(opts) + get_latest_github_release('Microsoft', 'TypeScript', opts) + end end end diff --git a/lib/docs/scrapers/underscore.rb b/lib/docs/scrapers/underscore.rb index 004a09e9..a09b4acb 100644 --- a/lib/docs/scrapers/underscore.rb +++ b/lib/docs/scrapers/underscore.rb @@ -20,5 +20,10 @@ module Docs © 2009–2018 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
Licensed under the MIT License. HTML + + def get_latest_version(opts) + doc = fetch_doc('https://underscorejs.org/', opts) + doc.at_css('.version').content[1...-1] + end end end diff --git a/lib/docs/scrapers/vagrant.rb b/lib/docs/scrapers/vagrant.rb index 955746aa..7400fc8c 100644 --- a/lib/docs/scrapers/vagrant.rb +++ b/lib/docs/scrapers/vagrant.rb @@ -18,5 +18,10 @@ module Docs © 2010–2018 Mitchell Hashimoto
Licensed under the MPL 2.0 License. HTML + + def get_latest_version(opts) + contents = get_github_file_contents('hashicorp', 'vagrant', 'website/config.rb', opts) + contents.scan(/version\s+=\s+"([0-9.]+)"/)[0][0] + end end end diff --git a/lib/docs/scrapers/vue.rb b/lib/docs/scrapers/vue.rb index f92991f2..333ade1d 100644 --- a/lib/docs/scrapers/vue.rb +++ b/lib/docs/scrapers/vue.rb @@ -32,5 +32,9 @@ module Docs self.root_path = '/guide/index.html' self.initial_paths = %w(/api/index.html) end + + def get_latest_version(opts) + get_latest_github_release('vuejs', 'vue', opts) + end end end diff --git a/lib/docs/scrapers/vulkan.rb b/lib/docs/scrapers/vulkan.rb index e1fa4b4b..da5d696d 100644 --- a/lib/docs/scrapers/vulkan.rb +++ b/lib/docs/scrapers/vulkan.rb @@ -20,5 +20,10 @@ module Docs Licensed under the Creative Commons Attribution 4.0 International License.
Vulkan and the Vulkan logo are registered trademarks of the Khronos Group Inc. HTML + + def get_latest_version(opts) + tags = get_github_tags('KhronosGroup', 'Vulkan-Docs', opts) + tags[0]['name'][1..-1] + end end end diff --git a/lib/docs/scrapers/webpack.rb b/lib/docs/scrapers/webpack.rb index 255c1bb4..7af2cf11 100644 --- a/lib/docs/scrapers/webpack.rb +++ b/lib/docs/scrapers/webpack.rb @@ -68,5 +68,9 @@ module Docs Licensed under the MIT License. HTML end + + def get_latest_version(opts) + get_npm_version('webpack', opts) + end end end diff --git a/lib/docs/scrapers/yarn.rb b/lib/docs/scrapers/yarn.rb index 825a749f..c45695af 100644 --- a/lib/docs/scrapers/yarn.rb +++ b/lib/docs/scrapers/yarn.rb @@ -20,5 +20,9 @@ module Docs © 2016–present Yarn Contributors
Licensed under the BSD License. HTML + + def get_latest_version(opts) + get_latest_github_release('yarnpkg', 'yarn', opts) + end end end diff --git a/lib/docs/scrapers/yii.rb b/lib/docs/scrapers/yii.rb index d99d3ac7..eaabdbf0 100755 --- a/lib/docs/scrapers/yii.rb +++ b/lib/docs/scrapers/yii.rb @@ -34,5 +34,9 @@ module Docs options[:container] = '.grid_9' end + + def get_latest_version(opts) + get_latest_github_release('yiisoft', 'yii2', opts) + end end end diff --git a/lib/tasks/updates.thor b/lib/tasks/updates.thor new file mode 100644 index 00000000..c715f752 --- /dev/null +++ b/lib/tasks/updates.thor @@ -0,0 +1,315 @@ +class UpdatesCLI < Thor + # The GitHub user that is allowed to upload reports + UPLOAD_USER = 'devdocs-bot' + + # The repository to create an issue in when uploading the results + UPLOAD_REPO = 'freeCodeCamp/devdocs' + + def self.to_s + 'Updates' + end + + def initialize(*args) + require 'docs' + require 'progress_bar' + require 'terminal-table' + require 'date' + super + end + + desc 'check [--github-token] [--upload] [--verbose] [doc]...', 'Check for outdated documentations' + option :github_token, :type => :string + option :upload, :type => :boolean + option :verbose, :type => :boolean + def check(*names) + # Convert names to a list of Scraper instances + # Versions are omitted, if v10 is outdated than v8 is aswell + docs = names.map {|name| Docs.find(name.split(/@|~/)[0], false)}.uniq + + # Check all documentations for updates when no arguments are given + docs = Docs.all if docs.empty? + + opts = { + logger: logger + } + + if options.key?(:github_token) + opts[:github_token] = options[:github_token] + end + + with_progress_bar do |bar| + bar.max = docs.length + bar.write + end + + results = docs.map do |doc| + result = check_doc(doc, opts) + with_progress_bar(&:increment!) + result + end + + process_results(results) + rescue Docs::DocNotFound => error + logger.error(error) + logger.info('Run "thor docs:list" to see the list of docs.') + end + + private + + def check_doc(doc, opts) + logger.debug("Checking #{doc.name}") + + instance = doc.versions.first.new + scraper_version = instance.get_scraper_version(opts) + latest_version = instance.get_latest_version(opts) + + { + name: doc.name, + scraper_version: format_version(scraper_version), + latest_version: format_version(latest_version), + is_outdated: instance.is_outdated(scraper_version, latest_version) + } + rescue NotImplementedError + logger.warn("Couldn't check #{doc.name}, get_latest_version is not implemented") + error_result(doc, '`get_latest_version` is not implemented') + rescue => error + logger.error("Error while checking #{doc.name}\n#{error.full_message.strip}") + error_result(doc, error.message.gsub(/'/, '`')) + end + + def format_version(version) + str = version.to_s + + # If the version is numeric and greater than or equal to 1e9 it's probably a timestamp + return str if str.match(/^(\d)+$/).nil? or str.to_i < 1e9 + + DateTime.strptime(str, '%s').strftime('%F') + end + + def error_result(doc, reason) + { + name: doc.name, + error: reason + } + end + + def process_results(results) + successful_results = results.select {|result| result.key?(:is_outdated)} + failed_results = results.select {|result| result.key?(:error)} + + up_to_date_results = successful_results.select {|result| !result[:is_outdated]} + outdated_results = successful_results.select {|result| result[:is_outdated]} + + log_results(outdated_results, up_to_date_results, failed_results) + upload_results(outdated_results, up_to_date_results, failed_results) if options[:upload] + end + + # + # Result logging methods + # + + def log_results(outdated_results, up_to_date_results, failed_results) + log_failed_results(failed_results) unless failed_results.empty? + log_successful_results('Up-to-date', up_to_date_results) unless up_to_date_results.empty? + log_successful_results('Outdated', outdated_results) unless outdated_results.empty? + end + + def log_successful_results(label, results) + title = "#{label} documentations (#{results.length})" + headings = ['Documentation', 'Scraper version', 'Latest version'] + rows = results.map {|result| [result[:name], result[:scraper_version], result[:latest_version]]} + + table = Terminal::Table.new :title => title, :headings => headings, :rows => rows + puts table + end + + def log_failed_results(results) + title = "Documentations that could not be checked (#{results.length})" + headings = %w(Documentation Reason) + rows = results.map {|result| [result[:name], result[:error]]} + + table = Terminal::Table.new :title => title, :headings => headings, :rows => rows + puts table + end + + # + # Upload methods + # + + def upload_results(outdated_results, up_to_date_results, failed_results) + # We can't create issues without a GitHub token + unless options.key?(:github_token) + logger.error("Please specify a GitHub token with the public_repo permission for #{UPLOAD_USER} with the --github-token parameter") + return + end + + logger.info('Uploading the results to a new GitHub issue') + + logger.info('Checking if the GitHub token belongs to the correct user') + user = github_get('/user') + + # Only allow the DevDocs bot to upload reports + unless user['login'] == UPLOAD_USER + logger.error("Only #{UPLOAD_USER} is supposed to upload the results to a new issue. The specified github token is not for #{UPLOAD_USER}.") + return + end + + logger.info('Creating a new GitHub issue') + + issue = results_to_issue(outdated_results, up_to_date_results, failed_results) + created_issue = github_post("/repos/#{UPLOAD_REPO}/issues", issue) + + logger.info('Checking if the previous issue is still open') + + search_params = { + q: "Documentation versions report in:title author:#{UPLOAD_USER} is:issue repo:#{UPLOAD_REPO}", + sort: 'created', + order: 'desc' + } + + matching_issues = github_get('/search/issues', search_params) + previous_issue = matching_issues['items'].find {|item| item['number'] != created_issue['number']} + + if previous_issue.nil? + logger.info('No previous issue found') + log_upload_success(created_issue) + else + logger.info('Commenting on the previous issue') + + comment = "This report was superseded by ##{created_issue['number']}." + github_post("/repos/#{UPLOAD_REPO}/issues/#{previous_issue['number']}/comments", {body: comment}) + if previous_issue['closed_at'].nil? + logger.info('Closing the previous issue') + github_patch("/repos/#{UPLOAD_REPO}/issues/#{previous_issue['number']}", {state: 'closed'}) + log_upload_success(created_issue) + else + logger.info('The previous issue has already been closed') + log_upload_success(created_issue) + end + end + end + + def results_to_issue(outdated_results, up_to_date_results, failed_results) + results = [ + successful_results_to_markdown('Outdated', outdated_results), + successful_results_to_markdown('Up-to-date', up_to_date_results), + failed_results_to_markdown(failed_results) + ] + + results_str = results.select {|result| !result.nil?}.join("\n\n") + travis_str = ENV['TRAVIS'].nil? ? '' : "\n\nThis issue was created by Travis CI build [##{ENV['TRAVIS_BUILD_NUMBER']}](#{ENV['TRAVIS_BUILD_WEB_URL']})." + + title = "Documentation versions report for #{Date.today.strftime('%B %Y')}" + body = <<-MARKDOWN +## What is this? + +This is an automatically created issue which contains information about the version status of the documentations available on DevDocs. The results of this report can be used by maintainers when updating outdated documentations. + +Maintainers can close this issue when all documentations are up-to-date. The issue is also automatically closed when the next report is created.#{travis_str} + +## Results + +The #{outdated_results.length + up_to_date_results.length + failed_results.length} documentations are divided as follows: +- #{outdated_results.length} that #{outdated_results.length == 1 ? 'is' : 'are'} outdated +- #{up_to_date_results.length} that #{up_to_date_results.length == 1 ? 'is' : 'are'} up-to-date (patch updates are ignored) +- #{failed_results.length} that could not be checked + MARKDOWN + + { + title: title, + body: body.strip + "\n\n" + results_str + } + end + + def successful_results_to_markdown(label, results) + return nil if results.empty? + + title = "#{label} documentations (#{results.length})" + headings = ['Documentation', 'Scraper version', 'Latest version'] + rows = results.map {|result| [result[:name], result[:scraper_version], result[:latest_version]]} + + results_to_markdown(title, headings, rows) + end + + def failed_results_to_markdown(results) + return nil if results.empty? + + title = "Documentations that could not be checked (#{results.length})" + headings = %w(Documentation Reason) + rows = results.map {|result| [result[:name], result[:error]]} + + results_to_markdown(title, headings, rows) + end + + def results_to_markdown(title, headings, rows) + "
\n#{title}\n\n#{create_markdown_table(headings, rows)}\n
" + end + + def create_markdown_table(headings, rows) + header = headings.join(' | ') + separator = '-|' * headings.length + body = rows.map {|row| row.join(' | ')} + + header + "\n" + separator[0...-1] + "\n" + body.join("\n") + end + + def log_upload_success(created_issue) + logger.info("Successfully uploaded the results to #{created_issue['html_url']}") + end + + # + # HTTP utilities + # + + def github_get(endpoint, params = {}) + github_request(endpoint, {method: :get, params: params}) + end + + def github_post(endpoint, params) + github_request(endpoint, {method: :post, body: params.to_json}) + end + + def github_patch(endpoint, params) + github_request(endpoint, {method: :patch, body: params.to_json}) + end + + def github_request(endpoint, opts) + url = "https://api.github.com#{endpoint}" + + # GitHub token authentication + opts[:headers] = { + Authorization: "token #{options[:github_token]}" + } + + # GitHub requires the Content-Type to be application/json when a body is passed + if opts.key?(:body) + opts[:headers]['Content-Type'] = 'application/json' + end + + logger.debug("Making a #{opts[:method]} request to #{url}") + response = Docs::Request.run(url, opts) + + # response.success? is false if the response code is 201 + # GitHub returns 201 Created after an issue is created + if response.success? || response.code == 201 + JSON.parse(response.body) + else + logger.error("Couldn't make a #{opts[:method]} request to #{url} (response code #{response.code})") + nil + end + end + + # A utility method which ensures no progress bar is shown when stdout is not a tty + def with_progress_bar(&block) + return unless $stdout.tty? + @progress_bar ||= ::ProgressBar.new + block.call @progress_bar + end + + def logger + @logger ||= Logger.new($stdout).tap do |logger| + logger.level = options[:verbose] ? Logger::DEBUG : Logger::INFO + logger.formatter = proc {|severity, datetime, progname, msg| "[#{severity}] #{msg}\n"} + end + end +end