SiteHealth

⚠️ Project is still experimental, API will change (a lot) without notice.

Crawl a site and check various health indicators, such as:

• Server errors
• HTTP errors
• Invalid HTML/XML/JSON
• Missing HTML title/description
• Missing image alt-attribute
• Google Pagespeed

Installation

Add this line to your application's Gemfile:

gem "site_health"

And then execute:

$ bundle

Or install it yourself as:

$ gem install site_health

Usage

CLI usage.

Crawl and check site

nurse = SiteHealth.check("https://example.com")

Check list of URLs

nurse = SiteHealth.check_urls(["https://example.com"])

Write raw JSON result to file

nurse = SiteHealth.check("https://example.com")
json = JSON.pretty_generate(nurse.journal)

File.write("result.json", json)

Each issue

SiteHealth.check_urls(urls) do |nurse|
  nurse.clerk do |clerk|
    clerk.every_issue { |issue| puts "#{issue.severity}, #{issue.title}" }
  end
end

Simple issue reports

nurse = SiteHealth.check("https://example.com")
report = SiteHealth::IssuesReport.new(nurse.issue) do |r|
  r.fields = %i[url title detail] # issue fields
  r.select { |issue| issue.url.include?('blog/') }
end

report.to_a
report.to_csv
report.to_json

Event handlers

urls = ["https://example.com"]
nurse = SiteHealth.check_urls(urls) do |nurse|
  nurse.clerk do |clerk|
    clerk.every_journal do |journal, page|
      time_in_seconds = journal[:runtime_in_seconds]
      puts "Found page #{page.title} - #{page.url} (checks took #{time_in_seconds})"
    end

    clerk.every_check do |check|
      puts "Ran check: #{check.name}"
    end

    clerk.every_failed_url do |url|
      puts "Failed to fetch: #{url}"
    end
  end
end

Write page speed summary CSV

nurse = SiteHealth.check("https://example.com")
summary = SiteHealth::PageSpeedSummarizer.new(nurse.journal)
File.write("page_size_summary.csv", summary.to_csv)

Configuration

All configuration is optional.

SiteHealth.configure do |config|
  # Override default checkers
  config.checkers = [:json_syntax, :html]

  # Configure logger
  config.logger = Logger.new(STDOUT).tap do |logger|
    logger.progname = 'SiteHealth'
    logger.level = Logger::INFO
  end

  # Configure HTMLProofer
  config.html_proofer do |proofer_config|
    proofer_config.log_level = :info
    proofer_config.check_opengraph = false
  end

  # Configure W3C HTML/CSS validator
  config.w3c_validators do |w3c_config|
    w3c_config.css_uri = 'http://localhost:8888/check'
    w3c_config.html_uri = 'http://localhost:8888/check'
  end
end

Load non-default checkers:

A few of the non-default checkers available in this gem require 3rd-party dependencies which aren't installed by default.

Checker name	Gem
google_page_speed	google-api-client
html_proofer	html-proofer
w3c_html	w3c_validators
w3c_css	w3c_validators

If you intend to use any of those checkers make sure to install the gem first. For example to use the google_page_speed checker add google-api-client to your Gemfile or install it manually with gem install google-api-client. Then you register the checker for use.

SiteHealth.config.register_checker :google_page_speed
# LoadError is raised if google-api-client is *not* installed

Add your own checker:

class ProfanityChecker < SiteHealth::Checker
  name "profanity"
  types %i[html json xml css javascript]

  def check
    add_data(profanity: {
      damn: page.body.include?(" damn "),
      shit: page.body.include?(" shit ")
    })
  end
end

# Then register it
SiteHealth.configure do |config|
  config.register_checker ProfanityChecker
end

CLI

Usage: site_health --help
        --url=val0
        --fields=priority,title,url  Issue fields to include - by default all fields are included
        --output=result.csv          Output path, .csv or .json
        --stats-output=stats.csv     Stats output path, .csv or .json
        --[no-]progress              Print progress while running to STDOUT
    -h, --help                       How to use

Development

After checking out the repo, run bin/setup to install dependencies. Then, run bundle exec rake to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/buren/site_health.

License

The gem is available as open source under the terms of the MIT License.

---

TODO

• Good way to render result/reports data
• Improve logger support
• Checkers
  • canonical URL
  • http vs https links
  • links matching a pattern