Validating outside the browser - command line tool

[nataliecarey]

Hi guys, I've released a command line tool for validating AMP pages. I'm working on an AMP implementation for a client and we've found the browser console validator useful for exploring but not so good for automation. I've published a tool to NPM https://www.npmjs.com/package/amp-validator which gives the validation results available on the command line. For example:

amp-validator examples/everything.amp.html

examples/everything.amp.html failed validation
 -- Line 174:2 MANDATORY_ATTR_MISSING video-id
 -- Line 178:2 MANDATORY_ATTR_MISSING video-id
 -- Line 343:2 DISALLOWED_TAG amp-font
 -- Line 1:0 MANDATORY_ATTR_MISSING A11Y: Element AMP-IMG#img0 must have "role" attribute due to "tap" action
 -- Line 1:0 MANDATORY_ATTR_MISSING A11Y: Element AMP-IMG#img0 must have "tabindex" attribute due to "tap" action.

It works with local files and remote URLs and you can call it with multiple URLs/Files at once. It exits with a success status if all pages validate and a failure status if any pages don't validate.

It can give a JSON output with the param --output=json - this gives various details that might be useful in a build step:

{
  "examples/everything.amp.html": {
    "errors": [
      {
        "line": 174,
        "char": 2,
        "reason": "MANDATORY_ATTR_MISSING video-id"
      },
      {
        "line": 178,
        "char": 2,
        "reason": "MANDATORY_ATTR_MISSING video-id"
      },
      {
        "line": 343,
        "char": 2,
        "reason": "DISALLOWED_TAG amp-font"
      },
      {
        "line": 1,
        "char": 0,
        "reason": "MANDATORY_ATTR_MISSING A11Y: Element AMP-IMG#img0 must have \"role\" attribute due to \"tap\" action"
      },
      {
        "line": 1,
        "char": 0,
        "reason": "MANDATORY_ATTR_MISSING A11Y: Element AMP-IMG#img0 must have \"tabindex\" attribute due to \"tap\" action."
      }
    ],
    "ampVersion": {
      "precise": 1447372082206,
      "releaseDate": "2015-11-12T23:48:02.206Z",
      "declared": "v0"
    },
    "success": false
  }
}

Next steps are:

1. Release an online version at http://ampvalidator.com/
2. Get feedback for command line and web versions
3. Discuss "provider-specific validation requirements"
4. Change headless browser? Currently this implementation uses Zombie JS which requires node-gyp, I think this might be an obstacle for some who would want to use it, it may benefit from a port to another language or use a browser which doesn't require gyp (phantom? something else?).

What do I mean by "provider-specific validation requirements"? My client cares about being in Google's AMP demo. Speaking with Google shows that there are additional requirements on top of a valid AMP document, I'd like to add --google as an opt-in to Google's particular requirements. I presume there are already other rules being decided by other companies, it would be good if teams can validate against all the requirements in one place - it may be wishful thinking but if we can start gathering the various provider-specific requirements that we know about it should help.

Feedback is welcome, feature requests can be discussed and bugs raised on the Github https://github.com/dorightdigital/amp-validator

I hope it's a tool you can benefit from.

[cramforce]

Very cool! We're in the process of open sourcing the validator and we were definitely planning to publish it on NPM. This would be JS only, so likely a bit faster and easier to install.

One nice thing about your solution is that it is guaranteed to be in-sync with the prod validator (since it is using it). It comes at a cost of spinning up a browser and talking to the internet.

Would be great to work together on that!

[cramforce]

Since it is super relevant: The core validator is now open source: #951

[nataliecarey]

Wonderful, I'll see if I can rid myself of the headless browser.

[cramforce]

@matcarey Your solution has one big benefit: It uses the live validator without needing to update from npm. It would be neat to keep an option that downloads the rules from the internet. Its a tradeoff but nice if people have the option.

[cramforce]

Basically our plan would be to push a minor release to NPM every time we push the web validator.

[nataliecarey]

True. I'll take a look at the core validator and see what the options are.

I suppose I'm treating this more like an integration test than a unit test - does my AMP page meet the current validation requirements of the AMP validator. That means it would fail if I've broken it or if a breaking change was (presumably accidentally) released.

I wonder whether it's helpful to opt into a particular SHA/tag for validation, or if feeling brave validate against HEAD. I could then clone or download the raw from github and validate against that.

At some stage we should take implementation options back to https://github.com/dorightdigital/amp-validator/issues for more fine grained discussion :)

[pietergreyling]

FYI, I submitted a related pull request: "Add new validator tools subfolder and doc for building a command-line validator on Mac OS X #1554": #1554

[ericlindley-g]

@pietergreyling — are there any additional items left, or does #1554 cover this issue?

[nataliecarey]

Looks like it does, I'll have a play to be sure. I'll close this one on the assumption it's all good and I'll comment elsewhere if I think something's missing.