Support for saving requests to HAR file

[asg017]

Playwright has support for HAR files, which saves all network requests during a session to a custom file format.

The API is awkward but we can use route_from_har. We'll need to add update=True to save requests to the file, instead of "serveing" them.

There's a few more options to:

• update_content: "embed"|"attach". Maybe --har-content=embed?
• update_mode: "full"|"minimal". Maybe --har-mode=full?

[simonw]

https://playwright.dev/python/docs/release-notes#version-123 added this in June 2022 https://github.com/microsoft/playwright-python/releases/tag/v1.23.0

context = browser.new_context(record_har_path="github.har.zip")
# ... do stuff ...
context.close()

[simonw]

Got a prototype working! I had to edit the har.har JSON to get this to just show the entries:

https://lite.datasette.io/?install=datasette-pretty-json&json=https://gist.githubusercontent.com/simonw/0af14ac477dddbd0fb34a4334394cf4e/raw/cd5c4ada5affe88eacb1b25968f05ef30b993621/har.json#/data/har

[simonw]

I built this as a separate command:

shot-scraper har https://datasette.io/

That will save it to datasette-io.har.zip (renaming to avoid clobbering existing files) - you can use the -o/--output option to specify somewhere else to save it.

[simonw]

% shot-scraper har --help
Usage: shot-scraper har [OPTIONS] URL

  Record a HAR file for the specified page

  Usage:

      shot-scraper har https://datasette.io/

Options:
  -a, --auth FILENAME    Path to JSON authentication context file
  -o, --output FILE      HAR filename
  -j, --javascript TEXT  Execute this JS prior to taking the snapshot
  --timeout INTEGER      Wait this many milliseconds before failing
  --log-console          Write console.log() to stderr
  --fail                 Fail with an error code if a page returns an HTTP
                         error
  --skip                 Skip pages that return HTTP errors
  --bypass-csp           Bypass Content-Security-Policy
  --auth-password TEXT   Password for HTTP Basic authentication
  --auth-username TEXT   Username for HTTP Basic authentication
  --help                 Show this message and exit.

[simonw]

Is there an argument for adding this as an option to other commands such as shot-scraper javascript as well? Since this new command supports a -j/--javascript option for executing extra JavaScript I'm not sure that the other commands would benefit from a --record-har option, but maybe they would?

[simonw]

This option actually has no effect:

shot-scraper har https://datasette.io/ --javascript 'document.title="mess with the DOM first"'

Because the HAR has already been recorded by the time the JavaScript executes.

[simonw]

Having shot-scraper multi able to store HAR files would be neat.

[simonw]

Got Claude to write me a very neat pytest fixture for running a localhost web server: https://gist.github.com/simonw/360b520fdb82d48c669db575cf74b9f4

[simonw]

Blogged about this here: https://simonwillison.net/2025/Feb/13/shot-scraper/