automation-extra-interception-proxy

Simple way to play with a site requests and responses.

Tested in puppeteer/chromium only!

Breaking changes in v1:

Cause: major shift of focus from proxying to convenient api for interception api

removed enableLegacyCookieHandling
RequestMode.native is now default
removed default handlers in requestHandlers - you need to add it manually
removed options gotHooks, proxy, agent - moved to buildGotHttpHandler

Known issues in managed mode

Grease chips are missing.
Some headers are missing.
[CORS] OPTIONS requests (preflight requests) are missing before the actual request will be executed.
[CORS] Headers are close to being correct, but they're not.
WebSockets will be handled by the browser (IP leak may occur if you are using a proxy in the package but not in Puppeteer itself).
Optimization can be bad on high load.

Installing

Using npm

npm install automation-extra-interception-proxy

Using yarn

yarn add automation-extra-interception-proxy

Why?

This package solves next problems:

1. Traffic sniffing

Time to time required to reach information from the browser request. By default you can reach easily only to headers information. If you want to just read all responses you also can do that but time to time it will throw errors by one of next reasons.

At first page can be already closed and then your code will throw an error.

At second some sites using service workers for requesting some information. Unfortunately you cant handle this situation without manual requesting and then converting to puppeteer.

2. Data manipulation

If you want just adjust some requests or responses you should do that manually.

Example. You want get original request/response and do some adjustments. This package will help do that easily. You just getting what you want by single function call.

3. Set proxy

Yes, puppeteer already have a proxy support throw additional process arguments. But you should manually maintain proxy credentials each request(?, not sure). Also you cant use socks proxy(?, not sure).

4. Asynchronous decisions for requests

Even with cooperative mode you can not make your decisions asynchronously. Here you can chain of handlers with will proceed request decision one by one. Also you can say that this is latest decision and no need to ask another handlers in the chain. Also in one handler you can can adjust request/response for the next one.

Motivation

We live in the world where almost each website have internal api. When you are looking at the network tab in Chrome DevTools its easy to handle where and what. Data already yours but you cant just get what you want. But you have to fight for the information you desired for. So lets fight together!

API

wrapPage
- Parameters
IConfig
continue
ignoreResponseBodyIfPossible
flushLocal
- Parameters
recordError
- Parameters
recordInternalError
- Parameters
recordWarning
- Parameters
RequestMode
- ignore
- native
- managed
RequestStage
IRequestOptions
- method
- url
- headers
- body
_bodyError
IAbortReason
ConfigurableMixin
applyLoggableMixin
- Parameters
NetworkMixin
InterceptionProxyRequest
- Parameters
UserListener
_OriginalRequestStateManager
- Parameters

wrapPage

src/index.ts:22-24

Add interception ability to the page (sample)

Parameters

page Puppeteer.Page Page for future interceptions
config IConfig?

Returns Promise<InterceptionProxyPageConfig>

IConfig

src/interfaces/base.ts:37-85

Plugin configuration object

cooperativePriority

src/interfaces/base.ts:47-47

Puppeteer' "Cooperative Intercept Mode" priority

This package using own way to manage cooperation

Use only if you know what it does

[Read more]

Type: (undefined | number)

requestMode

src/interfaces/base.ts:57-57

ignore - Plugin will do nothing about original request

native - Plugin will just listen to the original request/response data and all requests will fulfilled by puppeteer itself. But some plugin functionality can be unavailable.

managed - Plugin will do all requests by requestHandlers or by himself. All plugin features will be available.

Default - managed

Type: RequestMode

logger

src/interfaces/base.ts:61-61

You can handle all plugins messages

Type: any

timeout

src/interfaces/base.ts:65-65

Request timeout in milliseconds(actual execution only)

Type: number

nativeContinueIfPossible

src/interfaces/base.ts:72-72

If you didn't changed request or response, let puppeteer handle this request by himself

Default: false

Type: boolean

ignoreResponseBodyIfPossible

src/interfaces/base.ts:82-82

If you did not use the plugin' response object it will not retrieve response from puppeteer for better performance

Applies for native mode only

Type: boolean

continue

src/interfaces/classes.ts:42-42

Will send gathered response back to the puppeteer immediately

If response not collected yet will call getResponse first.

Returns Promise<void>

ignoreResponseBodyIfPossible

src/interfaces/mixins.ts:14-14

src/interfaces/network.ts:11-11

Plugin will do nothing about original request

Type: string

native

src/interfaces/network.ts:16-16

Plugin will just listen to the original request/response data and all requests will fulfilled by puppeteer itself. But some plugin functionality can be unavailable.

Type: string

managed

src/interfaces/network.ts:20-20

Plugin will do all requests by himself. All plugin features will be available.

src/interfaces/network.ts:72-100

Plugin' request options. The request have significant difference with Puppeteer' request.

Can be modified. All changes will be applied to the actual Puppeteer' request and will be executed

method

src/interfaces/network.ts:78-78

Request method.

If request were executed you will unable to change this property.

Type: Method

url

src/interfaces/network.ts:85-85

Request url.

If request were executed you will unable to change this property.

Type: string

headers

src/interfaces/network.ts:92-92

Request headers.

If request were executed you will unable to change this property.

Type: Headers

body

src/interfaces/network.ts:99-99

Request body.

If request were executed you will unable to change this property.

Type: (string | Buffer | undefined)

_bodyError

src/interfaces/network.ts:108-108

Type: string

IAbortReason

src/interfaces/network.ts:129-129

This option will override the response

aborted - An operation was aborted (due to user action).
accessdenied - Permission to access a resource, other than the network, was denied.
addressunreachable - The IP address is unreachable. This usually means that there is no route to the specified host or network.
blockedbyclient - The client chose to block the request.
blockedbyresponse - The request failed because the response was delivered along with requirements which are not met ('X-Frame-Options' and 'Content-Security-Policy' ancestor checks, for instance).
connectionaborted - A connection timed out as a result of not receiving an ACK for data sent.
connectionclosed - A connection was closed (corresponding to a TCP FIN).
connectionfailed - A connection attempt failed.
connectionrefused - A connection attempt was refused.
connectionreset - A connection was reset (corresponding to a TCP RST).
internetdisconnected - The Internet connection has been lost.
namenotresolved - The host name could not be resolved.
timedout - An operation timed out.
failed - A generic failure occurred.

Type: ErrorCode

ConfigurableMixin

src/mixins/ConfigurableMixin.ts:89-229

Extends ConfigurableMixinBase

Plugin general configuration.

applyLoggableMixin

src/mixins/LoggableMixin.ts:13-48

Parameters

base any

Returns any

NetworkMixin

src/mixins/NetworkMixin.ts:7-52

Extends base

Expecting this.stage in runtime

InterceptionProxyRequest

src/classes/Request.ts:38-216

Extends RequestBase

Plugin' request. The request have significant difference with Puppeteer' request.

Parameters

initial INewRequestInitialArgs
requestOptions IRequestOptions

UserListener

src/classes/_OriginalRequestStateManager.ts:6-6

Type: function (response: (Puppeteer.HTTPResponse | null)): void

_OriginalRequestStateManager

src/classes/_OriginalRequestStateManager.ts:13-59

keep in track frow state

Parameters

``
``

Request handlers

Data url

Dependencies: data-urls

npm install data-urls

import { dataUrlHandler } from 'automation-extra-interception-proxy/handlers/dataUrl`

Http by got

Dependencies: got

npm install got

import { buildGotHttpHandler } from 'automation-extra-interception-proxy/handlers/gotHttp`

Samples

single page interception

/**
 * This example will show how to enable interceptions for single page.
 *
 * This code will get some wallpaper image urls from bing.com
 *
 * This code could be broken if their behavior was changed.
 */

// require libs
const puppeteer = require('puppeteer');
const InterceptionUtils = require('automation-extra-interception-proxy');

// do everything async
(async () => {

    // launch some browser
    const browser = await puppeteer.launch({
        headless: false,
    });

    // get some page
    const page = await browser.newPage();

    // attach interception commands
    await InterceptionUtils.wrapPage(page, {
        requestMode: "managed",
    });

    // create promise callback for async processing
    let callback;
    const promise = new Promise((resolve) => { callback = resolve; });

    // add some listener
    page.interceptions.addRequestListener('bing-images', async request => {

        // filter anything else
        if (request.url !== 'https://www.bing.com/hp/api/model') {
            // just letting you know that we got something else here
            console.log('Ignoring', request.url.slice(0, 50));
            return
        }

        // get response data
        const response = await request.getResponse();

        // grab data directly from their api response
        const apiData = response.json;

        // doing anything you like
        const imageUrls = apiData.MediaContents.map(({ ImageContent }) =>
            `https://www.bing.com${ImageContent.Image.Url}`);

        // back to async thread
        callback(imageUrls);

    }); // end of listener

    // goto to our destination and wait for the response
    const [imageUrls] = await Promise.all([
        promise,
        page.goto('https://www.bing.com/'),
    ]);

    // print our image urls
    console.log('imageUrls', imageUrls);

    // not necessary: cleaning our listener
    page.interceptions.deleteLocalRequestListener('bing-images');

    // closing browser
    await browser.close();

})(); // ent of our thread

Open other samples

Troubleshooting

Cookies does not work

Probably you're using old version of puppeteer. Try you upgrade first.

In case if you don't want to or cookies still does not work switch the package to v0.9.0 or below and enable enableLegacyCookieHandling.

Does cors requests are broken?

Yes, the implementation is still raw.

TODO:

handlers
- move to separate packages
- documentation
- support Grease cipher - as another handler
finalize cors managed requests - need to pass cors test
add tests
- plugin flow
documentation
- improve docs command
- describe wrapPage
- describe InterceptionProxyPlugin class
websocket support

Name		Name	Last commit message	Last commit date
Latest commit History 40 Commits
samples		samples
src		src
test		test
testServer		testServer
.gitignore		.gitignore
.node-version		.node-version
LICENSE		LICENSE
README.md		README.md
build.sh		build.sh
jest-puppeteer.config.js		jest-puppeteer.config.js
jest.config.js		jest.config.js
package.json		package.json
references.md		references.md
test.dev.js		test.dev.js
tsconfig.json		tsconfig.json

License

utyfua/automation-extra-interception-proxy

Folders and files

Latest commit

History

Repository files navigation

automation-extra-interception-proxy

Table of Contents

Breaking changes in v1:

Known issues in managed mode

Installing

Why?

1. Traffic sniffing

2. Data manipulation

3. Set proxy

4. Asynchronous decisions for requests

Motivation

API

Table of Contents

wrapPage

Parameters

IConfig

cooperativePriority

requestMode

logger

timeout

nativeContinueIfPossible

ignoreResponseBodyIfPossible

continue

ignoreResponseBodyIfPossible

flushLocal

Parameters

recordError

Parameters

recordInternalError

Parameters

recordWarning

Parameters

RequestMode

ignore

native

managed

RequestStage

gotRequest

sentRequest

gotResponse

sentResponse

closed

IRequestOptions

method

url

headers

body

_bodyError

IAbortReason

ConfigurableMixin

applyLoggableMixin

Parameters

NetworkMixin

InterceptionProxyRequest

Parameters

UserListener

_OriginalRequestStateManager

Parameters

Request handlers

Data url

Http by got

Samples

single page interception

Open other samples

Troubleshooting

Cookies does not work

Does cors requests are broken?

TODO:

License

About

Topics

Resources

License

Stars