Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore/bump chromium webgl+kerberos #42751

Merged
merged 32 commits into from
Aug 27, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
7866be2
WIP: Adding libs for webgl
Jun 27, 2019
83a9d4d
WIP Adding swiftshader libs to chromium
Jun 28, 2019
3e68905
WIP: Adding missing binaries for webgl in chromium
Jul 1, 2019
87d1528
Use pipes for communication with chrome to avoid networking snafus
Jul 26, 2019
0951604
Merge remote-tracking branch 'upstream/master'
Jul 29, 2019
b35b6ff
Merge remote-tracking branch 'upstream/master'
Jul 30, 2019
02b5257
Merge remote-tracking branch 'upstream/master'
Aug 1, 2019
b4995c5
Merge remote-tracking branch 'upstream/master'
Aug 5, 2019
33f2a0b
Bumps puppeteer in prep for new chromium build + types and better @ty…
Aug 6, 2019
ac9088a
Remove ignore
Aug 6, 2019
26deffa
Removing of final @ts-ignore now that we have types
Aug 6, 2019
48c3134
README updates
Aug 6, 2019
e580990
Merge branch 'chore/roll-chromium-webgl' into chore/bump-chromium-web…
Aug 6, 2019
6810a47
Merge remote-tracking branch 'upstream/master' into chore/bump-chromi…
Aug 9, 2019
491be4e
Merge remote-tracking branch 'upstream/master'
Aug 9, 2019
680e0a5
Merge remote-tracking branch 'upstream/master'
Aug 9, 2019
91458db
Fixing binding issues
Aug 9, 2019
530d485
Merge remote-tracking branch 'upstream/master'
Aug 12, 2019
69a9b13
Merge branch 'master' into chore/bump-chromium-webgl-kerberos
Aug 12, 2019
42e4306
Merge remote-tracking branch 'upstream/master' into chore/bump-chromi…
Aug 13, 2019
7761486
Fixing maps integration wrt reporting + conditional pipes for puppeteer
Aug 13, 2019
96f0f41
Merge remote-tracking branch 'upstream/master' into chore/bump-chromi…
Aug 21, 2019
8ffbbfc
Adding new deps to the windows build
Aug 26, 2019
199b5cc
New s3 builds
Aug 26, 2019
f80ba30
Merge remote-tracking branch 'upstream/master' into chore/bump-chromi…
Aug 26, 2019
d184029
Merge remote-tracking branch 'upstream/master' into chore/bump-chromi…
Aug 26, 2019
bbf0e8f
Checksums for updated linux build
Aug 26, 2019
10e0648
Merge remote-tracking branch 'upstream/master' into chore/bump-chromi…
Aug 27, 2019
c929c59
Moving types out of puppeteer file and into core puppeteer module
Aug 27, 2019
aefc12f
launch => puppeteerLaunch
Aug 27, 2019
a0a13b3
Maps comment about render loading in reporting
Aug 27, 2019
59fd444
Clarify how reporting uses hooks and events for viz
Aug 27, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions x-pack/build_chromium/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,13 +86,13 @@ In windows, at least, you will need to do a number of extra steps:

Find the sha of the Chromium commit you wish to build. Most likely, you want to build the Chromium revision that is tied to the version of puppeteer that we're using.

Find the Chromium revision (modify the following command to be wherever you have the kibana source installed):
Find the Chromium revision (run in kibana's working directory):

- `cat ~/dev/elastic/kibana/x-pack/node_modules/puppeteer-core/package.json | grep chromium_revision`
- `cat node_modules/puppeteer-core/package.json | grep chromium_revision`
- Take the revision number from that, and tack it to the end of this URL: https://crrev.com
- (For example: https://crrev.com/637110)
- (For example, puppeteer@1.19.0 has rev (674921): https://crrev.com/674921)
- Grab the SHA from there
- (For example, rev 637110 has sha 2fac04abf6133ab2da2846a8fbd0e97690722699)
- (For example, rev 674921 has sha 312d84c8ce62810976feda0d3457108a6dfff9e6)

Note: In Linux, you should run the build command in tmux so that if your ssh session disconnects, the build can keep going. To do this, just type `tmux` into your terminal to hop into a tmux session. If you get disconnected, you can hop back in like so:

Expand All @@ -102,9 +102,9 @@ Note: In Linux, you should run the build command in tmux so that if your ssh ses

To run the build, replace the sha in the following commands with the sha that you wish to build:

- Mac: `python ~/chromium/build_chromium/build.py 2fac04abf6133ab2da2846a8fbd0e97690722699`
- Linux: `python ~/chromium/build_chromium/build.py 2fac04abf6133ab2da2846a8fbd0e97690722699`
- Windows: `python c:\chromium\build_chromium\build.py 2fac04abf6133ab2da2846a8fbd0e97690722699`
- Mac: `python ~/chromium/build_chromium/build.py 312d84c8ce62810976feda0d3457108a6dfff9e6`
- Linux: `python ~/chromium/build_chromium/build.py 312d84c8ce62810976feda0d3457108a6dfff9e6`
- Windows: `python c:\chromium\build_chromium\build.py 312d84c8ce62810976feda0d3457108a6dfff9e6`

## Artifacts

Expand Down
12 changes: 11 additions & 1 deletion x-pack/build_chromium/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,23 @@ def archive_file(name):
# must be bundled with the Chromium executable.
if platform.system() == 'Linux':
archive_file('headless_shell')
archive_file(os.path.join('swiftshader', 'libEGL.so'))
archive_file(os.path.join('swiftshader', 'libGLESv2.so'))

elif platform.system() == 'Windows':
archive_file('headless_shell.exe')
archive_file('dbghelp.dll')
archive_file('icudtl.dat')
archive_file(os.path.join('swiftshader', 'libEGL.dll'))
archive_file(os.path.join('swiftshader', 'libEGL.dll.lib'))
archive_file(os.path.join('swiftshader', 'libGLESv2.dll'))
archive_file(os.path.join('swiftshader', 'libGLESv2.dll.lib'))

elif platform.system() == 'Darwin':
archive_file('headless_shell')
archive_file('Helpers/chrome_crashpad_handler')
archive_file('libswiftshader_libEGL.dylib')
archive_file('libswiftshader_libGLESv2.dylib')
archive_file(path.join('Helpers', 'chrome_crashpad_handler'))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is existing, but stands out a little, and I'm wondering why we have it in the Darwin build

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to look again. The binary fails to start wo it, however I'm not sure as to why


archive.close()

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,15 @@ import { ExitFullScreenButton } from 'ui/exit_full_screen';
import { getIndexPatternsFromIds } from '../../index_pattern_util';
import { ES_GEO_FIELD_TYPE } from '../../../common/constants';
import { i18n } from '@kbn/i18n';
import uuid from 'uuid/v4';

const RENDER_COMPLETE_EVENT = 'renderComplete';

export class GisMap extends Component {

state = {
isInitialLoadRenderTimeoutComplete: false,
domId: uuid(),
geoFields: [],
}

Expand All @@ -47,6 +51,21 @@ export class GisMap extends Component {
this._clearRefreshTimer();
}

// Reporting uses both a `data-render-complete` attribute and a DOM event listener to determine
// if a visualization is done loading. The process roughly is:
// - See if the `data-render-complete` attribute is "true". If so we're done!
// - If it's not, then reporting injects a listener into the browser for a custom "renderComplete" event.
// - When that event is fired, we snapshot the viz and move on.
// Failure to not have the dom attribute, or custom event, will timeout the job.
// See x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/wait_for_render.ts for more.
_onInitialLoadRenderComplete = () => {
const el = document.querySelector(`[data-dom-id="${this.state.domId}"]`);
joelgriffith marked this conversation as resolved.
Show resolved Hide resolved

if (el) {
el.dispatchEvent(new CustomEvent(RENDER_COMPLETE_EVENT, { bubbles: true }));
}
}

_loadGeoFields = async (nextIndexPatternIds) => {
if (_.isEqual(nextIndexPatternIds, this._prevIndexPatternIds)) {
// all ready loaded index pattern ids
Expand Down Expand Up @@ -118,9 +137,10 @@ export class GisMap extends Component {
() => {
if (this._isMounted) {
this.setState({ isInitialLoadRenderTimeoutComplete: true });
this._onInitialLoadRenderComplete();
}
},
1000
5000
);
}

Expand All @@ -135,6 +155,8 @@ export class GisMap extends Component {
mapInitError,
} = this.props;

const { domId } = this.state;

if (mapInitError) {
return (
<div data-render-complete data-shared-item>
Expand Down Expand Up @@ -179,6 +201,7 @@ export class GisMap extends Component {
<EuiFlexGroup
gutterSize="none"
responsive={false}
data-dom-id={domId}
data-render-complete={this.state.isInitialLoadRenderTimeoutComplete}
data-shared-item
>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
*/

import open from 'opn';
import { Page, SerializableOrJSHandle, EvaluateFn } from 'puppeteer';
import { parse as parseUrl } from 'url';
import { Page, SerializableOrJSHandle, EvaluateFn } from 'puppeteer';
import { ViewZoomWidthHeight } from '../../../../export_types/common/layouts/layout';
import { LevelLogger } from '../../../../server/lib';
import {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,13 @@
import fs from 'fs';
import os from 'os';
import path from 'path';
// @ts-ignore
import puppeteer from 'puppeteer-core';
import { Browser, Page, LaunchOptions } from 'puppeteer';
import rimraf from 'rimraf';
import * as Rx from 'rxjs';
import { map, share, mergeMap, filter, partition, ignoreElements, tap } from 'rxjs/operators';
import { InnerSubscriber } from 'rxjs/internal/InnerSubscriber';

import { puppeteerLaunch } from '../puppeteer';
import { LevelLogger as Logger } from '../../../lib/level_logger';
import { HeadlessChromiumDriver } from '../driver';
import { args, IArgOptions } from './args';
Expand Down Expand Up @@ -63,23 +62,21 @@ export class HeadlessChromiumDriverFactory {
proxyConfig: this.browserConfig.proxy,
});

return puppeteer
.launch({
userDataDir,
executablePath: this.binaryPath,
ignoreHTTPSErrors: true,
args: chromiumArgs,
env: {
TZ: browserTimezone,
},
} as LaunchOptions)
.catch((error: Error) => {
logger.warning(
`The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports: [${error}]`
);
logger.warning(`See Chromium's log output at "${getChromeLogLocation(this.binaryPath)}"`);
return null;
});
return puppeteerLaunch({
userDataDir,
executablePath: this.binaryPath,
ignoreHTTPSErrors: true,
args: chromiumArgs,
env: {
TZ: browserTimezone,
},
} as LaunchOptions).catch((error: Error) => {
logger.warning(
`The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports: [${error}]`
);
logger.warning(`See Chromium's log output at "${getChromeLogLocation(this.binaryPath)}"`);
return null;
});
}

create({
Expand Down Expand Up @@ -109,8 +106,8 @@ export class HeadlessChromiumDriverFactory {
let browser: Browser;
let page: Page;
try {
browser = await puppeteer.launch({
pipe: true,
browser = await puppeteerLaunch({
pipe: !this.browserConfig.inspect,
userDataDir,
executablePath: this.binaryPath,
ignoreHTTPSErrors: true,
joelgriffith marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -126,7 +123,6 @@ export class HeadlessChromiumDriverFactory {
// which can cause the job to fail even if we bump timeouts in
// the config. Help alleviate errors like
// "TimeoutError: waiting for selector ".application" failed: timeout 30000ms exceeded"
// @ts-ignore outdated typedefs for puppteer
page.setDefaultTimeout(this.queueTimeout);

this.logger.debug(`Browser driver factory created`);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,23 @@ export const paths = {
packages: [
{
platforms: ['darwin', 'freebsd', 'openbsd'],
archiveFilename: 'chromium-2fac04a-darwin.zip',
archiveChecksum: '36814b1629457aa178b4ecdf6cc1bc5f',
rawChecksum: '9b40e2efa7f4f1870835ee4cdaf1dd51',
archiveFilename: 'chromium-312d84c-darwin.zip',
archiveChecksum: '020303e829745fd332ae9b39442ce570',
rawChecksum: '101dfea297c5818a7a3f3317a99dde02',
binaryRelativePath: 'headless_shell-darwin/headless_shell',
},
{
platforms: ['linux'],
archiveFilename: 'chromium-2fac04a-linux.zip',
archiveChecksum: '5cd6b898a35f9dc0ba6f49d821b8a2a3',
rawChecksum: 'b3fd218d3c3446c388da4e6c8a82754c',
archiveFilename: 'chromium-312d84c-linux.zip',
archiveChecksum: '15ba9166a42f93ee92e42217b737018d',
rawChecksum: '3455db62ea4bd2d6e891e9155313305a',
binaryRelativePath: 'headless_shell-linux/headless_shell',
},
{
platforms: ['win32'],
archiveFilename: 'chromium-2fac04a-windows.zip',
archiveChecksum: '1499a4d5847792d59b9c1a8ab7dc8b94',
rawChecksum: '08b48d2f3d23c4bc8b58779ca4a7b627',
archiveFilename: 'chromium-312d84c-windows.zip',
archiveChecksum: '3e36adfb755dacacc226ed5fd6b43105',
rawChecksum: 'ec7aa6cfecb172129474b447311275ec',
binaryRelativePath: 'headless_shell-windows\\headless_shell.exe',
},
],
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

joelgriffith marked this conversation as resolved.
Show resolved Hide resolved
import puppeteer from 'puppeteer';
// @ts-ignore lacking typedefs which this module fixes
import puppeteerCore from 'puppeteer-core';

export const puppeteerLaunch: (
opts?: puppeteer.LaunchOptions
) => Promise<puppeteer.Browser> = puppeteerCore.launch.bind(puppeteerCore);
joelgriffith marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 2 additions & 2 deletions x-pack/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
"@types/pngjs": "^3.3.1",
"@types/prop-types": "^15.5.3",
"@types/proper-lockfile": "^3.0.1",
"@types/puppeteer": "^1.12.4",
"@types/puppeteer": "^1.19.0",
"@types/react": "^16.8.0",
"@types/react-dom": "^16.8.0",
"@types/react-redux": "^6.0.6",
Expand Down Expand Up @@ -305,7 +305,7 @@
"prop-types": "^15.6.0",
"proper-lockfile": "^3.2.0",
"puid": "1.0.7",
"puppeteer-core": "^1.13.0",
"puppeteer-core": "^1.19.0",
"raw-loader": "3.1.0",
"react": "^16.8.0",
"react-apollo": "^2.1.4",
Expand Down
16 changes: 8 additions & 8 deletions yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -3801,10 +3801,10 @@
resolved "https://registry.yarnpkg.com/@types/proper-lockfile/-/proper-lockfile-3.0.1.tgz#dd770a2abce3adbcce3bd1ed892ce2f5f17fbc86"
integrity sha512-ODOjqxmaNs0Zkij+BJovsNJRSX7BJrr681o8ZnNTNIcTermvVFzLpz/XFtfg3vNrlPVTJY1l4e9h2LvHoxC1lg==

"@types/puppeteer@^1.12.4":
version "1.12.4"
resolved "https://registry.yarnpkg.com/@types/puppeteer/-/puppeteer-1.12.4.tgz#8388efdb0b30a54a7e7c4831ca0d709191d77ff1"
integrity sha512-aaGbJaJ9TuF9vZfTeoh876sBa+rYJWPwtsmHmYr28pGr42ewJnkDTq2aeSKEmS39SqUdkwLj73y/d7rBSp7mDQ==
"@types/puppeteer@^1.19.0":
version "1.19.0"
resolved "https://registry.yarnpkg.com/@types/puppeteer/-/puppeteer-1.19.0.tgz#59f0050bae019cee7c3af2bb840a25892a3078b6"
integrity sha512-Db9LWOuTm2bR/qgPE7PQCmnsCQ6flHdULuIDWTks8YdQ/SGHKg5WGWG54gl0734NDKCTF5MbqAp2qWuvBiyQ3Q==
dependencies:
"@types/node" "*"

Expand Down Expand Up @@ -22510,10 +22510,10 @@ punycode@^1.2.4, punycode@^1.4.1:
resolved "https://registry.yarnpkg.com/punycode/-/punycode-1.4.1.tgz#c0d5a63b2718800ad8e1eb0fa5269c84dd41845e"
integrity sha1-wNWmOycYgArY4esPpSachN1BhF4=

puppeteer-core@^1.13.0:
version "1.13.0"
resolved "https://registry.yarnpkg.com/puppeteer-core/-/puppeteer-core-1.13.0.tgz#f8001851e924e6e9ef6e9fae1778c3ab87c3f307"
integrity sha512-8MypjWVHu2EEdtN2HxhCsTtIYdJgiCcbGpHoosv265fzanfOICC2/DadLZq6/Qc/OKsovQmjkO+2vKMrV3BRfA==
puppeteer-core@^1.19.0:
version "1.19.0"
resolved "https://registry.yarnpkg.com/puppeteer-core/-/puppeteer-core-1.19.0.tgz#3c3f98edb5862583e3a9c19cbc0da57ccc63ba5c"
integrity sha512-ZPbbjUymorIJomHBvdZX5+2gciUmQtAdepCrkweHH6rMJr96xd/dXzHgmYEOBMatH44SmJrcMtWkgsLHJqT89g==
dependencies:
debug "^4.1.0"
extract-zip "^1.6.6"
Expand Down