Skip to content

Commit

Permalink
Add Lychee Link Checker into OSD
Browse files Browse the repository at this point in the history
1. Fix broken links in OSD
2. Generate lycheeexcude list to filter out false negative warnings from test files or external links
3. Add TODO items for internal unavaiable links
4. Integrate with doc link service change.

Signed-off-by: Zuocheng Ding <zding817@gmail.com>
  • Loading branch information
zuochengding committed Nov 19, 2021
1 parent 9047044 commit 75398f0
Show file tree
Hide file tree
Showing 172 changed files with 662 additions and 265 deletions.
31 changes: 31 additions & 0 deletions .github/workflows/doc_links_checker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

name: Doc Link Checker

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
linkchecker:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Load Excludes
run: |
LYCHEE_EXCLUDE=$(sed -e :a -e 'N;s/\n/ --exclude /;ta' .lycheeexclude)
echo "LYCHEE_EXCLUDE=$LYCHEE_EXCLUDE" >> $GITHUB_ENV
- name: Lychee Link Checker
id: lychee
uses: lycheeverse/lychee-action@master
with:
args: --accept=200,403,429 --exclude ${{ env.LYCHEE_EXCLUDE }} --exclude-mail "**/*.html" "**/*.md" "**/*.txt" "**/*.json" "**/*.js" "**/*.ts" "**/*.tsx"
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Fail if there were link errors
run: exit ${{ steps.lychee.outputs.exit_code }}
117 changes: 117 additions & 0 deletions .lycheeexclude
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Local or predefined end points
http://localhost
https://localhost
http://127.0.0.1/
https://127.0.0.1/
http://127.0.0.1:10002/bar
http://127.0.0.1:10002/
http://opensearch
https://opensearch
https://opensearch-dashboards
http://opensearch-dashboards
https://opensearch.internal.org/
https://maps.search-services.aws.a2z.com/
https://tiles.maps.search-services.aws.a2z.com/
https://telemetry.opensearch.org/
https://telemetry-staging.opensearch.org/
https://api.github.com/repos/opensearch-project/OpenSearch-Dashboards/
file:///*
git://*
https://www.opensearch.org/guide/en/kibana/current/tutorial-visualizing.html
https://www.opensearch.org/guide/en/kibana/current/tutorial-load-dataset.html

# Dummy urls in tests
http://domain.
http://www.domain.
http://somehost
https://somehost
http://some.host
https://some-host.com/
https://other.some-host.com/
http://test:user@somehost/
https://some.another.host/
http://noone.nowhere.none/
http://bar
http://foo
http://test.com/
https://files.foobar/
https://tiles.foobar/
https://1.1.1.1:9200/
http://192.168.1.1:1234/
http://9.8.7.6/
http://1.2.3.4/
http://8.8.8.8/
https://path.to/
https://example.com/
http://example.com/
https://example.org/
http://some-url/
http://buildurl/
https://dryrun/
https://url/
http://url/
http://notfound.svg/
https://validurl/
https://myopensearch-dashboardsdomain.com
http://myopensearch-dashboardsdomain.com
https://other-opensearch-dashboards.external:8080/
http://myotherdomain.com:5601/
http://myopensearch-dashboardsdomain.com:5601/
https://your-cdn-host.com/
http://not-your-opensearch-dashboards.com/
http://www.mysite.com/
http://myserver.mydomain.com:5601/
https://myexternaldep.com/
http://notlocalhost/
http://site.com/
http://node-b/
http://node-a:9200/
https://node-c/
https://elsewhere
https://opensearch:changeme@example.com:9200
http://test:user@somehost/
https://mycloudinstance:9200/
https://dev-url.co/
https://extenal.org/_search
http://plugins.example.com/
https://build-url/
http://test.com/
https://path.to/
http://site.com/
http://not-your-opensearch-dashboards.com/
http://evil.com/
https://www.opensearch.org/cool/path
https://www.opensearch.org/redirect
http://www.opensearch.org/painlessDocs
https://www.opensearch.org/subscriptions
https://www.opensearch.org/opensearch-maps-service
https://www.hostedgraphite.com/

# External urls
https://www.zeek.org/
http://google.com/
https://api.worldbank.org/
https://vega.github.io/
http://twitter.com/
https://twitter.com/
https://storage.googleapis.com/
http://tools.ietf.org/
https://github.com/
http://github.com/
http://jsperf
https://jsperf
https://gist.githubusercontent.com/
https://nodejs.org/
https://www.npmjs.com/
https://microsoft.github.io/
http://api.worldbank.org/
https://f1542b814f674090afd914960583265f.apm.us-central1.gcp.cloud.es.io/
http://www.matthewcopeland.me/
http://threedubmedia.googlecode.com/*
https://developer.mozilla.org/
https://a.tile.openstreetmap.org/
http://www.creedthoughts.gov
https://media-for-the-masses.theacademyofperformingartsandscience.org/
6 changes: 6 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ Say that you would want to debug a test in CI group 1, you can run the following

This will print of an address, to which you could open your chrome browser on your instance and navigate to `chrome://inspect/#devices` and inspect the functional test runner `scripts/functional_tests.js`.

### Additional checks
Make sure you run lint checker before submitting a pull request. To run lint checker:
`node scripts/precommit_hook.js --fix`

Please ensure that you don't introduce any broken links accidently. For any intentional broken link (e.g. dummy url in unit test), you can add it to `.lycheeexclude` allow list specifically.

# Continuous Integration
Automated testing is provided with Jenkins for Continuous Integration. Jenkins enables developers to build, deploy, and automate projects and provides us to run groups of tests quickly. CI groups are ran from the [Jenkinsfile](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/Jenkinsfile).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ const { TextHighlightRules } = ace.acequire('ace/mode/text_highlight_rules');
const oop = ace.acequire('ace/lib/oop');

export const OpenSearchSqlHighlightRules = function (this: any) {
// See https://www.opensearch.org/guide/en/elasticsearch/reference/current/sql-commands.html
// See https://opensearch.org/docs/latest/search-plugins/sql/index/
const keywords =
'describe|between|in|like|not|and|or|desc|select|from|where|having|group|by|order' +
'asc|desc|pivot|for|in|as|show|columns|include|frozen|tables|escape|limit|rlike|all|distinct|is';

const builtinConstants = 'true|false';

// See https://www.opensearch.org/guide/en/elasticsearch/reference/current/sql-syntax-show-functions.html
// See https://opensearch.org/docs/latest/search-plugins/sql/index/
const builtinFunctions =
'avg|count|first|first_value|last|last_value|max|min|sum|kurtosis|mad|percentile|percentile_rank|skewness' +
'|stddev_pop|sum_of_squares|var_pop|histogram|case|coalesce|greatest|ifnull|iif|isnull|least|nullif|nvl' +
Expand All @@ -56,7 +56,7 @@ export const OpenSearchSqlHighlightRules = function (this: any) {
'|ltrim|octet_length|position|repeat|replace|right|rtrim|space|substring|ucase|cast|convert|database|user|st_astext|st_aswkt' +
'|st_distance|st_geometrytype|st_geomfromtext|st_wkttosql|st_x|st_y|st_z|score';

// See https://www.opensearch.org/guide/en/elasticsearch/reference/current/sql-data-types.html
// See https://opensearch.org/docs/latest/search-plugins/sql/index/
const dataTypes =
'null|boolean|byte|short|integer|long|double|float|half_float|scaled_float|keyword|text|binary|date|ip|object|nested|time' +
'|interval_year|interval_month|interval_day|interval_hour|interval_minute|interval_second|interval_year_to_month' +
Expand Down
2 changes: 1 addition & 1 deletion packages/osd-dev-utils/certs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The password used for both of these is "storepass". Other copies are also provid

## Certificate generation

[OpenSearch cert-util](https://www.opensearch.org/guide/en/elasticsearch/reference/current/certutil.html) and [OpenSSL](https://www.openssl.org/) were used to generate these certificates. The following commands were used from the root directory of OpenSearch:
[OpenSearch cert-util](https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/) and [OpenSSL](https://www.openssl.org/) were used to generate these certificates. The following commands were used from the root directory of OpenSearch:

```
# Generate the PKCS #12 keystore for a CA, valid for 50 years
Expand Down
2 changes: 1 addition & 1 deletion packages/osd-logging/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ is still in `core` for now.
- [Log level](#log-level)
- [Layouts](#layouts)

The way logging works in OpenSearch Dashboards is inspired by `log4j 2` logging framework used by [Elasticsearch](https://www.opensearch.org/guide/en/elasticsearch/reference/current/settings.html#logging).
The way logging works in OpenSearch Dashboards is inspired by `log4j 2` logging framework used by [OpenSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#logging).
The main idea is to have consistent logging behaviour (configuration, log format etc.) across the entire Elastic Stack
where possible.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@
"_cluster/health",
"_cluster/health/{indices}"
],
"documentation": "https://www.opensearch.org/guide/en/elasticsearch/reference/master/cluster-health.html"
"documentation": "https://opensearch.org/docs/latest/opensearch/rest-api/cluster-health/"
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"cluster.health":{
"documentation":{
"url":"https://www.opensearch.org/guide/en/elasticsearch/reference/master/cluster-health.html",
"url":"https://opensearch.org/docs/latest/opensearch/rest-api/cluster-health/",
"description":"Returns basic information about the health of the cluster."
},
"stability":"stable",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,22 @@ You can fix this problem by:
https://on.cypress.io/element-is-animating
Because this error occurred during a `after each` hook we are skipping the remaining tests in the current suite: `timeline flyout button`
at cypressErr (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:146621:16)
at cypressErrByPath (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:146630:10)
at Object.throwErrByPath (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:146593:11)
at Object.ensureElementIsNotAnimating (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:137560:24)
at ensureNotAnimating (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127434:13)
at runAllChecks (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127522:9)
at retryActionability (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127542:16)
at tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
at Function.Promise.attempt.Promise.try (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:6339:29)
at tryFn (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:140680:21)
at whenStable (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:140715:12)
at http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:140259:16
at tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7000:31)
at Promise._settlePromise (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7102:10)]]></failure>
at cypressErr (https://example.com/__cypress/runner/cypress_runner.js:146621:16)
at cypressErrByPath (https://example.com/__cypress/runner/cypress_runner.js:146630:10)
at Object.throwErrByPath (https://example.com/__cypress/runner/cypress_runner.js:146593:11)
at Object.ensureElementIsNotAnimating (https://example.com/__cypress/runner/cypress_runner.js:137560:24)
at ensureNotAnimating (https://example.com/__cypress/runner/cypress_runner.js:127434:13)
at runAllChecks (https://example.com/__cypress/runner/cypress_runner.js:127522:9)
at retryActionability (https://example.com/__cypress/runner/cypress_runner.js:127542:16)
at tryCatcher (https://example.com/__cypress/runner/cypress_runner.js:9065:23)
at Function.Promise.attempt.Promise.try (https://example.com/__cypress/runner/cypress_runner.js:6339:29)
at tryFn (https://example.com/__cypress/runner/cypress_runner.js:140680:21)
at whenStable (https://example.com/__cypress/runner/cypress_runner.js:140715:12)
at https://example.com/__cypress/runner/cypress_runner.js:140259:16
at tryCatcher (https://example.com/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (https://example.com/__cypress/runner/cypress_runner.js:7000:31)
at Promise._settlePromise (https://example.com/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (https://example.com/__cypress/runner/cypress_runner.js:7102:10)]]></failure>
</testcase>
</testsuite>
</testsuites>
Original file line number Diff line number Diff line change
Expand Up @@ -335,8 +335,8 @@ it('rewrites cypress reports with minimal changes', async () => {
You can fix this problem by:
- Passing \`{force: true}\` which disables all error checking
@@ -46,5 +37,5 @@
at Promise._settlePromise (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7102:10)]]›‹/failure›
at Promise._settlePromise (https://example.com/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (https://example.com/__cypress/runner/cypress_runner.js:7102:10)]]›‹/failure›
‹/testcase›
‹/testsuite›
-‹/testsuites›
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ export const opensearchTestConfig = new (class OpenSearchTestConfig {
}

getUrlParts() {
// Allow setting one complete TEST_OPENSEARCH_URL for opensearch like https://elastic:changeme@myCloudInstance:9200
// Allow setting one complete TEST_OPENSEARCH_URL for opensearch like https://opensearch:changeme@example.com:9200
if (process.env.TEST_OPENSEARCH_URL) {
const testOpenSearchUrl = url.parse(process.env.TEST_OPENSEARCH_URL);
return {
Expand Down
6 changes: 5 additions & 1 deletion src/core/public/chrome/chrome_service.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,11 @@ export class ChromeService {
defaultMessage="Support for Internet Explorer will be dropped in future versions of this software, please check {link}."
values={{
link: (
<EuiLink target="_blank" href="https://www.opensearch.org/support/matrix" external>
<EuiLink
target="_blank"
href={docLinks.links.opensearchDashboards.browser}
external
>
<FormattedMessage
id="core.chrome.browserDeprecationLink"
defaultMessage="the support matrix on our website"
Expand Down
6 changes: 0 additions & 6 deletions src/core/public/chrome/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,6 @@
* GitHub history for details.
*/

// export const OPENSEARCH_DASHBOARDS_FEEDBACK_LINK =
// 'https://www.opensearch.org/products/kibana/feedback?blade=kibanafeedback';
// export const OPENSEARCH_DASHBOARDS_ASK_OPENSEARCH_LINK =
// 'https://www.opensearch.org/products/kibana/ask-elastic?blade=kibanaaskelastic';
// export const GITHUB_CREATE_ISSUE_LINK = 'https://github.com/opensearch-project/OpenSearch-Dashboards/issues/new/choose';

export const OPENSEARCH_DASHBOARDS_FEEDBACK_LINK = 'https://github.com/opensearch-project';
export const OPENSEARCH_DASHBOARDS_ASK_OPENSEARCH_LINK = 'https://github.com/opensearch-project';
export const GITHUB_CREATE_ISSUE_LINK =
Expand Down
2 changes: 1 addition & 1 deletion src/core/public/chrome/ui/header/header_help_menu.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ export type ChromeHelpExtensionMenuDiscussLink = EuiButtonEmptyProps & {
linkType: 'discuss';
/**
* URL to discuss page.
* i.e. `https://discuss.opensearch.org/c/${appName}`
* i.e. https://discuss.opendistrocommunity.dev/
*/
href: string;
};
Expand Down
Loading

0 comments on commit 75398f0

Please sign in to comment.