Dashboard support for ES7 / Pbench server 0.71 #114

dbutenhof · 2020-12-18T22:25:26Z

Revise dashboard Elasticsearch queries to be compatible with Elasticsearch 7 syntax in order to work with 0.71 Pbench server, which now indexes to Elasticsearch 7.

Of special note: this also switches the dashboard over to use the two Pbench server "native query" APIs I've implemented, replacing both the query services and the fetch model effect transformations for queryControllers/fetchControllers and queryMonthIndices/fetchMonthIndices.

Also, during testing I found inconsistencies in the search behavior that seem to have been pre-existing bugs; e.g., the search state passed on to fetchIterationSamples didn't contain the id property required by that code to construct the queries. (Pay special attention to these changes.)

The dashboard actually runs on my 0.71 server VM and the RDU2 elasticsearch instance. The yarn test passes, with a warning for the SiderMenu test which apparently isn't including a key prop in a component list: not my change, and I decided not to mess with it. The e2e test script passes locally; hopefully will also on TravisCI.

The new endpoints.js should look something like this:

window.endpoints = {
  elasticsearch: 'http://elasticsearch.intlab.perf-infra.lab.eng.rdu2.redhat.com',
  results: 'http://dhcp31-170.perf.lab.eng.bos.redhat.com:8901',
  graphql: 'http://graphql.perf.lab.eng.bos.redhat.com:9466',
  prefix: 'drb.',
  result_index: 'v5.result-data-sample.',
  result_data_index: 'v5.result-data.',
  run_index: 'v6.run-data.',
  run_toc_index: 'v6.run-toc.',
  pbench_server: 'http://dhcp31-170.perf.lab.eng.bos.redhat.com:8001/api/v1',
};

Note that I changed the placeholders for the index strings to be a bit more useful. Ideally, we'll be able to convert the dashboard over to a server-driven configuration like my #2024 endpoint configuration API, which would remove all of this complexity.

Also of note is my Server API Architecture document, which has before/after payload samples and additional notes.

src/services/dashboard.js

dbutenhof · 2021-01-06T16:43:32Z

So Travis failed at yarn install ... this is clearly a Travis environmental difference that doesn't appear to have anything to do with my changes. Any ideas?

warning file-loader@2.0.0: Invalid bin field for "file-loader".
warning umi-url-pnp-loader@1.1.2: Invalid bin field for "umi-url-pnp-loader".
info fsevents@2.3.1: The platform "linux" is incompatible with this module.
info "fsevents@2.3.1" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@1.2.13: The platform "linux" is incompatible with this module.
info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
error postcss@8.2.2: The engine "node" is incompatible with this module. Expected version "^10 || ^12 || >=14". Got "13.14.0"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
yarn install failed!

src/models/search.js

portante · 2021-01-06T18:10:34Z

public/endpoints.js

+  run_index: 'vn.run-data.',
+  run_toc_index: 'vn.run-toc.',
+  result_index: 'vn.result-data-sample.',
+  result_data_index: 'vn.result-data.',


These five above don't seem like end points. This seems like what we want the initial server load to give the dashboard.

Yeah, "endpoints" might be better termed "server configuration". More obvious now that I split the two main indices. In my configuration API I put these in a "metadata" section separate from the "endpoints" (URI) section.

Hey @dbutenhof, I suppose this file is in .gitignore

Hey @dbutenhof, I suppose this file is in .gitignore

Yes, it is; although for some reason that pattern isn't working for me. Which in this case is fine as I had to add additional configuration keys.

Wait, if public/endpoints.js is in .gitignore, why are we not removing it instead of changing it in the PR?

Actually, that's a good point; .gitignore is only for untracked files. Maybe we should really have a public/endpoints.example.js that's tracked and updated, with the build looking for an untracked public/endpoints.js that's in .gitignore.

Hmm; that, unfortunately, breaks the current end-to-end testing because the search test requires endpoints.js placeholders that match the internal test framework placeholders. (This is actually a problem if you run the tests with a "real" endpoints.js, too.) It's probably possible to get npm/yarn/umi to "build" the example file as the endpoints.js to use, but I don't know enough about the capabilities and structure of package.json to do that.

I'll take a look into possibly using a mocked endpoint file for enabling e2e testing without the requirement of endpoints.js. @dbutenhof we should update the fields present in the README.md file at the root of the project directory.

portante · 2021-01-08T12:59:55Z

public/endpoints.js

+  run_index: 'vn.run-data.',
+  run_toc_index: 'vn.run-toc.',
+  result_index: 'vn.result-data-sample.',
+  result_data_index: 'vn.result-data.',


Wait, if public/endpoints.js is in .gitignore, why are we not removing it instead of changing it in the PR?

gurbirkalsi

Travis CI is failing due to an incompatible version of node running the test environment.

error postcss@8.2.3: The engine "node" is incompatible with this module. Expected version "^10 || ^12 || >=14". Got "13.14.0"

We should consider upgrading the node environment to >=14 in .travis.yml to resolve this issue.

gurbirkalsi · 2021-01-11T20:51:24Z

public/endpoints.js

+  run_index: 'vn.run-data.',
+  run_toc_index: 'vn.run-toc.',
+  result_index: 'vn.result-data-sample.',
+  result_data_index: 'vn.result-data.',


I'll take a look into possibly using a mocked endpoint file for enabling e2e testing without the requirement of endpoints.js. @dbutenhof we should update the fields present in the README.md file at the root of the project directory.

gurbirkalsi · 2021-01-11T20:55:45Z

src/models/search.js

@@ -55,18 +55,37 @@ export default {
      searchResults.resultCount = response.hits.total;


It seems resultCount is being assigned an object opposed to an integer specified from the total field. This is causing [Object] [object] to be rendered in the Search page component once a search query has been made.

Thanks. Somehow I missed that, both visually and comparing my Postman ES7 output... hits.total is an object {value: 1, relation: 'eq'} ... I want response.hits.total.value here.

gurbirkalsi

LGTM, just needs a rebase to resolve TravisCI issues!

dbutenhof · 2021-01-14T16:37:45Z

Well, this is interesting. I had done some brief experimentation with renaming endpoints.js to work with the intent of the .gitignore. And apparently when I pushed my rebase, I ended up actually deleting it... not just from github but from my local repo. Which, of course, breaks the e2e tests.

So now I've got to restore it to match the internal mocking. It'll be a minute...

dbutenhof · 2021-01-14T19:29:28Z

I suppose it's worth a note here that when this PR is merged, the dashboard master will no longer work with Pbench server code earlier than master ... the current production and staging servers are 0.69 and run with Elasticsearch V1.

Should we cut a dashboard branch before merging this? And possibly even updating it to bump the dashboard version?

dbutenhof · 2021-01-14T21:18:23Z

I suppose it's worth a note here that when this PR is merged, the dashboard master will no longer work with Pbench server code earlier than master ... the current production and staging servers are 0.69 and run with Elasticsearch V1.

Should we cut a dashboard branch before merging this? And possibly even updating it to bump the dashboard version?

FYI: After consulting with Gurbir, I've updated the dashboard version to 3.0.0 for this change.

dbutenhof self-assigned this Dec 18, 2020

dbutenhof added bug Something isn't working enhancement New feature or request labels Dec 18, 2020

dbutenhof force-pushed the api branch 2 times, most recently from 4e89c20 to a8c21fb Compare January 4, 2021 21:17

portante suggested changes Jan 5, 2021

View reviewed changes

src/services/dashboard.js Outdated Show resolved Hide resolved

dbutenhof force-pushed the api branch from a8c21fb to 43c40f3 Compare January 6, 2021 16:16

dbutenhof changed the title ~~Preliminary Dashboard support for ES7~~ Dashboard support for ES7 / Pbench server 0.71 Jan 6, 2021

dbutenhof marked this pull request as ready for review January 6, 2021 16:44

dbutenhof requested review from portante, anishaswain and npalaska January 6, 2021 16:44

dbutenhof assigned gurbirkalsi and unassigned gurbirkalsi Jan 6, 2021

portante suggested changes Jan 6, 2021

View reviewed changes

dbutenhof force-pushed the api branch from 43c40f3 to e8b3b73 Compare January 6, 2021 19:20

dbutenhof requested review from aquibbaig, gurbirkalsi, riya-17 and portante January 6, 2021 19:23

portante suggested changes Jan 8, 2021

View reviewed changes

gurbirkalsi suggested changes Jan 11, 2021

View reviewed changes

dbutenhof force-pushed the api branch from e8b3b73 to d8343d8 Compare January 12, 2021 21:54

dbutenhof requested review from gurbirkalsi and portante January 13, 2021 12:54

gurbirkalsi approved these changes Jan 14, 2021

View reviewed changes

dbutenhof force-pushed the api branch from d8343d8 to 3e17c2b Compare January 14, 2021 15:32

dbutenhof force-pushed the api branch from 3e17c2b to 496646c Compare January 14, 2021 16:46

dbutenhof requested a review from gurbirkalsi January 14, 2021 19:26

Preliminary Dashboard support for ES7

53da6d5

dbutenhof force-pushed the api branch from 496646c to 53da6d5 Compare January 14, 2021 21:02

gurbirkalsi merged commit f78ef01 into distributed-system-analysis:master Jan 14, 2021

dbutenhof deleted the api branch January 15, 2021 12:37

dbutenhof linked an issue Jul 26, 2021 that may be closed by this pull request

Upgrade Elasticsearch to v7 #71

Closed

dbutenhof mentioned this pull request Jul 26, 2021

Upgrade Elasticsearch to v7 #71

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dashboard support for ES7 / Pbench server 0.71 #114

Dashboard support for ES7 / Pbench server 0.71 #114

dbutenhof commented Dec 18, 2020 •

edited

Loading

dbutenhof commented Jan 6, 2021 •

edited

Loading

portante Jan 6, 2021

dbutenhof Jan 6, 2021

aquibbaig Jan 7, 2021 •

edited

Loading

dbutenhof Jan 7, 2021

portante Jan 8, 2021

dbutenhof Jan 8, 2021

dbutenhof Jan 8, 2021

gurbirkalsi Jan 11, 2021

portante Jan 8, 2021

gurbirkalsi left a comment

gurbirkalsi Jan 11, 2021

gurbirkalsi Jan 11, 2021

dbutenhof Jan 12, 2021

dbutenhof Jan 12, 2021

gurbirkalsi left a comment

dbutenhof commented Jan 14, 2021

dbutenhof commented Jan 14, 2021

dbutenhof commented Jan 14, 2021

		@@ -55,18 +55,37 @@ export default {
		searchResults.resultCount = response.hits.total;

Dashboard support for ES7 / Pbench server 0.71 #114

Dashboard support for ES7 / Pbench server 0.71 #114

Conversation

dbutenhof commented Dec 18, 2020 • edited Loading

dbutenhof commented Jan 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aquibbaig Jan 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gurbirkalsi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gurbirkalsi left a comment

Choose a reason for hiding this comment

dbutenhof commented Jan 14, 2021

dbutenhof commented Jan 14, 2021

dbutenhof commented Jan 14, 2021

dbutenhof commented Dec 18, 2020 •

edited

Loading

dbutenhof commented Jan 6, 2021 •

edited

Loading

aquibbaig Jan 7, 2021 •

edited

Loading