Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

20211011 zrq hdbscan config #586

Closed
wants to merge 64 commits into from
Closed

20211011 zrq hdbscan config #586

wants to merge 64 commits into from

Conversation

Zarquan
Copy link
Collaborator

@Zarquan Zarquan commented Oct 20, 2021

  1. New naming scheme for configurations zeppelin-{cpu}.{mem}-spark-{n}.{cpu}.{mem}.yml
  2. New Ansible config for HDBSCAN, 55 core 90G memory for Zeppelin, 6x 27 core 45G memory for Spark.
  3. Notes on setting up the system for debugging.
  4. Notes for writing the 2022 resource request.

stvoutsin and others added 30 commits September 21, 2021 12:04
@stvoutsin
Added some notes on testing with Benchmarking tool
5d73750
@stvoutsin
Add some more notes on testing with benchmarking tool
4ebb9fd
@stvoutsin
Added playbook for running a benchmark
16fdc28
@stvoutsin
Add test to deployment script
df06564
@stvoutsin
Update create all script, add optional deploytype as param (set as te…
946260d 
…st to run benchmark)
@stvoutsin
Added notes on running a test deploy with benchmarker
f5f80f2
@stvoutsin
Merge branch 'wfau:master' into issue-benchmarking
97af358
@stvoutsin
Change repository location of testing tool
ae3d5a1
@stvoutsin
Fixed Typo
8a3fa33
@stvoutsin
Merge branch 'wfau:master' into issue-benchmarking
899754c
@stvoutsin
Add notes on running a deploy with the automated benchmarking
4cfc742
@stvoutsin
Notes on running a test deploy on large cluster
a7c4eed
@stvoutsin
Update medium test notes
18753d7
@stvoutsin
Update large test notes
823c1ad
@Zarquan
Notes and tests for HDBSCAN config
f4b4a50
@Zarquan
Notes and tests for HDBSCAN config
70b15aa
@Zarquan
Merge branch '20211011-zrq-hdbscan-config' of github.com:Zarquan/agla…
6e2998a 
…is into 20211011-zrq-hdbscan-config
@Zarquan
Remove deprecated config settings (issue wfau#584)
c5cec9e
@stvoutsin
Upgrade Hadoop / Zeppelin & Spark versions / Fix issue with pip lib i…
97ba06d 
…nstallation
@stvoutsin
Added exclude interpreter setting in Zeppelin (Needed in Zeppelin 0.1…
9d1bbc6 
…0.0 to exclude interp.)
@stvoutsin
Added Benchmark running Ansible script
42e1e47
@stvoutsin
Added notes on testing upgraded version
c43f0ed
@stvoutsin
Set default interpreter as spark / Write to hadoop dir as root / Excl…
5cd5755 
…ude all unsupported interpreters
@stvoutsin
Added notes for another test with upgraded versions
926facf
@stvoutsin
Change medium-04 config to use upgraded versions
1bc51f9
@stvoutsin
Fix Job Manager UI issue in new Zeppelin
c190938
@stvoutsin
Changes to benchmark setup config based on review
eeeae66
@stvoutsin
Fixing issue with missing libraries
0d91081
@stvoutsin
Added benchmark config to this repo
2142d65
@stvoutsin
Change location of notebook config for benchmarking to this repo
412b613
stvoutsin
stvoutsin reviewed Oct 22, 2021
View reviewed changes
Copy link
Collaborator

@stvoutsin stvoutsin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a look and changes look fine, do we want to approve this first or #590 ? I'm wondering what the easiest way to do it is regarding potential merge conflicts

@Zarquan
Copy link
Collaborator Author

Zarquan commented Oct 22, 2021

I'm wondering what the easiest way to do it is regarding potential merge conflicts
As far as I can tell the two PRs don't have any overlapping files, so there shouldn't be a conflict.
The only reason for asking for a rebase on #583 was because there was the possibility of the same change being applied twice.
This branch isn't urgent, it can wait until after we merge #583.

Zarquan and others added 20 commits December 1, 2021 06:14
@Zarquan
Merge pull request wfau#583 from stvoutsin/issue-benchmarking
cc05970 
Updates to Ansible scripts to include an (optional) benchmark test
@stvoutsin
Disable iPython in zeppelin interpreter
f358293
@Zarquan
Merge pull request wfau#590 from stvoutsin/issue-upgrade-spark-3
8c90c72 
Upgrade Spark / Hadoop / Zeppelin (Issue wfau#535)
@Zarquan
....
ab159eb
@Zarquan
Notes and tests for HDBSCAN config
68b2732
@Zarquan
Remove deprecated config settings (issue wfau#584)
a48ae90
@Zarquan
Notes on debugging
d6f1629
@Zarquan
Notes on resource requirements
55da716
@Zarquan
New config naming scheme zeppelin-{cpu}.{mem}-spark-{n}.{cpu}.{mem}.yml
81680d1
@Zarquan
Merge branch '20211011-zrq-hdbscan-config' of github.com:Zarquan/agla…
45e52bb 
…is into 20211011-zrq-hdbscan-config
@Zarquan
Removed accidental commit from master
3d451f3
@Zarquan
Notes and tests for HDBSCAN config
4234f0b
@Zarquan
Remove deprecated config settings (issue wfau#584)
dc7ba43
@Zarquan
Notes on debugging
8238493
@Zarquan
Notes on resource requirements
5990da4
@Zarquan
New config naming scheme zeppelin-{cpu}.{mem}-spark-{n}.{cpu}.{mem}.yml
03222f6
@Zarquan
Notes and tests for HDBSCAN config
de428f3
@Zarquan
Remove deprecated config settings (issue wfau#584)
6209d29
@Zarquan
New config naming scheme zeppelin-{cpu}.{mem}-spark-{n}.{cpu}.{mem}.yml
f34df2e
@Zarquan
Merge branch '20211011-zrq-hdbscan-config' of github.com:Zarquan/agla…
d353805 
…is into 20211011-zrq-hdbscan-config
@Zarquan
Copy link
Collaborator Author

Zarquan commented Dec 3, 2021

Rebased the branch, PR got confusinh.
Closing this PR and starting a new one.

@Zarquan Zarquan closed this Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stvoutsin stvoutsin stvoutsin left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Double the Zeppelin node
2 participants
@Zarquan @stvoutsin