#157 finalize v2 spark infrastructure #321
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
10 tasks
cwoods-cpointe
commented
Sep 9, 2024
| |S3 Local | `s3local-deploy-v2` |https://github.com/boozallen/aissemble/blob/{git-tree}/extensions/extensions-helm/aissemble-localstack-chart/README.md[S3 Local,role=external,window=_blank] | ||
| |Configuration Store | `configuration-store-deploy-v2` |https://github.com/boozallen/aissemble/blob/{git-tree}/extensions/extensions-helm/aissemble-configuration-store-chart/README.md[Configuration Store,role=external,window=_blank] | ||
| |Spark Operator | `aissemble-spark-operator-deploy-v2` |https://github.com/boozallen/aissemble/blob/{git-tree}/extensions/extensions-helm/aissemble-spark-operator-chart/README.md[Spark Operator,role=external,window=_blank] | ||
| |Application | Fermenter Profile | ReadMe |
There was a problem hiding this comment.
i: Only code format changes
cwoods-cpointe
commented
Sep 9, 2024
| @@ -11,61 +11,61 @@ helm install mlflow oci://ghcr.io/boozallen/aissemble-mlflow-chart --version <AI | |||
| # Properties | |||
| The following properties are inherited from the base [MLflow chart](https://github.com/bitnami/charts/blob/main/bitnami/mlflow/README.md), but with updated default values. | |||
|
|
|||
| | Property | Description | Default | | |||
There was a problem hiding this comment.
i: Only code format changes
cwoods-cpointe
commented
Sep 9, 2024
| @@ -21,6 +30,9 @@ sparkApp: | |||
| - name: ivy-cache | |||
| persistentVolumeClaim: | |||
| claimName: ivy-cache | |||
| - name: spark-events | |||
There was a problem hiding this comment.
I: Spark events were previously saved to s3. Changed to save to a PVC
There was a problem hiding this comment.
Should we include this new property in the README properties section?
There was a problem hiding this comment.
Good question. There is a section on the shared Ivy cache which is similarly a pvc. I can add a section to the readme for Spark-events
cwoods-cpointe
commented
Sep 9, 2024
| @@ -37,17 +37,17 @@ aissemble-spark-operator-chart: | |||
| webhook.enable: false | |||
| ``` | |||
|
|
|||
| | Property | Description | Required Override | Default | | |||
There was a problem hiding this comment.
i: Only code format changes
cwoods-cpointe
commented
Sep 9, 2024
| supplementalVolumeMounts: | ||
| - name: configurations | ||
| mountPath: "/configurations" | ||
| readOnly: true |
There was a problem hiding this comment.
i: resolved issue where the configuration-store PVC was not read only
JeffreyRoss
approved these changes
Sep 9, 2024
...extensions-helm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/README.md
Outdated
Show resolved
Hide resolved
...extensions-helm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/README.md
Outdated
Show resolved
Hide resolved
carter-cundiff
approved these changes
Sep 9, 2024
cwoods-cpointe
force-pushed
the
feature/spark-infrastructure-v2-default
branch
from
3f6ef81 to
a173fd1
Compare
Default all new applications to use spark-infrastructure to the V2 charts. Mark any V1 users with a depreciation warning. Provide migration instructions for going from V1 -> V2. Update spark-events to be saved to a more dynamic PVC instead of a S3 bucket. Signed-off-by: Peter McClonski <mcclonski_peter@bah.com> #300 Resolve several issues with the V2 charts Correct the Spark-event PV host node mount point. Update the spark configmap name to be in line with our documentaion (-conf -> -config). Corrected a bug in the Configuration-store PVC being writable instead of read only.
cwoods-cpointe
force-pushed
the
feature/spark-infrastructure-v2-default
branch
from
42ec405 to
20ecf37
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Default all new applications to use spark-infrastructure to the V2 charts. Mark any V1 users with a depreciation warning. Provide migration instructions for going from V1 -> V2.
Update spark-events to be saved to a more dynamic PVC instead of a S3 bucket.
Signed-off-by: Peter McClonski mcclonski_peter@bah.com
#300 Resolve several issues with the V2 charts
Correct the Spark-event PV host node mount point. Update the spark configmap name to be in line with our documentaion (-conf -> -config).
Corrected a bug in the Configuration-store PVC being writable instead of read only.