Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a unified docker image for all Spark applications #161

Closed
dreamtalen opened this issue Feb 8, 2023 · 2 comments
Closed

Create a unified docker image for all Spark applications #161

dreamtalen opened this issue Feb 8, 2023 · 2 comments
Assignees
@tushartathgur
Labels
enhancement New feature or request
Milestone
Theia v0.5 release

Comments

@dreamtalen
Copy link

All of our Spark applications, including Policy Recommendation and Throughput Anomaly Detection, are built on the gcr.io/spark-operator/spark-py docker image and have their dependent libraries installed on top.

However, this image has a relatively large size of around 1GB. To save disk space on the user's node and prepare for the addition of more Spark applications in the future, we aim to create a unified docker image for all of these applications.
@dreamtalen dreamtalen added the enhancement New feature or request label Feb 8, 2023
@elton-furtado elton-furtado added this to the Theia v0.6 release milestone Feb 8, 2023
@tushartathgur tushartathgur self-assigned this Feb 13, 2023
@tushartathgur tushartathgur mentioned this issue Feb 28, 2023
Merged
@tushartathgur
Copy link
Contributor

Also restructure spark and other common constants from both TAD and NPR controllers to utils.go

@tushartathgur tushartathgur mentioned this issue Mar 2, 2023
Closed
tushartathgur referenced this issue in tushartathgur/theia Mar 2, 2023
Unified Dockerfile Image for Spark Jobs
2a222be 
Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
@tushartathgur tushartathgur mentioned this issue Mar 2, 2023
Merged
tushartathgur referenced this issue in tushartathgur/theia Mar 2, 2023
Unified Dockerfile Image for Spark Jobs
8cda64f 
Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
tushartathgur referenced this issue in tushartathgur/theia Mar 3, 2023
Unified Dockerfile Image for Spark Jobs
bff9182 
    Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.
    Unified Dockerimage now uses 1.4 GB of image space than earlier
2.4GB combined
    Spark Jobs constants have been moved under utils.go in order to
avoid duplicacy of constants

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
tushartathgur referenced this issue in tushartathgur/theia Mar 4, 2023
startTime: 2022-08-24T10:45:57Z
470ad43 
Unified Dockerfile Image for Spark Jobs

    Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.
    Unified Dockerimage now uses 1.4 GB of image space than earlier
2.4GB combined
    Spark Jobs constants have been moved under utils.go in order to
avoid duplicacy of constants

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
tushartathgur referenced this issue in tushartathgur/theia Mar 4, 2023
startTime: 2022-08-24T10:45:57Z
bd70023 
Unified Dockerfile Image for Spark Jobs

    Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.
    Unified Dockerimage now uses 1.4 GB of image space than earlier
2.4GB combined
    Spark Jobs constants have been moved under utils.go in order to
avoid duplicacy of constants

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
tushartathgur referenced this issue in tushartathgur/theia Mar 7, 2023
Unified Dockerfile Image for Spark Jobs
05d77bd 
    Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.
    Unified Dockerimage now uses 1.4 GB of image space than earlier
2.4GB combined
    Spark Jobs constants have been moved under utils.go in order to
avoid duplicacy of constants

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
tushartathgur added a commit that referenced this issue Mar 8, 2023
@tushartathgur
Unified Dockerfile Image for Spark Jobs (#174)
4ddf56b 
* Unified Dockerfile Image for Spark Jobs

    Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.
    Unified Dockerimage now uses 1.4 GB of image space than earlier
2.4GB combined
    Spark Jobs constants have been moved under utils.go in order to
avoid duplicacy of constants

Solve #161

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>

* Addressed comments for Unified Dockerfile

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>

---------

Signed-off-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
Co-authored-by: Tushar Tathgur <tathgurt@tathgurtFLVDL.vmware.com>
@tushartathgur
Copy link
Contributor

Code merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tushartathgur tushartathgur

Labels
enhancement New feature or request
Projects
None yet
Milestone
Theia v0.5 release
Development

When branches are created from issues, their pull requests are automatically linked.

161-create-a-unified-docker-image-for-all-spark-applications tushartathgur/theia
3 participants
@dreamtalen @elton-furtado @tushartathgur