Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge develop-1.9.0 into master for release purpose (#738)
* fix issue, can not upload file on namenode and change fixed value to variable on istio gateway (#601) * fix issue, can not upload file on namenode Signed-off-by: stone-wlg <stone_wlg@163.com> * fix issue, change fixed value to variable on istio gateway Signed-off-by: stone-wlg <stone_wlg@163.com> * add default log level for fateflow service (#609) * Make the tutorial up to date with 1.7.2 on master branch (#570) * optimize the tutorial and make sure the example python scripts on notebook can run Signed-off-by: Chen Jing <jingch@vmware.com> * Change the wrong version in tutorial and remove a useless file Signed-off-by: Chen Jing <jingch@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> Signed-off-by: Sun Xia <asdfsx@gmail.com> * Update contributing document (#575) * Update the contributing.md document for license and how to build, test. Signed-off-by: Layne Peng <playne@vmware.com> * Add no-code way contributing part. Signed-off-by: Layne Peng <playne@vmware.com> * Update the license up to 2022. Signed-off-by: Layne Peng <playne@vmware.com> Signed-off-by: Sun Xia <asdfsx@gmail.com> * export variable cacheType Signed-off-by: Sun Xia <asdfsx@gmail.com> * fix bug Signed-off-by: Sun Xia <asdfsx@gmail.com> * add default log level for fateflow service Signed-off-by: Sun Xia <asdfsx@gmail.com> Co-authored-by: Chen <paulkim23@163.com> Co-authored-by: Chen Jing <jingch@vmware.com> Co-authored-by: Layne Peng <playne@vmware.com> * fix issue, fateflow logging error (#610) Signed-off-by: stone-wlg <stone_wlg@163.com> * Resolve the log flood issue of kubefate pod (#612) * Resolve the log flood issue of kubefate pod Signed-off-by: Chen Jing <jingch@vmware.com> * Use TCP for the kubefate probe Signed-off-by: Chen Jing <jingch@vmware.com> * Make it possible to upload dataset which is more than 100M in k8s spark deployment (#614) * Change hdfs datanode to statefulset and fix the rabbitmq example Signed-off-by: Chen Jing <jingch@vmware.com> * Optimize the name of the template Signed-off-by: Chen Jing <jingch@vmware.com> * Make the replicas of datanode configurable Signed-off-by: Chen Jing <jingch@vmware.com> * Make sure in docker compose mode, we can upload a dataset > 100M size (#617) * add the hdfs datanode number to 3 Signed-off-by: Chen Jing <jingch@vmware.com> * add more binary file types to avoid auto-convertion on a linux machine Signed-off-by: Chen Jing <jingch@vmware.com> * fix issues Signed-off-by: Chen Jing <jingch@vmware.com> * fix: Remove variable 'ingressHost' that is no longer in use (#620) Signed-off-by: magic-hya <huangya@asiainfo.com> * fix the condition: any issue of pulsar docker compose file (#627) Signed-off-by: Chen Jing <jingch@vmware.com> * add/bind federatedml dir/volumn in eggroll containers (#629) Signed-off-by: gxcuit <gxcuit@163.com> Co-authored-by: Chen Jing <jingch@vmware.com> * optmized the pr template (#630) * optmized the pr template Signed-off-by: Chen Jing <jingch@vmware.com> * fix typo Signed-off-by: Chen Jing <jingch@vmware.com> * optimize the docs: part I (#646) Signed-off-by: Chen Jing <jingch@vmware.com> * Feat: add docker build of fate_test (#637) * add docker build of fate_test Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * update doc Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Fix fateflow's model_local_cache path (#639) * Fix fateflow's model_local_cache path Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * fix fateflow's jobs cache path Signed-off-by: Chenlong Ma <chenlongm@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> * Update the docs of Slim FATE (#632) * Update the docs of Slim FATE Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Add comparison between different backends Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Update docker-deploy/README.md accurate expression Co-authored-by: Chen Jing <jingch@vmware.com> Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Add English documentation for Introduction_to_Backend_Architecture Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Update docker-deploy/README.md Co-authored-by: Chen Jing <jingch@vmware.com> Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Fix broken URLs Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * resolve comments Signed-off-by: Chenlong Ma <chenlongm@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> * Do not let datanodes to share one pvc (#650) * Do not let datanodes to share one pvc Signed-off-by: Chen Jing <jingch@vmware.com> * fix the and grammar Signed-off-by: Chen Jing <jingch@vmware.com> * Fix an format issue Signed-off-by: Chen Jing <jingch@vmware.com> * Fix an un expected typo Signed-off-by: Chen Jing <jingch@vmware.com> * Support the validation for cluster.yaml (#636) * add ValuesTemplateExample to HelmChart Model Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add Vadd getValueTemplateExample api Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add Vvalidation implement and tests Signed-off-by: hang lv <xlv20@fudan.edu.cn> * req and resp in cli side Signed-off-by: hang lv <xlv20@fudan.edu.cn> * reqauto generated doc Signed-off-by: hang lv <xlv20@fudan.edu.cn> * bug fix Signed-off-by: hang lv <xlv20@fudan.edu.cn> * remove test case from local file Signed-off-by: hang lv <xlv20@fudan.edu.cn> * bug fix Signed-off-by: hang lv <xlv20@fudan.edu.cn> * bug fix Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Optimize the docs: part II (#654) * Optimize the docs: part II Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * fix change deployment to statefulset in nodemanager (#653) * fix change deployment to statefulset in nodemanager Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * remove default value of nodeSelector tolerations and affinity Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * optimize the docs: part III (#655) * optimize the docs: part III Signed-off-by: Chen Jing <jingch@vmware.com> * Fix a mistake Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * feat add ssh port configuration in docker deploy (#656) Signed-off-by: magic-hya <huangya@asiainfo.com> * Fix the wrong example link of Notebook (#659) Signed-off-by: Chenyang Gao <gps949@outlook.com> change the example link from http://party9999.fateboard.example.com to http://party9999.notebook.example.com Co-authored-by: Chen Jing <jingch@vmware.com> * Fix the typo of README.txt (#660) Signed-off-by: Chenyang Gao <gps949@outlook.com> fixed the mistake spell of "shown" Co-authored-by: Chen Jing <jingch@vmware.com> * modify eggroll manager ports (#661) Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Remove service url scheme hardcode in cli (#663) Signed-off-by: Chenyang Gao <gps949@outlook.com> There's hardcode scheme(http://) of KubeFATE service url in Kubefate cli, so that it can only support service without TLS (http & ws). It's neither graceful nor security. This commit removes the hardcode and add a new option in the config.yaml. (Modified the README accordingly) * fix configuration PATH of job_default_config.yaml (#667) Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Reduce the necessary size of the spark base image (#669) Signed-off-by: Chen Jing <jingch@vmware.com> * Signed-off-by: Chenyang Gao <gps949@outlook.com> (#670) Add doc about how to config TLS kubefate service Few days ago, I made a PR to remove the hardcode URL scheme in the cli. To give more help to the guys want to deploy the kubefate service with a TLS, I add a markdown doc as tutorial. It's not so detail. But may be a very first step for the ones may be interested in. * alert user if the modules not match backend (#674) * alert user if the modules not match backend Signed-off-by: hang lv <xlv20@fudan.edu.cn> * bug fix for UT Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add supportbundle function to cli (#672) * add supportbundle function to cli Signed-off-by: hang lv <xlv20@fudan.edu.cn> * split supportbundle module into 3 parts Signed-off-by: hang lv <xlv20@fudan.edu.cn> * bug fix and more field filled Signed-off-by: hang lv <xlv20@fudan.edu.cn> * correct spell of comment * detail ingress on rules, ports and address Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Merge branch 'fix-issue-597' of github.com:n063h/KubeFATE into fix-issue-597 Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Unit Test for k8sclient and supportbundle (#676) * add supportbundle function to cli Signed-off-by: hang lv <xlv20@fudan.edu.cn> * split supportbundle module into 3 parts Signed-off-by: hang lv <xlv20@fudan.edu.cn> * bug fix and more field filled Signed-off-by: hang lv <xlv20@fudan.edu.cn> * correct spell of comment * detail ingress on rules, ports and address Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Merge branch 'fix-issue-597' of github.com:n063h/KubeFATE into fix-issue-597 Signed-off-by: hang lv <xlv20@fudan.edu.cn> * unit test for k8sclient Signed-off-by: hang lv <xlv20@fudan.edu.cn> * unit test for supportbundle Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Support defining resource requirements for rabbitmq, pulsar and nodemanager (#675) * Make resource customizable for eggroll, pulsar and rabbitmq Signed-off-by: Chen Jing <jingch@vmware.com> * Improvement Signed-off-by: Chen Jing <jingch@vmware.com> * fix typo Signed-off-by: Chen Jing <jingch@vmware.com> * remove warning on chartVersion (#678) Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Add the proposal for rolling upgrade (#679) Signed-off-by: Chen Jing <jingch@vmware.com> * Fix a typo everywhere (#680) Signed-off-by: Chen Jing <jingch@vmware.com> * Optimize the proposal (#683) Signed-off-by: Chen Jing <jingch@vmware.com> * Change the deployments to statefulSets for whoever has a PVC (#681) * Change the deployments to statefulSets for whoever has a PVC Signed-off-by: Chen Jing <jingch@vmware.com> * Fix issues Signed-off-by: Chen Jing <jingch@vmware.com> * Fix a mistake of the chart Signed-off-by: Chen Jing <jingch@vmware.com> * add service names for the yaml files of the sts Signed-off-by: Chen Jing <jingch@vmware.com> * set the default replicas for 2 for node manager, set default storage for pulsar Signed-off-by: Chen Jing <jingch@vmware.com> * fix typo Signed-off-by: Chen Jing <jingch@vmware.com> * add default replicas for hdfs datanode Signed-off-by: Chen Jing <jingch@vmware.com> * add default values for hdfs Signed-off-by: Chen Jing <jingch@vmware.com> * fix typo Signed-off-by: Chen Jing <jingch@vmware.com> * Part 1: Support getting the list of the scripts by the new/old versions (#686) * Support getting the list of the scripts by the versions Signed-off-by: Chen Jing <jingch@vmware.com> * fix issues Signed-off-by: Chen Jing <jingch@vmware.com> * address comments Signed-off-by: Chen Jing <jingch@vmware.com> * fix an error msg Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * chart name should be fate not FATE Signed-off-by: Chen Jing <jingch@vmware.com> * fix an issue Signed-off-by: Chen Jing <jingch@vmware.com> * Part 2: Implement the upgrade manager (#688) * add the docker file and the helm chart for fate upgrade manager Signed-off-by: Chen Jing <jingch@vmware.com> * add image pull policy for the template Signed-off-by: Chen Jing <jingch@vmware.com> * optimize the job name to make it include the versions Signed-off-by: Chen Jing <jingch@vmware.com> * make the default version a normal version string Signed-off-by: Chen Jing <jingch@vmware.com> * move the docker file stuff to fate builder Signed-off-by: Chen Jing <jingch@vmware.com> * Part 3: Fum shutdown flow (#690) * initial checkin Signed-off-by: Chen Jing <jingch@vmware.com> * update helm chart Signed-off-by: Chen Jing <jingch@vmware.com> * update helm chart Signed-off-by: Chen Jing <jingch@vmware.com> * update helm chart Signed-off-by: Chen Jing <jingch@vmware.com> * Change clusterrole to role Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * fix issues Signed-off-by: Chen Jing <jingch@vmware.com> * Kubefate support rolling upgrade (#691) * initial checkin Signed-off-by: Chen Jing <jingch@vmware.com> * initial checkin Signed-off-by: Chen Jing <jingch@vmware.com> * Implemetation, happy path test passed Signed-off-by: Chen Jing <jingch@vmware.com> * Fix self-review mistakes Signed-off-by: Chen Jing <jingch@vmware.com> * submit go mod and go sum Signed-off-by: Chen Jing <jingch@vmware.com> * fix UT Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * make the code beautiful (#694) Signed-off-by: Chen Jing <jingch@vmware.com> * Part 1 : fixed an issue which caused fate to not work with istio (#696) * fixed an issue which caused fate to not work with istio Signed-off-by: hang lv <xlv20@fudan.edu.cn> * annotate rewriteAppHTTPProbers only if istio enabled Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Fix the issue found when try to uprade to v1.9.0 (#697) Signed-off-by: Chen Jing <jingch@vmware.com> * Fix the issue that we failed to stop upgrading from v1.7.1 (#698) Signed-off-by: Chen Jing <jingch@vmware.com> * fix time difference between host and container (#689) Signed-off-by: Zhang Hui <zhang_hui0019@163.com> Co-authored-by: Chen Jing <jingch@vmware.com> * Add the doc for upgrade (#700) * Add the doc for upgrade Signed-off-by: Chen Jing <jingch@vmware.com> * Polish the doc Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * clean up the build code in KubeFATE (#701) * clean up the build code in KubeFATE Signed-off-by: Chen Jing <jingch@vmware.com> * address comment Signed-off-by: Chen Jing <jingch@vmware.com> * Support TLS for eggroll: p2p mode (#703) Signed-off-by: Chen Jing <jingch@vmware.com> * [2/3][3/3] Support TLS for eggroll: exchange mode and docker-compose (#705) * Support TLS for eggroll: exchange mode Signed-off-by: Chen Jing <jingch@vmware.com> * fix comments Signed-off-by: Chen Jing <jingch@vmware.com> * set swagger default auth (#710) Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add tls support for gateways with istio (#709) * add tls support for gateways with istio Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add guide and fix credential name Signed-off-by: hang lv <xlv20@fudan.edu.cn> * modified guid Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Update docs/Secure_Gateways_with_Istio Co-authored-by: Chen Jing <jingch@vmware.com> * Update Eggroll_with_TLS.md (#711) Fix some issue about the doc * Feat: add images select for docker-compose and chart (#706) * docker-compose supports selecting images according to configuration Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * k8s support images select Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * update docs Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * update for comments Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * fix serving-tag Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * fix typo Signed-off-by: Chenlong Ma <chenlongm@vmware.com> Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Create document of deploying FATE cluster with admin role in certain namespace (#718) * Add document on how to deploy and use kubefate in k8s mode when users only has admin roles in certain namespaces(#708) Signed-off-by: fwenqian <fwenqian@vmware.com> * Update some description in Deploy_FATE_Cluster_with_Admin_Role_in_Certain_Namespace.md Signed-off-by: fwenqian <fwenqian@vmware.com> * Update docs/Deploy_FATE_Cluster_with_Admin_Role_in_Certain_Namespace.md Signed-off-by: fwenqian <fwenqian@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> * Update error messages when listing namespaces or getting node ip fails with admin role in particular namespace. (#721) Update error messages when listing namespaces or getting node ip fails. Signed-off-by: fwenqian <fwenqian@vmware.com> * Feat: KubeFATE support FATE_Serving v2.1.6 (#722) * Feat: KubeFATE support FATE-Serving v2.1.6 Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Fix: Serving api return "serviceId is not bind model" after pod restarted Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * fix volumes path Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * fix: zookeeper cannot create directory '/bitnami/zookeeper/data': Permission denied Signed-off-by: Chenlong Ma <chenlongm@vmware.com> Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Fix chart format (#720) * fix chart format to meet validation requirement Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add skippedkeys Signed-off-by: hang lv <xlv20@fudan.edu.cn> Signed-off-by: hang lv <xlv20@fudan.edu.cn> Co-authored-by: Chen Jing <jingch@vmware.com> * Fix validate issue (#714) * error message if chart not found Signed-off-by: hang lv <xlv20@fudan.edu.cn> * change pattern for validation Signed-off-by: hang lv <xlv20@fudan.edu.cn> * check module&backend Signed-off-by: hang lv <xlv20@fudan.edu.cn> * UT for modules&backend validation Signed-off-by: hang lv <xlv20@fudan.edu.cn> Signed-off-by: hang lv <xlv20@fudan.edu.cn> Co-authored-by: Chen Jing <jingch@vmware.com> * remove imageTag checks & leave it to default configured in values.yaml (#723) * remove imageTag checks & leave it to default configured in values.yaml Signed-off-by: fwenqian <fwenqian@vmware.com> * remove all imageTag related content Signed-off-by: fwenqian <fwenqian@vmware.com> Signed-off-by: fwenqian <fwenqian@vmware.com> Co-authored-by: Chen Jing <jingch@vmware.com> * Fix validation issue (#729) * bug fix Signed-off-by: hang lv <xlv20@fudan.edu.cn> * log skippedError Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add more test and fix backend validation bug Signed-off-by: hang lv <xlv20@fudan.edu.cn> Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Fix validate issue (#732) * yaml storage default setting corrected Signed-off-by: hang lv <xlv20@fudan.edu.cn> * fix a bug which cause list obj returns no error Signed-off-by: hang lv <xlv20@fudan.edu.cn> * allow empty value & add test Signed-off-by: hang lv <xlv20@fudan.edu.cn> Signed-off-by: hang lv <xlv20@fudan.edu.cn> * Support FATE v1.9.0 (#712) * Support FATE v1.9.0 Signed-off-by: Chen Jing <jingch@vmware.com> * Fix mistakes Signed-off-by: Chen Jing <jingch@vmware.com> * add a 1.9 config in the cm of python Signed-off-by: Chen Jing <jingch@vmware.com> * bump kubefate version Signed-off-by: Chen Jing <jingch@vmware.com> * support spark 3 Signed-off-by: Chen Jing <jingch@vmware.com> * change the port of probe of hadoop Signed-off-by: Chen Jing <jingch@vmware.com> * change the port of probe of datanode Signed-off-by: Chen Jing <jingch@vmware.com> * update pulsar version per Webank's request Signed-off-by: Chen Jing <jingch@vmware.com> * fix a bug of service_conf Signed-off-by: Chen Jing <jingch@vmware.com> * fix the bug of spark-local Signed-off-by: Chen Jing <jingch@vmware.com> * fix the bug of spark-local Signed-off-by: Chen Jing <jingch@vmware.com> * fix the bug of spark-local Signed-off-by: Chen Jing <jingch@vmware.com> * fix the bug of spark-local Signed-off-by: Chen Jing <jingch@vmware.com> * fix the docker_deploy file Signed-off-by: Chen Jing <jingch@vmware.com> * add 2 job configs for docker compose Signed-off-by: Chen Jing <jingch@vmware.com> * fix little issues Signed-off-by: Chen Jing <jingch@vmware.com> * address comments Signed-off-by: Chen Jing <jingch@vmware.com> * address comments Signed-off-by: Chen Jing <jingch@vmware.com> * typo Signed-off-by: Chen Jing <jingch@vmware.com> Signed-off-by: Chen Jing <jingch@vmware.com> * Fix issue 719 (#731) * allow users to customize pulsar env Signed-off-by: hang lv <xlv20@fudan.edu.cn> * fix format error Signed-off-by: hang lv <xlv20@fudan.edu.cn> * allow users to customize pulsar confs Signed-off-by: hang lv <xlv20@fudan.edu.cn> * add/delete some conf in yaml Signed-off-by: hang lv <xlv20@fudan.edu.cn> * update pulsa mappings to doc Signed-off-by: hang lv <xlv20@fudan.edu.cn> * update doc Signed-off-by: hang lv <xlv20@fudan.edu.cn> Signed-off-by: hang lv <xlv20@fudan.edu.cn> * update docs for rm backend (#733) * update docs for rm backend Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Revert its Signed-off-by: Chenlong Ma <chenlongm@vmware.com> Signed-off-by: Chenlong Ma <chenlongm@vmware.com> * Update the tutorial doc (#734) * Update the tutorial doc Signed-off-by: Chen Jing <jingch@vmware.com> * address comments Signed-off-by: Chen Jing <jingch@vmware.com> * modify ip address Signed-off-by: Chen Jing <jingch@vmware.com> Signed-off-by: Chen Jing <jingch@vmware.com> * fix a little typo (#737) Signed-off-by: Chen Jing <jingch@vmware.com> Signed-off-by: Chen Jing <jingch@vmware.com> Signed-off-by: stone-wlg <stone_wlg@163.com> Signed-off-by: Chen Jing <jingch@vmware.com> Signed-off-by: magic-hya <huangya@asiainfo.com> Signed-off-by: Chenlong Ma <chenlongm@vmware.com> Signed-off-by: hang lv <xlv20@fudan.edu.cn> Signed-off-by: fwenqian <fwenqian@vmware.com> Co-authored-by: stone.wlg <stone_wlg@163.com> Co-authored-by: Sun Xia <asdfsx@gmail.com> Co-authored-by: Layne Peng <playne@vmware.com> Co-authored-by: magic-hya <huangya@asiainfo.com> Co-authored-by: gxcuit <gxcuit@163.com> Co-authored-by: owlet42 <owlet42@126.com> Co-authored-by: hang lv <xlv20@fudan.edu.cn> Co-authored-by: Chenyang Gao <gps949@outlook.com> Co-authored-by: Zhang Hui <zhang_hui0019@163.com> Co-authored-by: fwenqian <50017882+Loaking@users.noreply.github.com>
- Loading branch information
11 people
authored
Aug 30, 2022
1 parent
d710a7d
commit a7d8839