Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update 2.1.0 #5515

Merged
merged 118 commits into from
Mar 6, 2024
Merged

Update 2.1.0 #5515

merged 118 commits into from
Mar 6, 2024

Conversation

mgqa34
Copy link
Contributor

@mgqa34 mgqa34 commented Feb 29, 2024 •
edited
Loading

Major Features and Improvements

Arch

  • Some bugs fixed for spark computing engine

Component

  • Unified IO keys naming format for all components
  • Add LLMLoader to support running FATE-LLM v2.0 with pipeline

OSX

  • Compatible with eggroll-v2.x

EggRoll

  • add 2.x api backport support
  • bug fix

FATE-Flow

Major Features and Improvements

  • Improved the display issue of output data.
  • Enhanced the PyPI package: configuration files have been relocated to the user's home directory, and the relative paths for uploading data are based on the user's home directory.
  • Added support for running FATE algorithms with Spark + Hadoop.

Bug-Fix

  • Fixed an issue where failed tasks could not be retried.
  • Fixed an issue where the system couldn't run when the task cores exceeded the system total cores.

FATE-Client

  • Pipeline: add supports for fate-llm 2.0
    • newly added LLMModelLoader, LLMDatasetLoader, LLMDataFuncLoader
    • newly added configuration parsing of seq2seq_runner and ot_runner
  • Pipeline: unified input interface of components

FATE-LLM

  • Adapt to fate-v2.0 framework:
    • Migrate parameter-efficient fine-tuning training methods and models.
    • Migrate Standard Offsite-Tuning and Extended Offsite-Tuning（Federated Offsite-Tuning+)
    • Newly trainer，dataset, data_processing function design
  • New FedKSeed Federated Tuning Algorithm: train large language models in a federated learning setting with extremely low communication cost

FATE-Test

  • Add Support for Job Runtime Configuration

sagewe and others added 30 commits January 5, 2024 19:11
@sagewe
partitoned before some action
454c9cd 
Signed-off-by: sagewe <wbwmat@gmail.com>
@sagewe
add subtractbykey
a255582 
Signed-off-by: sagewe <wbwmat@gmail.com>
@sagewe
fix: add trace for override method
ff8ed20 
Signed-off-by: sagewe <wbwmat@gmail.com>
@sagewe
fix: union preserve partition num
a247077 
Signed-off-by: sagewe <wbwmat@gmail.com>
@sagewe
fix: join and subtractByKey preserve partition num
90ba860 
Signed-off-by: sagewe <wbwmat@gmail.com>
@dylan-fan
fix join and subtractByKey
407945c 
Signed-off-by: dylan-fan <289765648@qq.com>
@mgqa34
dataframe: fix row retrieval depends deteministic partition
569aae8 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@Xiong-Li-github
init
ac6779c 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@mgqa34
dataframe: fix row retrieval as key is unorered when using spark
16e131c 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@dylan-fan
Merge branch 'bugfix-2.0.0-rc-spark-partitioned' of https://github.co…
8379cbb 
…m/FederatedAI/FATE into bugfix-2.0.0-rc-spark-partitioned

� Conflicts:
�	python/fate/arch/computing/backends/spark/_table.py
@Xiong-Li-github
v1.x
ffa59f7 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@dylan-fan
fix spark bug
ee7e3c2 
Signed-off-by: dylan-fan <289765648@qq.com>
@sagewe
chore: format with black
e201615 
Signed-off-by: sagewe <wbwmat@gmail.com>
@sagewe
revert join
be8fdd3 
Signed-off-by: sagewe <wbwmat@gmail.com>
@sagewe
Merge pull request #5462 from FederatedAI/spark-fix-2.1-dy
6edf87b 
Spark fix 2.1 dy
@dylan-fan
fix spark bug for sample api
9ae9d57 
Signed-off-by: dylan-fan <289765648@qq.com>
@dylan-fan
Merge remote-tracking branch 'origin/spark-fix-2.1-dy' into spark-fix…
dcad51e 
…-2.1-dy
@dylan-fan
Merge pull request #5463 from FederatedAI/spark-fix-2.1-dy
fc838f5 
Spark fix 2.1 dy
@dylan-fan
Merge pull request #5439 from FederatedAI/bugfix-2.0.0-rc-spark-parti…
c5ce454 
…tioned

bugfix 2.0.0 rc spark partitioned
@Xiong-Li-github
create v2 package
0e85473 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@Xiong-Li-github
add v3
0cf14f3 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@dylan-fan
add osx timeout config
6591f72 
Signed-off-by: dylan-fan <289765648@qq.com>
@dylan-fan
Merge pull request #5465 from FederatedAI/spark-fix-2.1-dy
82550be 
add osx timeout config
@sagewe
fix: parallel computing blocks by lmdb write in shuffle without reduce
58ceec5 
since lmdb not allow using multiple writer by default, when shuffle without reduce, we need to
1. write shuffled values to immediate output table for each (input partition, output_partition) pairs
2. copy (*, output_partition) to output

Signed-off-by: sagewe <wbwmat@gmail.com>
@Xiong-Li-github
new v2 module
9f64e6e 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@Xiong-Li-github
init v3
fd377fe 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@Xiong-Li-github
add v3
b41517f 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@Xiong-Li-github
update v3
d59cf27 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@Xiong-Li-github
update v2&v3
f63a324 
Signed-off-by: v_wbxiongli <740332065@qq.com>
@Xiong-Li-github
update service.sh & package
960eaae 
Signed-off-by: v_wbxiongli <740332065@qq.com>
mgqa34 added 4 commits March 1, 2024 14:05
@mgqa34
update performance doc
f06460e 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@mgqa34
Merge pull request #5517 from FederatedAI/feature-2.1.0-update_dag_doc
a733743 
update performance doc
@mgqa34
fix performance doc
a43bcfd 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@mgqa34
update performance doc
629bd84 
Signed-off-by: mgqa34 <mgq3374541@163.com>
dylan-fan
dylan-fan approved these changes Mar 4, 2024
View reviewed changes
Copy link
Collaborator

@dylan-fan dylan-fan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

LaynePeng
LaynePeng approved these changes Mar 4, 2024
View reviewed changes
Copy link
Contributor

@LaynePeng LaynePeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jarviszeng-zjc
Copy link
Contributor

LGTM

snowzjx
snowzjx approved these changes Mar 4, 2024
View reviewed changes
Copy link

@snowzjx snowzjx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

hainingzhang
hainingzhang approved these changes Mar 4, 2024
View reviewed changes
Copy link
Collaborator

@hainingzhang hainingzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

mgqa34 and others added 17 commits March 5, 2024 09:15
@mgqa34
update readme of doc
c721d72 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@talkingwallace
Fix agg type in component Signed-off-by: weijingchen <talkingwallace@…
dd66419 
…sohu.com>

Signed-off-by: cwj <talkingwallace@sohu.com>
@talkingwallace
Update codes Signed-off-by: weijingchen <talkingwallace@sohu.com>
0699682 
Signed-off-by: cwj <talkingwallace@sohu.com>
@talkingwallace
fix Signed-off-by: weijingchen <talkingwallace@sohu.com>
7fc67d2 
Signed-off-by: cwj <talkingwallace@sohu.com>
@nemirorox
rename examples pipeline doc
52e8e9d 
Signed-off-by: Yu Wu <yolandawu131@gmail.com>
@mgqa34
Merge pull request #5528 from FederatedAI/dev-2.1.0-nn-agg-fix
4638dcd 
Fix agg type in component Signed-off-by: weijingchen <talkingwallace@…
@mgqa34
Merge pull request #5529 from FederatedAI/feature-2.1.0-union-input
005d9a1 
rename examples pipeline doc
@mgqa34
fix data link interface
addc368 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@mgqa34
fix tutorial
8a9baac 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@mgqa34
Merge pull request #5527 from FederatedAI/feature-2.1.0-update_dag_doc
ba6d1a0 
update readme of doc
@talkingwallace
update bq Signed-off-by: weijingchen <talkingwallace@sohu.com>
ed51d35 
Signed-off-by: cwj <talkingwallace@sohu.com>
@talkingwallace
fix doc Signed-off-by: weijingchen <talkingwallace@sohu.com>
57b0fba 
Signed-off-by: cwj <talkingwallace@sohu.com>
@mgqa34
Merge pull request #5533 from FederatedAI/dev-2.1.0-bq-fix
1ebee50 
update bq Signed-off-by: weijingchen <talkingwallace@sohu.com>
@nemirorox
benchmark config adopt newer param format for sklearn
caf7b5d 
Signed-off-by: Yu Wu <yolandawu131@gmail.com>
@mgqa34
Merge pull request #5535 from FederatedAI/feature-2.1.0-bq-sklearn-up…
6ec71cf 
…date

benchmark config adopt newer param format for sklearn
@mgqa34
update submodules of fate-v2.1.0
bfb1dab 
Signed-off-by: mgqa34 <mgq3374541@163.com>
@mgqa34
Merge pull request #5537 from FederatedAI/feature-2.1.0-update_submod…
545a326 
…ules

update submodules of fate-v2.1.0
@mgqa34 mgqa34 merged commit fe7674c into master Mar 6, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hainingzhang hainingzhang hainingzhang approved these changes

@dylan-fan dylan-fan dylan-fan approved these changes

@LaynePeng LaynePeng LaynePeng approved these changes

@snowzjx snowzjx snowzjx approved these changes

@jarviszeng-zjc jarviszeng-zjc Awaiting requested review from jarviszeng-zjc

@vistakk vistakk Awaiting requested review from vistakk

@excellent-lixl excellent-lixl Awaiting requested review from excellent-lixl

Assignees

@talkingwallace talkingwallace

@mgqa34 mgqa34

@sagewe sagewe

@nemirorox nemirorox

@zhihuiwan zhihuiwan

@Xiong-Li-github Xiong-Li-github

Labels
arch Architecture related bug Something isn't working enhancement New feature or request examples
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.