Code

feat(job): custom job scheduler#880

Code

Merged

tianweidut merged 119 commits intostar-whale:mainstar-whale/starwhale:mainfrom goldenxinxing:feat/job_3goldenxinxing/starwhale:feat/job_3Copy head branch name to clipboardAug 18, 2022

+1,711-1,030

Commits

Commits on Aug 2, 2022

• draft new job scheduler

  

  goldenxinxing

  authored and

  gaoxinxing

  committed
  c72ce28

  Copy full SHA for c72ce28

• tune example for mnist

  

  goldenxinxing

  authored and

  gaoxinxing

  committed
  adada06

  Copy full SHA for adada06

• update mnist model file

  

  gaoxinxing

  committed
  0832876

  Copy full SHA for 0832876

• refactor(eval):job dag

  

  gaoxinxing

  committed
  5d3d41e

  Copy full SHA for 5d3d41e

• fix yaml dump problem

  

  gaoxinxing

  committed
  c2f7653

  Copy full SHA for c2f7653

• run eval with existing version; single task execute

  

  gaoxinxing

  committed
  7522875

  Copy full SHA for 7522875

• dataset uri sent by param

  

  gaoxinxing

  committed
  53fa451

  Copy full SHA for 53fa451

• optimise entrypoint

  

  gaoxinxing

  committed
  58c9402

  Copy full SHA for 58c9402

• code format

  

  goldenxinxing

  authored and

  gaoxinxing

  committed
  d8d9c3a

  Copy full SHA for d8d9c3a

• feat:add default handler for evaluation

  

  gaoxinxing

  committed
  9849301

  Copy full SHA for 9849301

• update ignore

  

  gaoxinxing

  committed
  e7c2408

  Copy full SHA for e7c2408

• add default handler for eval

  

  gaoxinxing

  committed
  d1db3fd

  Copy full SHA for d1db3fd

Commits on Aug 3, 2022

• change job values to list

  

  gaoxinxing

  committed
  78a51e4

  Copy full SHA for 78a51e4

• job parser for controller

  

  gaoxinxing

  committed
  525ad32

  Copy full SHA for 525ad32

• job parser for controller

  

  gaoxinxing

  committed
  594439e

  Copy full SHA for 594439e

Commits on Aug 4, 2022

• controller:extract job content during upload;tune task entity(doing)

  

  gaoxinxing

  committed
  639ce3f

  Copy full SHA for 639ce3f

• change eval handler to model

  

  gaoxinxing

  committed
  f867dd7

  Copy full SHA for f867dd7

Commits on Aug 5, 2022

• change eval handler to model

  

  gaoxinxing

  committed
  23a2f76

  Copy full SHA for 23a2f76

• default pipeline use datastore(doing)

  

  gaoxinxing

  committed
  c42f496

  Copy full SHA for c42f496

• draft new job scheduler

  

  goldenxinxing

  authored and

  gaoxinxing

  committed
  236ffb9

  Copy full SHA for 236ffb9

• tune example for mnist

  

  goldenxinxing

  authored and

  gaoxinxing

  committed
  40fec7c

  Copy full SHA for 40fec7c

• model change

  

  gaoxinxing

  committed
  8c5c5d5

  Copy full SHA for 8c5c5d5

• refactor(eval):job dag

  

  gaoxinxing

  committed
  0620c74

  Copy full SHA for 0620c74

• fix yaml dump problem

  

  gaoxinxing

  committed
  47beade

  Copy full SHA for 47beade

• run eval with existing version; single task execute

  

  gaoxinxing

  committed
  18cc832

  Copy full SHA for 18cc832

• dataset uri sent by param

  

  gaoxinxing

  committed
  0ed6e20

  Copy full SHA for 0ed6e20

• optimise entrypoint

  

  gaoxinxing

  committed
  011c583

  Copy full SHA for 011c583

• code format

  

  goldenxinxing

  authored and

  gaoxinxing

  committed
  d626837

  Copy full SHA for d626837

• feat:add default handler for evaluation

  

  gaoxinxing

  committed
  22ddec6

  Copy full SHA for 22ddec6

• update ignore

  

  gaoxinxing

  committed
  9f49100

  Copy full SHA for 9f49100

• add default handler for eval

  

  gaoxinxing

  committed
  240a3c7

  Copy full SHA for 240a3c7

• change job values to list

  

  gaoxinxing

  committed
  743724e

  Copy full SHA for 743724e

• job parser for controller

  

  gaoxinxing

  committed
  42f927a

  Copy full SHA for 42f927a

• job parser for controller

  

  gaoxinxing

  committed
  556e5b3

  Copy full SHA for 556e5b3

• controller:extract job content during upload;tune task entity(doing)

  

  gaoxinxing

  committed
  b328f50

  Copy full SHA for b328f50

• change eval handler to model

  

  gaoxinxing

  committed
  ba8e0c9

  Copy full SHA for ba8e0c9

• change eval handler to model

  

  gaoxinxing

  committed
  bcd4fb4

  Copy full SHA for bcd4fb4

• default pipeline use datastore(doing)

  

  gaoxinxing

  committed
  aa9b3c7

  Copy full SHA for aa9b3c7

• Merge remote-tracking branch 'gxx-fork/feat/job_3' into feat/job_3

  Show description for fe2f362
  

  gaoxinxing

  committed
  fe2f362

  Copy full SHA for fe2f362

Commits on Aug 8, 2022

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  c508398

  Copy full SHA for c508398

• default pipeline support datastore

  

  gaoxinxing

  committed
  862725c

  Copy full SHA for 862725c

• fix:datastore dir bug; add manual dump method

  

  gaoxinxing

  committed
  271f9f9

  Copy full SHA for 271f9f9

Commits on Aug 9, 2022

• fix:datastore scan can't found results

  

  gaoxinxing

  committed
  e56a687

  Copy full SHA for e56a687

• feat:default ppl handler support datastore

  

  gaoxinxing

  committed
  f214d8f

  Copy full SHA for f214d8f

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  1d4ac2d

  Copy full SHA for 1d4ac2d

• code format

  

  gaoxinxing

  committed
  cf562b2

  Copy full SHA for cf562b2

• optimise code style

  

  gaoxinxing

  committed
  8d50d78

  Copy full SHA for 8d50d78

Commits on Aug 10, 2022

• add multi processing communicate:pipe

  

  gaoxinxing

  committed
  7598aeb

  Copy full SHA for 7598aeb

• code format

  

  gaoxinxing

  committed
  e2c1159

  Copy full SHA for e2c1159

Commits on Aug 11, 2022

• multiprocessing use duplex pipe communication

  

  gaoxinxing

  committed
  f5dfcdc

  Copy full SHA for f5dfcdc

• default eval step use simple way to coding

  

  gaoxinxing

  committed
  ca82147

  Copy full SHA for ca82147

Commits on Aug 12, 2022

• tune default ppl handler impl

  

  gaoxinxing

  committed
  f4b3ce3

  Copy full SHA for f4b3ce3

• skip eval unittest

  

  gaoxinxing

  committed
  e9c39ca

  Copy full SHA for e9c39ca

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  4a359b0

  Copy full SHA for 4a359b0

• fix controller var name bug

  

  gaoxinxing

  committed
  14b4a1c

  Copy full SHA for 14b4a1c

• controller unit test ignore

  

  gaoxinxing

  committed
  0e33165

  Copy full SHA for 0e33165

• conflict resolve

  

  gaoxinxing

  committed
  154e129

  Copy full SHA for 154e129

• fix mypy check error

  

  gaoxinxing

  committed
  e8d90b6

  Copy full SHA for e8d90b6

• fix code check style problem

  

  gaoxinxing

  committed
  6009ffa

  Copy full SHA for 6009ffa

• skip unit test

  goldenxinxing

  committed
  36a62d3

  Copy full SHA for 36a62d3

Commits on Aug 15, 2022

• unit test code fix

  

  gaoxinxing

  committed
  a850a21

  Copy full SHA for a850a21

• code format

  

  gaoxinxing

  committed
  1e5ebdd

  Copy full SHA for 1e5ebdd

• rollback mnist model file

  

  gaoxinxing

  committed
  3cf3126

  Copy full SHA for 3cf3126

• datastore key support blank

  

  gaoxinxing

  committed
  d1027f7

  Copy full SHA for d1027f7

• rollback metric key

  

  gaoxinxing

  committed
  8710e67

  Copy full SHA for 8710e67

• change comment

  

  gaoxinxing

  committed
  ef3a4ca

  Copy full SHA for ef3a4ca

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  b110d6e

  Copy full SHA for b110d6e

• remove some comment

  

  gaoxinxing

  committed
  78df99a

  Copy full SHA for 78df99a

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  2f80fd9

  Copy full SHA for 2f80fd9

• update e2e test script

  

  gaoxinxing

  committed
  085fcb8

  Copy full SHA for 085fcb8

• update e2e test script

  

  gaoxinxing

  committed
  f4f2894

  Copy full SHA for f4f2894

• update comment

  

  gaoxinxing

  committed
  febd85b

  Copy full SHA for febd85b

• update some usage advice

  

  gaoxinxing

  committed
  3a9dfae

  Copy full SHA for 3a9dfae

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  35dd002

  Copy full SHA for 35dd002

• skip test for ppl

  

  gaoxinxing

  committed
  33e0d62

  Copy full SHA for 33e0d62

• fix code check problem

  

  gaoxinxing

  committed
  b3f7496

  Copy full SHA for b3f7496

• fix code check problem

  

  gaoxinxing

  committed
  47fb644

  Copy full SHA for 47fb644

• fix code check problem

  

  gaoxinxing

  committed
  9c84f9e

  Copy full SHA for 9c84f9e

• fix code check problem

  

  gaoxinxing

  committed
  59ad081

  Copy full SHA for 59ad081

Commits on Aug 16, 2022

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  6f79405

  Copy full SHA for 6f79405

• fix code check problem

  

  gaoxinxing

  committed
  49a7b4c

  Copy full SHA for 49a7b4c

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  d447d56

  Copy full SHA for d447d56

• add logger for datastore

  

  gaoxinxing

  committed
  ae4149c

  Copy full SHA for ae4149c

• fit new datastore scan api

  

  gaoxinxing

  committed
  3ee1cc4

  Copy full SHA for 3ee1cc4

• support new dataset

  

  gaoxinxing

  committed
  6d4f9be

  Copy full SHA for 6d4f9be

Commits on Aug 17, 2022

• tune schedule way

  

  gaoxinxing

  committed
  1e10c73

  Copy full SHA for 1e10c73

• fix:datastore dump problem at multiprocessing pool

  

  gaoxinxing

  committed
  5a4759f

  Copy full SHA for 5a4759f

• standalone debug completed

  

  gaoxinxing

  committed
  68a3f62

  Copy full SHA for 68a3f62

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  52c6cbc

  Copy full SHA for 52c6cbc

• add ppl unit test

  

  gaoxinxing

  committed
  34606ec

  Copy full SHA for 34606ec

• fix bug for get metrics api

  

  gaoxinxing

  committed
  4ab200c

  Copy full SHA for 4ab200c

• add ppl unit test

  

  gaoxinxing

  committed
  b6ad82d

  Copy full SHA for b6ad82d

• use multithread

  

  gaoxinxing

  committed
  154e13d

  Copy full SHA for 154e13d

• fix unit test bug

  

  gaoxinxing

  committed
  331bc7e

  Copy full SHA for 331bc7e

Commits on Aug 18, 2022

• fix unit test bug

  

  gaoxinxing

  committed
  6016193

  Copy full SHA for 6016193

• code format

  

  gaoxinxing

  committed
  617b87f

  Copy full SHA for 617b87f

• skip test

  

  gaoxinxing

  committed
  9587ccc

  Copy full SHA for 9587ccc

• set concurrency for pool

  

  gaoxinxing

  committed
  0048e2c

  Copy full SHA for 0048e2c

• fix share object bug

  

  gaoxinxing

  committed
  d253ad5

  Copy full SHA for d253ad5

• use Protocol for Callback

  

  gaoxinxing

  committed
  cd780ec

  Copy full SHA for cd780ec

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  98fe7f2

  Copy full SHA for 98fe7f2

• code format

  

  gaoxinxing

  committed
  7f9ba31

  Copy full SHA for 7f9ba31

• @step use consts var

  

  gaoxinxing

  committed
  9cf6ad6

  Copy full SHA for 9cf6ad6

• code format

  

  gaoxinxing

  committed
  8e8571a

  Copy full SHA for 8e8571a

• step dependency -> needs

  

  gaoxinxing

  committed
  517d94b

  Copy full SHA for 517d94b

• remove unused init method

  

  gaoxinxing

  committed
  762b376

  Copy full SHA for 762b376

• extract common function to util package

  

  gaoxinxing

  committed
  00df15e

  Copy full SHA for 00df15e

• rename for entrypoint

  

  gaoxinxing

  committed
  0a66293

  Copy full SHA for 0a66293

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  7aecd17

  Copy full SHA for 7aecd17

• simply check

  

  gaoxinxing

  committed
  1dba50c

  Copy full SHA for 1dba50c

• use util's function for dump

  

  gaoxinxing

  committed
  200c6f7

  Copy full SHA for 200c6f7

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  0029a33

  Copy full SHA for 0029a33

• update requirement

  

  gaoxinxing

  committed
  623ff84

  Copy full SHA for 623ff84

• code format

  

  gaoxinxing

  committed
  4bf606a

  Copy full SHA for 4bf606a

• Merge branch 'main' into feat/job_3

  goldenxinxing

  authored
  f42beed

  Copy full SHA for f42beed

• job yaml optimise

  

  gaoxinxing

  committed
  7ae9fab

  Copy full SHA for 7ae9fab

• log format

  

  gaoxinxing

  committed
  ad19dca

  Copy full SHA for ad19dca

• optimise list

  

  gaoxinxing

  committed
  77fd490

  Copy full SHA for 77fd490

• code format

  

  gaoxinxing

  committed
  b990ba5

  Copy full SHA for b990ba5