Support multiphase in PAIK8S #2368
Conversation
merge master
merge master
Update evolution doc (microsoft#1493)
merge master
merge master
merge master
augment pylintrc (microsoft#1643)
fix console.log (microsoft#1636)
merge master
merge master
merge master
merge master
Filter prune algo implementation (microsoft#1655)
merge master
merge master
merge master
merge master
merge master
merge master
merge master
merge master
merge master
merge master
Support monitor mode when creating or resuming a new experiment (microsoft#1933)
Add test for documentation build (microsoft#1924)
fix pipeline status badge (microsoft#1942)
merge master
merge master
merge master
merge master
merge master
merge master
merge master
Fix trialkeeper flush (microsoft#2174)
merge master
merge master
mege master
merge master
merge master
merge master
| if (trialJobDetail === undefined) { | ||
| throw new Error(`updateTrialJob failed: ${trialJobId} not found`); | ||
| } | ||
| // Write file content ( parameter.cfg ) to working folders | ||
| await this.writeParameterFile(trialJobDetail.logPath, form.hyperParameters); |
There was a problem hiding this comment.
why write to logPath instead of trail job's working directory?
There was a problem hiding this comment.
This path will synchronize parameter files in local to docker container, it is the working directory for PAI in docker container.
There was a problem hiding this comment.
then is trialjob.workingDirectory useful now?
There was a problem hiding this comment.
In PAITrainingService, there are two directories. One is workingDirectory, this directory stores nniManager logs and datastore data, the default path is located in ~/nni/experiments/{experimentId}. Another path is logPath, which could synchronize data with docker container, store output data from trial and parameter files, the default path is located in {nniManagerNFSMountPath}/{experimentId}/{trialId}. The parameter.cfg file should be put in logPath folder instead of workingDirectory .
There was a problem hiding this comment.
@SparkSnail so why not merge the two directory together? according to my understanding in local mode there is only one directory right?
There was a problem hiding this comment.
As discussed offline, NNI maintains workingDirectory and logPath in PAI mode. This structure maybe need refactor, this pr will keep current logic temporarily.
|
please enable multiphase test case in pai and pai-windows pipeline |
fixed. |
|
@SparkSnail is there UT or IT for testing multiphase on paiK8S? |
Yes, pipeline has enabled multiphase examples on paiK8S in this pr. |
No description provided.