Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make pending timeout customizable #1268

Merged
merged 2 commits into from
May 5, 2019
Merged

make pending timeout customizable #1268

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bdb11c9
make pending timeout customizable
cheyang May 1, 2019
3b911d1
fix the description of arg
cheyang May 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions components/arena/docker/arena_launcher.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,10 @@ def main(argv=None):
parser.add_argument('--tensorboard-image', type=str, default='tensorflow/tensorflow:1.12.0')
parser.add_argument('--timeout-hours', type=int,
default=200,
help='Time in hours to wait for the Job submitted by arena to complete')
help='Time in minutes to wait for the Job submitted by arena to complete')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be hours, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because training always takes more than 1 hour.

parser.add_argument('--pending-timeout-minutes', type=int,
default=360,
help='Time in hours to wait for the Job submitted by arena from pending to running')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The var says --pending-timeout-minutes, whereas the descriptions says hours?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for spotting it！ Fix it.

# parser.add_argument('--command', type=str)
parser.add_argument('--output-dir', type=str, default='')
parser.add_argument('--output-data', type=str, default='None')
Expand Down Expand Up @@ -321,7 +324,8 @@ def main(argv=None):

# wait for job done
# _wait_job_done(fullname, job_type, datetime.timedelta(minutes=timeout_hours))
_wait_job_running(fullname, job_type, datetime.timedelta(minutes=30))
pending_timeout_minutes = args.pending_timeout_minutes
_wait_job_running(fullname, job_type, datetime.timedelta(minutes=pending_timeout_minutes))

rc = _job_logging(fullname, job_type)
logging.info("rc: {0}".format(rc))
Expand Down
6 changes: 3 additions & 3 deletions components/arena/python/arena/_arena_distributed_tf_op.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def estimator_op(name, image, command,
evaluator=False, evaluator_cpu_limit=0, evaluator_memory_limit=0,
env=[], data=[], sync_source=None,
metrics=['Train-accuracy:PERCENTAGE'],
arena_image='cheyang/arena_launcher:v0.5',
arena_image='cheyang/arena_launcher:v0.6',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these images be moved to a more neutral docker registry name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with your suggestions. I'd like to move it to a neutral docker registry name. I will keep it in mind and try to find. Thanks.

timeout_hours=240):

"""This function submits Distributed TFJob in Estimator mode.
Expand Down Expand Up @@ -62,7 +62,7 @@ def parameter_servers_op(name, image, command, env, data, sync_source, annotatio
tensorboard,
worker_port, ps_port,
metrics=['Train-accuracy:PERCENTAGE'],
arena_image='cheyang/arena_launcher:v0.5',
arena_image='cheyang/arena_launcher:v0.6',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above

timeout_hours=240):

"""This function submits Distributed TFJob in Parameter Servers mode.
Expand Down Expand Up @@ -97,7 +97,7 @@ def distributed_tf_op(name, image, command, env=[], data=[], sync_source=None,
ps_port=22224,
tensorboard=False,
metrics=['Train-accuracy:PERCENTAGE'],
arena_image='cheyang/arena_launcher:v0.5',
arena_image='cheyang/arena_launcher:v0.6',
timeout_hours=240):
"""This function submits Distributed TFJob in Distributed mode.
Expand Down
2 changes: 1 addition & 1 deletion components/arena/python/arena/_arena_mpi_op.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def mpi_job_op(name, image, command, workers=1, gpus=0, cpu_limit=0, memory_limi
rdma=False,
tensorboard=False, tensorboard_image=None,
metrics=['Train-accuracy:PERCENTAGE'],
arenaImage='cheyang/arena_launcher:v0.5',
arenaImage='cheyang/arena_launcher:v0.6',
timeout_hours=240):
"""This function submits MPI Job, it can run Allreduce-style Distributed Training.
Expand Down
2 changes: 1 addition & 1 deletion components/arena/python/arena/_arena_standalone_op.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def standalone_job_op(name, image, command, gpus=0, cpu_limit=0, memory_limit=0,
tensorboard=False, tensorboard_image=None,
data=[], sync_source=None, annotations=[],
metrics=['Train-accuracy:PERCENTAGE'],
arena_image='cheyang/arena_launcher:v0.5',
arena_image='cheyang/arena_launcher:v0.6',
timeout_hours=240):

"""This function submits a standalone training Job
Expand Down