model_main.py - eats a lot of cpu, doesn't use that match gpu, trains slowly #5719

anonym24 · 2018-11-08T08:17:48Z

It never starts any training, but just loads my CPU by 90%

python object_detection\model_main.py --pipeline_config_path=object_detection\training\ssd_mobilenet_v1_coco.config --model_dir=object_detection\images --num_train_steps=50000 --sample_1_of_n_eval_examples=1 --alsologtostderr
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W1108 10:06:58.010473  8132 tf_logging.py:125] Forced number of epochs for all eval validations to be 1.
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W1108 10:06:58.010473  8132 tf_logging.py:125] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x0000023D23E837B8>) includes params argument, but params are not passed to Estimator.
W1108 10:06:58.026091  8132 tf_logging.py:125] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x0000023D23E837B8>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W1108 10:06:58.162420  8132 tf_logging.py:125] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
W1108 10:06:58.271738  8132 tf_logging.py:125] From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
WARNING:tensorflow:From C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W1108 10:06:58.412351  8132 tf_logging.py:125] From C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\core\preprocessor.py:1207: calling squeeze (from tensorflow.python.ops.array_ops) with squeeze_dims is deprecated and will be removed in a future version.
Instructions for updating:
Use the `axis` argument instead
W1108 10:06:58.568533  8132 tf_logging.py:125] From C:\tensorflow1\models\research\object_detection\core\preprocessor.py:1207: calling squeeze (from tensorflow.python.ops.array_ops) with squeeze_dims is deprecated and will be removed in a future version.
Instructions for updating:
Use the `axis` argument instead
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:148: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
W1108 10:06:59.552674  8132 tf_logging.py:125] From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:148: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
2018-11-08 10:07:17.951339: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-11-08 10:07:18.276994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7465
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2018-11-08 10:07:18.281700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-08 10:07:24.611608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-08 10:07:24.616268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2018-11-08 10:07:24.617424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2018-11-08 10:07:24.618709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4727 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)

System information

What is the top-level directory of the model you are using: http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): pip tensorflow-gpu https://www.tensorflow.org/install/pip
TensorFlow version (use command below): 1.12.0
Bazel version (if compiling from source): -
CUDA/cuDNN version: 9.0/7
GPU model and memory: GTX 1060 6GB

The text was updated successfully, but these errors were encountered:

anonym24 · 2018-11-08T08:18:46Z

Actually after some minutes it started doing something

anonym24 · 2018-11-08T08:25:43Z

but it failed with TypeError: can't pickle dict_values objects

anonym24 · 2018-11-08T08:26:21Z

logs:

creating index...
index created!
2018-11-08 10:21:47.762637: W tensorflow/core/framework/op_kernel.cc:1261] Invalid argument: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[1], [1,300,300,3], [1,300,300,3], [1,2], [1,3], [1,100], [1,100,4], [1,100,30], [1,100,30], [1,100], [1,100], [1,100], [1]], output_types=[DT_INT32, DT_FLOAT, DT_UINT8, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x/_1406}} = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1656_...st/sub_5/x", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\evaluation.py", line 274, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1312, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[node IteratorGetNext (defined at object_detection\model_main.py:105)  = IteratorGetNext[output_shapes=[[1], [1,300,300,3], [1,300,300,3], [1,2], [1,3], [1,100], [1,100,4], [1,100,30], [1,100,30], [1,100], [1,100], [1,100], [1]], output_types=[DT_INT32, DT_FLOAT, DT_UINT8, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x/_1406}} = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1656_...st/sub_5/x", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x)]]

Caused by op 'IteratorGetNext', defined at:
  File "object_detection\model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection\model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 460, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1484, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1517, in _call_model_fn_eval
    input_fn, model_fn_lib.ModeKeys.EVAL)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1075, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\util.py", line 110, in parse_input_fn_result
    result = iterator.get_next()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 421, in get_next
    name=name)), self._output_types,
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2108, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): End of sequence
         [[node IteratorGetNext (defined at object_detection\model_main.py:105)  = IteratorGetNext[output_shapes=[[1], [1,300,300,3], [1,300,300,3], [1,2], [1,3], [1,100], [1,100,4], [1,100,30], [1,100,30], [1,100], [1,100], [1,100], [1]], output_types=[DT_INT32, DT_FLOAT, DT_UINT8, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x/_1406}} = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1656_...st/sub_5/x", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x)]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


         [[{{node PyFunc_3}} = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
         [[{{node cond_7/Detections_Left_Groundtruth_Right/7/_2001}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_228_cond_7/Detections_Left_Groundtruth_Right/7", tensor_type=DT_STRING, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "object_detection\model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection\model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 467, in _evaluate
    output_dir=self.eval_dir(name))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1591, in _evaluate_run
    config=self._session_config)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\evaluation.py", line 274, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 783, in __exit__
    self._close_internal(exception_type)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 816, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 942, in end
    self._final_ops, feed_dict=self._final_ops_feed_dict)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


         [[node PyFunc_3 (defined at C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py:356)  = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
         [[{{node cond_7/Detections_Left_Groundtruth_Right/7/_2001}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_228_cond_7/Detections_Left_Groundtruth_Right/7", tensor_type=DT_STRING, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'PyFunc_3', defined at:
  File "object_detection\model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection\model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 460, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1484, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1520, in _call_model_fn_eval
    features, labels, model_fn_lib.ModeKeys.EVAL, config)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1195, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "C:\tensorflow1\models\research\object_detection\model_lib.py", line 414, in model_fn
    eval_config, category_index.values(), eval_dict)
  File "C:\tensorflow1\models\research\object_detection\eval_util.py", line 681, in get_eval_metric_ops_for_evaluators
    eval_dict))
  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 356, in get_estimator_eval_metric_ops
    first_value_op = tf.py_func(first_value_func, [], tf.float32)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 457, in py_func
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 281, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\gen_script_ops.py", line 132, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


         [[node PyFunc_3 (defined at C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py:356)  = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
         [[{{node cond_7/Detections_Left_Groundtruth_Right/7/_2001}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_228_cond_7/Detections_Left_Groundtruth_Right/7", tensor_type=DT_STRING, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

anonym24 · 2018-11-08T08:54:29Z

mb this will help #4780 (comment)

I'm trying to start it again now...

but again, model_train.py takes so much time until it starts training or fails with some errors

seems, model_train.py was implemented very badly, legacy train.py was starting much faster

but now I have to wait forever until it starts and mb fails afterwards... so annoying

anonym24 · 2018-11-08T09:00:26Z

Ok I edited model_train.py

But now it just doesn't do anything:

0% everywhere (cpu, disk, gpu)

anonym24 · 2018-11-08T09:22:50Z

Finally it started to do something after pressed ctr+c.. (not sure what though, it doesn't show every step)

And it eats so much CPU (I have Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz and GPU again is GTX 1060 6GB)

name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7465
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2018-11-08 10:35:47.051199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-08 10:35:47.913900: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-08 10:35:47.916794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2018-11-08 10:35:47.918458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2018-11-08 10:35:47.920196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4727 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-11-08 10:46:13.217802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-08 11:09:05.091090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-08 11:09:05.093468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2018-11-08 11:09:05.094978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2018-11-08 11:09:05.096597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4727 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=4.04s).
Accumulating evaluation results...
DONE (t=1.43s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.010
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.032
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.005
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.024
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.136
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.016
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.020
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.020
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.005
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.039
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.208
2018-11-08 11:13:14.082005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-08 11:13:14.085614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-08 11:13:14.088468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2018-11-08 11:13:14.089702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2018-11-08 11:13:14.091008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4727 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=3.55s).
Accumulating evaluation results...
DONE (t=0.95s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.010
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.032
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.003
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.023
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.136
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.016
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.020
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.020
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.005
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.037
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.214

I see now some checkpoints, but they are created so slowly

Or is faster_rcnn_inception_v2_pets.config + faster_rcnn_inception_v2_coco_2018_01_28 combination just much faster than ssd_mobilenet_v1_coco.config + ssd_mobilenet_v1_coco_2018_01_28

363325971 · 2018-11-08T09:48:03Z

logs:

creating index...
index created!
2018-11-08 10:21:47.762637: W tensorflow/core/framework/op_kernel.cc:1261] Invalid argument: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[1], [1,300,300,3], [1,300,300,3], [1,2], [1,3], [1,100], [1,100,4], [1,100,30], [1,100,30], [1,100], [1,100], [1,100], [1]], output_types=[DT_INT32, DT_FLOAT, DT_UINT8, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x/_1406}} = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1656_...st/sub_5/x", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\evaluation.py", line 274, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1312, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[node IteratorGetNext (defined at object_detection\model_main.py:105)  = IteratorGetNext[output_shapes=[[1], [1,300,300,3], [1,300,300,3], [1,2], [1,3], [1,100], [1,100,4], [1,100,30], [1,100,30], [1,100], [1,100], [1,100], [1]], output_types=[DT_INT32, DT_FLOAT, DT_UINT8, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x/_1406}} = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1656_...st/sub_5/x", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x)]]

Caused by op 'IteratorGetNext', defined at:
  File "object_detection\model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection\model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 460, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1484, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1517, in _call_model_fn_eval
    input_fn, model_fn_lib.ModeKeys.EVAL)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1075, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\util.py", line 110, in parse_input_fn_result
    result = iterator.get_next()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 421, in get_next
    name=name)), self._output_types,
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2108, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): End of sequence
         [[node IteratorGetNext (defined at object_detection\model_main.py:105)  = IteratorGetNext[output_shapes=[[1], [1,300,300,3], [1,300,300,3], [1,2], [1,3], [1,100], [1,100,4], [1,100,30], [1,100,30], [1,100], [1,100], [1,100], [1]], output_types=[DT_INT32, DT_FLOAT, DT_UINT8, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x/_1406}} = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1656_...st/sub_5/x", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Postprocessor/BatchMultiClassNonMaxSuppression/PadOrClipBoxList/sub_5/x)]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


         [[{{node PyFunc_3}} = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
         [[{{node cond_7/Detections_Left_Groundtruth_Right/7/_2001}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_228_cond_7/Detections_Left_Groundtruth_Right/7", tensor_type=DT_STRING, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "object_detection\model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection\model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 467, in _evaluate
    output_dir=self.eval_dir(name))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1591, in _evaluate_run
    config=self._session_config)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\evaluation.py", line 274, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 783, in __exit__
    self._close_internal(exception_type)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 816, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 942, in end
    self._final_ops, feed_dict=self._final_ops_feed_dict)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


         [[node PyFunc_3 (defined at C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py:356)  = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
         [[{{node cond_7/Detections_Left_Groundtruth_Right/7/_2001}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_228_cond_7/Detections_Left_Groundtruth_Right/7", tensor_type=DT_STRING, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'PyFunc_3', defined at:
  File "object_detection\model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection\model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 460, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1484, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1520, in _call_model_fn_eval
    features, labels, model_fn_lib.ModeKeys.EVAL, config)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\estimator\estimator.py", line 1195, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "C:\tensorflow1\models\research\object_detection\model_lib.py", line 414, in model_fn
    eval_config, category_index.values(), eval_dict)
  File "C:\tensorflow1\models\research\object_detection\eval_util.py", line 681, in get_eval_metric_ops_for_evaluators
    eval_dict))
  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 356, in get_estimator_eval_metric_ops
    first_value_op = tf.py_func(first_value_func, [], tf.float32)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 457, in py_func
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 281, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\gen_script_ops.py", line 132, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
    op_def=op_def)
  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "C:\Users\Admin\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\script_ops.py", line 206, in __call__
    ret = func(*args)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
    self._metrics = self.evaluate()

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 207, in evaluate
    self._detection_boxes_list)

  File "C:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "C:\Users\Admin\AppData\Local\Continuum\anaconda3\envs\tensorflow1\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


         [[node PyFunc_3 (defined at C:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py:356)  = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
         [[{{node cond_7/Detections_Left_Groundtruth_Right/7/_2001}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_228_cond_7/Detections_Left_Groundtruth_Right/7", tensor_type=DT_STRING, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

how did you fix this issue?

anonym24 · 2018-11-08T09:53:08Z

@363325971 in this comment I pasted a link for a solution #5719 (comment)

it's old issue and it has been already solved

@363325971 and please short your comment
just make a reference to my comment, there is no need to quote the whole logcat

363325971 · 2018-11-08T10:02:41Z

@anonym24 thanks

anonym24 · 2018-11-09T10:52:11Z

seems issue was in big value for batch_size (default is 24)
when I set batch_size to 1 it stopped eating a lot of CPU and training started to work much faster (as it was with train.py)

that's why faster_rcnn_inception_v2_pets.config was working ok (fast enough) because it had 1 value by default

WendieSun · 2018-11-13T04:58:04Z

Should we see loss after running model_main.py? I only see precision and recall metrics

NorwayLobster · 2018-11-28T06:37:47Z

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.010 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.032 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.003 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.023 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.136 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.016 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.020 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.020 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.005 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.037 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.214

Hello, my result is almost the same as yours above. do you know why the recall and precision is so low?

NorwayLobster · 2018-11-28T06:38:27Z

@anonym24

Twikkxx · 2019-02-18T01:39:43Z

Hello guys, I'm training an object detection with my custom dataset and deploy it in a raspberry pi does the batch size affect the accuracy of the model when deployed to the pi ?

Cavan09 · 2019-03-20T14:29:15Z

@anonym24
Did you ever happen to find out why you had to press ctrl + c to continue training? I seem to be getting that issue periodically while training.

fengwuxuan · 2019-05-14T09:01:45Z

@anonym24 thanks

google-ml-butler · 2022-09-14T20:00:26Z

Are you satisfied with the resolution of your issue?
Yes
No

anonym24 closed this as completed Nov 8, 2018

anonym24 reopened this Nov 8, 2018

anonym24 changed the title ~~model_main.py loads/starts forever (cpu 90%)~~ model_main.py - TypeError: can't pickle dict_values objects Nov 8, 2018

anonym24 mentioned this issue Nov 8, 2018

Are you planning to try "model_main.py" again instead of "train.py" EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10#184

Open

anonym24 changed the title ~~model_main.py - TypeError: can't pickle dict_values objects~~ model_main.py - doesn't do anything later after start (cpu, disk, gpu - 0% usage) Nov 8, 2018

anonym24 mentioned this issue Nov 8, 2018

SSD MobileNet train fails with Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>) #5715

Closed

anonym24 changed the title ~~model_main.py - doesn't do anything later after start (cpu, disk, gpu - 0% usage)~~ model_main.py - eats a lot of cpu, doesn't use that match gpu, trains slowly Nov 8, 2018

anonym24 mentioned this issue Nov 9, 2018

Error using SSD-mobilenet-v1-coco: Incompatible shapes: [3,1917] vs. [16,1] #5391

Closed

anonym24 mentioned this issue Nov 9, 2018

SSD model with mobilenet feature extractor is not converging . #2193

Closed

tensorflowbutler assigned robieta Nov 10, 2018

anonym24 mentioned this issue Nov 12, 2018

Which config should I use to train model for Android (covert later to .tflite format)? EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10#185

Closed

robieta removed their assignment Feb 6, 2020

ravikyram added models:research models that come under research directory type:support labels Jul 10, 2020

ravikyram assigned tombstone, jch1 and pkulzc Jul 10, 2020

jaeyounkim added models:research:odapi ODAPI and removed models:research models that come under research directory labels Jun 25, 2021

saberkun closed this as completed Sep 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

model_main.py - eats a lot of cpu, doesn't use that match gpu, trains slowly #5719

model_main.py - eats a lot of cpu, doesn't use that match gpu, trains slowly #5719

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018

anonym24 commented Nov 8, 2018

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

363325971 commented Nov 8, 2018

anonym24 commented Nov 8, 2018 •

edited

Loading

363325971 commented Nov 8, 2018

anonym24 commented Nov 9, 2018 •

edited

Loading

WendieSun commented Nov 13, 2018

NorwayLobster commented Nov 28, 2018

NorwayLobster commented Nov 28, 2018

Twikkxx commented Feb 18, 2019

Cavan09 commented Mar 20, 2019

fengwuxuan commented May 14, 2019

google-ml-butler bot commented Sep 14, 2022

model_main.py - eats a lot of cpu, doesn't use that match gpu, trains slowly #5719

model_main.py - eats a lot of cpu, doesn't use that match gpu, trains slowly #5719

Comments

anonym24 commented Nov 8, 2018 • edited Loading

System information

anonym24 commented Nov 8, 2018 • edited Loading

anonym24 commented Nov 8, 2018

anonym24 commented Nov 8, 2018

anonym24 commented Nov 8, 2018 • edited Loading

anonym24 commented Nov 8, 2018 • edited Loading

anonym24 commented Nov 8, 2018 • edited Loading

363325971 commented Nov 8, 2018

anonym24 commented Nov 8, 2018 • edited Loading

363325971 commented Nov 8, 2018

anonym24 commented Nov 9, 2018 • edited Loading

WendieSun commented Nov 13, 2018

NorwayLobster commented Nov 28, 2018

NorwayLobster commented Nov 28, 2018

Twikkxx commented Feb 18, 2019

Cavan09 commented Mar 20, 2019

fengwuxuan commented May 14, 2019

google-ml-butler bot commented Sep 14, 2022

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 8, 2018 •

edited

Loading

anonym24 commented Nov 9, 2018 •

edited

Loading