Compatibility issues with Python3 in the refactored API #4982

AakashKumarNain · 2018-08-02T09:55:21Z

System information

What is the top-level directory of the model you are using: Object Detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):16.04
TensorFlow installed from (source or binary):Binary
TensorFlow version (use command below):1.8
Bazel version (if compiling from source):N/A
CUDA/cuDNN version:9.0
GPU model and memory:N/A
Exact command to reproduce:N/A

Describe the problem

With the refactored API, which was done after the Open Image Challenge was hosted on Kaggle, there are too many incompatibilities issues when using the API with Python3. Two of the sample errors that I faced today while trying to run the mobilenet version of RetinaNet are:

models/research/object_detection/models/feature_map_generators.py", line 225, in fpn_top_down_feature_maps
reversed(zip(output_feature_map_keys, output_feature_maps_list)))
TypeError: argument to reversed() must be a sequence

models/research/object_detection/model_lib.py", line 282, in model_fn
    losses = [loss_tensor for loss_tensor in losses_dict.itervalues()]
AttributeError: 'dict' object has no attribute 'itervalues'

Both the issues are because of the syntax error if you run it with Python3. The first error can be resolved by using the line reversed(list(zip(output_feature_map_keys, output_feature_maps_list))))at 225 in feature_map_generators.py and the second error is because of the fact that itervalues() was removed in Python3

Can you please look into all the files once again?

The text was updated successfully, but these errors were encountered:

fernandorovai · 2018-08-02T18:36:53Z

I am facing similar issue while training:
models/research/object_detection/model_lib.py", line 395, in model_fn
eval_metric_ops = {str(k): v for k, v in eval_metric_ops.iteritems()}
AttributeError: 'dict' object has no attribute 'iteritems'

Python3 does not have iteritems() anymore, should use items() instead.

fernandorovai · 2018-08-02T20:44:48Z

Another problem related to Python3:
models/research/object_detection/utils/object_detection_evaluation.py", line 290, in evaluate
category_name = unicode(category_name, 'utf-8')
NameError: name 'unicode' is not defined

replace this: category_name = unicode(category_name, 'utf-8')
to this: category_name = str(category_name, 'utf-8')

AakashKumarNain · 2018-08-06T06:09:58Z

If I run the API with Python2, as per these instructions given in the docs:

python object_detection/model_main.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${MODEL_DIR} \
    --num_train_steps=${NUM_TRAIN_STEPS} \
    --num_eval_steps=${NUM_EVAL_STEPS} \
    --alsologtostderr

I cannot see the training steps progress in my terminal.

netanel-s · 2018-08-06T08:12:05Z

AakashKumarNain, add tf.logging.set_verbosity(tf.logging.INFO) to model_main after the imports.

cjr0106 · 2018-09-01T09:16:42Z

Did you meet the problem:
AttributeError: module 'json.encoder' has no attribute 'FLOAT_REPR'
when you run the coco_tools_test.py in tensorflow/tf.models/models/research/object_detection/metrics/coco_tools_test.py

yu-jingrui · 2018-09-07T10:08:17Z

@fernandorovai I met with the same issue with iteritems(). After changing it to items() there is some other error:

INFO:tensorflow:Loading and preparing annotation results...
2018-09-07 11:58:48.515029: W tensorflow/core/framework/op_kernel.cc:1263] Invalid argument: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "/usr/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File ".../tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File ".../tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File ".../tensorflow/models/research/object_detection/metrics/coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "/usr/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
    return fn(*args)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "/usr/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File ".../tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File ".../tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File ".../tensorflow/models/research/object_detection/metrics/coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "/usr/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


	 [[Node: PyFunc_1 = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_3", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
	 [[Node: map_1/while/TensorArrayWrite/TensorArrayWriteV3/_1553 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_623_map_1/while/TensorArrayWrite/TensorArrayWriteV3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](^_cloopmap_1/while/NextIteration_2/_1443)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../tensorflow/models/research/object_detection/model_main.py", line 105, in <module>
    tf.app.run()
  File "/usr/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/yjin/DL/tensorflow/models/research/object_detection/model_main.py", line 101, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
    return executor.run()
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/training.py", line 590, in run
    return self.run_local()
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/training.py", line 691, in run_local
    saving_listeners=saving_listeners)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/estimator.py", line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/estimator.py", line 1145, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/estimator.py", line 1173, in _train_model_default
    saving_listeners)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/estimator.py", line 1451, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 583, in run
    run_metadata=run_metadata)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1059, in run
    run_metadata=run_metadata)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1150, in run
    raise six.reraise(*original_exc_info)
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1135, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1215, in run
    run_metadata=run_metadata))
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 464, in after_run
    if self._save(run_context.session, global_step):
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 489, in _save
    if l.after_save(session, step):
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/training.py", line 497, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/training.py", line 517, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/training.py", line 884, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/estimator.py", line 470, in evaluate
    output_dir=self.eval_dir(name))
  File "/usr/lib/python3.7/site-packages/tensorflow/python/estimator/estimator.py", line 1501, in _evaluate_run
    config=self._session_config)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/evaluation.py", line 212, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 695, in __exit__
    self._close_internal(exception_type)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 727, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 824, in end
    self._final_ops, feed_dict=self._final_ops_feed_dict)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/usr/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: can't pickle dict_values objects
Traceback (most recent call last):

  File "/usr/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File ".../tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 339, in first_value_func
    self._metrics = self.evaluate()

  File ".../tensorflow/models/research/object_detection/metrics/coco_evaluation.py", line 193, in evaluate
    self._detection_boxes_list)

  File ".../tensorflow/models/research/object_detection/metrics/coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "/usr/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects

sraimund · 2018-09-07T15:31:53Z

@fisheess Your error and a fix has been reported in #4780

yu-jingrui · 2018-09-10T08:54:04Z

@sraimund Thanks. It works.

wt-huang · 2018-11-03T00:43:54Z

Closing as this is resolved

lighTQ · 2019-03-13T07:30:44Z

Did you meet the problem:
AttributeError: module 'json.encoder' has no attribute 'FLOAT_REPR'
when you run the coco_tools_test.py in tensorflow/tf.models/models/research/object_detection/metrics/coco_tools_test.py

Did you reslove this question?

stiv-yakovenko · 2019-07-13T12:12:37Z

I still have this problem.

wander1985 · 2019-07-17T16:20:14Z

Another problem related to Python3:
models/research/object_detection/utils/object_detection_evaluation.py", line 290, in evaluate
category_name = unicode(category_name, 'utf-8')
NameError: name 'unicode' is not defined

replace this: category_name = unicode(category_name, 'utf-8') to this: category_name = str(category_name, 'utf-8')

I am new to object detection api. What works for me is to make the replacement in "object_detection_evaluation.py" under "x:...\Lib\site-packages\object_detection-0.1-py3.7.egg\object_detection". I don't know the reason, but it also works.

AakashKumarNain changed the title ~~Compatibilities issues with Python3 in the refactored API~~ Compatibility issues with Python3 in the refactored API Aug 2, 2018

tensorflowbutler assigned nealwu Aug 3, 2018

apacha mentioned this issue Aug 3, 2018

[object_detection] Python 3 incompatibilities and other errors #4996

Closed

pkulzc mentioned this issue Aug 29, 2018

NameError: name 'unicode' is not defined in object_detection/utils/object_detection_evaluation.py #5203

Closed

wt-huang closed this as completed Nov 3, 2018

gulingfengze mentioned this issue Nov 7, 2018

NameError: name 'unicode' is not defined on object_detection_evaluation.py #5369

Closed

gedefet added a commit to gedefet/models that referenced this issue Nov 14, 2018

Fixed as per issue tensorflow#4982

688242d

hendrixcosta mentioned this issue Sep 13, 2019

Adicionar arquivos do satcomum erpbrasil/erpbrasil.base#3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility issues with Python3 in the refactored API #4982

Compatibility issues with Python3 in the refactored API #4982

AakashKumarNain commented Aug 2, 2018 •

edited

Loading

fernandorovai commented Aug 2, 2018

fernandorovai commented Aug 2, 2018

AakashKumarNain commented Aug 6, 2018

netanel-s commented Aug 6, 2018

cjr0106 commented Sep 1, 2018

yu-jingrui commented Sep 7, 2018

sraimund commented Sep 7, 2018

yu-jingrui commented Sep 10, 2018

wt-huang commented Nov 3, 2018

lighTQ commented Mar 13, 2019

stiv-yakovenko commented Jul 13, 2019

wander1985 commented Jul 17, 2019

Compatibility issues with Python3 in the refactored API #4982

Compatibility issues with Python3 in the refactored API #4982

Comments

AakashKumarNain commented Aug 2, 2018 • edited Loading

System information

Describe the problem

fernandorovai commented Aug 2, 2018

fernandorovai commented Aug 2, 2018

AakashKumarNain commented Aug 6, 2018

netanel-s commented Aug 6, 2018

cjr0106 commented Sep 1, 2018

yu-jingrui commented Sep 7, 2018

sraimund commented Sep 7, 2018

yu-jingrui commented Sep 10, 2018

wt-huang commented Nov 3, 2018

lighTQ commented Mar 13, 2019

stiv-yakovenko commented Jul 13, 2019

wander1985 commented Jul 17, 2019

AakashKumarNain commented Aug 2, 2018 •

edited

Loading