Running Detect on Edge TPU gives error "ImportError: generic_type: type "InterpreterWrapper" is already registered!"

[phodgers]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Detection

Bug

C:\Users\Philip\Downloads\yolov5>python detect.py --weights yolov5s-int8-edgetpu.tflite
detect: weights=['yolov5s-int8-edgetpu.tflite'], source=data\images, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs\detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False
YOLOv5 v6.0-96-g8df64a9 torch 1.10.0+cu113 CUDA:0 (NVIDIA GeForce GTX 950A, 2048MiB)

Loading yolov5s-int8-edgetpu.tflite for TensorFlow Edge TPU inference...
Traceback (most recent call last):
File "C:\Users\Philip\Downloads\yolov5\detect.py", line 244, in
main(opt)
File "C:\Users\Philip\Downloads\yolov5\detect.py", line 239, in main
run(**vars(opt))
File "C:\Users\Philip\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Philip\Downloads\yolov5\detect.py", line 79, in run
model = DetectMultiBackend(weights, device=device, dnn=dnn)
File "C:\Users\Philip\Downloads\yolov5\models\common.py", line 338, in init
import tflite_runtime.interpreter as tfli
File "C:\Users\Philip\AppData\Local\Programs\Python\Python39\lib\site-packages\tflite_runtime\interpreter.py", line 41, in
from tflite_runtime import _pywrap_tensorflow_interpreter_wrapper as _interpreter_wrapper
ImportError: generic_type: type "InterpreterWrapper" is already registered!

Environment

YOLOv5 v6.0-96-g8df64a9
WIndows 10

Minimal Reproducible Example

python export.py --weights yolov5s.pt --include tflite --int8
mv yolov5s-int8.tflite yolov5s-int8-edgetpu.tflite
python detect.py --weights yolov5s-int8-edgetpu.tflite

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@phodgers thanks for the bug report! I searched online but no answers:
https://stackoverflow.com/questions/68721613/tflite-runtime-interpreter-py-is-already-registered-on-windows-10

Does this happen with default tflite export also or only with --int8?

[phodgers]

This seems to happen only when the 'edgetpu' label is added in the file name, so when it is switching over for Edge TPU inferencing. If I run the same file with original name yolov5s-int8.tflite it inferences on the CPU.

C:\Users\Philip\Downloads\yolov5>python detect.py --weights yolov5s-int8.tflite
detect: weights=['yolov5s-int8.tflite'], source=data\images, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs\detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False
YOLOv5 v6.0-96-g8df64a9 torch 1.10.0+cu113 CUDA:0 (NVIDIA GeForce GTX 950A, 2048MiB)

Loading yolov5s-int8.tflite for TensorFlow Lite inference...
image 1/2 C:\Users\Philip\Downloads\yolov5\data\images\bus.jpg: 640x640 4 class0s, 1 class5, Done. (55.688s)
image 2/2 C:\Users\Philip\Downloads\yolov5\data\images\zidane.jpg: 640x640 2 class0s, 1 class27, Done. (55.239s)
Speed: 760.3ms pre-process, 55463.3ms inference, 2.0ms NMS per image at shape (1, 3, 640, 640)
Results saved to runs\detect\exp14

[glenn-jocher]

@phodgers yes edgetpu filenames use a different route:

yolov5/models/common.py

Lines 335 to 348 in 8df64a9

	elif tflite: # https://www.tensorflow.org/lite/guide/python#install_tensorflow_lite_for_python
	if 'edgetpu' in w.lower():
	LOGGER.info(f'Loading {w} for TensorFlow Edge TPU inference...')
	import tflite_runtime.interpreter as tfli
	delegate = {'Linux': 'libedgetpu.so.1', # install https://coral.ai/software/#edgetpu-runtime
	'Darwin': 'libedgetpu.1.dylib',
	'Windows': 'edgetpu.dll'}[platform.system()]
	interpreter = tfli.Interpreter(model_path=w, experimental_delegates=[tfli.load_delegate(delegate)])
	else:
	LOGGER.info(f'Loading {w} for TensorFlow Lite inference...')
	interpreter = tf.lite.Interpreter(model_path=w) # load TFLite model
	interpreter.allocate_tensors() # allocate
	input_details = interpreter.get_input_details() # inputs
	output_details = interpreter.get_output_details() # outputs

[phodgers]

I have confirmed this on both Windows and Raspberry Pi systems. It seems line 321 of yolov5/models/common.py

import tensorflow as tf

causes the issue, when there is both Tensorflow and Tensoflow Lite on the system. So perhaps this section of code needs refactored when dealing with TFLITE files.

[glenn-jocher]

@phodgers I looked a little closer into this. I am able to reproduce an error on a fresh Colab notebook:

Loading yolov5s-int8-edgetpu.tflite for TensorFlow Edge TPU inference...
Traceback (most recent call last):
  File "detect.py", line 244, in <module>
    main(opt)
  File "detect.py", line 239, in main
    run(**vars(opt))
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 79, in run
    model = DetectMultiBackend(weights, device=device, dnn=dnn)
  File "/content/yolov5/models/common.py", line 338, in __init__
    import tflite_runtime.interpreter as tfli
ModuleNotFoundError: No module named 'tflite_runtime'

But the user is expected to have installed TFLite Edge TPU dependencies indicated in the comments in this section before attempting inference, i.e. on L335 and L339:

yolov5/models/common.py

Lines 335 to 342 in f17c86b

	elif tflite: # https://www.tensorflow.org/lite/guide/python#install_tensorflow_lite_for_python
	if 'edgetpu' in w.lower():
	LOGGER.info(f'Loading {w} for TensorFlow Edge TPU inference...')
	import tflite_runtime.interpreter as tfli
	delegate = {'Linux': 'libedgetpu.so.1', # install https://coral.ai/software/#edgetpu-runtime
	'Darwin': 'libedgetpu.1.dylib',
	'Windows': 'edgetpu.dll'}[platform.system()]
	interpreter = tfli.Interpreter(model_path=w, experimental_delegates=[tfli.load_delegate(delegate)])

[phodgers]

@glenn-jocher Yes, your error indicates you also need to install TFLite. But when you install TFLite, you then get the error I posted at the top (when trying to use detect.py with edgetpu inference). I found that I could successfully run the detect.py for edgetpu by commenting out line 321, since for the detection, edgetpu only needs TFLite. So I think there needs to be some refactoring to use TF library for when exporting (used for .TFLITE export, according to requirements.txt) and TFLite library for detection, but not to try to refer to both in same script since it brings up that error about the InterpreterWrapper already being registered.

[glenn-jocher]

@phodgers oh ok I understand now. So you're saying we need to scope the tensorflow import better. Please submit a PR with the improved import scoping you had in mind, thank you!

Please see our ✅ Contributing Guide to get started.