Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error during Export of TfRecords #970

Closed
efcy opened this issue Dec 17, 2019 · 5 comments · Fixed by #993 or #1567
Closed

Error during Export of TfRecords #970

efcy opened this issue Dec 17, 2019 · 5 comments · Fixed by #993 or #1567
Assignees
@zhiltsov-max
Labels
bug Something isn't working
Milestone
1.0.0-beta.1

Comments

@efcy
Copy link
Contributor

efcy commented Dec 17, 2019

When exporting a dataset as TF Detection API TFRecord via the GUI the following error shows up.

Could not export dataset for the task 14
Error: Request failed with status code 500. "Work-horse process was terminated unexpectedly (waitpid returned 134)".

I tried multiple datasets with different bounding box annotations.

Commit: 2ec4b32

Does anybody has an idea on how I can proceed debugging this issue?
@zhiltsov-max zhiltsov-max self-assigned this Dec 17, 2019
@zhiltsov-max
Copy link
Contributor

Hi, the simplest variant is to try to export a Datumaro dataset and then just export to tfrecord via Datumaro CLI:

# download Datumaro dataset
# unpack and enter the directory
python -m virtualenv venv
. ./venv/bin/activate
pip install ./datumaro/
pip install -r ./cvat/utils/cli/requirements.txt
PYTHONPATH=":" datum project export -f 'tf_detection_api' -d export

@nmanovic nmanovic added the bug Something isn't working label Dec 17, 2019
@nmanovic nmanovic added this to the 1.0.0 - Beta milestone Dec 17, 2019
@efcy
Copy link
Contributor Author

efcy commented Dec 19, 2019 •
edited
Loading

@zhiltsov-max when i try your approach I get this output:

2019-12-19 14:38:05,915 INFO: Loading 'task_15_images' source contents...
2019-12-19 14:38:05,916 INFO: Loading own dataset...
2019-12-19 14:38:05,916 ERROR: __init__() got an unexpected keyword argument 'cmdline_args'
Traceback (most recent call last):
  File "./datum", line 8, in <module>
    sys.exit(main())
  File "/home/benji/my_test_datumaro/task_test-2019_12_18_13_42_54-datumaro_project/datumaro/datumaro/__init__.py", line 86, in main
    return command(command_args)
  File "/home/benji/my_test_datumaro/task_test-2019_12_18_13_42_54-datumaro_project/datumaro/datumaro/cli/project/__init__.py", line 316, in main
    return args.command(args)
  File "/home/benji/my_test_datumaro/task_test-2019_12_18_13_42_54-datumaro_project/datumaro/datumaro/cli/project/__init__.py", line 154, in export_command
    cmdline_args=args.extra_args)
  File "/home/benji/my_test_datumaro/task_test-2019_12_18_13_42_54-datumaro_project/datumaro/datumaro/components/project.py", line 527, in export
    converter = self.env.make_converter(output_format, **converter_kwargs)
  File "/home/benji/my_test_datumaro/task_test-2019_12_18_13_42_54-datumaro_project/datumaro/datumaro/components/project.py", line 243, in make_converter
    return self.converters.get(name)(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'cmdline_args'

@zhiltsov-max
Copy link
Contributor

zhiltsov-max commented Dec 20, 2019 •
edited
Loading

@BenjiSchlotter, thanks, the problem is clear now. It is easy to fix, you can just add such an argument to the class constructor at datumaro/components/convertes/tfrecord.py, if you don't want to wait for the upstream fix.

@zhiltsov-max zhiltsov-max mentioned this issue Dec 20, 2019
Merged
@efcy
Copy link
Contributor Author

efcy commented Dec 24, 2019

After the fix i now get a StackOverflow Error:

(venv) benji@karpathy:~/bla$ PYTHONPATH=":" datum project export -f 'tf_detection_api' -d export
2019-12-24 12:14:55,304 INFO: Loading the project...
Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fbc0f79c740 (most recent call first):
  File "/usr/lib/python3.6/collections/__init__.py", line 872 in __init__
  File "/usr/lib/python3.6/configparser.py", line 1149 in _unify_values
  File "/usr/lib/python3.6/configparser.py", line 781 in get
  File "/home/benji/bla/venv/lib/python3.6/site-packages/git/config.py", line 84 in assure_data_present
  File "/usr/lib/python3.6/configparser.py", line 803 in _get
  File "/usr/lib/python3.6/configparser.py", line 809 in _get_conv
  File "/usr/lib/python3.6/configparser.py", line 829 in getboolean
  File "/home/benji/bla/venv/lib/python3.6/site-packages/git/config.py", line 84 in assure_data_present
  File "/home/benji/bla/venv/lib/python3.6/site-packages/git/repo/base.py", line 188 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/git/repo/base.py", line 929 in init
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 109 in init
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 101 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 186 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 644 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/importers/datumaro.py", line 18 in __call__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 647 in make_dataset
  File "/home/benji/bla/venv/lib/python3.6/site-packages/datumaro/components/project.py", line 340 in __init__
  ...
Aborted (core dumped)

@efcy
Copy link
Contributor Author

efcy commented Dec 24, 2019

But the fix did not adress my original problem. The problem with exporting via the GUI was that tensorflow can't be imported on my machine due to missing AVX instructions. I had to build tensorflow myself and replace it in the docker container. Maybe a better error handling would be helpful for others as well.

@zhiltsov-max zhiltsov-max mentioned this issue Dec 27, 2019
Merged
@zhiltsov-max zhiltsov-max mentioned this issue May 22, 2020
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhiltsov-max zhiltsov-max

Labels
bug Something isn't working
Projects
None yet
Milestone
1.0.0-beta.1
3 participants
@efcy @zhiltsov-max @nmanovic