Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage not working for TensorFlow Model call function #856

Open
NathanDotTo opened this issue Oct 11, 2019 · 9 comments
Open

Coverage not working for TensorFlow Model call function #856

NathanDotTo opened this issue Oct 11, 2019 · 9 comments
Labels
exotic Unusual execution environment

Comments

@NathanDotTo
Copy link

This problem was first reported at https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FPY-38366

1 - I am running a unit test that is invoking the call(...) function of a TensorFlow model. I can see the trace message, that I have placed in the function, printed so I know the function is being called.
2 - The coverage analysis shows that code in red, as though it was not invoked.
3 - It should show green as it was invoked.

See test_model.py in https://github.com/Data-Science-Projects/demo-routenet.

See also the requirements.txt in that project.

@NathanDotTo NathanDotTo added the bug Something isn't working label Oct 11, 2019
@nedbat
Copy link
Owner

nedbat commented Oct 14, 2019

I've never used TensorFlow before. I'm going to guess that the execution of Python code within call(...) doesn't trigger the Python trace function, but I don't know.

Can you provide very very explicit reproduction steps? Include the version of Python, the commands to run to install everything, and the commands to run the program. Thanks.

@NathanDotTo
Copy link
Author

Python 3.7.4 on macOS 10.14.6.

The steps so far are:

git clone https://github.com/Data-Science-Projects/demo-routenet.git
cd demo-routenet/bin
. ./create_routenet_venv.sh
cd ../tests/unit
pytest -s test_model.py --cov

Since I am looking the results in PyCharm, I am not quite sure how to recreate what I see in PyCharm at the CLI. What seems to be missing, in PyCharm, is any indication that the call function in the src/routenet/model/routenet_model.py code is covered. It is clear that the code is called, as one can see the output from the print statement.

@nedbat
Copy link
Owner

nedbat commented Oct 26, 2019

Thanks, this lets me reproduce the problem. I can see that line 65 in routenet_model.py is reported as uncovered. I'll dig into it.

@NathanDotTo
Copy link
Author

NathanDotTo commented Oct 26, 2019 via email

@nedbat
Copy link
Owner

nedbat commented Oct 27, 2019

@NathanDotTo this was an interesting one! It turns out that TensorFlow is making a transformed copy of your code, and then running it. So your original file isn't actually executed. In a way, coverage.py was right! I've opened tensorflow/tensorflow#33759 with TensorFlow to see what we can do about it.

@NathanDotTo
Copy link
Author

NathanDotTo commented Oct 27, 2019 via email

@Zminghua
Copy link

Zminghua commented Oct 31, 2019

I have a similar question, when I debug tensor2tensor and set a breakpoint in t2t_model.py at line 316, the debugger doesn't break as if there is no breakpoint. Then I run the code step by step, in the end, I also found a temporary Python file in /tmp/.

  • tensorflow 1.15
  • python 3.7

@nedbat
Copy link
Owner

nedbat commented Oct 31, 2019

@Zminghua debugging is not a coverage.py concern, but sharing my new-found expertise, it should work if you put a "pdb.set_trace()" line in your t2t_model.py file. That line will then be copied to the temporary file, and the debugger will break. You will be in a slightly strange world, since your file has been changed, but you can at least be in the debugger.

@Zminghua
Copy link

Zminghua commented Nov 1, 2019

@nedbat Thanks for your reminding. I have learned that it is AutoGraph conversion in Tensorflow keras module. After I added a "pdb.set_trace()" line, I was really in a slightly strange world. However, I have modified tensorflow source code to prevent the conversion. Thank you very very very much.

@nedbat nedbat added exotic Unusual execution environment and removed bug Something isn't working labels Jan 15, 2020
gsakkis added a commit to TileDB-Inc/TileDB-ML that referenced this issue Feb 9, 2022
gsakkis added a commit to TileDB-Inc/TileDB-ML that referenced this issue Feb 9, 2022
gsakkis added a commit to TileDB-Inc/TileDB-ML that referenced this issue Feb 10, 2022
gsakkis added a commit to TileDB-Inc/TileDB-ML that referenced this issue Feb 10, 2022
gsakkis added a commit to TileDB-Inc/TileDB-ML that referenced this issue Feb 10, 2022
gsakkis added a commit to TileDB-Inc/TileDB-ML that referenced this issue Feb 14, 2022
* Factor out the common bits of PyTorch & Tensorflow batch classes to BaseDenseBatch & BaseSparseBatch

* Inline _ensure_attr

* Test both the public TensorflowTileDBDataset and the private _generator to work around nedbat/coveragepy#856

* Factor out the PyTorch & Tensorflow generator logic to _batch_utils.tensor_generator

* Change Tuple to Sequence or Iterator in signatures

* Rename idx parameter to perm_idxs

* Project-wide rename of *attribute_names to *attrs

* Optimization/fix: ensure batch_offsets doesn't extend beyond the size of the current buffer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
exotic Unusual execution environment
Projects
None yet
Development

No branches or pull requests

3 participants