Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

basic readme example failing with errors with latest torch nightlies #9

Closed
chauhang opened this issue Jan 28, 2024 · 2 comments · Fixed by #10
Closed

basic readme example failing with errors with latest torch nightlies #9

chauhang opened this issue Jan 28, 2024 · 2 comments · Fixed by #10

Comments

@chauhang
Copy link

Not to run the basic examples described in the readme -- it is failing with the latest nightlies torch==2.3.0.dev20240127+cu118
Output with errors:

TORCH_LOGS="+dynamo" python compile_depf.py
/opt/conda/envs/nllm3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/opt/conda/envs/nllm3/lib/python3.10/site-packages/depyf/explain/enable_debugging.py:153: UserWarning: You are trying to debug `torch.compile`. Please make sure the code runs multiple times to cover all the possible branches.
  warnings.warn((
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing toy_example /home/ubuntu/compile/compile_depf.py:5
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] Stack (most recent call last):
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/home/ubuntu/compile/compile_depf.py", line 20, in <module>
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     main()
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/home/ubuntu/compile/compile_depf.py", line 14, in main
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     toy_example(torch.randn(10), torch.randn(10))
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 443, in _fn
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     return fn(*args, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 605, in catch_errors
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     return callback(frame, cache_entry, hooks, frame_state)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 748, in _convert_frame
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     result = inner_convert(frame, cache_entry, hooks, frame_state)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 390, in _convert_frame_assert
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     return _compile(
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/contextlib.py", line 79, in inner
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     return func(*args, **kwds)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 650, in _compile
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     guarded_code = compile_inner(code, one_graph, hooks, transform)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 248, in time_wrapper
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     r = func(*args, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 531, in compile_inner
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     out_code = transform_code_object(code, transform)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     transformations(instructions, code_options)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 155, in _fn
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     return fn(*args, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 478, in transform
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     tracer = InstructionTranslator(
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2032, in __init__
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     _step_logger()(
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]   File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/logging.py", line 55, in log
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO]     logger.log(level, "Step %s: %s", step, msg, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-28 01:08:50,003] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/ubuntu/compile/compile_depf.py:5 in toy_example ()
[2024-01-28 01:08:50,003] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]     @torch.compile
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/ubuntu/compile/compile_depf.py:7 in toy_example (toy_example)
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         x = a / (torch.abs(a) + 1)
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST a []
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [LazyVariableTracker()]
[2024-01-28 01:08:50,051] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR abs [LazyVariableTracker(), PythonModuleVariable(<module 'torch' from '/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2024-01-28 01:08:50,052] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST a [LazyVariableTracker(), TorchInGraphFunctionVariable(<built-in method abs of type object at 0x7f7f874c6000>)]
[2024-01-28 01:08:50,052] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [LazyVariableTracker(), TorchInGraphFunctionVariable(<built-in method abs of type object at 0x7f7f874c6000>), LazyVariableTracker()]
[2024-01-28 01:08:50,052] [0/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_a_ L['a']
[2024-01-28 01:08:50,053] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['a'] (10,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], constraint_sizes=[None], tensor_source=LocalSource(local_name='a', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
[2024-01-28 01:08:50,055] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), TensorVariable()]
[2024-01-28 01:08:50,055] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2024-01-28 01:08:50,057] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_TRUE_DIVIDE None [TensorVariable(), TensorVariable()]
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/ubuntu/compile/compile_depf.py:8 in toy_example (toy_example)
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         if b.sum() < 0:
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST b []
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sum [LazyVariableTracker()]
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_b_ L['b']
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['b'] (10,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], constraint_sizes=[None], tensor_source=LocalSource(local_name='b', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
[2024-01-28 01:08:50,059] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), sum)]
[2024-01-28 01:08:50,060] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable()]
[2024-01-28 01:08:50,060] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE COMPARE_OP < [TensorVariable(), ConstantVariable(int)]
[2024-01-28 01:08:50,061] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 38 [TensorVariable()]
[2024-01-28 01:08:50,062] [0/0] torch._dynamo.symbolic_convert: [DEBUG] generic_jump triggered compile
[2024-01-28 01:08:50,062] [0/0] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='generic_jump TensorVariable()', user_stack=[<FrameSummary file /home/ubuntu/compile/compile_depf.py, line 8 in toy_example>], graph_break=True)
Traceback (most recent call last):
  File "/home/ubuntu/compile/compile_depf.py", line 20, in <module>
    main()
  File "/home/ubuntu/compile/compile_depf.py", line 14, in main
    toy_example(torch.randn(10), torch.randn(10))
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 443, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 605, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 748, in _convert_frame
    result = inner_convert(frame, cache_entry, hooks, frame_state)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 390, in _convert_frame_assert
    return _compile(
  File "/opt/conda/envs/nllm3/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 677, in _compile
    raise InternalTorchDynamoError(str(e)).with_traceback(
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 650, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 248, in time_wrapper
    r = func(*args, **kwargs)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 531, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 155, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 496, in transform
    tracer.run()
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2125, in run
    super().run()
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 787, in run
    and self.step()
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 750, in step
    getattr(self, inst.opname)(inst)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 379, in inner
    self.output.compile_subgraph(
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 918, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "/opt/conda/envs/nllm3/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1071, in compile_and_call_fx_graph
    graph_code_log.debug("%s", lazy_format_graph_code(name, gm))
  File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/depyf/explain/patched_lazy_format_graph_code.py", line 34, in patched_lazy_format_graph_code
    fn.__code__ = scope[fn.__name__].__code__
torch._dynamo.exc.InternalTorchDynamoError: _lazy_forward

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner               0
@youkaichao youkaichao mentioned this issue Jan 28, 2024
Merged
@youkaichao
Copy link
Member

@chauhang Hey, thanks for your interest. Can you try the latest code? It should work now. The problem you mentioned should be caused by LazyGraphModule introduced several days ago.

@youkaichao youkaichao reopened this Jan 28, 2024
@youkaichao
Copy link
Member

Actually that PR pytorch/pytorch#117911 was just merged yesterday. So thank you @chauhang for pointing out the issue so timely :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix for LazyGraphModule thuml/depyf
2 participants
@chauhang @youkaichao