torch.compile error
Hi, i have this problem when i am running torch.compile code snippet:
Traceback (most recent call last):
File "/data/gemma_torch/torch_compile.py", line 40, in
outputs = model.generate(**model_inputs, past_key_values=past_key_values, do_sample=True, temperature=1.0, max_new_tokens=128)
...
torch._dynamo.exc.Unsupported: reconstruct: UserDefinedObjectVariable(HybridCache)
from user code:
File "/home/.local/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 1111, in forward
return CausalLMOutputWithPast(
Versions:
torch: 2.3.1
transformers 4.42.4
python: 3.10
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0
GPU: NVIDIA L40S
Did anyone encounter this problem? What can i try to get it running?
Hi
@LD-inform
, It seems, there is issue with the installed torch version 2.3.1 . Could you please try again by upgrading the torch library to the latest version 2.5.1 using !pip install -U torch and let us know if the issue still persists.