runtime error

Exit code: 1. Reason: ... .......... .......... .......... 99% 259M 0s 212800K .......... .......... .......... .......... .......... 99% 323M 0s 212850K .......... .......... .......... .......... .......... 99% 335M 0s 212900K .......... .......... .......... .......... .......... 99% 302M 0s 212950K .......... .......... .......... .......... .......... 99% 324M 0s 213000K .......... ....... 100% 271M=4.2s 2024-09-13 19:35:21 (49.7 MB/s) - ‘assets/vqgan_jax_strongaug.ckpt’ saved [218130007/218130007] lr_scheduler.pth nnet.pth nnet_ema.pth optimizer.pth step.pth Working with z of shape (1, 256, 16, 16) = 65536 dimensions. Strict load Restored from assets/vqgan_jax_strongaug.ckpt xformers not available, will use pytorch attention instead 2024-09-13 19:35:22.367 | DEBUG | libs.uvit_t2i_vq:__init__:200 - codebook size in nnet: 1024 num vis tokens: 256 2024-09-13 19:35:28.895 | DEBUG | libs.uvit_t2i_vq:__init__:200 - codebook size in nnet: 1024 num vis tokens: 256 2024-09-13 19:35:35.580 | INFO | utils:initialize_train_state:178 - nnet has 505693313 parameters 2024-09-13 19:35:36.846 | INFO | utils:resume:151 - resume from assets/ckpts/cc3m-285000.ckpt 2024-09-13 19:35:36.846 | INFO | utils:load:119 - load from assets/ckpts/cc3m-285000.ckpt Traceback (most recent call last): File "app.py", line 150, in <module> train_state.resume(ckpt_root=config.resume_root) File "/home/user/app/utils.py", line 152, in resume self.load(ckpt_path) File "/home/user/app/utils.py", line 120, in load self.step = torch.load(os.path.join(path, 'step.pth'), map_location='cpu') File "/home/user/.pyenv/versions/3.8.16/lib/python3.8/site-packages/torch/serialization.py", line 795, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/home/user/.pyenv/versions/3.8.16/lib/python3.8/site-packages/torch/serialization.py", line 1002, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) EOFError: Ran out of input

Container logs: