2024-05-09 17:31:56,758 INFO StreamThr :72 [internal.py:wandb_internal():86] W&B internal server running at pid: 72, started at: 2024-05-09 17:31:56.757220 2024-05-09 17:31:56,760 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status 2024-05-09 17:31:56,771 INFO WriterThread:72 [datastore.py:open_for_write():85] open: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/run-a805570edd5eddcfb7deb3a583d7b4c2.wandb 2024-05-09 17:31:56,773 DEBUG SenderThread:72 [sender.py:send():379] send: header 2024-05-09 17:31:56,858 DEBUG SenderThread:72 [sender.py:send():379] send: run 2024-05-09 17:31:56,862 INFO SenderThread:72 [sender.py:_maybe_setup_resume():758] checking resume status for mila-language-drift/trl/a805570edd5eddcfb7deb3a583d7b4c2 2024-05-09 17:31:57,321 INFO SenderThread:72 [dir_watcher.py:__init__():211] watching files in: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files 2024-05-09 17:31:57,321 INFO SenderThread:72 [sender.py:_start_run_threads():1121] run started: a805570edd5eddcfb7deb3a583d7b4c2 with start time 1715275916.76161 2024-05-09 17:31:57,321 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:31:57,323 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:31:57,328 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: check_version 2024-05-09 17:31:57,329 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: check_version 2024-05-09 17:31:57,433 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: run_start 2024-05-09 17:31:57,443 DEBUG HandlerThread:72 [system_info.py:__init__():31] System info init 2024-05-09 17:31:57,443 DEBUG HandlerThread:72 [system_info.py:__init__():46] System info init done 2024-05-09 17:31:57,443 INFO HandlerThread:72 [system_monitor.py:start():181] Starting system monitor 2024-05-09 17:31:57,444 INFO SystemMonitor:72 [system_monitor.py:_start():145] Starting system asset monitoring threads 2024-05-09 17:31:57,444 INFO HandlerThread:72 [system_monitor.py:probe():201] Collecting system info 2024-05-09 17:31:57,445 INFO SystemMonitor:72 [interfaces.py:start():190] Started cpu monitoring 2024-05-09 17:31:57,445 INFO SystemMonitor:72 [interfaces.py:start():190] Started disk monitoring 2024-05-09 17:31:57,446 INFO SystemMonitor:72 [interfaces.py:start():190] Started gpu monitoring 2024-05-09 17:31:57,446 INFO SystemMonitor:72 [interfaces.py:start():190] Started memory monitoring 2024-05-09 17:31:57,447 INFO SystemMonitor:72 [interfaces.py:start():190] Started network monitoring 2024-05-09 17:31:57,563 DEBUG HandlerThread:72 [system_info.py:probe():195] Probing system 2024-05-09 17:31:57,566 DEBUG HandlerThread:72 [system_info.py:_probe_git():180] Probing git 2024-05-09 17:31:57,588 DEBUG HandlerThread:72 [system_info.py:_probe_git():188] Probing git done 2024-05-09 17:31:57,588 DEBUG HandlerThread:72 [system_info.py:probe():240] Probing system done 2024-05-09 17:31:57,588 DEBUG HandlerThread:72 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35', 'python': '3.11.2', 'heartbeatAt': '2024-05-09T17:31:57.563065', 'startedAt': '2024-05-09T17:31:56.729897', 'docker': None, 'cuda': None, 'args': ('--bf16', '--dataset_eval_split', 'validation', '--dataset_name', 'vwxyzjn/summarize_from_feedback_tldr_3_filtered_oai_preprocessing_1706381144', '--ddp_find_unused_parameters', 'False', '--eval_steps', '0.2', '--evaluation_strategy', 'steps', '--gradient_accumulation_steps', '4', '--gradient_checkpointing', 'False', '--hub_model_id', 'mnoukhov/pythia410m-sft-tldr', '--learning_rate', '3e-6', '--logging_steps', '100', '--lr_scheduler_type', 'cosine', '--max_seq_length', '580', '--model_name', 'EleutherAI/pythia-410m-deduped', '--num_train_epochs', '1', '--per_device_eval_batch_size', '8', '--per_device_train_batch_size', '32', '--push_to_hub', '--report_to', 'wandb', '--task_type', 'tldr', '--output_dir', '/home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2'), 'state': 'running', 'program': '/home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/sft.py', 'codePath': 'results/a805570edd5eddcfb7deb3a583d7b4c2/code/sft.py', 'git': {'remote': 'git@github.com:mnoukhov/trl.git', 'commit': '7906781c51d4720a3eb07d2e523d714799a48997'}, 'email': 'mnoukhov@gmail.com', 'root': '/home/toolkit/trl', 'host': '40084257-ebc6-4abd-94a1-8b674b58783e', 'username': '_toolchain', 'executable': '/home/toolkit/.conda/envs/trl/bin/python', 'cpu_count': 128, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2294.3337304687498, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3393.5, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3390.299, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.658, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3385.351, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3382.543, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.137, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3380.021, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3383.662, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.681, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3384.723, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}, {'current': 2250.0, 'min': 1500.0, 'max': 2250.0}], 'disk': {'total': 1757.7545127868652, 'used': 1371.7691802978516}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 2015.6588516235352}} 2024-05-09 17:31:57,589 INFO HandlerThread:72 [system_monitor.py:probe():211] Finished collecting system info 2024-05-09 17:31:57,589 INFO HandlerThread:72 [system_monitor.py:probe():214] Publishing system info 2024-05-09 17:31:57,589 DEBUG HandlerThread:72 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2024-05-09 17:31:57,591 DEBUG HandlerThread:72 [system_info.py:_save_pip():67] Saving pip packages done 2024-05-09 17:31:57,592 DEBUG HandlerThread:72 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment 2024-05-09 17:31:58,298 DEBUG HandlerThread:72 [system_info.py:_save_conda():86] Saving conda packages done 2024-05-09 17:31:58,302 INFO HandlerThread:72 [system_monitor.py:probe():216] Finished publishing system info 2024-05-09 17:31:58,323 INFO Thread-12 :72 [dir_watcher.py:_on_file_created():271] file/dir created: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/conda-environment.yaml 2024-05-09 17:31:58,323 INFO Thread-12 :72 [dir_watcher.py:_on_file_created():271] file/dir created: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-metadata.json 2024-05-09 17:31:58,323 INFO Thread-12 :72 [dir_watcher.py:_on_file_created():271] file/dir created: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/requirements.txt 2024-05-09 17:31:58,324 INFO Thread-12 :72 [dir_watcher.py:_on_file_created():271] file/dir created: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:31:58,359 DEBUG SenderThread:72 [sender.py:send():379] send: files 2024-05-09 17:31:58,359 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-metadata.json with policy now 2024-05-09 17:31:58,366 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:31:58,366 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:31:58,521 DEBUG SenderThread:72 [sender.py:send():379] send: telemetry 2024-05-09 17:31:58,521 DEBUG SenderThread:72 [sender.py:send():379] send: config 2024-05-09 17:31:58,522 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:31:58,522 DEBUG SenderThread:72 [sender.py:send():379] send: telemetry 2024-05-09 17:31:58,522 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:31:58,522 WARNING SenderThread:72 [sender.py:send_metric():1326] Seen metric with glob (shouldn't happen) 2024-05-09 17:31:58,522 DEBUG SenderThread:72 [sender.py:send():379] send: telemetry 2024-05-09 17:31:59,324 INFO Thread-12 :72 [dir_watcher.py:_on_file_created():271] file/dir created: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:00,268 INFO wandb-upload_0:72 [upload_job.py:push():131] Uploaded file /tmp/tmpnrozw_h4wandb/eg88w50g-wandb-metadata.json 2024-05-09 17:32:01,326 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:02,113 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:05,330 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:07,332 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:07,821 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:09,334 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:13,337 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:13,366 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:32:13,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:32:13,531 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:15,339 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:17,340 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:18,796 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:21,345 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:23,347 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:24,176 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:25,348 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:28,366 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:32:28,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:32:29,352 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:29,459 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:30,353 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/config.yaml 2024-05-09 17:32:31,356 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:33,358 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:34,652 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:37,361 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:39,363 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:39,930 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:41,364 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:43,366 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:32:43,367 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:43,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:32:45,079 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:47,370 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:49,372 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:50,105 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:51,373 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:55,255 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:32:55,377 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:57,378 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:32:57,447 DEBUG SystemMonitor:72 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2024-05-09 17:32:57,452 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:32:58,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:32:58,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:32:59,380 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:00,461 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:01,382 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:05,385 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:05,676 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:07,387 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:09,389 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:10,830 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:13,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:33:13,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:33:13,392 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:15,395 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:16,050 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:17,397 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:21,351 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:21,399 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:23,401 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:25,403 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:26,566 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:27,455 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:33:28,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:33:28,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:33:29,406 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:31,409 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:31,856 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:33,412 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:35,414 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:37,071 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:39,417 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:41,418 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:42,316 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:43,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:33:43,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:33:43,421 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:47,424 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:47,490 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:49,426 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:51,428 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:52,801 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:55,431 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:57,433 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:33:57,502 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:33:57,921 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:33:58,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:33:58,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:33:59,435 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:01,436 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:03,066 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:05,439 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:07,441 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:08,146 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:09,443 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:13,223 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:13,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:34:13,367 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:34:13,446 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:15,448 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:17,450 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:18,261 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:19,452 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:23,456 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:23,487 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:25,458 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:27,460 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:27,503 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:34:28,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:34:28,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:34:28,519 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:29,462 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:33,466 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:33,793 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:35,468 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:37,469 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:38,978 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:41,473 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:43,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:34:43,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:34:43,475 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:44,124 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:45,477 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:49,286 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:49,481 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:51,483 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:53,485 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:54,474 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:55,487 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:34:57,507 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:34:58,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:34:58,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:34:59,483 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:34:59,491 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:01,493 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:03,496 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:04,804 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:07,499 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:09,502 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:09,839 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:11,504 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:13,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:35:13,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:35:13,506 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:14,986 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:17,510 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:19,519 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:20,257 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:21,521 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:25,398 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:25,525 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:27,511 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:35:27,528 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:28,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:35:28,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:35:29,530 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:30,472 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:31,533 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:35,536 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:35,707 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:37,547 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:40,863 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:41,550 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:43,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:35:43,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:35:43,553 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:45,554 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:46,037 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:49,561 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:51,294 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:51,562 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:53,564 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:55,567 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:35:56,482 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:35:57,513 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:35:58,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:35:58,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:35:59,571 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:01,572 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:01,679 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:03,574 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:06,876 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:07,577 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:09,579 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:11,581 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:11,881 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:13,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:36:13,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:36:13,583 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:16,990 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:17,587 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:19,589 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:20,283 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:36:20,285 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:36:20,286 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:36:20,286 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:36:20,286 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:36:20,286 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:36:20,286 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:36:20,288 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:36:20,589 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:36:21,590 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:22,290 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:22,592 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:25,594 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:27,459 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:27,515 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:36:27,596 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:28,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:36:28,368 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:36:29,608 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:32,873 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:33,613 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:35,615 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:37,617 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:38,018 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:38,618 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/config.yaml 2024-05-09 17:36:41,621 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:43,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:36:43,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:36:43,622 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:43,958 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:45,624 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:47,626 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:49,166 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:51,629 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:53,631 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:54,461 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:55,634 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:36:57,518 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:36:58,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:36:58,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:36:59,488 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:36:59,637 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:00,639 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:02,641 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:04,927 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:06,645 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:08,647 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:10,102 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:10,649 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:13,368 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:37:13,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:37:14,652 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:15,121 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:16,654 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:18,656 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:20,224 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:20,658 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:24,662 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:25,413 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:26,664 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:27,521 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:37:28,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:37:28,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:37:28,666 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:30,531 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:32,670 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:34,672 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:35,886 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:36,674 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:38,677 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:41,015 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:42,680 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:43,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:37:43,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:37:44,682 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:46,214 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:46,684 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:50,688 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:51,403 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:52,690 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:54,692 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:56,586 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:37:56,695 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:37:57,524 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:37:58,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:37:58,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:38:00,698 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:01,693 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:02,700 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:04,703 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:06,843 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:08,706 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:10,708 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:12,060 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:12,710 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:13,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:38:13,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:38:14,712 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:17,145 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:18,716 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:20,718 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:22,202 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:22,720 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:26,724 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:27,437 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:27,527 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:38:28,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:38:28,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:38:28,726 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:30,728 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:32,580 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:32,731 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:36,734 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:37,772 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:38,737 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:40,739 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:43,045 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:43,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:38:43,369 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:38:44,742 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:46,744 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:48,109 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:48,745 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:52,748 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:53,223 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:54,750 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:56,752 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:38:57,531 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:38:58,370 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:38:58,370 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:38:58,529 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:38:58,754 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:02,757 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:03,650 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:04,759 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:06,761 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:08,764 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:10,764 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:12,766 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:13,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:39:13,370 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:39:14,041 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:14,768 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:16,772 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:19,187 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:20,779 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:22,781 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:24,298 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:24,783 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:27,533 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:39:28,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:39:28,370 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:39:28,787 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:29,470 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:30,789 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:32,792 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:34,776 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:36,794 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:38,797 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:39,880 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:40,799 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:42,802 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:43,370 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:39:43,370 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:39:45,142 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:46,806 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:48,808 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:50,273 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:50,812 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:54,815 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:55,450 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:39:56,817 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:39:57,536 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:39:58,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:39:58,370 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:39:58,820 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:00,545 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:00,821 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:02,823 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:04,825 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:05,646 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:06,826 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:08,828 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:10,781 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:10,830 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:12,832 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:13,369 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:40:13,370 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:40:14,834 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:15,817 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:16,836 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:18,839 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:20,838 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:20,842 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:22,844 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:24,846 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:25,924 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:26,848 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:27,539 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:40:28,372 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:40:28,372 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:40:28,850 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:30,853 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:30,952 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:32,856 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:34,862 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:36,012 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:36,864 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:38,867 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:40,869 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:41,048 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:42,870 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:43,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:40:43,373 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:40:44,872 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:46,082 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:46,874 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:48,876 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:50,878 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:51,219 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:52,880 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:53,620 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:40:53,622 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:40:53,622 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:40:53,623 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:40:53,623 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 17:40:53,623 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:40:53,623 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:40:53,626 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:40:53,881 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:40:54,882 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:56,884 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:40:57,185 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:40:57,542 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:40:58,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:40:58,373 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:41:00,887 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:02,380 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:02,889 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:04,891 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:07,563 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:08,894 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:10,896 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:12,700 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:12,898 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:13,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:41:13,373 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:41:16,902 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:17,950 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:18,905 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:18,905 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/config.yaml 2024-05-09 17:41:20,907 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:22,910 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:23,108 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:26,913 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:27,546 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:41:28,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:41:28,373 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:41:28,536 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:28,915 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:30,917 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:33,562 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:34,921 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:36,924 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:37,560 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:41:37,561 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:41:37,562 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:41:37,564 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:41:37,924 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:41:38,564 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:38,925 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:40,927 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:43,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:41:43,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:41:44,462 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:44,931 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:46,933 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:48,935 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:50,117 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:52,938 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:54,939 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:55,272 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:41:56,942 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:41:57,549 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:41:58,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:41:58,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:42:00,614 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:00,945 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:02,947 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:04,950 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:05,888 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:08,953 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:10,955 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:11,072 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:12,957 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:13,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:42:13,373 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:42:14,959 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:16,157 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:18,963 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:20,965 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:21,331 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:22,968 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:26,484 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:26,971 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:27,552 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:42:28,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:42:28,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:42:28,974 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:30,976 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:31,625 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:34,980 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:36,847 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:36,982 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:38,985 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:40,987 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:42,014 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:43,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:42:43,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:42:44,991 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:46,993 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:47,152 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:48,995 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:52,379 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:52,999 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:55,001 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:57,003 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:42:57,464 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:42:57,556 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:42:58,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:42:58,373 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:42:59,005 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:02,675 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:03,009 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:05,011 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:07,013 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:07,947 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:11,017 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:13,021 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:13,157 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:13,373 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:43:13,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:43:15,023 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:18,299 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:19,027 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:21,033 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:23,035 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:23,477 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:25,038 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:27,558 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:43:28,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:43:28,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:43:28,487 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:29,042 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:31,044 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:33,047 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:33,854 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:37,050 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:39,052 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:39,142 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:41,055 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:43,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:43:43,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:43:44,491 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:45,058 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:47,060 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:49,062 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:49,526 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:51,064 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:54,650 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:43:55,068 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:57,071 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:43:57,561 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:43:58,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:43:58,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:43:59,073 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:00,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:03,076 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:05,078 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:05,525 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:07,081 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:09,083 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:10,642 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:13,086 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:13,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:44:13,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:44:15,088 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:15,783 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:17,091 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:20,941 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:21,095 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:23,097 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:25,100 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:26,072 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:27,102 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:27,565 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:44:28,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:44:28,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:44:31,106 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:31,293 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:33,108 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:35,110 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:36,516 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:39,114 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:41,116 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:41,610 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:43,118 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:43,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:44:43,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:44:45,120 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:46,771 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:49,123 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:51,125 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:52,007 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:53,127 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:57,102 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:44:57,131 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:44:57,567 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:44:58,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:44:58,374 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:44:59,133 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:01,135 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:02,288 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:05,138 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:07,140 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:07,553 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:09,142 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:11,144 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:12,771 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:13,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:45:13,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:45:15,148 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:17,151 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:18,081 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:19,156 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:23,159 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:23,257 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:25,162 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:27,172 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:27,570 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:45:28,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:45:28,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:45:28,546 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:31,176 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:33,179 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:33,718 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:35,181 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:38,883 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:39,185 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:41,187 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:43,189 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:43,374 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:45:43,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:45:44,101 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:45,192 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:49,195 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:49,444 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:51,197 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:53,200 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:54,659 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:45:57,204 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:45:57,376 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:45:57,378 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:45:57,379 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:45:57,381 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:45:57,573 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:45:58,205 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:45:58,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:45:58,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:45:59,206 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:00,018 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:01,208 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:05,212 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:05,320 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:07,214 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:09,222 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:10,376 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:13,226 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:13,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:46:13,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:46:15,228 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:15,533 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:17,230 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:19,233 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:20,782 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:23,237 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:25,240 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:26,002 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:27,242 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:27,577 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:46:28,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:46:28,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:46:31,150 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:31,246 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:33,249 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:35,253 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:36,360 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:39,256 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:41,262 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:41,599 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:43,265 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:43,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:46:43,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:46:46,883 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:47,268 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:49,271 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:51,272 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:52,068 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:53,274 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:57,278 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:46:57,579 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:46:57,581 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:46:58,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:46:58,375 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:46:59,280 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:01,282 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:03,208 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:03,284 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:07,288 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:08,459 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:09,292 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:11,294 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:13,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:47:13,376 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:47:13,512 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:15,297 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:17,300 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:18,655 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:19,302 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:21,304 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:23,780 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:25,308 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:27,309 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:27,582 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:47:28,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:47:28,376 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:47:29,312 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:29,506 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:33,316 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:35,004 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:35,318 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:37,319 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:39,321 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:40,167 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:43,324 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:43,376 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:47:43,376 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:47:45,326 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:45,489 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:47,328 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:50,525 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:51,332 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:53,334 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:55,336 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:47:55,815 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:47:57,586 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:47:58,375 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:47:58,376 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:47:59,344 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:00,914 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:01,349 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:03,351 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:05,353 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:05,966 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:09,356 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:11,088 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:11,358 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:13,360 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:13,376 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:48:13,376 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:48:16,316 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:17,363 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:19,366 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:21,368 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:21,470 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:23,370 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:26,665 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:27,373 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:27,588 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:48:28,376 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:48:28,376 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:48:29,375 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:31,377 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:31,729 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:33,380 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:36,938 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:37,383 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:39,385 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:41,387 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:42,095 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:43,376 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:48:43,377 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:48:45,390 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:47,345 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:47,392 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:49,395 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:51,397 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:52,393 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:53,400 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:55,403 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:57,405 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:48:57,453 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:48:57,591 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:48:58,377 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:48:58,377 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:48:59,407 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:01,409 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:02,456 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:03,411 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:05,413 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:07,416 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:07,504 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:09,418 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:11,419 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:12,530 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:13,377 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:49:13,377 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:49:13,421 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:15,423 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:17,425 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:17,614 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:19,428 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:21,438 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:22,671 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:23,440 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:25,442 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:27,444 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:27,595 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:49:27,762 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:28,377 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:49:28,377 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:49:29,449 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:31,451 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:32,764 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:33,454 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:35,456 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:37,458 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:37,814 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:39,460 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:41,462 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:42,923 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:43,377 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:49:43,378 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:49:43,465 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:45,467 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:46,073 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:49:46,074 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:49:46,075 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:49:46,077 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:49:46,468 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:46,469 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:49:47,470 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:48,078 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:51,474 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:53,339 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:53,476 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:55,478 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:49:57,597 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:49:58,377 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:49:58,378 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:49:58,378 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:49:59,483 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:01,485 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:03,487 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:03,723 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:05,489 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:08,917 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:09,493 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:11,496 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:13,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:50:13,383 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:50:13,498 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:14,482 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:17,501 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:19,505 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:19,779 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:21,508 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:23,510 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:24,961 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:27,514 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:27,601 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:50:28,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:50:28,383 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:50:29,517 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:30,139 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:31,519 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:35,351 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:35,523 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:37,525 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:39,528 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:40,425 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:43,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:50:43,383 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:50:43,531 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:45,502 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:45,533 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:47,535 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:49,538 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:50,833 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:53,541 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:55,543 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:56,076 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:50:57,549 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:50:57,603 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:50:58,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:50:58,383 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:51:01,160 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:01,553 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:03,555 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:05,557 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:06,421 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:09,560 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:11,562 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:11,752 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:13,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:51:13,383 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:51:13,564 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:14,423 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:51:14,424 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:51:14,424 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:51:14,427 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:51:14,565 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:51:15,566 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:16,567 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:16,981 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:19,571 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:21,573 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:22,120 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:23,575 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:27,348 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:27,580 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:27,606 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:51:28,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:51:28,384 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:51:29,582 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:31,585 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:32,514 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:35,589 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:37,591 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:37,843 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:39,594 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:42,978 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:43,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:51:43,383 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:51:43,597 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:45,599 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:47,601 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:48,113 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:49,603 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:53,346 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:51:53,607 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:55,609 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:57,610 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:51:57,612 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:51:58,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:51:58,384 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:51:58,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:01,615 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:03,617 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:03,769 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:05,619 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:08,933 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:09,622 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:11,624 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:13,383 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:52:13,384 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:52:13,626 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:14,134 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:15,628 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:19,481 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:19,632 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:21,634 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:23,636 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:24,487 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:27,612 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:52:27,640 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:28,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:52:28,384 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:52:29,642 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:29,737 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:31,643 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:33,645 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:34,967 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:37,650 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:39,651 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:40,036 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:41,653 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:43,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:52:43,384 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:52:45,078 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:45,656 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:47,658 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:49,659 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:50,218 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:51,661 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:55,400 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:52:55,665 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:57,616 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:52:57,667 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:52:58,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:52:58,384 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:52:59,670 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:00,572 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:03,674 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:05,676 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:05,885 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:07,678 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:11,006 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:11,682 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:13,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:53:13,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:53:13,684 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:15,685 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:16,091 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:17,688 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:21,691 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:22,079 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:23,694 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:25,696 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:27,145 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:27,619 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:53:27,698 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:28,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:53:28,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:53:31,701 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:32,314 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:33,705 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:35,707 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:37,612 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:39,710 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:41,712 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:42,794 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:43,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:53:43,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:53:43,714 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:45,718 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:47,877 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:49,723 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:51,725 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:53,004 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:53,726 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:57,622 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:53:57,730 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:53:58,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:53:58,385 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:53:58,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:53:59,732 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:01,734 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:03,419 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:03,736 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:07,740 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:08,493 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:09,742 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:11,744 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:13,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:54:13,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:54:14,471 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:15,747 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:17,750 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:19,753 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:19,824 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:21,755 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:24,933 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:25,759 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:27,625 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:54:27,761 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:28,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:54:28,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:54:29,764 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:30,477 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:33,767 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:35,649 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:35,770 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:37,772 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:39,774 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:40,840 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:43,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:54:43,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:54:43,778 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:45,780 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:45,872 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:47,782 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:51,100 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:51,785 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:53,788 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:55,790 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:56,335 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:54:57,629 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:54:57,793 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:54:58,384 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:54:58,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:55:01,679 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:01,796 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:03,799 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:05,802 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:06,848 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:09,805 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:11,808 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:12,077 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:13,385 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:55:13,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:55:13,810 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:17,303 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:17,814 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:19,817 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:21,819 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:22,505 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:25,822 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:27,633 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:55:27,635 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:27,824 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:28,385 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:55:28,385 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:55:29,827 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:31,829 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:32,984 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:32,997 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:55:32,999 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:55:32,999 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:55:33,003 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:55:33,831 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:55:34,842 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:35,855 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:38,004 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:43,005 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:43,385 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:55:43,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:55:48,456 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:54,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:55:55,902 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:57,634 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:55:57,904 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:55:58,385 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:55:58,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:55:59,475 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:01,907 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:03,909 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:04,643 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:05,911 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:07,913 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:09,721 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:10,916 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:12,918 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:13,385 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:56:13,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:56:14,920 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:14,983 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:18,924 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:20,171 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:20,926 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:22,928 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:24,931 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:25,281 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:27,638 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:56:28,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:56:28,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:56:28,934 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:30,564 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:30,942 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:32,944 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:35,758 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:36,947 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:38,949 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:40,885 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:40,951 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:43,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:56:43,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:56:44,955 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:46,102 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:46,957 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:48,959 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:50,965 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:51,330 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:54,969 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:56,421 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:56:56,971 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:56:57,642 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:56:58,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:56:58,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:56:58,974 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:01,545 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:02,977 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:04,979 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:06,571 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:06,981 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:08,983 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:11,653 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:12,986 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:13,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:57:13,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:57:14,988 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:16,855 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:16,990 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:20,993 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:22,084 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:22,995 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:24,998 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:27,000 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:27,274 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:27,643 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:57:28,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:57:28,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:57:31,003 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:32,394 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:33,005 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:35,008 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:37,010 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:37,470 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:41,013 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:42,718 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:43,015 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:43,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:57:43,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:57:45,017 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:47,923 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:49,020 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:51,023 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:53,025 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:53,038 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:57,029 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:57:57,646 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:57:58,039 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:57:58,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:57:58,386 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:57:59,031 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:01,033 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:03,035 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:03,183 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:05,037 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:07,039 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:08,315 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:09,041 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:11,043 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:13,045 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:13,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:58:13,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:13,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:58:15,047 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:17,049 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:18,455 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:19,050 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:21,054 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:23,056 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:23,497 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:25,058 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:27,060 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:27,650 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:58:28,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:58:28,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:58:28,518 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:29,063 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:31,065 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:33,067 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:33,645 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:35,070 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:37,072 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:38,664 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:39,075 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:41,077 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:43,079 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:43,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:58:43,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:58:43,713 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:45,081 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:47,084 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:48,771 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:49,086 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:51,089 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:53,091 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:53,775 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:58:55,093 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:55,427 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 17:58:55,429 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 17:58:55,430 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 17:58:55,432 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 17:58:56,094 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 17:58:57,095 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:57,653 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:58:58,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:58:58,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:58:59,097 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:58:59,565 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:01,099 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:05,103 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:05,146 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:07,105 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:09,107 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:10,450 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:13,110 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:13,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:59:13,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:59:15,112 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:15,606 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:17,114 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:20,794 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:21,118 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:23,120 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:25,122 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:25,860 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:27,124 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:27,657 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:59:28,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:59:28,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:59:31,077 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:31,128 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:33,130 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:35,133 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:36,398 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:39,136 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:41,138 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:41,641 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:43,140 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:43,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:59:43,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:59:46,712 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:47,143 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:49,145 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:51,148 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:51,926 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:53,150 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:57,153 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 17:59:57,181 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 17:59:57,659 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 17:59:58,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 17:59:58,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 17:59:59,155 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:01,158 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:02,321 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:05,161 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:07,163 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:07,640 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:09,165 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:12,928 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:13,168 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:13,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:00:13,387 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:00:15,170 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:17,172 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:18,100 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:21,175 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:23,176 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:23,179 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:25,179 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:27,181 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:27,662 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:00:28,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:00:28,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:00:28,388 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:31,185 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:33,187 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:33,584 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:35,190 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:38,732 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:39,193 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:41,195 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:43,197 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:43,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:00:43,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:00:44,512 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:47,200 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:49,203 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:49,818 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:51,205 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:53,208 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:55,045 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:00:57,211 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:00:57,667 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:00:58,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:00:58,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:00:59,213 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:00,112 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:01,215 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:05,218 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:05,313 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:07,220 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:07,833 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 18:01:07,835 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 18:01:07,836 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 18:01:07,838 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 18:01:08,221 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 18:01:09,223 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:10,482 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:11,225 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:13,387 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:01:13,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:01:15,228 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:15,577 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:17,231 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:19,233 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:20,662 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:23,236 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:25,239 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:25,895 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:27,241 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:27,669 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:01:28,388 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:01:28,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:01:29,243 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:31,080 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:33,246 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:35,249 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:36,257 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:37,251 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:41,255 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:41,631 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:43,257 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:43,388 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:01:43,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:01:45,259 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:46,905 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:49,263 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:51,266 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:52,029 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:53,268 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:55,270 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:01:57,182 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:01:57,672 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:01:58,388 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:01:58,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:01:59,274 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:01,280 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:02,198 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:03,283 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:07,286 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:07,455 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:09,289 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:11,291 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:12,702 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:13,388 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:02:13,388 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:02:15,294 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:17,297 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:18,021 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:19,299 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:23,174 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:23,303 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:25,305 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:27,307 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:27,675 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:02:28,367 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:28,396 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:02:28,396 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:02:29,310 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:33,314 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:34,343 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:35,317 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:37,320 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:39,322 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:39,347 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:43,325 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:43,393 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:02:43,393 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:02:44,536 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:45,327 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:47,330 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:49,332 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:49,571 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:53,336 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:54,730 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:02:55,338 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:57,341 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:02:57,678 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:02:58,394 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:02:58,394 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:03:00,080 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:01,345 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:03,348 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:05,349 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:05,361 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:09,353 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:10,461 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:11,355 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:13,358 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:13,394 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:03:13,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:03:15,615 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:17,361 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:19,363 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:20,783 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:21,365 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:23,367 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:25,915 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:27,370 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:27,681 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:03:28,394 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:03:28,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:03:29,372 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:31,118 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:31,373 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:35,378 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:36,251 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:37,380 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:39,382 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:41,322 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:41,384 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:43,394 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:03:43,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:03:45,387 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:46,412 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:47,389 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:49,391 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:51,707 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:53,396 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:55,398 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:57,034 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:03:57,400 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:03:57,684 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:03:58,394 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:03:58,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:04:01,404 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:02,287 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:03,406 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:05,408 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:07,409 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:07,410 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:11,413 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:12,434 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:13,395 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:04:13,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:04:13,415 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:15,418 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:17,680 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:19,421 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:21,423 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:22,969 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:23,426 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:25,428 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:27,739 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:04:28,395 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:04:28,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:04:28,543 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:29,431 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:31,433 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:33,435 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:34,517 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:37,439 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:39,442 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:39,764 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:41,443 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:43,395 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:04:43,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:04:45,447 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:45,535 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:47,449 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:49,451 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:50,823 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:53,455 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:55,457 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:55,992 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:04:57,459 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:04:57,741 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:04:58,395 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:04:58,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:05:01,126 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:01,463 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:03,465 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:05,467 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:06,337 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:07,469 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:11,472 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:11,539 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:13,395 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:05:13,395 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:05:13,474 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:15,476 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:16,824 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:19,480 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:21,482 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:21,896 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:23,484 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:25,486 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:27,030 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:27,043 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 18:05:27,045 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 18:05:27,045 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 18:05:27,047 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 18:05:27,488 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 18:05:27,744 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:05:28,410 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:05:28,411 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:05:29,490 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:31,492 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:32,266 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:33,494 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:37,386 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:37,498 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:39,500 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:41,502 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:42,613 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:43,404 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:05:43,404 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:05:43,504 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:47,508 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:47,853 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:49,509 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:51,513 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:53,012 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:55,516 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:57,518 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:05:57,748 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:05:58,086 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:05:58,404 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:05:58,404 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:05:59,520 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:01,522 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:03,137 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:05,525 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:07,527 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:08,193 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:09,529 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:13,348 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:13,404 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:06:13,404 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:06:13,533 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:15,535 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:17,538 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:18,422 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:19,540 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:23,543 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:23,561 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:25,546 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:27,548 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:27,750 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:06:28,404 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:06:28,404 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:06:28,715 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:31,551 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:33,553 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:34,714 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:35,556 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:37,558 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:39,845 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:41,562 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:43,404 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:06:43,404 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:06:43,564 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:45,141 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:45,566 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:49,569 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:50,204 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:51,572 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:53,574 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:55,204 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:06:55,576 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:57,578 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:06:57,753 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:06:58,404 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:06:58,404 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:06:59,581 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:00,337 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:01,582 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:03,584 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:05,442 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:05,587 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:07,589 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:09,591 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:10,485 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:11,593 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:13,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:07:13,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:07:13,595 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:15,503 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:15,597 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:17,599 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:19,601 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:20,597 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:21,603 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:23,605 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:25,607 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:25,652 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:27,610 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:27,756 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:07:28,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:07:28,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:07:29,611 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:30,680 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:31,613 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:33,615 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:35,618 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:35,727 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:37,621 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:39,623 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:40,789 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:41,625 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:43,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:07:43,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:07:43,627 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:45,629 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:45,792 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:47,597 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 18:07:47,599 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 18:07:47,600 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 18:07:47,602 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 18:07:47,631 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:47,631 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 18:07:48,632 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:49,633 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:51,166 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:51,637 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:55,640 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:56,349 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:07:57,642 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:07:57,759 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:07:58,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:07:58,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:07:59,645 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:01,605 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:01,646 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:05,649 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:06,740 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:07,651 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:09,653 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:11,923 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:13,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:08:13,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:08:13,657 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:15,658 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:17,660 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:17,789 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:19,662 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:23,064 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:23,666 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:25,668 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:27,670 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:27,764 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:08:28,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:08:28,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:08:28,517 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:31,673 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:33,675 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:34,087 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:35,678 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:37,680 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:39,173 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:41,683 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:43,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:08:43,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:08:43,685 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:44,356 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:45,687 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:49,414 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:49,690 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:51,693 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:53,695 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:54,513 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:55,697 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:08:57,765 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:08:58,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:08:58,405 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:08:59,652 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:08:59,701 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:01,703 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:03,705 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:04,923 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:07,709 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:09,711 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:09,955 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:11,713 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:13,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:09:13,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:09:13,715 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:15,152 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:17,719 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:19,721 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:20,454 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:21,723 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:25,570 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:25,726 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:27,727 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:27,768 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:09:28,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:09:28,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:09:29,729 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:30,573 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:33,732 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:35,735 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:36,017 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:37,738 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:39,740 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:41,288 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:43,405 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:09:43,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:09:43,743 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:45,746 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:46,499 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:47,748 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:51,703 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:51,753 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:53,755 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:55,757 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:09:56,867 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:09:57,772 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:09:58,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:09:58,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:09:59,761 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:01,763 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:01,997 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:03,765 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:05,768 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:07,136 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:09,772 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:11,774 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:12,365 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:13,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:10:13,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:10:13,778 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:17,568 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:17,781 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:19,784 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:21,786 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:22,767 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:25,790 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:27,775 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:10:27,778 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:27,791 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:28,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:10:28,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:10:29,793 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:31,795 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:33,190 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:35,800 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:37,802 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:38,429 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:39,805 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:43,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:10:43,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:10:43,548 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:43,672 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 18:10:43,673 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 18:10:43,674 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 18:10:43,676 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 18:10:43,808 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:43,808 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 18:10:44,809 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:45,811 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:47,813 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:48,866 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:51,818 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:53,820 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:54,023 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:10:55,822 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:57,778 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:10:57,825 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:10:58,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:10:58,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:10:59,222 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:01,829 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:03,831 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:04,370 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:05,833 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:09,678 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:09,837 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:11,839 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:13,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:11:13,406 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:11:13,840 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:14,834 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:17,844 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:19,846 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:19,943 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:21,849 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:23,851 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:25,058 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:27,781 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:11:27,855 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:28,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:11:28,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:11:29,857 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:30,158 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:31,859 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:33,861 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:35,356 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:37,866 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:39,868 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:40,548 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:41,870 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:43,406 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:11:43,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:11:45,712 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:45,874 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:47,876 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:49,878 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:50,918 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:53,883 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:55,885 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:56,043 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:11:57,785 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:11:57,888 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:11:58,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:11:58,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:11:59,891 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:01,226 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:03,894 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:05,896 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:06,425 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:07,899 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:11,651 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:11,901 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:13,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:12:13,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:12:13,904 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:15,906 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:16,803 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:19,910 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:21,848 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:21,912 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:23,914 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:25,916 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:27,012 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:27,787 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:12:28,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:12:28,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:12:29,919 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:31,921 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:32,110 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:33,923 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:35,926 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:37,150 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:39,930 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:41,932 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:42,447 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:43,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:12:43,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:12:43,934 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:47,586 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:47,936 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:49,938 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:51,942 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:52,825 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:55,945 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:57,791 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:12:57,914 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:12:57,948 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:12:58,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:12:58,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:12:59,950 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:01,952 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:03,069 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:05,956 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:07,958 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:09,051 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:09,961 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:11,963 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:13,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:13:13,407 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:13:14,584 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:15,967 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:17,969 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:19,971 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:20,446 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:23,975 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:25,586 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:25,977 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:27,796 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:13:27,979 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:28,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:13:28,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:13:29,982 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:30,774 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:33,986 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:35,989 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:36,007 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:37,990 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:41,149 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:41,994 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:43,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:13:43,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:13:43,996 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:45,998 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:46,301 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:48,000 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:51,936 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:52,005 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:54,007 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:56,009 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:13:56,986 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:13:57,797 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:13:58,407 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:13:58,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:14:00,014 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:02,016 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:02,088 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:04,018 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:06,020 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:07,287 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:10,024 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:12,026 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:12,498 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:13,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:14:13,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:14:14,028 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:17,800 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:18,032 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:20,034 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:22,036 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:22,916 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:26,040 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:27,800 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:14:28,029 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:28,042 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:28,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:14:28,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:14:30,044 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:32,046 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:33,162 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:36,051 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:38,054 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:38,335 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:40,056 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:42,058 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:43,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:14:43,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:14:43,481 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:46,062 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:48,064 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:48,612 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:50,066 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:53,832 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:14:54,070 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:56,072 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:57,804 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:14:58,075 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:14:58,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:14:58,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:14:58,840 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:01,496 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 18:15:01,497 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 18:15:01,497 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 18:15:01,500 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 18:15:02,079 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:02,079 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 18:15:03,080 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:04,082 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:04,089 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:06,084 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:08,086 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:09,266 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:12,090 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:13,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:15:13,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:15:14,092 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:14,449 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:16,094 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:19,588 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:20,098 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:22,100 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:24,102 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:24,820 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:26,105 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:27,807 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:15:28,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:15:28,408 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:15:29,872 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:30,108 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:32,111 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:32,139 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: partial_history 2024-05-09 18:15:32,141 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 18:15:32,142 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 18:15:32,142 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 18:15:32,142 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 18:15:32,142 DEBUG SenderThread:72 [sender.py:send():379] send: metric 2024-05-09 18:15:32,143 DEBUG SenderThread:72 [sender.py:send():379] send: history 2024-05-09 18:15:32,143 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: summary_record 2024-05-09 18:15:32,145 INFO SenderThread:72 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end 2024-05-09 18:15:33,112 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:33,112 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/wandb-summary.json 2024-05-09 18:15:34,119 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:35,145 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:40,149 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:41,140 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/config.yaml 2024-05-09 18:15:43,409 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:15:43,409 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:15:45,365 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:46,144 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:48,146 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:50,148 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:50,395 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:52,150 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:54,153 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:55,553 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:15:56,155 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:57,810 DEBUG SenderThread:72 [sender.py:send():379] send: stats 2024-05-09 18:15:58,157 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:15:58,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:15:58,409 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:16:00,160 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:00,630 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:16:02,161 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:04,163 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:05,893 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:16:06,165 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:08,167 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:10,170 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:11,060 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: status_report 2024-05-09 18:16:12,172 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log 2024-05-09 18:16:13,408 DEBUG HandlerThread:72 [handler.py:handle_request():144] handle_request: stop_status 2024-05-09 18:16:13,409 DEBUG SenderThread:72 [sender.py:send_request():406] send_request: stop_status 2024-05-09 18:16:14,174 INFO Thread-12 :72 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/toolkit/trl/results/a805570edd5eddcfb7deb3a583d7b4c2/code/wandb/run-20240509_173156-a805570edd5eddcfb7deb3a583d7b4c2/files/output.log