[2025-04-22 04:16:17,607] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-22 04:16:18,861] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-22 04:16:19,106] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-22 04:16:19,282] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-22 04:16:19,284] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-22 04:16:19,442] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO 04-22 04:16:19 __init__.py:190] Automatically detected platform cuda.
[2025-04-22 04:16:20,490] [INFO] [comm.py:652:init_distributed] cdb=None
INFO 04-22 04:16:20 __init__.py:190] Automatically detected platform cuda.
INFO 04-22 04:16:20 __init__.py:190] Automatically detected platform cuda.
INFO 04-22 04:16:21 __init__.py:190] Automatically detected platform cuda.
INFO 04-22 04:16:21 __init__.py:190] Automatically detected platform cuda.
INFO 04-22 04:16:21 __init__.py:190] Automatically detected platform cuda.
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
[2025-04-22 04:16:21,777] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-22 04:16:21,896] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-22 04:16:21,896] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2025-04-22 04:16:22,069] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-22 04:16:22,081] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-22 04:16:22,221] [INFO] [comm.py:652:init_distributed] cdb=None
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
[2025-04-22 04:16:23,659] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
[2025-04-22 04:16:23,865] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO 04-22 04:16:25 __init__.py:190] Automatically detected platform cuda.
INFO 04-22 04:16:25 __init__.py:190] Automatically detected platform cuda.
[2025-04-22 04:16:26,459] [INFO] [comm.py:652:init_distributed] cdb=None
[2025-04-22 04:16:26,504] [INFO] [comm.py:652:init_distributed] cdb=None
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
using:  <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
[2025-04-22 04:16:40,428] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,429] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,429] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,429] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,429] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,430] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,484] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:40,557] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
2025-04-22 04:16:40.701 n116-043-044:107010:107010 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.701 n116-043-044:107010:107010 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.702 n116-043-044:107010:107010 [0] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.702 n116-043-044:107010:107010 [0] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.719 n116-043-044:107010:107010 [0] NCCL INFO NCCL LOG Enabled
NCCL version 2.21.5-2+cuda12.4
2025-04-22 04:16:40.720 n116-043-044:107015:107015 [5] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107016:107016 [6] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107017:107017 [7] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107015:107015 [5] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107016:107016 [6] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107015:107015 [5] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107017:107017 [7] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107015:107015 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.720 n116-043-044:107016:107016 [6] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107016:107016 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.720 n116-043-044:107017:107017 [7] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107017:107017 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.720 n116-043-044:107012:107012 [2] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107012:107012 [2] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107014:107014 [4] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107012:107012 [2] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107012:107012 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.720 n116-043-044:107014:107014 [4] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107014:107014 [4] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107014:107014 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.720 n116-043-044:107011:107011 [1] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107013:107013 [3] NCCL INFO cudaDriverVersion 12040
2025-04-22 04:16:40.720 n116-043-044:107011:107011 [1] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107013:107013 [3] NCCL INFO NCCL LOG Enabled
2025-04-22 04:16:40.720 n116-043-044:107011:107011 [1] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107011:107011 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.720 n116-043-044:107013:107013 [3] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET6
2025-04-22 04:16:40.720 n116-043-044:107013:107013 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to =eth0
2025-04-22 04:16:40.722 n116-043-044:107016:107016 [6] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.722 n116-043-044:107014:107014 [4] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.722 n116-043-044:107017:107017 [7] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.722 n116-043-044:107015:107015 [5] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.722 n116-043-044:107013:107013 [3] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.722 n116-043-044:107011:107011 [1] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.722 n116-043-044:107012:107012 [2] NCCL INFO Bootstrap : Using eth0:fdbd:dccd:cdc1:1368:0:368::<0>
2025-04-22 04:16:40.723 n116-043-044:107016:107016 [6] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.723 n116-043-044:107014:107014 [4] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.723 n116-043-044:107015:107015 [5] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.723 n116-043-044:107017:107017 [7] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.723 n116-043-044:107013:107013 [3] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.723 n116-043-044:107011:107011 [1] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.723 n116-043-044:107012:107012 [2] NCCL INFO NET/Plugin: No plugin found (gcp-fastrak)
2025-04-22 04:16:40.740 n116-043-044:107013:107013 [3] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107015:107015 [5] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107014:107014 [4] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107012:107012 [2] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107016:107016 [6] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107017:107017 [7] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107015:107015 [5] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107014:107014 [4] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107013:107013 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107012:107012 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107017:107017 [7] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107016:107016 [6] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107014:107014 [4] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107015:107015 [5] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107013:107013 [3] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107014:107014 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107015:107015 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107014:107014 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.740 n116-043-044:107017:107017 [7] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107013:107013 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107015:107015 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.740 n116-043-044:107016:107016 [6] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107017:107017 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107013:107013 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.740 n116-043-044:107012:107012 [2] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107016:107016 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107017:107017 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.740 n116-043-044:107016:107016 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.740 n116-043-044:107012:107012 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107012:107012 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:40.740 n116-043-044:107011:107011 [1] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:40.740 n116-043-044:107011:107011 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107011:107011 [1] NCCL INFO NET/Plugin: Loaded net plugin FasTrak (v7)
2025-04-22 04:16:40.740 n116-043-044:107011:107011 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
2025-04-22 04:16:40.740 n116-043-044:107011:107011 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (>= v5). ncclCollNetPlugin symbols v4 and lower are not supported.
2025-04-22 04:16:41.188 n116-043-044:107010:108352 [0] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.188 n116-043-044:107010:108352 [0] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.193 n116-043-044:107015:108353 [5] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.193 n116-043-044:107015:108353 [5] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.206 n116-043-044:107016:108358 [6] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.206 n116-043-044:107016:108358 [6] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.215 n116-043-044:107011:108359 [1] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.215 n116-043-044:107011:108359 [1] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.217 n116-043-044:107014:108355 [4] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.217 n116-043-044:107014:108355 [4] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.217 n116-043-044:107017:108354 [7] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.217 n116-043-044:107017:108354 [7] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.224 n116-043-044:107013:108356 [3] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.224 n116-043-044:107013:108356 [3] NCCL INFO Using network FasTrak
2025-04-22 04:16:41.228 n116-043-044:107012:108357 [2] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:16:41.228 n116-043-044:107012:108357 [2] NCCL INFO Using network FasTrak
2025-04-22 04:16:42.517 n116-043-044:107016:108358 [6] NCCL INFO DMA-BUF is available on GPU device 6
2025-04-22 04:16:42.519 n116-043-044:107010:108352 [0] NCCL INFO DMA-BUF is available on GPU device 0
2025-04-22 04:16:42.560 n116-043-044:107015:108353 [5] NCCL INFO DMA-BUF is available on GPU device 5
2025-04-22 04:16:42.583 n116-043-044:107014:108355 [4] NCCL INFO DMA-BUF is available on GPU device 4
2025-04-22 04:16:42.595 n116-043-044:107011:108359 [1] NCCL INFO DMA-BUF is available on GPU device 1
2025-04-22 04:16:42.598 n116-043-044:107017:108354 [7] NCCL INFO DMA-BUF is available on GPU device 7
2025-04-22 04:16:42.607 n116-043-044:107012:108357 [2] NCCL INFO DMA-BUF is available on GPU device 2
2025-04-22 04:16:42.608 n116-043-044:107013:108356 [3] NCCL INFO DMA-BUF is available on GPU device 3
2025-04-22 04:16:42.609 n116-043-044:107013:108356 [3] NCCL INFO ncclCommInitRank comm 0x10ec6240 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId c000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107012:108357 [2] NCCL INFO ncclCommInitRank comm 0xfeca910 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId b000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107014:108355 [4] NCCL INFO ncclCommInitRank comm 0xfdc8140 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 84000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107011:108359 [1] NCCL INFO ncclCommInitRank comm 0x1042b540 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 5000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107015:108353 [5] NCCL INFO ncclCommInitRank comm 0x10c641c0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 85000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107010:108352 [0] NCCL INFO ncclCommInitRank comm 0xf80cf60 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 4000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107017:108354 [7] NCCL INFO ncclCommInitRank comm 0xf646140 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 8c000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:42.609 n116-043-044:107016:108358 [6] NCCL INFO ncclCommInitRank comm 0x11d0a1d0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 8b000 commId 0x6ec617b8d4362414 - Init START
2025-04-22 04:16:43.274 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.277 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.293 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.299 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.300 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.302 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.304 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.309 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.832 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.838 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.860 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.868 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.870 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.874 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.874 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:43.879 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.387 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.433 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.435 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.492 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.498 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.500 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.525 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.544 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:44.739 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.117 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.206 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.246 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.271 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.272 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.301 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.373 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.540 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.763 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.841 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.843 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.869 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.881 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.900 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:45.955 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.105 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.314 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.446 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.448 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.451 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.465 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.474 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.579 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.768 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.799 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:46.932 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.088 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.102 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.107 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.125 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.127 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.454 n116-043-044:107013:108356 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.458 n116-043-044:107013:108356 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.467 n116-043-044:107011:108359 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.471 n116-043-044:107011:108359 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.501 n116-043-044:107013:108356 [3] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.501 n116-043-044:107011:108359 [1] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO ==========================================
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO Setting affinity for GPU 3 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:16:47.503 n116-043-044:107013:108356 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO ==========================================
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.503 n116-043-044:107011:108359 [1] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.504 n116-043-044:107011:108359 [1] NCCL INFO Setting affinity for GPU 1 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:16:47.504 n116-043-044:107011:108359 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.511 n116-043-044:107013:108356 [3] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.512 n116-043-044:107011:108359 [1] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.533 n116-043-044:107017:108354 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.536 n116-043-044:107017:108354 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.579 n116-043-044:107017:108354 [7] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO ==========================================
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:16:47.581 n116-043-044:107017:108354 [7] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.590 n116-043-044:107017:108354 [7] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.606 n116-043-044:107014:108355 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.610 n116-043-044:107014:108355 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.634 n116-043-044:107012:108357 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.637 n116-043-044:107012:108357 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.643 n116-043-044:107014:108355 [4] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.644 n116-043-044:107014:108355 [4] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.644 n116-043-044:107014:108355 [4] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO ==========================================
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:16:47.645 n116-043-044:107014:108355 [4] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.647 n116-043-044:107010:108352 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.651 n116-043-044:107010:108352 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.653 n116-043-044:107014:108355 [4] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.665 n116-043-044:107012:108357 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO ==========================================
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO Setting affinity for GPU 2 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:16:47.666 n116-043-044:107012:108357 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.667 n116-043-044:107016:108358 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.670 n116-043-044:107016:108358 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.673 n116-043-044:107010:108352 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.674 n116-043-044:107012:108357 [2] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO ==========================================
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO Setting affinity for GPU 0 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:16:47.675 n116-043-044:107010:108352 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.676 n116-043-044:107015:108353 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:16:47.677 n116-043-044:107016:108358 [6] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO ==========================================
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:16:47.678 n116-043-044:107016:108358 [6] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.680 n116-043-044:107015:108353 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107016:108358 [6] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.686 n116-043-044:107015:108353 [5] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to PIX
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107010:108352 [0] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO ==========================================
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:16:47.688 n116-043-044:107015:108353 [5] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO comm 0xf646140 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO comm 0x11d0a1d0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO comm 0xf80cf60 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO comm 0xfeca910 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO comm 0x10c641c0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO comm 0xfdc8140 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO comm 0x1042b540 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 00 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 01 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 00 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 02 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 01 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 00 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 00 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 02 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 0 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 03 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 03 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 12 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 04 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 1 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 01 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 13 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107013:108356 [3] NCCL INFO comm 0x10ec6240 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 01 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 04 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 05 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 00 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 2 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 02 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 02 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 14 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 03 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 05 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 06 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 01 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 03 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 3 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 04 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 06 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 15 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 07 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 05 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 02 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 07 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 4 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 04 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107013:108356 [3] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 08 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 06 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 03 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 08 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 16 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 04 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 05 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 07 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 09 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 09 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 5 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 05 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 06 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 08 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 10 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 10 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 09 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 17 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 06 : 3 -> 4 -> 5
2025-04-22 04:16:47.696 n116-043-044:107012:108357 [2] NCCL INFO Ring 07 : 1 -> 2 -> 3
2025-04-22 04:16:47.696 n116-043-044:107013:108356 [3] NCCL INFO Ring 00 : 2 -> 3 -> 4
2025-04-22 04:16:47.696 n116-043-044:107010:108352 [0] NCCL INFO Tree 6 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107015:108353 [5] NCCL INFO Ring 11 : 4 -> 5 -> 6
2025-04-22 04:16:47.696 n116-043-044:107016:108358 [6] NCCL INFO Ring 11 : 5 -> 6 -> 7
2025-04-22 04:16:47.696 n116-043-044:107011:108359 [1] NCCL INFO Tree 6 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.696 n116-043-044:107017:108354 [7] NCCL INFO Ring 10 : 6 -> 7 -> 0
2025-04-22 04:16:47.696 n116-043-044:107014:108355 [4] NCCL INFO Ring 07 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 08 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 12 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 12 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 18 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 13 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 13 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 01 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 18 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 11 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 14 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 08 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 09 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 7 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 14 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 09 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 7 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 02 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 12 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 10 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 15 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 10 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 19 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 03 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 16 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 15 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 19 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 11 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 13 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 11 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 8 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 17 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 12 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 04 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 16 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 18 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 8 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 12 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 17 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 14 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 20 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 13 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 05 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 19 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 20 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 18 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 13 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 06 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 15 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 14 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 14 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 07 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 9 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 20 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 9 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 19 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 16 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 15 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 15 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 21 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 16 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 08 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 21 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 21 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 20 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 17 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 16 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 10 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 17 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 21 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 22 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 17 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 18 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 09 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 10 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 18 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 22 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 10 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 22 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Ring 23 : 5 -> 6 -> 7
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 18 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 19 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 22 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 19 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 11 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Ring 23 : 4 -> 5 -> 6
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 19 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 11 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 20 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 11 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 20 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Tree 23 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 21 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 20 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 12 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Tree 23 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 21 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 21 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 22 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 22 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Ring 23 : 1 -> 2 -> 3
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 13 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 22 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107016:108358 [6] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 4.
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Ring 23 : 3 -> 4 -> 5
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Ring 23 : 6 -> 7 -> 0
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 00/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 14 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 00 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107015:108353 [5] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 01 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 15 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 01/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 02 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 16 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107012:108357 [2] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 03 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 02/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107017:108354 [7] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 17 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107014:108355 [4] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 04 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 03/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 18 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 05 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 04/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 19 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 06 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 20 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 05/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 07 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 21 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 06/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 08 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 22 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 07/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 09 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Ring 23 : 2 -> 3 -> 4
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 10 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 08/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 11 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 09/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 12 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 13 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 10/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 14 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 11/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 15 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107013:108356 [3] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 16 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 12/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 17 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 13/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 18 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 14/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 19 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 20 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 15/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 21 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 16/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 22 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 17/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Ring 23 : 0 -> 1 -> 2
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 18/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 19/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 20/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 21/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107011:108359 [1] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 22/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Channel 23/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 00 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 01 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 02 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 03 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 04 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 05 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 06 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 07 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 08 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 09 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 10 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 11 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 12 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 13 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 14 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 15 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 16 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 17 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 18 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 19 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 20 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 21 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 22 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Ring 23 : 7 -> 0 -> 1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO NCCL_BUFFSIZE set by environment to 4194304.
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO NCCL_P2P_NVL_CHUNKSIZE set by environment to 1048576.
2025-04-22 04:16:47.697 n116-043-044:107010:108352 [0] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:16:47.715 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 00/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.715 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 00/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.715 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 00/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.715 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 00/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.715 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 00/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.715 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 01/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.715 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 01/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 01/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 01/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 01/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 00/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 02/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 02/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 00/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 02/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.716 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 00/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 02/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 01/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 02/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 03/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 03/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 01/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 03/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 01/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 03/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 04/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 02/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 03/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.717 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 04/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 02/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 04/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 02/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 04/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 05/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 04/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 03/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 05/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 03/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 05/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 05/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.718 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 03/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 06/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 05/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 04/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 06/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 04/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 06/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 06/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 04/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 06/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 07/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 07/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.719 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 05/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 05/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 07/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 07/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 05/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 07/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 08/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 08/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 06/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 06/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 08/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.720 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 08/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 06/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 08/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 09/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 09/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 07/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 09/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 07/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 09/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 07/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 09/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 10/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 10/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.721 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 08/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 10/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 08/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 10/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 08/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 11/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 10/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 11/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 09/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 11/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 11/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 09/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.722 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 09/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 12/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 11/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 12/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 10/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 12/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 12/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 10/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 10/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 13/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 12/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 13/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 11/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 13/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.723 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 13/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 11/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 11/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 14/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 13/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 14/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 12/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 14/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 14/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 12/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 12/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 15/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.724 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 14/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 15/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 13/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 15/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 15/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 13/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 13/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 16/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 15/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 16/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 16/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 16/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.725 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 14/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 14/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 17/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 14/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 16/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 17/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 17/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 17/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 15/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 18/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 15/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 15/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.726 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 17/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 18/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 18/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 18/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 16/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 19/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 16/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 18/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 16/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 19/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 19/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 19/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.727 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 17/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 20/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 19/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 17/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 17/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 20/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 20/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 20/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 18/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 21/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 20/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 18/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.728 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 18/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 21/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 21/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 21/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 19/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 22/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 21/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 19/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 19/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 22/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 22/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 22/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 20/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.729 n116-043-044:107016:108358 [6] NCCL INFO Ring Channel 23/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 22/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 20/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 20/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107012:108357 [2] NCCL INFO Ring Channel 23/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107015:108353 [5] NCCL INFO Ring Channel 23/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107017:108354 [7] NCCL INFO Ring Channel 23/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 21/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107014:108355 [4] NCCL INFO Ring Channel 23/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 21/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.730 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 21/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.731 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 22/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.731 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 22/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.731 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 22/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:47.734 n116-043-044:107011:108359 [1] NCCL INFO Ring Channel 23/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:16:47.737 n116-043-044:107010:108352 [0] NCCL INFO Ring Channel 23/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:16:47.738 n116-043-044:107013:108356 [3] NCCL INFO Ring Channel 23/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.371 n116-043-044:107017:108354 [7] NCCL INFO Connected all rings
2025-04-22 04:16:48.371 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 00/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.371 n116-043-044:107016:108358 [6] NCCL INFO Connected all rings
2025-04-22 04:16:48.371 n116-043-044:107015:108353 [5] NCCL INFO Connected all rings
2025-04-22 04:16:48.385 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 01/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.388 n116-043-044:107011:108359 [1] NCCL INFO Connected all rings
2025-04-22 04:16:48.388 n116-043-044:107010:108352 [0] NCCL INFO Connected all rings
2025-04-22 04:16:48.390 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 02/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.393 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 03/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.393 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 04/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.393 n116-043-044:107012:108357 [2] NCCL INFO Connected all rings
2025-04-22 04:16:48.393 n116-043-044:107013:108356 [3] NCCL INFO Connected all rings
2025-04-22 04:16:48.394 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 05/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.394 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 06/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.395 n116-043-044:107014:108355 [4] NCCL INFO Connected all rings
2025-04-22 04:16:48.395 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 07/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.395 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 08/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.396 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 09/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.397 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 10/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.397 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 11/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.398 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 12/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.399 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 13/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.399 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 14/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.400 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 15/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.400 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 16/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.401 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 17/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.402 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 18/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.402 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 19/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.403 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 20/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.404 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 21/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.405 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 22/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.405 n116-043-044:107017:108354 [7] NCCL INFO Tree Channel 23/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:16:48.406 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 00/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.406 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 00/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.406 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 00/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.407 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 01/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.407 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 01/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.407 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 02/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.407 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 02/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.407 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 01/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.407 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 00/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.408 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 03/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.408 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 02/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.408 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 04/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.409 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 03/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.409 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 05/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.409 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 04/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.409 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 06/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.409 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 05/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 07/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 03/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 06/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 01/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 08/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 04/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 07/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 02/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.410 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 09/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 05/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 08/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 03/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 10/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 09/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 06/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 04/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.411 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 11/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 10/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 05/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 07/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 12/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 00/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 11/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 00/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 06/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 08/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.412 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 13/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 01/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 12/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 01/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 07/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 09/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 14/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 02/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 13/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.413 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 08/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 02/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 10/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 15/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 03/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 14/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 09/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 11/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 16/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 03/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 04/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 15/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 10/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.414 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 12/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 17/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 05/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 16/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 04/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 11/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 13/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 18/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 06/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 17/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 12/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 05/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 14/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.415 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 19/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 07/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 18/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 13/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 06/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 15/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 20/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 08/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 19/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 14/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 16/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 21/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 07/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.416 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 09/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 20/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 15/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 17/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 22/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 10/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 08/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 21/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 16/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107015:108353 [5] NCCL INFO Tree Channel 23/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 18/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.417 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 11/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 22/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 09/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 17/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 19/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 12/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107011:108359 [1] NCCL INFO Tree Channel 23/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 18/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 10/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 20/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 13/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.418 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 19/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.419 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 11/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.419 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 21/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.419 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 14/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.421 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 20/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.422 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 22/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.422 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 12/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.422 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 15/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.424 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 21/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.425 n116-043-044:107016:108358 [6] NCCL INFO Tree Channel 23/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:16:48.425 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 16/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.425 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 13/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.427 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 22/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.429 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 17/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.429 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 14/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.430 n116-043-044:107013:108356 [3] NCCL INFO Tree Channel 23/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:16:48.432 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 15/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.434 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 18/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.435 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 16/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.438 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 19/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.447 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 17/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.450 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 20/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.453 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 21/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.458 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 22/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.459 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 18/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.466 n116-043-044:107012:108357 [2] NCCL INFO Tree Channel 23/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:16:48.469 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 19/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.476 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 20/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.485 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 21/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.495 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 22/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.505 n116-043-044:107014:108355 [4] NCCL INFO Tree Channel 23/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:16:48.804 n116-043-044:107017:108354 [7] NCCL INFO Connected all trees
2025-04-22 04:16:48.804 n116-043-044:107017:108354 [7] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.804 n116-043-044:107017:108354 [7] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.804 n116-043-044:107017:108354 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.804 n116-043-044:107017:108354 [7] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.914 n116-043-044:107010:108352 [0] NCCL INFO Connected all trees
2025-04-22 04:16:48.914 n116-043-044:107010:108352 [0] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.914 n116-043-044:107010:108352 [0] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.914 n116-043-044:107010:108352 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.914 n116-043-044:107010:108352 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.948 n116-043-044:107015:108353 [5] NCCL INFO Connected all trees
2025-04-22 04:16:48.948 n116-043-044:107016:108358 [6] NCCL INFO Connected all trees
2025-04-22 04:16:48.948 n116-043-044:107015:108353 [5] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.948 n116-043-044:107016:108358 [6] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.948 n116-043-044:107015:108353 [5] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.948 n116-043-044:107016:108358 [6] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.948 n116-043-044:107015:108353 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.948 n116-043-044:107015:108353 [5] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.948 n116-043-044:107016:108358 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.948 n116-043-044:107016:108358 [6] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.949 n116-043-044:107011:108359 [1] NCCL INFO Connected all trees
2025-04-22 04:16:48.950 n116-043-044:107011:108359 [1] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.950 n116-043-044:107011:108359 [1] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.950 n116-043-044:107011:108359 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.950 n116-043-044:107011:108359 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.951 n116-043-044:107012:108357 [2] NCCL INFO Connected all trees
2025-04-22 04:16:48.951 n116-043-044:107012:108357 [2] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.951 n116-043-044:107012:108357 [2] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.951 n116-043-044:107012:108357 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.951 n116-043-044:107012:108357 [2] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.951 n116-043-044:107014:108355 [4] NCCL INFO Connected all trees
2025-04-22 04:16:48.951 n116-043-044:107013:108356 [3] NCCL INFO Connected all trees
2025-04-22 04:16:48.951 n116-043-044:107014:108355 [4] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.951 n116-043-044:107014:108355 [4] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.951 n116-043-044:107014:108355 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.951 n116-043-044:107014:108355 [4] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.951 n116-043-044:107013:108356 [3] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:16:48.951 n116-043-044:107013:108356 [3] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:16:48.951 n116-043-044:107013:108356 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:16:48.951 n116-043-044:107013:108356 [3] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:16:48.980 n116-043-044:107015:108353 [5] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107011:108359 [1] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107016:108358 [6] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107017:108354 [7] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107012:108357 [2] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107010:108352 [0] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107013:108356 [3] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.980 n116-043-044:107014:108355 [4] NCCL INFO TUNER/Plugin: NCCL_TUNER_PLUGIN set to libnccl-tuner.so
2025-04-22 04:16:48.981 n116-043-044:107017:108354 [7] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107015:108353 [5] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107012:108357 [2] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107017:108354 [7] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107015:108353 [5] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107012:108357 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107013:108356 [3] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107017:108354 [7] NCCL INFO ncclCommInitRank comm 0xf646140 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 8c000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107016:108358 [6] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107010:108352 [0] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107015:108353 [5] NCCL INFO ncclCommInitRank comm 0x10c641c0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 85000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107012:108357 [2] NCCL INFO ncclCommInitRank comm 0xfeca910 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId b000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107011:108359 [1] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107013:108356 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107016:108358 [6] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107010:108352 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107011:108359 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107013:108356 [3] NCCL INFO ncclCommInitRank comm 0x10ec6240 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId c000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107010:108352 [0] NCCL INFO ncclCommInitRank comm 0xf80cf60 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 4000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107016:108358 [6] NCCL INFO ncclCommInitRank comm 0x11d0a1d0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 8b000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107011:108359 [1] NCCL INFO ncclCommInitRank comm 0x1042b540 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 5000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
2025-04-22 04:16:48.981 n116-043-044:107014:108355 [4] NCCL INFO TUNER/Plugin: Plugin name set by env to libnccl-net-gcp-fastrak.so
2025-04-22 04:16:48.981 n116-043-044:107014:108355 [4] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
2025-04-22 04:16:48.981 n116-043-044:107014:108355 [4] NCCL INFO ncclCommInitRank comm 0xfdc8140 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 84000 commId 0x6ec617b8d4362414 type default - Init COMPLETE
[2025-04-22 04:16:49,377] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 729, num_elems = 8.29B
[2025-04-22 04:16:54,185] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,193] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,203] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,218] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,221] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,275] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,398] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,422] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:16:54,701] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 1458, num_elems = 16.58B
[2025-04-22 04:17:06,556] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,565] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,572] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,584] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,587] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,615] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,618] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.15.4, git-hash=unknown, git-branch=unknown
[2025-04-22 04:17:06,618] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:06,631] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2025-04-22 04:17:06,633] [INFO] [logging.py:128:log_dist] [Rank 0] Creating ZeRO Offload
[2025-04-22 04:17:06,794] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 8
[2025-04-22 04:17:07,012] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2025-04-22 04:17:07,012] [INFO] [utils.py:782:see_memory_usage] MA 3.88 GB         Max_MA 6.8 GB         CA 7.53 GB         Max_CA 8 GB 
[2025-04-22 04:17:07,012] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 36.82 GB, percent = 2.0%
Parameter Offload: Total persistent parameters: 848896 in 368 params
[2025-04-22 04:17:07,340] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2025-04-22 04:17:07,340] [INFO] [utils.py:782:see_memory_usage] MA 3.88 GB         Max_MA 3.88 GB         CA 7.53 GB         Max_CA 8 GB 
[2025-04-22 04:17:07,340] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 36.83 GB, percent = 2.0%
[2025-04-22 04:17:07,342] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2025-04-22 04:17:07,342] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-04-22 04:17:07,342] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-04-22 04:17:07,342] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2025-04-22 04:17:07,342] [INFO] [config.py:1003:print]   amp_params ................... False
[2025-04-22 04:17:07,342] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fa974798510>
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   dump_state ................... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   global_rank .................. 0
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 1
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   gradient_clipping ............ 5.0
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   optimizer_name ............... None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   optimizer_params ............. None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   pld_params ................... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   train_batch_size ............. 8
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  1
[2025-04-22 04:17:07,343] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... False
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   world_size ................... 8
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  False
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=True, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2025-04-22 04:17:07,344] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 3
[2025-04-22 04:17:07,344] [INFO] [config.py:989:print_user_config]   json = {
    "fp16": {
        "enabled": false, 
        "loss_scale": 0, 
        "loss_scale_window": 1000, 
        "initial_scale_power": 16, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "zero_optimization": {
        "stage": 3, 
        "offload_optimizer": {
            "device": "none", 
            "pin_memory": true
        }, 
        "offload_param": {
            "device": "none", 
            "pin_memory": true
        }, 
        "overlap_comm": true, 
        "contiguous_gradients": true, 
        "sub_group_size": 1.000000e+09, 
        "reduce_bucket_size": "auto", 
        "stage3_prefetch_bucket_size": "auto", 
        "stage3_param_persistence_threshold": "auto", 
        "stage3_max_live_parameters": 1.000000e+09, 
        "stage3_max_reuse_distance": 1.000000e+09, 
        "stage3_gather_16bit_weights_on_model_save": true
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 5.0, 
    "steps_per_print": inf, 
    "train_batch_size": 8, 
    "train_micro_batch_size_per_gpu": 1, 
    "wall_clock_breakdown": false, 
    "zero_optimization.reduce_bucket_size": 1.284506e+07, 
    "zero_optimization.stage3_param_persistence_threshold": 3.584000e+04, 
    "zero_optimization.stage3_prefetch_bucket_size": 1.156055e+07
}
Parameter Offload: Total persistent parameters: 848896 in 368 params
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_05079.mp4
problem_id: 21317
prompt_length: 692
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_14051.png
problem_id: 156316
prompt_length: 280
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_1P7c8f5K1Xw.mp4
problem_id: 163789
prompt_length: 1062
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1105/5573667855.mp4
problem_id: 239381
prompt_length: 1049
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/479.png
problem_id: 220826
prompt_length: 320
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2109.07509_0.jpg
problem_id: 192930
prompt_length: 342
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/1026.png
problem_id: 180227
prompt_length: 171
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/b180ced8545f477d7d0f76981231df3e.png
problem_id: 221449
prompt_length: 135
Invalidate trace cache @ step 0 and module 1522: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([0.2176, 0.2041, 0.1959, 0.1838, 0.1694, 0.1514, 0.1362, 0.1201],
       device='cuda:5')tensor([0.2942, 1.2760, 0.2664, 0.2439, 0.2294, 0.2093, 0.1919, 0.1693],
       device='cuda:3')

tensor([0.2705, 1.2525, 0.2401, 0.2254, 0.2098, 0.1950, 0.1767, 0.1576],
       device='cuda:2')
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([0.2008, 0.1823, 0.1582, 0.1608, 0.1481, 0.1376, 0.1220, 0.1042],
       device='cuda:0')
tensor([0.3156, 0.2967, 0.2847, 0.2647, 0.2418, 0.2232, 0.1978, 0.1727],
       device='cuda:7')
tensor([0.3485, 0.3416, 1.3219, 0.2982, 1.2797, 0.2583, 1.2350, 1.2092],
       device='cuda:6')
tensor([0.3871, 0.3719, 0.3528, 0.3310, 0.3101, 0.2878, 0.2654, 0.2370],
       device='cuda:1')
tensor([0.4307, 1.4091, 0.3893, 0.3695, 0.3499, 0.3243, 0.2956, 0.2653],
       device='cuda:4')
{'completion_length': 82.171875, 'rewards/accuracy_reward': 0.109375, 'rewards/efficiency reward': 0.24791249632835388, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.35728752613067627, 'reward_std': 0.22488021850585938, 'kl': 2.34375, 'epoch': 0}
2025-04-22 04:21:10.791 n116-043-044:107016:114074 [6] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107014:114075 [4] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107010:114076 [0] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107016:114074 [6] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107014:114075 [4] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107013:114072 [3] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107010:114076 [0] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107012:114071 [2] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107011:114077 [1] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107013:114072 [3] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107017:114073 [7] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107012:114071 [2] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107011:114077 [1] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107015:114070 [5] NCCL INFO Using non-device net plugin version 7
2025-04-22 04:21:10.791 n116-043-044:107017:114073 [7] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107015:114070 [5] NCCL INFO Using network FasTrak
2025-04-22 04:21:10.791 n116-043-044:107014:114075 [4] NCCL INFO DMA-BUF is available on GPU device 4
2025-04-22 04:21:10.791 n116-043-044:107010:114076 [0] NCCL INFO DMA-BUF is available on GPU device 0
2025-04-22 04:21:10.791 n116-043-044:107016:114074 [6] NCCL INFO DMA-BUF is available on GPU device 6
2025-04-22 04:21:10.791 n116-043-044:107013:114072 [3] NCCL INFO DMA-BUF is available on GPU device 3
2025-04-22 04:21:10.791 n116-043-044:107012:114071 [2] NCCL INFO DMA-BUF is available on GPU device 2
2025-04-22 04:21:10.791 n116-043-044:107011:114077 [1] NCCL INFO DMA-BUF is available on GPU device 1
2025-04-22 04:21:10.791 n116-043-044:107017:114073 [7] NCCL INFO DMA-BUF is available on GPU device 7
2025-04-22 04:21:10.791 n116-043-044:107015:114070 [5] NCCL INFO DMA-BUF is available on GPU device 5
2025-04-22 04:21:10.792 n116-043-044:107010:114076 [0] NCCL INFO bootstrapSplit: comm 0x7f9dca311e00 parent 0xf80cf60 rank 0 nranks 8 color -934961569 key 0 prev 7 next 1 - DONE
2025-04-22 04:21:10.792 n116-043-044:107017:114073 [7] NCCL INFO bootstrapSplit: comm 0x7efdfa310bd0 parent 0xf646140 rank 7 nranks 8 color -934961569 key 7 prev 6 next 0 - DONE
2025-04-22 04:21:10.792 n116-043-044:107015:114070 [5] NCCL INFO bootstrapSplit: comm 0x7fda922ffe50 parent 0x10c641c0 rank 5 nranks 8 color -934961569 key 5 prev 4 next 6 - DONE
2025-04-22 04:21:10.792 n116-043-044:107013:114072 [3] NCCL INFO bootstrapSplit: comm 0x7f040a310d10 parent 0x10ec6240 rank 3 nranks 8 color -934961569 key 3 prev 2 next 4 - DONE
2025-04-22 04:21:10.792 n116-043-044:107012:114071 [2] NCCL INFO bootstrapSplit: comm 0x7fa392310990 parent 0xfeca910 rank 2 nranks 8 color -934961569 key 2 prev 1 next 3 - DONE
2025-04-22 04:21:10.792 n116-043-044:107016:114074 [6] NCCL INFO bootstrapSplit: comm 0x7f181e312490 parent 0x11d0a1d0 rank 6 nranks 8 color -934961569 key 6 prev 5 next 7 - DONE
2025-04-22 04:21:10.792 n116-043-044:107017:114073 [7] NCCL INFO ncclCommSplit comm 0x7efdfa310bd0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 8c000 parent 0xf646140 color -934961569 key 7 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107015:114070 [5] NCCL INFO ncclCommSplit comm 0x7fda922ffe50 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 85000 parent 0x10c641c0 color -934961569 key 5 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107010:114076 [0] NCCL INFO ncclCommSplit comm 0x7f9dca311e00 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 4000 parent 0xf80cf60 color -934961569 key 0 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107012:114071 [2] NCCL INFO ncclCommSplit comm 0x7fa392310990 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId b000 parent 0xfeca910 color -934961569 key 2 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107014:114075 [4] NCCL INFO bootstrapSplit: comm 0x7ef072312e50 parent 0xfdc8140 rank 4 nranks 8 color -934961569 key 4 prev 3 next 5 - DONE
2025-04-22 04:21:10.792 n116-043-044:107011:114077 [1] NCCL INFO bootstrapSplit: comm 0x7fae32314340 parent 0x1042b540 rank 1 nranks 8 color -934961569 key 1 prev 0 next 2 - DONE
2025-04-22 04:21:10.792 n116-043-044:107016:114074 [6] NCCL INFO ncclCommSplit comm 0x7f181e312490 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 8b000 parent 0x11d0a1d0 color -934961569 key 6 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107013:114072 [3] NCCL INFO ncclCommSplit comm 0x7f040a310d10 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId c000 parent 0x10ec6240 color -934961569 key 3 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107014:114075 [4] NCCL INFO ncclCommSplit comm 0x7ef072312e50 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 84000 parent 0xfdc8140 color -934961569 key 4 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:10.792 n116-043-044:107011:114077 [1] NCCL INFO ncclCommSplit comm 0x7fae32314340 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 5000 parent 0x1042b540 color -934961569 key 1 commId 0xddba01ec4ad6d20 - Init START
2025-04-22 04:21:11.535 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.537 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.560 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.568 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.571 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.587 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.592 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:11.613 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.055 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.126 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.140 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.182 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.211 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.211 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.248 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.260 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.643 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.701 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.761 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.786 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.791 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.885 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.943 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:12.946 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.146 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.150 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.333 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.406 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.461 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.538 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.568 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.647 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.825 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.829 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:13.982 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.076 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.122 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.158 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.184 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.236 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.391 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.395 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.546 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.643 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.692 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.732 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.783 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:14.888 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.030 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.070 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.119 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.291 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.407 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.479 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.622 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.650 n116-043-044:107014:114075 [4] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.653 n116-043-044:107014:114075 [4] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO ==========================================
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.654 n116-043-044:107014:114075 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.662 n116-043-044:107014:114075 [4] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.665 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.766 n116-043-044:107016:114074 [6] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO ==========================================
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.770 n116-043-044:107016:114074 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.777 n116-043-044:107016:114074 [6] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.806 n116-043-044:107017:114073 [7] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO ==========================================
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.810 n116-043-044:107017:114073 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.818 n116-043-044:107017:114073 [7] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.885 n116-043-044:107011:114077 [1] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO ==========================================
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.889 n116-043-044:107011:114077 [1] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.890 n116-043-044:107011:114077 [1] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.890 n116-043-044:107011:114077 [1] NCCL INFO Setting affinity for GPU 1 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.901 n116-043-044:107011:114077 [1] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.902 n116-043-044:107011:114077 [1] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.980 n116-043-044:107015:114070 [5] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:15.984 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO ==========================================
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:15.985 n116-043-044:107015:114070 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffffff,f0000000,000000ff,ffffffff,fff00000,00000000
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:15.992 n116-043-044:107015:114070 [5] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.005 n116-043-044:107013:114072 [3] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO ==========================================
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.009 n116-043-044:107013:114072 [3] NCCL INFO Setting affinity for GPU 3 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.017 n116-043-044:107013:114072 [3] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.027 n116-043-044:107012:114071 [2] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:16.030 n116-043-044:107010:114076 [0] NCCL INFO Could not find real path of /sys/class/pci_bus/fffffff/../../fffffff:ff:f
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO ==========================================
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.031 n116-043-044:107012:114071 [2] NCCL INFO Setting affinity for GPU 2 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO === System : maxBw 370.8 totalBw 370.8 ===
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO CPU/0-0 (1/1/2)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO + PCI[24.0] - PCI/0-2000 (10b5879610b58796)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-4000 (0)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-5000 (1)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-6000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-7000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO + PCI[24.0] - PCI/0-9000 (10b5879610b58796)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-b000 (2)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-c000 (3)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-d000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-e000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO + SYS[10.0] - CPU/1
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO CPU/0-1 (1/1/2)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO + PCI[24.0] - PCI/0-82000 (10b5879610b58796)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-84000 (4)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-85000 (5)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-86000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-87000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO + PCI[24.0] - PCI/0-89000 (10b5879610b58796)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-8b000 (6)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - GPU/0-8c000 (7)
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO                             + NVL[370.8] - NVS/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-8d000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO               + PCI[24.0] - NIC/0-8e000
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO + SYS[10.0] - CPU/0
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO ==========================================
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/4000 :GPU/0-4000 (0/5000.0/LOC) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/5000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (0/5000.0/LOC) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (0/5000.0/LOC) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (0/5000.0/LOC) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (2/24.0/PHB) CPU/0-1 (3/10.0/SYS) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/84000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (0/5000.0/LOC) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/85000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (0/5000.0/LOC) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/8B000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (0/5000.0/LOC) GPU/0-8c000 (2/370.8/NVL) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO GPU/8C000 :GPU/0-4000 (2/370.8/NVL) GPU/0-5000 (2/370.8/NVL) GPU/0-b000 (2/370.8/NVL) GPU/0-c000 (2/370.8/NVL) GPU/0-84000 (2/370.8/NVL) GPU/0-85000 (2/370.8/NVL) GPU/0-8b000 (2/370.8/NVL) GPU/0-8c000 (0/5000.0/LOC) NVS/0-0 (1/370.8/NVL) CPU/0-0 (3/10.0/SYS) CPU/0-1 (2/24.0/PHB) 
2025-04-22 04:21:16.035 n116-043-044:107010:114076 [0] NCCL INFO Setting affinity for GPU 0 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.039 n116-043-044:107012:114071 [2] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Pattern 1, crossNic 0, nChannels 12, bw 30.000000/30.000000, type NVL/PIX, sameChannels 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  0 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  1 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  2 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  3 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  4 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  5 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  6 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  7 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  8 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO  9 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO 10 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO 11 : GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO comm 0x7fda922ffe50 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO comm 0x7f181e312490 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO comm 0x7f9dca311e00 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO comm 0x7ef072312e50 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO comm 0x7efdfa310bd0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO comm 0x7f040a310d10 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO comm 0x7fa392310990 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO comm 0x7fae32314340 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 00 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 01 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 00 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 00 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 02 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 0 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 01 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 01 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 12 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 00 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 02 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 00 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 00 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 03 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 01 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 01 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 02 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 1 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 01 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 03 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 04 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 02 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 13 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 02 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 02 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 03 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 2 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 03 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 04 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 05 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 03 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 03 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 05 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 06 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 04 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 04 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 14 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 05 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 05 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 04 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 04 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 06 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 07 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 07 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 08 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 06 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 3 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 06 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 05 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 05 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 08 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 09 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 09 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 07 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 15 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 10 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 07 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 08 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 4 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 06 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 06 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 10 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 16 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 11 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 6 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 08 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 09 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 07 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 07 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 10 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 11 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 12 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 5 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 11 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 08 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 18 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 09 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 17 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 08 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 12 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 6 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 13 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 12 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 09 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 7 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 10 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 09 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 13 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 18 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 14 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 13 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 10 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 15 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 7 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 19 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 11 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 10 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 19 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 14 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 14 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 11 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 16 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 8 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 12 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 17 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 11 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 8 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 15 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 15 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 12 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 16 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 20 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 13 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 18 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 13 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 17 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 12 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 19 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 20 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 16 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 9 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 14 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 14 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 18 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 15 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 13 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 20 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 17 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 16 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 9 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 15 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 19 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 14 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 21 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 21 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 18 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 15 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 10 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 17 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 21 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 22 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 18 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 16 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 20 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 10 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 17 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 22 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 19 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 16 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 11 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 19 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 21 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 18 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Ring 23 : 4 -> 5 -> 6
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 20 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 22 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 20 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 17 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Tree 23 : -1 -> 0 -> 1/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 22 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 21 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 19 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Ring 23 : 1 -> 2 -> 3
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 21 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 11 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 18 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 22 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107015:114070 [5] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 20 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 00/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 22 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 19 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Tree 23 : 0 -> 1 -> 2/-1/-1
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Ring 23 : 3 -> 4 -> 5
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 21 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 01/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Ring 23 : 2 -> 3 -> 4
2025-04-22 04:21:16.043 n116-043-044:107012:114071 [2] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 22 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 20 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 02/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 21 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Ring 23 : 6 -> 7 -> 0
2025-04-22 04:21:16.043 n116-043-044:107014:114075 [4] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 22 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 03/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
2025-04-22 04:21:16.043 n116-043-044:107013:114072 [3] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Ring 23 : 5 -> 6 -> 7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 04/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107017:114073 [7] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 05/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 00 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107016:114074 [6] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 06/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 01 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 07/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 02 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 08/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 03 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 09/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 10/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 04 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 11/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 05 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 12/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 06 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 13/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 14/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 07 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 15/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 08 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 16/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 09 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 17/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 18/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 10 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 19/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 11 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 20/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 12 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 21/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 22/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 13 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Channel 23/24 :    0   1   2   3   4   5   6   7
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 14 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 00 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 15 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 01 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 16 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 02 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 03 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 17 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 04 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 18 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 05 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 06 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 19 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 07 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 08 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 20 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 09 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 21 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 10 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 11 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 22 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 12 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 13 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Ring 23 : 0 -> 1 -> 2
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 14 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 15 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 16 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107011:114077 [1] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 17 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 18 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 19 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 20 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 21 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 22 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Ring 23 : 7 -> 0 -> 1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
2025-04-22 04:21:16.043 n116-043-044:107010:114076 [0] NCCL INFO P2P Chunksize set to 524288
2025-04-22 04:21:16.061 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 00/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.061 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 00/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.061 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 00/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 01/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 00/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 01/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 01/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 00/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 02/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.062 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 01/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 02/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 02/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 01/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 03/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 02/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 03/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.063 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 03/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 02/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 04/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 03/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 04/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 04/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 03/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 05/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.064 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 04/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 05/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 05/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 04/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 06/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 05/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 06/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.065 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 06/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 05/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 07/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 06/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 00/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 07/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 07/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 00/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 06/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 08/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 07/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.066 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 01/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 08/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 00/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 08/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 01/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 07/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 09/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 08/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 02/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 09/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 01/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 09/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.067 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 02/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 08/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 10/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 09/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 03/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 10/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 02/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 10/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 03/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 11/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 09/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.068 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 10/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 04/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 11/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 03/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 04/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 11/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 12/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 10/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 11/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 05/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 12/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 04/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 05/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.069 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 12/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 13/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 11/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 12/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 06/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 13/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 05/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 06/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 13/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 14/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 12/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 13/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.070 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 07/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 14/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 06/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 07/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 14/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 15/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 13/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 14/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 08/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 15/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 07/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 08/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.071 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 15/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 16/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 15/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 14/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 09/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 16/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 08/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 16/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 09/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 17/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 16/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 15/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.072 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 10/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 17/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 17/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 09/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 10/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 18/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 17/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 11/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 16/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 18/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 18/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 10/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.073 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 11/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 19/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 18/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 12/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 17/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 19/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 19/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 11/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 12/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 20/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 19/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 13/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.074 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 18/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 20/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 20/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 12/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 13/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 21/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 20/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 14/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 19/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 21/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 21/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 13/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.075 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 14/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 22/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 21/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 15/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 20/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 22/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 22/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 14/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 15/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107017:114073 [7] NCCL INFO Ring Channel 23/0 : 7[7] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 22/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.076 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 16/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 21/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107010:114076 [0] NCCL INFO Ring Channel 23/0 : 0[0] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107014:114075 [4] NCCL INFO Ring Channel 23/0 : 4[4] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 15/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 16/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107013:114072 [3] NCCL INFO Ring Channel 23/0 : 3[3] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 17/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 22/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 17/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.077 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 16/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 18/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107011:114077 [1] NCCL INFO Ring Channel 23/0 : 1[1] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 18/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 17/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 19/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 19/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 18/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 20/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.078 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 20/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.080 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 19/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.080 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 21/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.082 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 21/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.083 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 20/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.083 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 22/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.085 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 22/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.086 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 21/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.086 n116-043-044:107016:114074 [6] NCCL INFO Ring Channel 23/0 : 6[6] -> 7[7] via P2P/IPC
2025-04-22 04:21:16.088 n116-043-044:107015:114070 [5] NCCL INFO Ring Channel 23/0 : 5[5] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.089 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 22/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.094 n116-043-044:107012:114071 [2] NCCL INFO Ring Channel 23/0 : 2[2] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.633 n116-043-044:107017:114073 [7] NCCL INFO Connected all rings
2025-04-22 04:21:16.633 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 00/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.636 n116-043-044:107016:114074 [6] NCCL INFO Connected all rings
2025-04-22 04:21:16.639 n116-043-044:107011:114077 [1] NCCL INFO Connected all rings
2025-04-22 04:21:16.639 n116-043-044:107010:114076 [0] NCCL INFO Connected all rings
2025-04-22 04:21:16.641 n116-043-044:107015:114070 [5] NCCL INFO Connected all rings
2025-04-22 04:21:16.641 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 01/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.645 n116-043-044:107014:114075 [4] NCCL INFO Connected all rings
2025-04-22 04:21:16.646 n116-043-044:107013:114072 [3] NCCL INFO Connected all rings
2025-04-22 04:21:16.646 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 02/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.646 n116-043-044:107012:114071 [2] NCCL INFO Connected all rings
2025-04-22 04:21:16.646 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 03/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.647 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 04/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.648 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 05/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.648 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 06/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.649 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 07/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.650 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 08/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.650 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 09/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.651 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 10/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.652 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 11/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.653 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 12/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.653 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 13/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.654 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 14/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.655 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 15/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.655 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 16/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.656 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 17/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.657 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 18/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.657 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 19/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.658 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 20/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.659 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 21/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.659 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 22/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.660 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 00/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.660 n116-043-044:107017:114073 [7] NCCL INFO Tree Channel 23/0 : 7[7] -> 6[6] via P2P/IPC
2025-04-22 04:21:16.660 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 01/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.661 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 00/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.661 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 00/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.661 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 01/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.661 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 02/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.661 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 01/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 02/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 03/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 02/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 00/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 00/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 03/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.662 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 04/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 00/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 03/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 01/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 01/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 04/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 05/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 01/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 04/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 02/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.663 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 02/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 05/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 06/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 02/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 05/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 03/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 03/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 06/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 07/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 03/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 06/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 04/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.664 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 04/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 07/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 04/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 08/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 07/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 05/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 05/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 08/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 05/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 09/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 08/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 06/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.665 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 06/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 09/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 06/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 10/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 09/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 07/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 07/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 10/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 07/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 11/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.666 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 10/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 08/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 08/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 11/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 08/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 12/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 11/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 09/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 09/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 12/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 09/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 13/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.667 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 12/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 10/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 10/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 13/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 10/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 14/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 13/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 11/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 11/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 14/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 11/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 15/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.668 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 14/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 12/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 12/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 15/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 12/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 16/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 15/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 13/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 13/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 16/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 13/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.669 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 17/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 16/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 14/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 14/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 17/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 14/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 18/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 17/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 15/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 15/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 18/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 15/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.670 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 19/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 18/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 16/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 16/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 19/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 16/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 20/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 19/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 17/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 17/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 20/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 17/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.671 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 21/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 18/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 20/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 18/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 21/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 18/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 22/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 19/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 21/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 19/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 22/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 19/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.672 n116-043-044:107016:114074 [6] NCCL INFO Tree Channel 23/0 : 6[6] -> 5[5] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 20/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 22/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 20/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107015:114070 [5] NCCL INFO Tree Channel 23/0 : 5[5] -> 4[4] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 20/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 21/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107011:114077 [1] NCCL INFO Tree Channel 23/0 : 1[1] -> 0[0] via P2P/IPC
2025-04-22 04:21:16.673 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 21/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.677 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 21/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.678 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 22/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.681 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 22/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.681 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 22/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:16.682 n116-043-044:107013:114072 [3] NCCL INFO Tree Channel 23/0 : 3[3] -> 2[2] via P2P/IPC
2025-04-22 04:21:16.694 n116-043-044:107014:114075 [4] NCCL INFO Tree Channel 23/0 : 4[4] -> 3[3] via P2P/IPC
2025-04-22 04:21:16.696 n116-043-044:107012:114071 [2] NCCL INFO Tree Channel 23/0 : 2[2] -> 1[1] via P2P/IPC
2025-04-22 04:21:17.135 n116-043-044:107017:114073 [7] NCCL INFO Connected all trees
2025-04-22 04:21:17.135 n116-043-044:107017:114073 [7] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.135 n116-043-044:107017:114073 [7] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.135 n116-043-044:107017:114073 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.135 n116-043-044:107017:114073 [7] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.194 n116-043-044:107010:114076 [0] NCCL INFO Connected all trees
2025-04-22 04:21:17.194 n116-043-044:107010:114076 [0] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.194 n116-043-044:107010:114076 [0] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.194 n116-043-044:107010:114076 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.194 n116-043-044:107010:114076 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.197 n116-043-044:107011:114077 [1] NCCL INFO Connected all trees
2025-04-22 04:21:17.197 n116-043-044:107011:114077 [1] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.197 n116-043-044:107011:114077 [1] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.197 n116-043-044:107011:114077 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.197 n116-043-044:107011:114077 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.199 n116-043-044:107012:114071 [2] NCCL INFO Connected all trees
2025-04-22 04:21:17.199 n116-043-044:107012:114071 [2] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.199 n116-043-044:107012:114071 [2] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.199 n116-043-044:107012:114071 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.199 n116-043-044:107012:114071 [2] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.199 n116-043-044:107013:114072 [3] NCCL INFO Connected all trees
2025-04-22 04:21:17.199 n116-043-044:107013:114072 [3] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.200 n116-043-044:107013:114072 [3] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.200 n116-043-044:107013:114072 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.200 n116-043-044:107013:114072 [3] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.203 n116-043-044:107014:114075 [4] NCCL INFO Connected all trees
2025-04-22 04:21:17.203 n116-043-044:107014:114075 [4] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.203 n116-043-044:107014:114075 [4] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.203 n116-043-044:107014:114075 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.203 n116-043-044:107014:114075 [4] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.203 n116-043-044:107015:114070 [5] NCCL INFO Connected all trees
2025-04-22 04:21:17.203 n116-043-044:107015:114070 [5] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.203 n116-043-044:107015:114070 [5] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.203 n116-043-044:107015:114070 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.203 n116-043-044:107015:114070 [5] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.205 n116-043-044:107016:114074 [6] NCCL INFO Connected all trees
2025-04-22 04:21:17.205 n116-043-044:107016:114074 [6] NCCL INFO NCCL_PROTO set by environment to Simple
2025-04-22 04:21:17.205 n116-043-044:107016:114074 [6] NCCL INFO NCCL_ALGO set by environment to Ring,Tree
2025-04-22 04:21:17.205 n116-043-044:107016:114074 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
2025-04-22 04:21:17.205 n116-043-044:107016:114074 [6] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
2025-04-22 04:21:17.232 n116-043-044:107016:114074 [6] NCCL INFO ncclCommSplit comm 0x7f181e312490 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 8b000 parent 0x11d0a1d0 color -934961569 key 6 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.232 n116-043-044:107014:114075 [4] NCCL INFO ncclCommSplit comm 0x7ef072312e50 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 84000 parent 0xfdc8140 color -934961569 key 4 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.232 n116-043-044:107010:114076 [0] NCCL INFO ncclCommSplit comm 0x7f9dca311e00 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 4000 parent 0xf80cf60 color -934961569 key 0 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.232 n116-043-044:107012:114071 [2] NCCL INFO ncclCommSplit comm 0x7fa392310990 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId b000 parent 0xfeca910 color -934961569 key 2 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.234 n116-043-044:107013:114072 [3] NCCL INFO ncclCommSplit comm 0x7f040a310d10 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId c000 parent 0x10ec6240 color -934961569 key 3 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.234 n116-043-044:107015:114070 [5] NCCL INFO ncclCommSplit comm 0x7fda922ffe50 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 85000 parent 0x10c641c0 color -934961569 key 5 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.234 n116-043-044:107011:114077 [1] NCCL INFO ncclCommSplit comm 0x7fae32314340 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 5000 parent 0x1042b540 color -934961569 key 1 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
2025-04-22 04:21:17.234 n116-043-044:107017:114073 [7] NCCL INFO ncclCommSplit comm 0x7efdfa310bd0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 8c000 parent 0xf646140 color -934961569 key 7 commId 0xddba01ec4ad6d20 type tp8 - Init COMPLETE
{'loss': 0.0939, 'grad_norm': 74.10909850115026, 'learning_rate': 1e-05, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_pvpfRBndHh8.mp4
problem_id: 75106
prompt_length: 1059
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/tides_2612.png
problem_id: 50084
prompt_length: 329
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_4023.png
problem_id: 46152
prompt_length: 279
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/10814.png
problem_id: 238083
prompt_length: 105
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/IconQA/images/54253.png
problem_id: 138500
prompt_length: 106
 path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/DVQA/images/bar_train_00066747.png
problem_id: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/1407.png87291
prompt_length: 323
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/Chart2Text/cauldron/chart2text/images/chart2text_00018322.png
problem_id: 8825
prompt_length: 290

problem_id: 73498
prompt_length: 125
Invalidate trace cache @ step 0 and module 2283: cache has only 0 modules
tensor([1.2118, 1.2079, 1.1954, 1.1830, 1.1712, 1.1560, 1.1371, 1.1210],
       device='cuda:7')
tensor([0.1869, 1.2051, 0.1929, 1.1752, 0.1622, 1.1459, 0.1347, 0.1143],
       device='cuda:2')
tensor([1.2086, 1.2855, 1.2699, 1.2535, 1.2376, 1.2133, 1.1946, 1.1743],
       device='cuda:5')
tensor([0.1702, 0.2074, 1.1923, 1.1823, 0.1669, 1.1506, 1.1357, 0.1166],
       device='cuda:0')
tensor([0.3638, 0.3902, 0.2474, 0.2315, 0.2103, 0.2960, 0.1762, 0.2709],
       device='cuda:1')
tensor([0.2880, 0.2950, 0.2783, 0.2640, 0.2470, 0.2289, 0.2119, 0.1849],
       device='cuda:4')
tensor([0.2767, 0.5446, 0.5085, 0.4888, 0.5027, 0.2372, 0.2149, 0.4399],
       device='cuda:6')
tensor([0.3036, 1.4249, 1.4027, 1.3815, 1.3581, 1.3331, 1.3030, 1.2746],
       device='cuda:3')
{'completion_length': 50.171875, 'rewards/accuracy_reward': 0.493817001581192, 'rewards/efficiency reward': 0.22554275393486023, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.7193597555160522, 'reward_std': 0.21964377164840698, 'kl': 0.404296875, 'epoch': 0.0}
{'loss': 0.0161, 'grad_norm': 7.983237822747673, 'learning_rate': 9.999999977182375e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/cf93f90fc6bccb3f21ae769b90ae7520.png
problem_id: 87132
prompt_length: 229
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_01349.mp4
problem_id: 166385
prompt_length: 731
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/3050.png
problem_id: 151311
prompt_length: 142
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_CEErZLiECYc.mp4
problem_id: 24634
prompt_length: 1049
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/113/RoLhMa9SHsI/split_3.mp4
problem_id: 91122
prompt_length: 1088
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Geometry3K/images/156.png
problem_id: 255906
prompt_length: 183
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/11288.png
problem_id: 16848
prompt_length: 225
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_LOINRpWGrlw.mp4
problem_id: 147562
prompt_length: 1147
Invalidate trace cache @ step 0 and module 3044: cache has only 0 modules
Error converting 'number_of_people' to float: could not convert string to float: 'number_of_people'
tensor([1.2360, 1.2844, 1.2686, 1.2533, 1.2324, 1.2071, 1.1902, 1.1702],
       device='cuda:5')
Error converting '45 employees' to float: could not convert string to float: '45 employees'
tensor([1.1661, 1.2030, 0.1922, 1.1768, 1.1617, 1.1476, 1.1356, 1.1175],
       device='cuda:7')
Error converting '' to float: could not convert string to float: ''
tensor([1.1780, 1.2276, 1.2149, 1.1977, 0.1819, 1.1690, 1.1511, 1.1329],
       device='cuda:2')
Error converting '' to float: could not convert string to float: ''
tensor([0.2202, 0.2182, 0.2051, 1.1956, 0.1818, 0.1664, 1.1517, 0.1330],
       device='cuda:6')
tensor([0.2804, 0.3753, 1.3534, 0.3385, 0.3189, 0.3002, 0.2676, 1.2452],
       device='cuda:1')
tensor([0.3058, 1.3884, 0.3715, 1.3484, 1.3278, 1.3040, 1.2769, 1.2480],
       device='cuda:4')
tensor([0.2645, 1.3934, 0.3733, 0.3549, 1.3321, 0.3088, 1.2816, 1.2572],
       device='cuda:0')
tensor([0.6570, 0.6617, 0.6698, 0.6087, 0.5931, 0.6142, 0.5569, 0.5658],
       device='cuda:3')
{'completion_length': 78.078125, 'rewards/accuracy_reward': 0.6040617823600769, 'rewards/efficiency reward': 0.24923691153526306, 'all_wrong': 0.125, 'all_correct': 0.0, 'reward': 0.8532986640930176, 'reward_std': 0.3353354334831238, 'kl': 1.0390625, 'epoch': 0.0}
{'loss': 0.0415, 'grad_norm': 35.481861560244724, 'learning_rate': 9.999999908729494e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/FigureQA/images/39638.png
problem_id: 171649
prompt_length: 303
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/A-OKVQA/images/507927.jpg
problem_id: 136335
prompt_length: 329
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_mouMMOraPyw.mp4
problem_id: 220692
prompt_length: 1063
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_2034.png
problem_id: 93632
prompt_length: 337
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1019/3477242531.mp4
problem_id: 188269
prompt_length: 1057
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_xIAq1OdWQM0.mp4
problem_id: 12553
prompt_length: 1053
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/0089/5379320378.mp4
problem_id: 235849
prompt_length: 1223
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/76afac85468ad73f932ea8dfcd6bb2f8.png
problem_id: 242147
prompt_length: 185
Invalidate trace cache @ step 0 and module 3805: cache has only 0 modules
Error converting '8 meters' to float: could not convert string to float: '8 meters'
tensor([1.2288, 1.2931, 1.2789, 1.2570, 1.2397, 1.2219, 1.1981, 1.1722],
       device='cuda:6')
tensor([1.2962, 1.3285, 1.3140, 1.2916, 1.2759, 1.2487, 1.2266, 1.1991],
       device='cuda:1')tensor([1.2607, 1.3943, 0.3717, 1.3533, 1.3306, 1.3117, 1.2844, 1.2559],
       device='cuda:7')

tensor([0.2219, 0.2046, 0.1809, 0.1801, 0.1676, 0.1550, 0.1405, 1.1234],
       device='cuda:0')
tensor([0.2758, 0.4019, 0.3813, 0.3581, 0.3317, 0.3103, 0.2838, 0.2531],
       device='cuda:2')
tensor([1.3450, 1.4029, 1.3832, 1.3590, 1.3357, 1.3102, 1.2788, 1.2550],
       device='cuda:3')
tensor([1.3509, 1.4248, 1.4022, 1.3874, 1.3587, 1.3321, 1.3050, 1.2702],
       device='cuda:4')
tensor([1.2996, 1.4037, 1.3835, 1.3609, 1.3400, 1.3169, 1.2957, 1.2600],
       device='cuda:5')
{'completion_length': 11.265625, 'rewards/accuracy_reward': 0.75, 'rewards/efficiency reward': 0.29319438338279724, 'all_wrong': 0.125, 'all_correct': 0.0, 'reward': 1.0431944131851196, 'reward_std': 0.11987271904945374, 'kl': 0.6171875, 'epoch': 0.0}
{'loss': 0.0247, 'grad_norm': 12.32774417235871, 'learning_rate': 9.999999794641359e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/97.png
problem_id: 97756
prompt_length: 325
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_8OA30kYcAUQ.mp4
problem_id: 3921
prompt_length: 1069
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/61d1f4548480ff5e325c625ea103bb15.png
problem_id: 231874
prompt_length: 80
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_Wti5fEgiYgc.mp4
problem_id: 69857
prompt_length: 1051
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/YYLKT.mp4
problem_id: 100460
prompt_length: 1006
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/23066.png
problem_id: 151630
prompt_length: 144
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/23270.png
problem_id: 110752
prompt_length: 317
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_insSTTMSC38.mp4
problem_id: 195795
prompt_length: 1125
Invalidate trace cache @ step 0 and module 4566: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
tensor([1.2512, 1.2290, 1.2171, 1.2026, 1.1865, 1.1744, 1.1577, 1.1404],
       device='cuda:2')
tensor([1.2416, 1.2880, 1.2746, 1.2596, 1.2426, 1.2208, 1.2051, 1.1843],
       device='cuda:6')
tensor([0.2405, 0.2098, 0.1951, 0.1790, 0.1625, 0.1464, 0.1339, 0.1125],
       device='cuda:7')
tensor([0.2358, 0.1952, 0.1823, 0.1758, 0.1597, 1.1461, 1.1339, 0.1176],
       device='cuda:5')
tensor([0.5367, 0.5919, 0.6577, 0.7239, 0.6579, 0.5981, 0.5636, 0.5016],
       device='cuda:3')
tensor([1.3017, 1.3780, 1.3562, 1.3377, 1.3192, 1.2923, 1.2690, 1.2400],
       device='cuda:1')
tensor([1.3544, 1.4087, 1.3940, 1.3676, 1.3465, 1.3230, 1.2927, 1.2661],
       device='cuda:4')
tensor([0.2766, 0.3832, 0.3622, 0.3482, 0.3236, 0.3004, 0.2764, 0.2475],
       device='cuda:0')
{'completion_length': 75.375, 'rewards/accuracy_reward': 0.5724137425422668, 'rewards/efficiency reward': 0.2525610327720642, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.8249748349189758, 'reward_std': 0.09693078696727753, 'kl': 239.0, 'epoch': 0.0}
{'loss': 9.5843, 'grad_norm': 5921.863888483461, 'learning_rate': 9.999999634917974e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_UoGoOznvKew.mp4
problem_id: 254142
prompt_length: 1051
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/EXAMS-V/9663cf6f-46b1-4432-b53b-9771da33ac35.png
problem_id: 112962
prompt_length: 402
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/319/_XxXWiOoyhY/split_3.mp4
problem_id: 96640
prompt_length: 1048
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_xnTWGIbp2j4.mp4
problem_id: 66994
prompt_length: 1041
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/b476d6b9d66a8c87ebd43bd5e58a9f9a.png
problem_id: 227
prompt_length: 89
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_22673.png
problem_id: 204587
prompt_length: 287
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_04170.mp4
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_7512.mp4
problem_id: 134769
prompt_length: 1093
problem_id: 208537
prompt_length: 748
Invalidate trace cache @ step 0 and module 5327: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
tensor([0.3891, 0.5423, 0.4449, 0.5191, 0.4888, 0.4164, 0.4155, 0.3631],
       device='cuda:2')
Error converting '' to float: could not convert string to float: ''
tensor([0.2237, 0.2071, 0.1766, 0.1665, 0.1649, 0.1372, 0.1208, 0.1128],
       device='cuda:3')
tensor([0.3265, 0.3847, 1.3646, 0.3485, 0.3242, 0.3057, 0.2801, 1.2523],
       device='cuda:0')
tensor([0.2638, 0.2496, 1.2353, 1.2203, 1.2061, 1.1867, 1.1736, 1.1561],
       device='cuda:4')
tensor([1.3101, 1.4141, 1.3931, 1.3682, 1.3505, 1.3203, 1.2927, 1.2655],
       device='cuda:6')
tensor([0.3606, 1.4299, 1.4096, 1.3905, 1.3651, 1.3356, 0.3051, 1.2731],
       device='cuda:7')
tensor([1.3169, 1.3740, 1.3543, 1.3407, 1.3134, 1.2945, 1.2666, 1.2437],
       device='cuda:1')
tensor([0.2871, 0.4052, 0.3822, 0.3618, 0.3395, 0.3153, 0.2939, 0.2645],
       device='cuda:5')
{'completion_length': 62.765625, 'rewards/accuracy_reward': 0.4894702136516571, 'rewards/efficiency reward': 0.29029011726379395, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.7797603011131287, 'reward_std': 0.20254206657409668, 'kl': 57.25, 'epoch': 0.0}
{'loss': 2.2992, 'grad_norm': 183132.62324709762, 'learning_rate': 9.999999429559339e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/f6bef16080a98451.png
problem_id: 222675
prompt_length: 290
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_07736.mp4
problem_id: 150483
prompt_length: 687
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/106/W882NYaaiEs/split_1.mp4
problem_id: 66392
prompt_length: 767
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/84b3fc5b08f24dba.jpg
problem_id: 234821
prompt_length: 303
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Z47QGlaQ1NE.mp4
problem_id: 142189
prompt_length: 1066
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_IZx-EMbylmM.mp4
problem_id: 112880
prompt_length: 1047
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/CLEVR-Math/images/CLEVR_train_027856.png
problem_id: 73461
prompt_length: 249
Invalidate trace cache @ step 0 and module 6088: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_d3tRR2uDqQ0.mp4
problem_id: 114383
prompt_length: 1049
Error converting '' to float: could not convert string to float: ''
tensor([0.2678, 0.2675, 0.2544, 0.2331, 0.2169, 0.1987, 0.1800, 0.1598],
       device='cuda:0')
tensor([0.2821, 0.2666, 0.2511, 0.2360, 0.2166, 0.2027, 0.1798, 0.1627],
       device='cuda:7')
tensor([0.2599, 0.3161, 0.2979, 0.2791, 0.2626, 0.2442, 0.2198, 0.1961],
       device='cuda:3')
tensor([0.4826, 0.6833, 0.5669, 0.5482, 0.5250, 0.5457, 0.5766, 0.5858],
       device='cuda:6')
tensor([0.2512, 1.3429, 1.3250, 1.3060, 1.2881, 1.2651, 1.2405, 1.2167],
       device='cuda:4')
tensor([0.3388, 1.3927, 0.3724, 1.3538, 0.3325, 1.3098, 1.2897, 0.2545],
       device='cuda:5')
tensor([1.3373, 1.4075, 1.3884, 1.3689, 1.3518, 1.3231, 1.2953, 1.2688],
       device='cuda:2')
tensor([1.3354, 1.4010, 1.3807, 1.3658, 1.3390, 1.3162, 1.2851, 1.2604],
       device='cuda:1')
{'completion_length': 18.046875, 'rewards/accuracy_reward': 0.45815473794937134, 'rewards/efficiency reward': 0.28356581926345825, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.7417205572128296, 'reward_std': 0.1480291187763214, 'kl': 0.91015625, 'epoch': 0.0}
{'loss': 0.0363, 'grad_norm': 13.133863328294094, 'learning_rate': 9.999999178565453e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_VYelZo6M3M0.mp4
problem_id: 38734
prompt_length: 1047
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/RoBUT_SQA/cauldron/robut_sqa/images/robut_sqa_00005507.png
problem_id: 53196
prompt_length: 284
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v1-3/train_val/v_E5bRptEXtq8.mp4
problem_id: 192776
prompt_length: 1050
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/adb27b96f96b9cdef864efb08e50488a.png
problem_id: 201464
prompt_length: 170
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_7983.mp4
problem_id: 139183
prompt_length: 1038
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/DP8U3.mp4
problem_id: 120602
prompt_length: 1012
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_zLZTqSaGxJo.mp4
problem_id: 257177
prompt_length: 1079
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2305.14176_2.jpg
problem_id: 178547
prompt_length: 381
Invalidate trace cache @ step 0 and module 6849: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([1.2687, 1.2597, 1.2474, 1.2328, 1.2148, 1.2022, 1.1856, 1.1682],
       device='cuda:6')
tensor([0.9488, 0.9820, 0.9714, 1.0335, 0.7765, 0.5735, 0.6938, 0.9911],
       device='cuda:4')
Error converting '' to float: could not convert string to float: ''
tensor([0.2064, 0.1950, 0.1897, 0.1664, 0.1534, 0.1487, 0.1305, 0.1185],
       device='cuda:3')
tensor([0.3332, 0.3747, 0.3575, 0.3375, 0.3230, 0.2922, 0.2680, 0.2385],
       device='cuda:2')
tensor([0.3569, 0.4284, 0.4080, 0.3831, 0.3624, 0.3339, 0.3113, 0.2759],
       device='cuda:1')
tensor([0.2988, 0.3854, 0.3672, 0.3471, 0.3244, 0.3034, 0.2763, 0.2488],
       device='cuda:0')
tensor([1.2964, 1.4031, 1.3809, 1.3603, 1.3357, 1.3093, 1.2880, 1.2552],
       device='cuda:7')
tensor([0.5358, 0.5375, 0.5084, 0.4866, 0.4332, 0.4946, 0.4616, 0.4433],
       device='cuda:5')
{'completion_length': 79.84375, 'rewards/accuracy_reward': 0.3607671856880188, 'rewards/efficiency reward': 0.2724192440509796, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6331864595413208, 'reward_std': 0.058496229350566864, 'kl': 31.5, 'epoch': 0.0}
{'loss': 1.2652, 'grad_norm': 3781.1288332746044, 'learning_rate': 9.999998881936324e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4V/Palmyra2.png
problem_id: 163331
prompt_length: 273
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/b9044e2bdc7587ac4083fd160cdc01d3.png
problem_id: 29361
prompt_length: 127
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_20437.png
problem_id: 63773
prompt_length: 312
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/3638.png
problem_id: 169915
prompt_length: 320
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/JP5NM.mp4
problem_id: 146244
prompt_length: 1106
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC4030533_F3_290784.jpg
problem_id: 212135
prompt_length: 176
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_2963.png
problem_id: 81101
prompt_length: 298
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/218/91z7e22XGy8/split_5.mp4
problem_id: 164047
prompt_length: 1054
Invalidate trace cache @ step 0 and module 7610: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([1.2506, 1.2009, 0.2106, 0.1792, 1.1786, 0.1627, 1.1451, 1.1313],
       device='cuda:5')
Error converting '' to float: could not convert string to float: ''
tensor([0.3097, 0.3497, 0.3283, 0.3096, 0.2904, 0.2697, 0.2405, 0.2155],
       device='cuda:0')
tensor([0.7380, 1.0561, 0.4325, 0.4162, 0.4012, 0.5017, 0.3770, 0.3912],
       device='cuda:4')
tensor([1.2566, 1.2538, 1.2401, 1.2247, 1.2073, 1.1941, 1.1766, 1.1578],
       device='cuda:2')
Error converting '' to float: could not convert string to float: ''
tensor([0.4496, 0.3412, 0.4014, 0.2858, 0.2944, 0.2619, 0.2965, 0.3166],
       device='cuda:1')
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([0.2321, 0.1921, 0.1864, 0.1654, 0.1533, 0.1400, 1.1277, 0.1097],
       device='cuda:7')
tensor([1.3097, 1.4339, 1.4183, 1.3901, 1.3761, 1.3505, 1.3130, 1.2903],
       device='cuda:6')
tensor([1.2700, 1.3953, 1.3772, 1.3534, 1.3322, 1.3070, 1.2852, 1.2500],
       device='cuda:3')
{'completion_length': 43.625, 'rewards/accuracy_reward': 0.5117361545562744, 'rewards/efficiency reward': 0.25706571340560913, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.7688018083572388, 'reward_std': 0.16857996582984924, 'kl': 53.75, 'epoch': 0.0}
{'loss': 2.1502, 'grad_norm': 2501.1068295386613, 'learning_rate': 9.999998539671949e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_S9alQwrQ-oo.mp4
problem_id: 195147
prompt_length: 1051
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/306/H4-p87JS-yA/split_1.mp4
problem_id: 129835
prompt_length: 690
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/LI5VF.mp4
problem_id: 52430
prompt_length: 1050
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/7247.png
problem_id: 186565
prompt_length: 141
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/52c3a99c460f05abf7e093829ee806a3.png
problem_id: 18408
prompt_length: 220
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_TS-p15iwQPc.mp4path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/9d66a7fd1ada6cae6216bcf6b3c02f8e.png
problem_id: 52201
prompt_length: 133

problem_id: 149231
prompt_length: 1134
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v1-3/train_val/v_99KPQLxH4cw.mp4
problem_id: 46953
prompt_length: 1057
Invalidate trace cache @ step 0 and module 8371: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''Error converting '' to float: could not convert string to float: ''

Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''Error converting '' to float: could not convert string to float: ''

tensor([0.2317, 0.1930, 0.1824, 0.1695, 0.1560, 0.1421, 0.1294, 0.1109],
       device='cuda:3')
Error converting '' to float: could not convert string to float: ''
tensor([0.1711, 0.2089, 0.1810, 0.1807, 0.1571, 0.1369, 0.1227, 0.1101],
       device='cuda:5')
tensor([1.2884, 1.3664, 1.3465, 1.3335, 1.3124, 1.2921, 0.2671, 1.2439],
       device='cuda:7')
tensor([0.1864, 0.2002, 0.1899, 0.1817, 0.1644, 0.1526, 0.1388, 0.1187],
       device='cuda:2')
tensor([0.3230, 0.4160, 0.3966, 0.3760, 0.3538, 0.3296, 0.3006, 0.2739],
       device='cuda:4')
tensor([1.3244, 1.4127, 1.4021, 1.3787, 1.3534, 1.3281, 1.3038, 1.2738],
       device='cuda:1')
tensor([1.3616, 1.4090, 1.3886, 0.3682, 1.3413, 0.3164, 1.2906, 1.2590],
       device='cuda:6')
tensor([1.3316, 1.4119, 1.3925, 1.3729, 1.3493, 1.3219, 1.3002, 1.2689],
       device='cuda:0')
{'completion_length': 251.8125, 'rewards/accuracy_reward': 0.453125, 'rewards/efficiency reward': 0.27183419466018677, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.7249591946601868, 'reward_std': 0.1352064311504364, 'kl': 92.0, 'epoch': 0.0}
{'loss': 3.6923, 'grad_norm': 322.0650231069469, 'learning_rate': 9.999998151772335e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_4345.png
problem_id: 9393
prompt_length: 325
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/IconQA/images/103454.png
problem_id: 176792
prompt_length: 106
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_06274.mp4
problem_id: 161960
prompt_length: 688
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/human_system_circulatory_3642.png
problem_id: 67271
prompt_length: 331
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/APJQ8.mp4
problem_id: 239665
prompt_length: 1059
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/A-OKVQA/images/577169.jpg
problem_id: 82210
prompt_length: 310
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/VA8NL.mp4
problem_id: 9232
prompt_length: 1068
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/8e3801b4da3404465e2353028a2a4167.png
problem_id: 174226
prompt_length: 194
Invalidate trace cache @ step 0 and module 9132: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2406, 0.3381, 0.3179, 0.2977, 0.2827, 0.2628, 0.2328, 0.2055],
       device='cuda:5')
tensor([1.2026, 1.2605, 1.2469, 1.2324, 1.2134, 1.1963, 1.1806, 1.1600],
       device='cuda:3')
tensor([0.3504, 0.3909, 0.3737, 0.3574, 0.3420, 0.3125, 1.0374, 0.2580],
       device='cuda:1')
tensor([0.2455, 0.2027, 0.1898, 0.1804, 0.1672, 0.1541, 0.1381, 0.1233],
       device='cuda:6')
tensor([0.2366, 0.1978, 0.1887, 0.1803, 1.1637, 0.1486, 0.1324, 0.1162],
       device='cuda:7')
tensor([1.3145, 1.3323, 1.3149, 1.2930, 1.2760, 1.2555, 1.2330, 1.2063],
       device='cuda:4')
tensor([0.2834, 0.4073, 1.3852, 1.3657, 0.3384, 1.3168, 1.2880, 0.2640],
       device='cuda:2')
tensor([1.3559, 1.4103, 1.3947, 1.3761, 1.3492, 1.3211, 1.2935, 1.2672],
       device='cuda:0')
{'completion_length': 38.921875, 'rewards/accuracy_reward': 0.46484375, 'rewards/efficiency reward': 0.26485538482666016, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.7296991348266602, 'reward_std': 0.16987848281860352, 'kl': 44.25, 'epoch': 0.0}
{'loss': 1.7654, 'grad_norm': 1089.8748984694862, 'learning_rate': 9.999997718237482e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/MapQA/images/map_3025.png
problem_id: 63691
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/31992.png
problem_id: 36245
prompt_length: 138
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/stratigraphy_9263.png
problem_id: 225848
prompt_length: 317
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/4324.png
problem_id: 131359
prompt_length: 321
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/206/8QCW0qWbNeI/split_2.mp4
problem_id: 57198
prompt_length: 1056
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/121e9fb8c5ee0f17f8da85827a996da1.png
problem_id: 174429
prompt_length: 188
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/711901da2725b3e3cd1972e12e2daad3.png
problem_id: 85486
prompt_length: 133
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_15970.png
problem_id: 211575
prompt_length: 241
Invalidate trace cache @ step 0 and module 9893: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '<num> 39 </num>' to float: could not convert string to float: '<num> 39 </num>'
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([1.2836, 1.2864, 0.2713, 1.2538, 1.2378, 1.2179, 1.1954, 1.1735],
       device='cuda:1')
tensor([0.2199, 0.2078, 0.1907, 0.1807, 0.1679, 0.1544, 0.1379, 0.1227],
       device='cuda:5')
tensor([0.2376, 0.3277, 0.3102, 0.2967, 0.2755, 0.2539, 0.2338, 0.2085],
       device='cuda:6')
Error converting '' to float: could not convert string to float: ''
tensor([0.2707, 1.2999, 0.2828, 1.2642, 0.2436, 1.2216, 1.2026, 1.1772],
       device='cuda:4')
tensor([0.2021, 0.1884, 0.1778, 0.1661, 0.1478, 0.1359, 0.1232, 0.1066],
       device='cuda:7')
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([0.2393, 0.1890, 0.1853, 0.1713, 0.1533, 0.1369, 0.1243, 0.1082],
       device='cuda:0')
tensor([1.2467, 1.3109, 1.2947, 1.2735, 1.2575, 0.2385, 1.2187, 0.1945],
       device='cuda:3')
tensor([1.3450, 1.4055, 1.3885, 1.3691, 1.3450, 1.3222, 1.2953, 1.2634],
       device='cuda:2')
{'completion_length': 59.484375, 'rewards/accuracy_reward': 0.40625, 'rewards/efficiency reward': 0.23020130395889282, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.636451244354248, 'reward_std': 0.19128462672233582, 'kl': 38.0, 'epoch': 0.0}
{'loss': 1.5192, 'grad_norm': 1044.1856677952796, 'learning_rate': 9.999997239067397e-06, 'epoch': 0.0}
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/PJO9R.mp4
problem_id: 179419
prompt_length: 1041 
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_5cBgF9T0N3U.mp4
problem_id: 222483
prompt_length: 1052
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_07501.mp4
problem_id: 188586
prompt_length: 736
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_iEa8Q-VeAH8.mp4
problem_id: 110972
prompt_length: 1059
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/MapQA/images/map_3276.png
problem_id: 28473
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/19463.png
problem_id: 170335
prompt_length: 143
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_8991.png
problem_id: 262058
prompt_length: 321
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D-gpt4v/ai2d_azuregpt_937.png
problem_id: 245669
prompt_length: 201
Invalidate trace cache @ step 0 and module 10654: cache has only 0 modules
tensor([0.1865, 0.2053, 0.1921, 0.1781, 0.1653, 0.1499, 0.1406, 0.1206],
       device='cuda:5')
tensor([1.3327, 1.2912, 1.2738, 1.2578, 1.2396, 1.2224, 1.2014, 1.1788],
       device='cuda:2')
tensor([0.3331, 0.3669, 0.3544, 0.3331, 0.3153, 0.2904, 0.2670, 0.2348],
       device='cuda:0')
tensor([0.2292, 0.2322, 0.3834, 0.2027, 0.1873, 0.1734, 0.3564, 0.3035],
       device='cuda:6')
tensor([0.3147, 1.4185, 0.4030, 0.3822, 0.3576, 0.3406, 0.3137, 0.2841],
       device='cuda:4')
tensor([0.2665, 0.3981, 0.3739, 0.3520, 0.3299, 0.2975, 0.2698, 0.2421],
       device='cuda:7')
tensor([1.3310, 1.3784, 1.3557, 1.3378, 0.3136, 1.2874, 1.2624, 1.2339],
       device='cuda:3')
tensor([1.2937, 1.3823, 1.3640, 1.3431, 1.3232, 1.2978, 1.2741, 1.2447],
       device='cuda:1')
{'completion_length': 80.703125, 'rewards/accuracy_reward': 0.38327541947364807, 'rewards/efficiency reward': 0.2771417498588562, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.6604171991348267, 'reward_std': 0.13055753707885742, 'kl': 121.0, 'epoch': 0.0}
{'loss': 4.834, 'grad_norm': 300.24024195530546, 'learning_rate': 9.999996714262083e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_cr2lbZ6or1Q.mp4
problem_id: 237829
prompt_length: 1057
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_1290.png
problem_id: 116366
prompt_length: 337
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/93f024cd5b188886daecf9a0fedb9d8a.png
problem_id: 127259
prompt_length: 262
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_8bDQ6ilIQpA.mp4
problem_id: 20837
prompt_length: 1053
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_C6BDilJJKgo.mp4
problem_id: 9475
prompt_length: 1052
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/EXAMS-V/01caf0fa-c696-40b6-99ac-f23053c72377.png
problem_id: 151674
prompt_length: 412
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Zw4illqWzFI.mp4
problem_id: 16029
prompt_length: 1010
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_aXlNqnuHa7U.mp4
problem_id: 146912
prompt_length: 1056
Invalidate trace cache @ step 0 and module 11415: cache has only 0 modules
tensor([1.2790, 0.2661, 1.2507, 0.2385, 1.2221, 0.2032, 0.1842, 0.1678],
       device='cuda:0')
tensor([0.3301, 0.3687, 0.3498, 0.3332, 0.3106, 0.2889, 0.2610, 0.2360],
       device='cuda:7')
tensor([0.2734, 0.2256, 0.2178, 0.2077, 0.1917, 0.1685, 1.1568, 0.1375],
       device='cuda:4')
tensor([0.2747, 1.3460, 1.3285, 1.3099, 1.2875, 1.2672, 1.2433, 1.2192],
       device='cuda:2')
tensor([1.3274, 1.3966, 1.3789, 1.3564, 1.3359, 1.3111, 1.2840, 1.2551],
       device='cuda:3')
tensor([1.2717, 1.3942, 1.3801, 1.3525, 1.3314, 1.3045, 1.2775, 1.2498],
       device='cuda:5')
tensor([1.3765, 1.4096, 1.3916, 1.3697, 1.3429, 1.3215, 1.2914, 1.2635],
       device='cuda:1')
tensor([0.5152, 0.6596, 0.6578, 0.6393, 0.5915, 0.5266, 0.4534, 0.5079],
       device='cuda:6')
{'completion_length': 113.421875, 'rewards/accuracy_reward': 0.583459734916687, 'rewards/efficiency reward': 0.2863892912864685, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.8698490858078003, 'reward_std': 0.18912255764007568, 'kl': 196.0, 'epoch': 0.0}
{'loss': 7.837, 'grad_norm': 1156.6613647527868, 'learning_rate': 9.999996143821546e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1609.03564_0.jpg
problem_id: 58649
prompt_length: 363
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1015/2774585614.mp4
problem_id: 85500
prompt_length: 1140
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/0c0ac55f447972cd.png
problem_id: 189047
prompt_length: 290
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextOCR/04889f6aaddad68f.png
problem_id: 87701
prompt_length: 285
path: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_5yEvWk3Fo_s.mp4
problem_id: 262519
prompt_length: 1066
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/5HPZ1.mp4path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D-gpt4v/ai2d_azuregpt_791.png
problem_id: 14748
prompt_length: 367

problem_id: 96749
prompt_length: 1050
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/V24A5.mp4
problem_id: 213941
prompt_length: 1134
Invalidate trace cache @ step 0 and module 12176: cache has only 0 modules
tensor([0.4542, 0.2781, 0.2597, 0.4963, 0.2284, 0.4592, 0.1904, 0.1709],
       device='cuda:4')
tensor([1.2752, 1.2685, 1.2535, 1.2346, 1.2168, 1.2015, 1.1806, 1.1615],
       device='cuda:1')
tensor([0.5768, 0.5505, 0.5722, 0.6120, 0.5066, 0.4343, 0.5046, 0.4835],
       device='cuda:2')
tensor([0.3451, 0.3097, 0.3338, 0.2930, 0.2611, 0.2429, 0.2395, 0.2345],
       device='cuda:5')
tensor([0.3094, 1.4133, 0.3922, 1.3680, 1.3501, 1.3218, 1.2959, 1.2686],
       device='cuda:0')
tensor([1.3419, 1.3965, 1.3833, 1.3537, 1.3352, 1.3139, 1.2837, 1.2557],
       device='cuda:3')
tensor([1.3487, 1.3998, 1.3802, 1.3645, 1.3406, 1.3116, 1.2889, 1.2586],
       device='cuda:6')
tensor([1.3812, 1.4028, 1.3840, 1.3609, 1.3394, 1.3189, 1.2870, 1.2593],
       device='cuda:7')
{'completion_length': 66.265625, 'rewards/accuracy_reward': 0.6506689786911011, 'rewards/efficiency reward': 0.28119078278541565, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.9318598508834839, 'reward_std': 0.10991593450307846, 'kl': 17.625, 'epoch': 0.0}
{'loss': 0.7057, 'grad_norm': 1222.4703106359689, 'learning_rate': 9.999995527745791e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/404/9zqwA2munL0/split_6.mp4
problem_id: 147858
prompt_length: 1079
path:path:  /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_-l5e1zVzQOQ.mp4/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/ChromeWriting/identity_6317.png

problem_id:problem_id:  165016150033

prompt_length:prompt_length:  100
1128
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/5d782a53ff52845c.jpg
problem_id: 45372
prompt_length: 300
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_19982.png
problem_id: 73550
prompt_length: 320
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1016/2833808682.mp4
problem_id: 2741
prompt_length: 1134
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/ChromeWriting/identity_2998.png
problem_id: 143412
prompt_length: 80
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/ea179b81bc932b5636687a50578cc006.png
problem_id: 63477
prompt_length: 62
Invalidate trace cache @ step 0 and module 12937: cache has only 0 modules
tensor([0.2377, 0.1953, 1.1791, 0.1682, 0.1532, 0.1371, 0.1210, 0.1083],
       device='cuda:6')
tensor([0.2470, 0.2080, 0.1902, 0.7144, 0.2381, 0.1420, 0.1277, 0.1067],
       device='cuda:0')
tensor([0.4662, 0.1975, 0.6327, 0.3972, 0.3772, 0.3665, 0.1286, 0.3308],
       device='cuda:7')
tensor([1.0551, 0.7619, 0.2997, 0.2963, 1.2594, 0.2397, 0.2861, 0.2751],
       device='cuda:3')
tensor([0.2473, 0.3000, 0.2826, 0.2648, 0.2438, 0.2238, 0.2013, 0.1775],
       device='cuda:2')
tensor([1.3073, 1.4198, 1.4012, 1.3770, 1.3564, 1.3306, 1.3054, 1.2777],
       device='cuda:1')
tensor([0.3049, 0.4042, 0.3843, 0.3688, 0.3427, 0.3144, 0.2878, 0.2584],
       device='cuda:4')
tensor([1.3237, 1.4000, 1.3754, 1.3558, 1.3353, 1.3119, 1.2807, 1.2542],
       device='cuda:5')
{'completion_length': 33.9375, 'rewards/accuracy_reward': 0.33665788173675537, 'rewards/efficiency reward': 0.25182390213012695, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.5884817242622375, 'reward_std': 0.16400042176246643, 'kl': 50.5, 'epoch': 0.0}
{'loss': 2.0203, 'grad_norm': 1086.0449867290524, 'learning_rate': 9.999994866034821e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/b7a1cb72-29de-4aa9-a416-b675e69cbc83.png
problem_id: 222184
prompt_length: 73
Invalidate trace cache @ step 0 and module 13698: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_h1H41xqecE4.mp4
problem_id: 242430
prompt_length: 1063
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_KyDS9Zg76Jg.mp4
problem_id: 151980
prompt_length: 1029
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_4DCxc6mO864.mp4
problem_id: 74080
prompt_length: 1055
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/A-OKVQA/images/267203.jpg
problem_id: 122451
prompt_length: 311
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/11207.png
problem_id: 153054
prompt_length: 333
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1011/3090695613.mp4
problem_id: 128172
prompt_length: 1135
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1012/4343935954.mp4
problem_id: 146430
prompt_length: 1141
tensor([1.3274, 1.3444, 1.3235, 1.3048, 1.2805, 1.2617, 1.2356, 1.2074],
       device='cuda:1')
tensor([0.2967, 0.2715, 0.2559, 0.2363, 0.2248, 0.2060, 0.1851, 0.1646],
       device='cuda:2')
tensor([0.1749, 0.5349, 0.4418, 0.5113, 0.1800, 0.3446, 0.1448, 0.4941],
       device='cuda:0')
tensor([0.3515, 0.4122, 0.4013, 0.3721, 0.3468, 0.3239, 0.2954, 0.2622],
       device='cuda:4')
tensor([1.2616, 1.3870, 1.3687, 1.3447, 1.3208, 1.3010, 1.2721, 1.2454],
       device='cuda:7')
tensor([1.3167, 1.4020, 1.3825, 1.3596, 1.3405, 1.3187, 1.2953, 1.2648],
       device='cuda:3')
tensor([1.3351, 1.3906, 1.3677, 1.3499, 1.3260, 1.2997, 1.2746, 1.2424],
       device='cuda:5')
tensor([1.2725, 1.4094, 1.3876, 1.3667, 1.3473, 1.3275, 1.2964, 1.2687],
       device='cuda:6')
{'completion_length': 36.5, 'rewards/accuracy_reward': 0.6470145583152771, 'rewards/efficiency reward': 0.29300716519355774, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.9400217533111572, 'reward_std': 0.06342214345932007, 'kl': 21.5, 'epoch': 0.0}
{'loss': 0.858, 'grad_norm': 10405.594505894527, 'learning_rate': 9.999994158688646e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/2340.png
problem_id: 45431
prompt_length: 160
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/IWMW1.mp4
problem_id: 109816
prompt_length: 1136
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/ZQKVY.mp4
problem_id: 154546
prompt_length: 1042
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_1093.png
problem_id: 57924
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_09309.mp4
problem_id: 167289
prompt_length: 690
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/RoBUT_SQA/cauldron/robut_sqa/images/robut_sqa_00007205.png
problem_id: 104842
prompt_length: 269
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/LWINQ.mp4
problem_id: 9964
prompt_length: 1051
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/FLM1T.mp4
problem_id: 200818
prompt_length: 1090
Invalidate trace cache @ step 0 and module 14459: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '266.5 ft' to float: could not convert string to float: '266.5 ft'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2390, 0.2147, 0.2029, 0.1875, 0.1705, 0.1598, 0.1406, 0.1252],
       device='cuda:7')
Error converting 'D' to float: could not convert string to float: 'D'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2651, 1.3383, 1.3241, 1.3021, 1.2833, 0.2611, 0.2412, 1.2155],
       device='cuda:2')
tensor([0.3188, 0.3526, 0.3377, 0.9189, 0.2958, 0.2755, 0.2513, 0.2273],
       device='cuda:3')
tensor([0.4310, 0.2441, 0.2094, 0.1956, 0.1827, 0.2173, 0.1496, 0.6582],
       device='cuda:5')
tensor([0.2830, 0.4036, 0.3878, 0.3657, 0.3462, 1.3214, 0.2948, 0.2619],
       device='cuda:0')
tensor([1.3440, 1.3947, 1.3782, 1.3543, 1.3311, 1.3100, 1.2856, 1.2551],
       device='cuda:4')
tensor([1.3326, 1.4268, 1.4067, 1.3873, 1.3657, 1.3352, 1.3119, 1.2855],
       device='cuda:1')
tensor([1.3521, 1.4087, 1.3926, 1.3681, 1.3449, 1.3185, 1.2928, 1.2591],
       device='cuda:6')
{'completion_length': 84.234375, 'rewards/accuracy_reward': 0.490427166223526, 'rewards/efficiency reward': 0.28836318850517273, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.7787903547286987, 'reward_std': 0.1840192675590515, 'kl': 16.625, 'epoch': 0.0}
{'loss': 0.6671, 'grad_norm': 12643.255979546262, 'learning_rate': 9.999993405707269e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/eee4a47a3e97e5e13791f8c820934c71.png
problem_id: 6874
prompt_length: 101
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_SRAtrvemI-I.mp4
problem_id: 262474
prompt_length: 1047
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_9903.png
problem_id: 220359
prompt_length: 309
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/FigureQA/images/99219.png
problem_id: 44614
prompt_length: 294
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_zryPDx9WL08.mp4
problem_id: 219317
prompt_length: 1062
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00001938.png
problem_id: 36917
prompt_length: 287
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/0a2b838a37193458c773f7a79fc5504e.png
problem_id: 54050
prompt_length:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1911.13197_0.jpg
problem_id:  198805
prompt_length: 371
89
Invalidate trace cache @ step 0 and module 15220: cache has only 0 modules
tensor([1.3493, 1.2621, 1.2469, 1.2373, 1.2189, 1.2026, 1.1854, 1.1610],
       device='cuda:6')
tensor([0.2530, 0.2175, 0.2059, 0.1890, 0.1693, 0.1585, 0.1396, 0.1240],
       device='cuda:0')
tensor([1.1152, 1.1605, 1.1900, 1.1286, 1.0645, 1.0504, 1.0775, 1.0585],
       device='cuda:5')
tensor([0.6379, 0.7256, 0.3388, 0.3200, 0.5632, 0.2776, 0.6284, 0.5554],
       device='cuda:4')
Error converting '3 > 2 > 4 > 1' to float: could not convert string to float: '3 > 2 > 4 > 1'
tensor([0.1829, 0.1994, 0.1888, 1.1733, 1.1602, 1.1414, 0.1291, 1.1117],
       device='cuda:7')
tensor([0.2679, 0.2606, 0.2482, 0.2307, 0.2170, 0.2002, 0.1828, 0.1657],
       device='cuda:3')
tensor([1.3362, 1.3949, 1.3806, 1.3584, 1.3296, 1.3102, 1.2821, 1.2550],
       device='cuda:2')
tensor([0.3546, 0.3959, 0.3761, 0.3550, 1.3291, 0.3049, 0.2792, 0.2461],
       device='cuda:1')
{'completion_length': 54.40625, 'rewards/accuracy_reward': 0.46700459718704224, 'rewards/efficiency reward': 0.24487186968326569, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.711876392364502, 'reward_std': 0.16032356023788452, 'kl': 36.5, 'epoch': 0.0}
{'loss': 1.4626, 'grad_norm': 682.0354433929341, 'learning_rate': 9.9999926070907e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/VQA-RAD/images/synpic49914.jpg
problem_id: 239291
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Zk4Ewz3v6aQ.mp4
problem_id: 231398
prompt_length: 1131
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_21856.png
problem_id: 216088
prompt_length: 282
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/6887748b38a55c554d5573478e72060c.png
problem_id: 250137
prompt_length: 63
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/7b80a6d38f3fd7987a7a8ea6308a4115.png
problem_id: 152834
prompt_length: 161
 path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/cd22cb78bdfd418f489034f416bdb186.png
problem_id: 1407
prompt_length: 211
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/Visual7W/cauldron/visual7w/images/visual7w_00003826.png
problem_id: 64568
prompt_length: 318path: 
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/722eeaf4e89a0171669125648ca8c3b7.png
problem_id: 222563
prompt_length: 276
Invalidate trace cache @ step 0 and module 15981: cache has only 0 modules
Error converting '4<answer>' to float: could not convert string to float: '4<answer>'
tensor([1.2388, 1.2034, 1.1914, 1.1775, 1.1639, 1.1497, 1.1322, 0.1156],
       device='cuda:2')
tensor([0.1963, 1.2138, 1.2009, 1.1904, 0.1757, 1.1608, 0.1440, 1.1307],
       device='cuda:5')
tensor([1.2115, 1.2372, 1.2201, 0.2100, 0.1931, 1.1730, 0.1639, 1.1436],
       device='cuda:6')
tensor([0.2520, 0.1899, 0.1791, 0.1655, 1.1485, 0.1338, 0.1213, 0.1060],
       device='cuda:3')
tensor([1.2324, 1.3310, 0.3165, 1.2953, 1.2727, 1.2489, 1.2252, 1.2007],
       device='cuda:0')
tensor([0.2445, 0.2948, 1.2788, 0.2654, 1.2411, 1.2234, 0.2034, 0.1811],
       device='cuda:7')
tensor([0.3717, 0.3094, 0.4151, 0.2608, 0.2850, 0.2229, 0.2226, 0.1926],
       device='cuda:1')
tensor([1.3254, 1.3969, 1.3784, 1.3583, 1.3357, 1.3115, 1.2885, 1.2643],
       device='cuda:4')
{'completion_length': 43.609375, 'rewards/accuracy_reward': 0.5673336982727051, 'rewards/efficiency reward': 0.22372688353061676, 'all_wrong': 0.125, 'all_correct': 0.0, 'reward': 0.7910605669021606, 'reward_std': 0.3443913161754608, 'kl': 63.5, 'epoch': 0.0}
{'loss': 2.54, 'grad_norm': 2721.2152439747842, 'learning_rate': 9.999991762838942e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/0741b1ea-5df7-40f9-9c1b-50713926958f.png
problem_id: 184023
prompt_length: 244
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/fbaeba0880a9d33c57785ce39f351ff0.png
problem_id: 242918
prompt_length: 102
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/f6d727dac436655fae4b35ec37922f2f.png
problem_id: 119782
prompt_length: 165
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Super-CLEVR/images/superCLEVR_new_001855.png
problem_id: 105575
prompt_length: 293
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/b411a51cb917d08d778b1928f3e8071f.png
problem_id: 105788
prompt_length: 134
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1001/5945234319.mp4path:
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_19188.png
problem_id: 175946
prompt_length: 298
problem_id: 218938
prompt_length: 1140
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_7ZcZTHLEcvo.mp4
problem_id: 85816
prompt_length: 1052
Invalidate trace cache @ step 0 and module 16742: cache has only 0 modules
tensor([0.2439, 0.3199, 0.3035, 0.2838, 0.2610, 0.2392, 0.2168, 0.1918],
       device='cuda:5')
tensor([0.2197, 1.2147, 0.2050, 0.1909, 0.1762, 1.1574, 0.1426, 0.1273],
       device='cuda:1')
tensor([0.2094, 0.2090, 0.1961, 0.1807, 0.1677, 0.1502, 0.1341, 0.1152],
       device='cuda:7')
tensor([0.6021, 0.2515, 0.3004, 1.1433, 0.5879, 0.6086, 0.2815, 0.8677],
       device='cuda:4')
Error converting 'D' to float: could not convert string to float: 'D'
tensor([0.2034, 0.2830, 0.2654, 0.2500, 0.2290, 0.2109, 0.1905, 0.1691],
       device='cuda:3')
tensor([1.1771, 1.2026, 0.1898, 0.1730, 0.1644, 1.1498, 0.1322, 0.1133],
       device='cuda:2')
tensor([0.2820, 1.3805, 1.3580, 1.3365, 1.3140, 1.2938, 1.2663, 1.2426],
       device='cuda:6')
tensor([1.3120, 1.3998, 1.3769, 1.3588, 1.3359, 1.3115, 1.2835, 1.2550],
       device='cuda:0')
{'completion_length': 36.171875, 'rewards/accuracy_reward': 0.35867607593536377, 'rewards/efficiency reward': 0.23053181171417236, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.5892078876495361, 'reward_std': 0.23025234043598175, 'kl': 288.0, 'epoch': 0.0}
{'loss': 11.4819, 'grad_norm': 240.10176902988388, 'learning_rate': 9.999990872952008e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/human_system_eye_6126.png
problem_id: 141650
prompt_length: 321
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/076b529fceb7b3f6980a8510c806264f.png
problem_id: 184243
prompt_length: 109
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/210/jGC0VhFCk5Y/split_6.mp4
problem_id: 44037
prompt_length: 1067
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_4396.png
problem_id: 10145
prompt_length: 328
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/a08475cdb3e99f02fac46c7b804f6ece.png
problem_id: 25796
prompt_length: 106
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/302/k6klZcQR2S0/split_1.mp4
problem_id: 121794
prompt_length: 1010
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_5wBo0Gd81-I.mp4
problem_id: 111831
prompt_length: 1082
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_CwvtmoTWNgQ.mp4
problem_id: 216407
prompt_length: 1051
Invalidate trace cache @ step 0 and module 17503: cache has only 0 modules
Error converting 'Cubic 4<answer>D' to float: could not convert string to float: 'Cubic 4<answer>D'
tensor([0.1798, 0.1986, 0.1815, 0.1697, 0.1550, 0.1429, 0.1258, 0.1104],
       device='cuda:1')
tensor([1.2092, 0.1995, 1.1877, 0.1762, 0.1590, 0.1418, 0.1297, 0.1129],
       device='cuda:3')
tensor([0.2266, 0.2760, 0.2626, 0.2481, 0.2279, 0.2098, 0.1881, 0.1663],
       device='cuda:4')
tensor([1.2811, 1.3201, 1.3048, 1.2842, 1.2647, 1.2400, 1.2173, 1.1903],
       device='cuda:5')
tensor([1.3406, 1.4001, 1.3785, 0.3612, 1.3393, 0.3140, 0.2851, 1.2592],
       device='cuda:0')
tensor([0.4642, 0.4871, 0.5288, 0.4867, 0.4331, 0.4434, 0.4011, 0.2048],
       device='cuda:2')
tensor([0.3433, 1.4252, 1.4015, 1.3801, 0.3601, 0.3310, 0.3066, 0.2755],
       device='cuda:6')
tensor([1.2679, 1.3919, 1.3762, 1.3579, 1.3358, 1.3084, 1.2852, 1.2534],
       device='cuda:7')
{'completion_length': 27.109375, 'rewards/accuracy_reward': 0.4257355332374573, 'rewards/efficiency reward': 0.26199984550476074, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.687735378742218, 'reward_std': 0.229683056473732, 'kl': 0.9921875, 'epoch': 0.0}
{'loss': 0.0397, 'grad_norm': 71.50028081041285, 'learning_rate': 9.999989937429903e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/Y91JG.mp4
problem_id: 47238
prompt_length: 1058
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_782xcTgNhZ0.mp4
problem_id: 63602
prompt_length: 1114
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/V34GF.mp4
problem_id: 6230
prompt_length: 1126
path: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1162/2467532859.mp4
problem_id: 191225
prompt_length: 1105
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/203/XlAZJB-uAO0/split_0.mp4
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_sfCfrWpHpu0.mp4
problem_id: 93094
prompt_length: 1051
problem_id: 149769
prompt_length: 1132
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/6FJBD.mp4
problem_id: 180576
prompt_length: 1054
Invalidate trace cache @ step 0 and module 18264: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_2931.png
problem_id: 227406
prompt_length: 307
Error converting 'The distance between the front of the children and the wall is approximately 13.30 inches' to float: could not convert string to float: 'The distance between the front of the children and the wall is approximately 13.30 inches'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.6283, 0.3035, 0.2889, 0.2674, 1.1764, 0.2272, 0.5794, 0.1804],
       device='cuda:4')
tensor([1.3582, 1.4062, 1.3882, 1.3631, 1.3406, 1.3184, 1.2862, 0.2579],
       device='cuda:6')
tensor([1.3621, 1.4067, 1.3857, 1.3649, 1.3453, 1.3212, 1.2949, 1.2656],
       device='cuda:5')
tensor([0.2644, 0.4005, 0.3838, 0.3600, 0.3384, 0.3145, 0.2867, 0.2589],
       device='cuda:1')
tensor([1.3407, 1.4034, 1.3815, 1.3617, 1.3372, 1.3137, 1.2869, 1.2545],
       device='cuda:0')
tensor([1.3380, 1.4002, 1.3822, 1.3604, 1.3369, 0.3165, 1.2880, 1.2547],
       device='cuda:7')
tensor([1.3475, 1.4085, 1.3958, 1.3727, 1.3523, 1.3261, 1.3026, 1.2759],
       device='cuda:2')
tensor([0.2688, 0.3952, 1.3773, 0.3542, 1.3353, 0.3103, 1.2805, 1.2545],
       device='cuda:3')
{'completion_length': 13.53125, 'rewards/accuracy_reward': 0.6825530529022217, 'rewards/efficiency reward': 0.3242897689342499, 'all_wrong': 0.125, 'all_correct': 0.0, 'reward': 1.006842851638794, 'reward_std': 0.22591063380241394, 'kl': 0.91015625, 'epoch': 0.0}
{'loss': 0.0365, 'grad_norm': 113.83858592582423, 'learning_rate': 9.999988956272635e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1103.2977_1.jpg
problem_id: 116036
prompt_length: 381
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/UVP3F.mp4
problem_id: 160346
prompt_length: 1067
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00005352.png
problem_id: 4767
prompt_length: 279
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_2xQVZorob1Q.mp4
problem_id: 129878
prompt_length: 1146
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/RJ27X.mp4
problem_id: 11962
prompt_length: 1011
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/325/1xjsM-I-KLI/split_8.mp4
problem_id: 169853
prompt_length: 811
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_09109.mp4
problem_id: 124542
prompt_length: 691
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_9077.png
problem_id: 175106
prompt_length: 316
Invalidate trace cache @ step 0 and module 19025: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.7671, 0.7869, 0.3149, 0.9837, 0.6174, 0.5943, 0.5735, 0.7755],
       device='cuda:6')
tensor([1.0741, 1.0959, 1.1915, 0.2330, 1.1060, 1.0861, 0.9555, 0.9931],
       device='cuda:4')
tensor([0.3274, 1.3657, 0.3476, 0.3263, 0.3103, 1.2854, 1.2609, 1.2368],
       device='cuda:0')
tensor([1.3451, 1.3970, 1.3796, 1.3593, 1.3362, 1.3120, 1.2855, 1.2597],
       device='cuda:5')
tensor([0.2699, 0.2536, 0.2384, 1.2265, 1.2080, 0.1867, 0.1750, 0.1558],
       device='cuda:1')
tensor([1.3622, 0.4390, 1.4247, 1.4007, 1.3807, 1.3599, 1.3299, 1.3029],
       device='cuda:7')
tensor([1.3356, 1.3944, 1.3765, 1.3527, 1.3270, 1.3076, 1.2742, 1.2484],
       device='cuda:3')
tensor([0.5625, 0.4939, 0.5351, 0.4748, 0.4984, 0.2512, 0.4757, 0.4486],
       device='cuda:2')
{'completion_length': 48.3125, 'rewards/accuracy_reward': 0.621063232421875, 'rewards/efficiency reward': 0.2907138168811798, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.911777138710022, 'reward_std': 0.25058653950691223, 'kl': 1.6796875, 'epoch': 0.0}
{'loss': 0.0671, 'grad_norm': 69.38702676243176, 'learning_rate': 9.999987929480216e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/6725bcd9-dc8d-4c43-8315-bd32b34c22fc.png
problem_id: 260484
prompt_length: 244
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4V/pierre-puvis-de-chavannes_thomas-alfred-jones.png
problem_id: 214354
prompt_length: 283
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/atomic_mass_number_9010.png
problem_id: 26557
prompt_length: 326
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/02DPI.mp4
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_13959.png
problem_id: 204233
prompt_length: 280
problem_id: 133925
prompt_length: 1050path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_cHYZPYLwvks.mp4
problem_id: 81385
prompt_length: 1055
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_2295.png
problem_id: 79052
prompt_length: 283

path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_WLQLE4NaNlI.mp4
problem_id: 208129
prompt_length: 1050
Invalidate trace cache @ step 0 and module 19786: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([1.2847, 1.2832, 1.2676, 1.2522, 1.2367, 1.2161, 1.1973, 0.1755],
       device='cuda:5')
tensor([0.3536, 0.3572, 0.2792, 0.2758, 0.2419, 0.2235, 0.2014, 0.1778],
       device='cuda:2')
tensor([0.2787, 0.2917, 0.2779, 0.2466, 0.2090, 0.3676, 0.3393, 0.2020],
       device='cuda:1')
tensor([0.7025, 0.6371, 0.5793, 0.2843, 0.6010, 0.5872, 0.5461, 0.5472],
       device='cuda:7')
tensor([0.2940, 0.3126, 0.5163, 0.5099, 0.4673, 0.2332, 0.4127, 0.3396],
       device='cuda:3')
tensor([0.3013, 0.3676, 0.3472, 0.3273, 1.3028, 1.2831, 1.2576, 1.2325],
       device='cuda:6')
tensor([0.3420, 1.4006, 0.3774, 1.3592, 1.3403, 1.3161, 1.2913, 1.2632],
       device='cuda:0')
tensor([1.3045, 1.4213, 1.4002, 1.3779, 1.3518, 1.3317, 1.2999, 1.2664],
       device='cuda:4')
{'completion_length': 38.234375, 'rewards/accuracy_reward': 0.45400989055633545, 'rewards/efficiency reward': 0.2752092778682709, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.7292191386222839, 'reward_std': 0.21711783111095428, 'kl': 0.640625, 'epoch': 0.0}
{'loss': 0.0257, 'grad_norm': 46.613534136590815, 'learning_rate': 9.99998685705265e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/CLEVR-Math/images/CLEVR_train_046310.png
problem_id: 173111
prompt_length: 254
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/127/0lkjIf5hZT4/split_8.mp4
problem_id: 32531
prompt_length: 1067
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/5IDF4.mp4
problem_id: 63952
prompt_length: 1045
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/GVLQA/data/flow/graph_visual/layout/fdp/hard/graph505.png
problem_id: 4016
prompt_length: 356
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/H3XC1.mp4
problem_id: 50235
prompt_length: 1140
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/IconQA/images/42717.png
problem_id: 94580
prompt_length: 82
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/A-OKVQA/images/500271.jpg
problem_id: 134704
prompt_length: 317
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_D2ggFcgEbFo.mp4
problem_id: 149417
prompt_length: 1064
Invalidate trace cache @ step 0 and module 20547: cache has only 0 modules
Error converting '<value> 9' to float: could not convert string to float: '<value> 9'
Error converting '' to float: could not convert string to float: ''
tensor([0.1996, 0.2034, 0.1925, 0.1773, 1.1638, 0.1459, 0.1276, 0.1132],
       device='cuda:5')
tensor([1.3165, 1.3481, 1.3307, 1.3081, 1.2911, 1.2684, 1.2425, 1.2167],
       device='cuda:0')
tensor([0.2452, 0.2589, 0.2408, 0.2251, 0.2060, 0.1953, 0.1745, 0.1564],
       device='cuda:3')
tensor([0.2311, 0.2860, 1.2676, 0.2510, 0.2321, 0.2158, 0.1957, 0.1707],
       device='cuda:1')
tensor([1.3173, 1.3883, 1.3727, 1.3510, 1.3297, 1.3124, 1.2814, 1.2516],
       device='cuda:7')
tensor([0.3435, 0.3910, 0.3687, 0.3491, 0.3220, 0.2999, 0.2731, 0.2408],
       device='cuda:4')
tensor([1.3314, 0.4069, 1.3914, 1.3655, 1.3436, 1.3195, 1.2904, 1.2601],
       device='cuda:6')
tensor([0.3038, 1.3901, 1.3680, 0.3476, 1.3229, 0.2964, 1.2719, 1.2403],
       device='cuda:2')
{'completion_length': 11.390625, 'rewards/accuracy_reward': 0.46875, 'rewards/efficiency reward': 0.275626540184021, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.7443764805793762, 'reward_std': 0.2188618779182434, 'kl': 0.98046875, 'epoch': 0.0}
{'loss': 0.0391, 'grad_norm': 17.80794812272755, 'learning_rate': 9.999985738989953e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/Visual7W/cauldron/visual7w/images/visual7w_00005241.png
problem_id: 248204
prompt_length: 313
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1028/5318941838.mp4
problem_id: 150334
prompt_length: 1044
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/QH3FK.mp4
problem_id: 111846
prompt_length: 1046
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_oXFKaTlg3p0.mp4
problem_id: 69332
prompt_length: 1075
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_ctzQu22aLLE.mp4
problem_id: 210582
prompt_length: 1073
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/MapQA/images/map_7121.png
problem_id: 192476
prompt_length: 306
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_09779.mp4
problem_id: 235994
prompt_length: 688
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_UWTpfygMUQw.mp4
problem_id: 97984
prompt_length: 1056
Invalidate trace cache @ step 0 and module 21308: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
tensor([0.2084, 0.2941, 0.2742, 0.2581, 0.2376, 0.2197, 0.1990, 0.1732],
       device='cuda:5')
tensor([1.3155, 1.3563, 1.3360, 1.3137, 1.2888, 1.2689, 1.2441, 1.2177],
       device='cuda:7')
Error converting 'There are 2 moving rubber objects' to float: could not convert string to float: 'There are 2 moving rubber objects'
tensor([1.2891, 0.3401, 0.3260, 1.3062, 0.2877, 0.2664, 0.2434, 0.2204],
       device='cuda:0')
tensor([0.3153, 0.4268, 0.4027, 0.3811, 1.3579, 1.3329, 1.3045, 1.2726],
       device='cuda:1')
tensor([1.3393, 1.4068, 1.3921, 1.3692, 1.3460, 1.3239, 1.2963, 1.2666],
       device='cuda:4')
tensor([0.2663, 0.3525, 0.3355, 0.3218, 0.3025, 0.2841, 0.2638, 0.2383],
       device='cuda:3')
tensor([1.3431, 1.4017, 1.3804, 1.3627, 1.3379, 1.3152, 1.2905, 1.2608],
       device='cuda:6')
tensor([1.3411, 1.4117, 1.3916, 1.3719, 1.3506, 1.3249, 1.2982, 1.2631],
       device='cuda:2')
{'completion_length': 9.78125, 'rewards/accuracy_reward': 0.59375, 'rewards/efficiency reward': 0.3098224997520447, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.9035725593566895, 'reward_std': 0.15546715259552002, 'kl': 0.8203125, 'epoch': 0.0}
{'loss': 0.0327, 'grad_norm': 25.994190025031088, 'learning_rate': 9.99998457529213e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_2oizmWFx4PA.mp4
problem_id: 136605
prompt_length: 1057
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/cycle_water_4927.png
problem_id: 249467
prompt_length: 268
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_2918.png
problem_id: 100990
prompt_length: 285
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextOCR/02a20e75172c4fbe.png
problem_id: 128838
prompt_length: 298
Invalidate trace cache @ step 0 and module 22069: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_aYrhuTGO440.mp4
problem_id: 259608
prompt_length: 1133
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_21381.png
problem_id: 220205
prompt_length: 78
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_K8OO5ExdKEU.mp4
problem_id: 205600
prompt_length: 1048
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/MapQA/images/map_7922.png
problem_id: 96106
prompt_length: 316
tensor([0.2483, 0.2616, 0.2505, 0.2343, 0.2183, 0.2027, 0.1827, 0.1608],
       device='cuda:7')
tensor([0.2982, 0.3399, 0.3088, 0.2910, 0.2687, 0.2480, 0.2234, 0.1991],
       device='cuda:0')
tensor([0.2861, 0.2875, 0.2746, 0.2588, 0.2346, 0.2164, 0.2026, 0.1785],
       device='cuda:6')
tensor([0.3348, 0.2138, 0.1992, 0.1841, 0.3352, 0.1557, 0.1360, 0.2821],
       device='cuda:2')
tensor([0.6423, 0.6416, 0.6476, 0.6307, 0.4918, 0.5105, 0.5237, 0.5100],
       device='cuda:3')
tensor([1.2908, 1.3763, 1.3601, 1.3389, 1.3173, 1.2919, 1.2664, 1.2423],
       device='cuda:5')
tensor([1.3510, 1.4192, 1.4007, 1.3819, 1.3597, 1.3300, 1.3036, 1.2765],
       device='cuda:4')
tensor([1.3570, 1.4123, 1.3943, 1.3753, 1.3524, 1.3259, 1.2985, 1.2702],
       device='cuda:1')
{'completion_length': 12.84375, 'rewards/accuracy_reward': 0.42198848724365234, 'rewards/efficiency reward': 0.2718730866909027, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6938616037368774, 'reward_std': 0.05182763934135437, 'kl': 0.75, 'epoch': 0.0}
{'loss': 0.0299, 'grad_norm': 7.297600914464108, 'learning_rate': 9.999983365959197e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_6273.png
problem_id: 23619
prompt_length: 310
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/1337.png
problem_id: 25535
prompt_length: 145
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/110/U10oixMc9_k/split_2.mp4
problem_id: 14883
prompt_length: 1054
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_3mTBrR4yOpg.mp4
problem_id: 200729
prompt_length: 1040
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/MapQA/images/map_8477.png
problem_id: 245451
prompt_length: 310
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_dgXPoqHrKRM.mp4
problem_id: 217852
prompt_length: 1068
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/edb2382ff318a4e52b08d0f88b8c8e70.png
problem_id: 112441
prompt_length: 83
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_04456.mp4
problem_id: 39116
prompt_length: 734
Invalidate trace cache @ step 0 and module 22830: cache has only 0 modules
Error converting 'This image is a close-up and it is not possible to accurately measure the distance between the illustrations and the texte in inches due to the size and scale of the photo. Please use a reliable scale or a ruler for an accurate measurement.' to float: could not convert string to float: 'This image is a close-up and it is not possible to accurately measure the distance between the illustrations and the texte in inches due to the size and scale of the photo. Please use a reliable scale or a ruler for an accurate measurement.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2521, 0.2101, 1.2007, 0.1835, 0.1717, 0.1522, 0.1362, 1.1217],
       device='cuda:1')
tensor([0.2225, 0.2238, 0.2065, 0.1880, 0.1773, 0.1601, 0.1450, 0.1205],
       device='cuda:0')
tensor([0.2090, 0.2949, 0.2796, 0.2586, 0.2408, 0.2204, 0.1995, 0.1755],
       device='cuda:2')
tensor([0.3133, 0.3272, 0.3115, 0.2889, 0.2695, 0.9971, 0.2267, 0.1998],
       device='cuda:4')
tensor([0.3022, 0.4116, 0.3897, 0.3712, 0.3524, 0.3241, 0.3036, 0.2759],
       device='cuda:6')
tensor([1.3277, 1.3990, 1.3776, 1.3539, 1.3286, 1.3060, 1.2795, 1.2522],
       device='cuda:5')
tensor([1.3034, 1.4196, 1.4011, 1.3750, 1.3507, 1.3257, 1.2970, 1.2723],
       device='cuda:7')
tensor([0.3020, 0.3781, 0.3646, 0.3429, 0.3199, 0.2959, 0.2704, 0.2404],
       device='cuda:3')
{'completion_length': 15.640625, 'rewards/accuracy_reward': 0.29296875, 'rewards/efficiency reward': 0.2741960883140564, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.5671648383140564, 'reward_std': 0.12334006279706955, 'kl': 0.87890625, 'epoch': 0.0}
{'loss': 0.0352, 'grad_norm': 11.04286527016537, 'learning_rate': 9.99998211099116e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_12264.png
problem_id: 125951
prompt_length: 318
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/ef975f7ce288d97de45a6f03b3792781.png
problem_id: 63471
prompt_length: 183
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/GVLQA/data/matching/graph_visual/layout/fdp/hard/graph1066.png
problem_id: 35015
prompt_length: 425
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4V/Hermitage.png
problem_id: 176953
prompt_length: 285
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_3857.mp4
problem_id: 185358
prompt_length: 1047
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/DVQA/images/bar_train_00051497.png
problem_id: 143792
prompt_length: 331
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_3934.png
problem_id: 41903
prompt_length: 311
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_9481.mp4
problem_id: 110300
prompt_length: 1069
Invalidate trace cache @ step 0 and module 23591: cache has only 0 modules
Error converting 'b = 15' to float: could not convert string to float: 'b = 15'
tensor([1.2054, 0.2836, 0.2686, 0.2526, 0.2353, 0.2221, 0.1991, 0.1811],
       device='cuda:0')
tensor([1.2302, 1.3252, 1.3019, 1.2836, 1.2652, 1.2430, 1.2228, 1.1963],
       device='cuda:1')
tensor([0.2195, 0.2053, 0.1932, 0.1778, 0.1643, 1.1516, 0.1349, 0.1214],
       device='cuda:4')
tensor([0.7885, 0.6689, 0.6164, 0.6270, 0.2853, 0.7193, 0.7112, 0.6808],
       device='cuda:7')
tensor([0.2644, 0.2579, 0.2485, 0.2306, 0.2177, 0.1980, 0.1792, 0.1610],
       device='cuda:3')
tensor([0.6166, 0.6656, 0.6239, 0.5142, 0.5719, 0.2506, 0.4868, 0.4775],
       device='cuda:5')
tensor([1.3403, 1.3776, 1.3665, 1.3422, 1.3223, 1.2982, 1.2715, 1.2481],
       device='cuda:6')
tensor([0.2995, 0.3934, 0.3777, 0.3488, 0.3326, 0.3087, 0.2861, 0.2510],
       device='cuda:2')
{'completion_length': 35.75, 'rewards/accuracy_reward': 0.3564385175704956, 'rewards/efficiency reward': 0.2609107196331024, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.6173492670059204, 'reward_std': 0.14367060363292694, 'kl': 0.671875, 'epoch': 0.0}
{'loss': 0.0269, 'grad_norm': 12.887679315293681, 'learning_rate': 9.999980810388033e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/0026/5986131986.mp4
problem_id: 165655
prompt_length: 1062
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/15451.png
problem_id: 18158
prompt_length: 113
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Geometry3K/images/1223.png
problem_id: 78039
prompt_length: 334
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_EWPWaPxEtQo.mp4
problem_id: 103204
prompt_length: 1080
 path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1099/8444144653.mp4
problem_id: 243103
prompt_length: 1050
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/421/D0STPAXDYxg/split_2.mp4
problem_id: 137108
prompt_length: 1139path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_6318.png
problem_id: 21024
prompt_length: 325

path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_uFzEDSEwiG8.mp4
problem_id: 110305
prompt_length: 1060
Invalidate trace cache @ step 0 and module 24352: cache has only 0 modules
Error converting '6131 + 1824 = 7955' to float: could not convert string to float: '6131 + 1824 = 7955'
tensor([0.2070, 0.1994, 0.1868, 0.1691, 0.1546, 0.1391, 0.1277, 0.1084],
       device='cuda:1')
tensor([1.2409, 0.3454, 0.8360, 0.3065, 1.1842, 0.3335, 0.6699, 0.2135],
       device='cuda:7')
tensor([1.2739, 1.2712, 1.2553, 1.2390, 1.2236, 1.2037, 1.1870, 0.1689],
       device='cuda:2')
tensor([1.3181, 1.3989, 1.3814, 1.3583, 1.3350, 0.3091, 1.2818, 1.2561],
       device='cuda:5')
tensor([1.2756, 1.4087, 1.3871, 1.3700, 1.3447, 1.3226, 1.2911, 1.2611],
       device='cuda:4')
tensor([0.3169, 0.3655, 1.3433, 0.3309, 0.3078, 0.2856, 0.2620, 0.2402],
       device='cuda:6')
tensor([1.2766, 1.3702, 1.3524, 1.3319, 1.3123, 1.2901, 1.2626, 1.2366],
       device='cuda:3')
tensor([0.3071, 0.4092, 0.3941, 0.3709, 0.3466, 0.3249, 0.3003, 0.2722],
       device='cuda:0')
{'completion_length': 11.421875, 'rewards/accuracy_reward': 0.5286369323730469, 'rewards/efficiency reward': 0.28627774119377136, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.8149147033691406, 'reward_std': 0.21331432461738586, 'kl': 0.65234375, 'epoch': 0.0}
{'loss': 0.026, 'grad_norm': 31.01342003704209, 'learning_rate': 9.999979464149827e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/323/_b5ZFYRGFDg/split_4.mp4
problem_id: 245355
prompt_length: 1079
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_3184.mp4
problem_id: 186505
prompt_length: 1049
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC8811295_F1_183375.jpg
problem_id: 132692
prompt_length: 156
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_8505.png
problem_id: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/31efb60c2e415ddf52ba8ec27c948bf8.png
problem_id: 177448
prompt_length: 120
221469
prompt_length: 328
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/425d976bf379fd529af5f1fbf031bc11.png
problem_id: 26037
prompt_length: 89
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_5448.png
problem_id: 202114
prompt_length: 74
Invalidate trace cache @ step 0 and module 25113: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/G2DIN.mp4
problem_id: 100100
prompt_length: 1044
Error converting '0.2 meters' to float: could not convert string to float: '0.2 meters'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([1.2345, 0.1923, 0.1827, 0.1676, 1.1530, 0.1423, 0.1238, 1.1083],
       device='cuda:5')
tensor([0.5226, 0.2181, 0.2017, 0.1863, 0.1689, 0.3295, 0.1385, 0.9408],
       device='cuda:7')
tensor([0.2388, 0.2183, 0.2059, 0.1910, 0.1746, 0.1585, 0.1425, 0.1244],
       device='cuda:6')
tensor([0.3106, 0.3951, 0.3034, 0.3222, 0.8887, 0.2802, 0.3020, 0.3246],
       device='cuda:0')
tensor([0.2567, 0.2495, 0.2335, 0.2154, 0.2072, 0.1902, 0.1712, 0.1520],
       device='cuda:3')
tensor([1.2635, 1.3927, 1.3752, 1.3542, 1.3298, 1.3058, 1.2813, 1.2525],
       device='cuda:1')
tensor([1.2900, 1.3942, 1.3735, 1.3577, 1.3344, 1.3091, 1.2867, 1.2631],
       device='cuda:2')
tensor([1.3493, 1.4104, 1.3928, 1.3721, 1.3456, 1.3256, 1.3010, 1.2692],
       device='cuda:4')
{'completion_length': 25.765625, 'rewards/accuracy_reward': 0.4572615921497345, 'rewards/efficiency reward': 0.24894379079341888, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.7062053680419922, 'reward_std': 0.15242700278759003, 'kl': 3.453125, 'epoch': 0.0}
{'loss': 0.138, 'grad_norm': 58.45558582125685, 'learning_rate': 9.999978072276555e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_5164.png
problem_id: 133030
prompt_length: 299
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/64.png
problem_id: 205912
prompt_length: 142
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/A-OKVQA/images/232546.jpg
problem_id: 85525
prompt_length: 334
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/UniGeo/images/448.png
problem_id: 73726
prompt_length: 170
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/7327808781e91847ebfbd9cda5020f2e.png
problem_id: 161336
prompt_length: 130
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Super-CLEVR/images/superCLEVR_new_000372.png
problem_id: 222091
prompt_length: 310
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_14350.png
problem_id: 212166
prompt_length: 316
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_61243.png
problem_id: 55048
prompt_length: 76
Invalidate trace cache @ step 0 and module 25874: cache has only 0 modules
Error converting '$$y = 5$$' to float: could not convert string to float: '$$y = 5$$'
tensor([0.2049, 0.1934, 0.1802, 0.1635, 0.1530, 0.1376, 0.1253, 0.1080],
       device='cuda:0')
tensor([1.3193, 1.3466, 1.3272, 1.3072, 1.2893, 1.2674, 1.2397, 1.2134],
       device='cuda:4')
tensor([0.2469, 0.3554, 0.3410, 0.3176, 0.2971, 0.2777, 0.2532, 0.2249],
       device='cuda:6')
tensor([0.2385, 0.2123, 0.1911, 0.1745, 0.1615, 0.1517, 0.1320, 0.1154],
       device='cuda:2')
tensor([0.2549, 0.2141, 1.2034, 1.1866, 1.1712, 1.1574, 1.1431, 1.1298],
       device='cuda:3')
tensor([0.2294, 0.3327, 0.3168, 0.3018, 0.2807, 0.2565, 0.2356, 0.2094],
       device='cuda:5')
tensor([0.3548, 0.2195, 0.2016, 0.1870, 0.2624, 0.3378, 0.2277, 0.2079],
       device='cuda:7')
tensor([0.2853, 0.2980, 1.2806, 0.2657, 1.2403, 1.2233, 1.2053, 1.1801],
       device='cuda:1')
{'completion_length': 11.234375, 'rewards/accuracy_reward': 0.3068181872367859, 'rewards/efficiency reward': 0.2223612666130066, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.5291794538497925, 'reward_std': 0.14949947595596313, 'kl': 0.71484375, 'epoch': 0.0}
{'loss': 0.0285, 'grad_norm': 19.577485138423288, 'learning_rate': 9.999976634768229e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_ivWTI2J_UnY.mp4
problem_id: 235192
prompt_length: 1131
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_AtS90VEur54.mp4
problem_id: 151205
prompt_length: 1041
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/22933b0c9ac4bc76fdc337f92308cc2b.png
problem_id: 186239
prompt_length: 115
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_14930.png
problem_id: 99308
prompt_length: 282
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/b5fdfa2bdfe4d227d83d0c9a8f3ffe19.png
problem_id: 242001
prompt_length: 114
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00001777.png
problem_id: 31426
prompt_length: 294
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D-gpt4v/ai2d_azuregpt_2059.png
problem_id: 75455
prompt_length: 358
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_Im5DzVxWGZI.mp4
problem_id: 245415
prompt_length: 1085
Invalidate trace cache @ step 0 and module 26635: cache has only 0 modules
Error converting '<36> <100>' to float: could not convert string to float: '<36> <100>'
Error converting '<' to float: could not convert string to float: '<'
tensor([0.1734, 0.2003, 0.1907, 0.1761, 1.1624, 0.1491, 0.1295, 0.1152],
       device='cuda:3')
tensor([1.1627, 1.2011, 1.1889, 1.1770, 1.1638, 1.1451, 1.1272, 1.1125],
       device='cuda:0')
tensor([1.2543, 1.2415, 1.1773, 1.2389, 1.1451, 1.2041, 1.1130, 1.1669],
       device='cuda:6')
tensor([0.3794, 0.4691, 0.3833, 0.4508, 0.3863, 0.3560, 0.3149, 0.3169],
       device='cuda:5')
tensor([0.3379, 0.4746, 0.5462, 0.5265, 0.5856, 0.4979, 0.4926, 0.5522],
       device='cuda:7')
tensor([0.2370, 0.3389, 0.3199, 0.3056, 0.2851, 0.2641, 0.2432, 0.2176],
       device='cuda:2')
tensor([0.2957, 0.3909, 0.3729, 0.3472, 0.3216, 0.3005, 0.2784, 1.2468],
       device='cuda:1')
tensor([0.3113, 0.3856, 0.3668, 0.3465, 0.3289, 0.3017, 0.2781, 0.2489],
       device='cuda:4')
{'completion_length': 37.703125, 'rewards/accuracy_reward': 0.32484227418899536, 'rewards/efficiency reward': 0.24577686190605164, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.5706191062927246, 'reward_std': 0.1227748915553093, 'kl': 5.40625, 'epoch': 0.0}
{'loss': 0.2168, 'grad_norm': 116.90593023594526, 'learning_rate': 9.999975151624861e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/PlotQA/images/86097.png
problem_id: 249651
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_yw1IZdbEzck.mp4
problem_id: 168974
prompt_length: 1162
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_1076.png
problem_id: 9831
prompt_length: 332
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/129.png
problem_id: 148715
prompt_length: 317
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/ChromeWriting/identity_3534.png
problem_id: 179118
prompt_length: 72
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_4600.png
problem_id: 49563
prompt_length: 311
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/ecf290556e2b2115652add716bceac68.png
problem_id: 94125
prompt_length: 181
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Oj_PLXsWxL4.mp4
problem_id: 234665
prompt_length: 1064
Invalidate trace cache @ step 0 and module 27396: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
tensor([0.1736, 0.2183, 0.2034, 0.1880, 0.3701, 0.1553, 0.1374, 0.1166],
       device='cuda:0')
tensor([0.2524, 0.3731, 0.3594, 0.3363, 0.3112, 0.2900, 0.2639, 0.2364],
       device='cuda:3')
tensor([1.2691, 1.2747, 1.2583, 1.2446, 1.2233, 1.2049, 1.1906, 1.1667],
       device='cuda:5')
tensor([0.1976, 1.2001, 1.1900, 1.1738, 1.1632, 1.1488, 1.1381, 1.1208],
       device='cuda:6')
tensor([0.2823, 1.2943, 1.2807, 0.2633, 0.2443, 1.2312, 0.2116, 1.1863],
       device='cuda:4')
tensor([0.7018, 0.5920, 0.4622, 0.2679, 0.3170, 0.5458, 0.5250, 0.4256],
       device='cuda:1')
tensor([1.2817, 1.3805, 1.3593, 1.3386, 1.3191, 1.2955, 1.2724, 1.2387],
       device='cuda:7')
tensor([0.3307, 0.4234, 0.4004, 0.3743, 0.3536, 0.3296, 0.2987, 0.2698],
       device='cuda:2')
{'completion_length': 9.8125, 'rewards/accuracy_reward': 0.45364874601364136, 'rewards/efficiency reward': 0.2533474862575531, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.7069962024688721, 'reward_std': 0.16079115867614746, 'kl': 1.25, 'epoch': 0.0}
{'loss': 0.0498, 'grad_norm': 132.6616224876674, 'learning_rate': 9.999973622846469e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/314/KmqfNQG1BXk/split_3.mp4
problem_id: 259172
prompt_length: 1071
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_2359.png
problem_id: 20281
prompt_length: 326
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_dM5v0U9-lbQ.mp4
problem_id: 76914
prompt_length: 1059
path: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/a51887558259c0cd.jpg
problem_id: 87342
prompt_length: 307
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/CLEVR-Math/images/CLEVR_train_013355.png
problem_id: 240309
prompt_length: 244
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/119/LqCDIssigHo/split_0.mp4
problem_id: 230041
prompt_length: 1054
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_18499.png
problem_id: 91524
prompt_length: 280
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/225/WiliCz0XYVU/split_4.mp4
problem_id: 148954
prompt_length: 1041
Invalidate trace cache @ step 0 and module 28157: cache has only 0 modules
tensor([1.2323, 1.3457, 1.3293, 1.3140, 0.2869, 1.2674, 1.2428, 1.2119],
       device='cuda:0')
tensor([0.2669, 0.3368, 0.3193, 0.3004, 0.2819, 0.2590, 0.2380, 0.2084],
       device='cuda:5')
tensor([0.3232, 0.3531, 0.3414, 0.3173, 0.2949, 0.2746, 0.2462, 0.2217],
       device='cuda:2')
tensor([0.4514, 0.3933, 0.4132, 0.3935, 0.3673, 0.3407, 0.3194, 0.3236],
       device='cuda:7')
tensor([1.2670, 1.3977, 1.3776, 1.3537, 1.3296, 1.3094, 1.2853, 1.2539],
       device='cuda:6')
tensor([1.3194, 1.3939, 1.3669, 1.3563, 1.3253, 1.3006, 1.2707, 1.2437],
       device='cuda:3')
tensor([1.3110, 1.3968, 1.3770, 1.3577, 1.3324, 1.3093, 1.2846, 1.2559],
       device='cuda:1')
tensor([1.3167, 1.4164, 1.3976, 1.3723, 1.3512, 1.3266, 1.2974, 1.2678],
       device='cuda:4')
{'completion_length': 17.328125, 'rewards/accuracy_reward': 0.625970184803009, 'rewards/efficiency reward': 0.3011762499809265, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.9271463751792908, 'reward_std': 0.08611433207988739, 'kl': 0.578125, 'epoch': 0.0}
{'loss': 0.023, 'grad_norm': 3.9561844445517127, 'learning_rate': 9.999972048433062e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/941043c62970a3bf.png
problem_id: 254759
prompt_length: 309
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2201.06587_2.jpg
problem_id: 234369
prompt_length: 341
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_349.png
problem_id: 67139
prompt_length: 327
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_5yrLDF_ZmN0.mp4
problem_id: 95230
prompt_length: 1129
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D-gpt4v/ai2d_azuregpt_2605.png
problem_id: 126016
prompt_length: 354
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/3405.png
problem_id: 61683
prompt_length: 105
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1903.08352_0.jpg
problem_id: 14416
prompt_length: 344
Invalidate trace cache @ step 0 and module 28918: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_u3nOmvgcs84.mp4
problem_id: 76991
prompt_length: 1060
Error converting 'Your answer is 42' to float: could not convert string to float: 'Your answer is 42'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([1.2789, 1.2857, 1.2665, 1.2558, 1.2358, 1.2159, 1.1961, 1.1764],
       device='cuda:6')
tensor([0.2108, 0.2044, 0.1912, 0.1779, 0.1612, 0.1452, 0.1323, 0.1173],
       device='cuda:7')
tensor([0.2602, 0.2833, 0.2701, 0.2516, 0.2348, 0.2203, 0.2050, 0.1811],
       device='cuda:2')
tensor([0.5676, 0.6018, 0.3331, 0.3159, 0.2936, 1.0278, 0.2490, 1.2146],
       device='cuda:5')
tensor([0.2794, 0.2707, 0.2554, 0.2372, 0.2219, 0.2043, 0.1834, 0.1617],
       device='cuda:4')
tensor([0.5694, 0.6036, 0.5178, 0.5091, 0.6405, 0.2025, 0.5398, 0.5255],
       device='cuda:0')
tensor([1.3369, 1.3958, 1.3780, 1.3614, 1.3352, 1.3114, 1.2875, 1.2588],
       device='cuda:1')
tensor([0.3324, 0.3983, 0.3785, 0.3553, 0.3365, 0.3072, 0.2827, 0.2537],
       device='cuda:3')
{'completion_length': 22.3125, 'rewards/accuracy_reward': 0.3217182755470276, 'rewards/efficiency reward': 0.2563019394874573, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.5780202150344849, 'reward_std': 0.09277733415365219, 'kl': 1.875, 'epoch': 0.0}
{'loss': 0.075, 'grad_norm': 83.90532688395449, 'learning_rate': 9.999970428384657e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00005233.png
problem_id: 137062
prompt_length: 136
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/6a1b3c2304d0f7b76263583fba31a42f.png
problem_id: 109501
prompt_length: 129
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/24188.png
problem_id: 35947
prompt_length: 76
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/K7E8L.mp4
problem_id: 119214
prompt_length: 1050
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_5611.png
problem_id: 106095
prompt_length: 281
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/4ccffc8a-64fa-4ecd-8af5-5c0c0e9a3b4d.png
problem_id: 170150
prompt_length: 297
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_Vq18yYl9czs.mp4
problem_id: 51213
prompt_length: 1058
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1208.0717_2.jpg
problem_id: 54168
prompt_length: 421
Invalidate trace cache @ step 0 and module 29679: cache has only 0 modules
tensor([0.2443, 0.2199, 0.2054, 0.1894, 0.1748, 0.1574, 0.1396, 0.1208],
       device='cuda:3')
tensor([0.6760, 0.7549, 0.7344, 0.7178, 0.2543, 0.6761, 0.6559, 0.3718],
       device='cuda:0')tensor([0.7189, 0.9871, 0.9758, 0.9598, 0.9417, 0.9259, 0.9123, 0.8929],
       device='cuda:4')

tensor([0.2598, 1.2455, 0.2281, 1.2164, 1.2002, 0.1793, 0.1664, 0.1485],
       device='cuda:1')
tensor([0.1931, 0.1927, 0.1808, 0.1689, 0.1550, 0.1401, 0.1246, 0.1107],
       device='cuda:2')
tensor([0.5874, 0.6310, 0.6413, 0.5823, 0.6293, 0.6188, 0.1817, 0.5211],
       device='cuda:6')
tensor([1.2575, 1.3865, 1.3716, 1.3468, 1.3280, 1.2995, 1.2765, 1.2483],
       device='cuda:7')
tensor([1.3280, 1.4053, 1.3870, 0.3610, 1.3373, 1.3214, 1.2828, 1.2541],
       device='cuda:5')
{'completion_length': 26.796875, 'rewards/accuracy_reward': 0.45696669816970825, 'rewards/efficiency reward': 0.23212930560112, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6890959739685059, 'reward_std': 0.17837759852409363, 'kl': 0.57421875, 'epoch': 0.0}
{'loss': 0.023, 'grad_norm': 80.36703531428392, 'learning_rate': 9.999968762701266e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/5UNMN.mp4
problem_id: 86770
prompt_length: 1053
Invalidate trace cache @ step 0 and module 30440: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/209/nfJ8Hsh7L_c/split_2.mp4
problem_id: 257238
prompt_length: 1009
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/CLEVR-Math/images/CLEVR_train_012685.png
problem_id: 224858
prompt_length: 248
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/230/p_UKXct35jY/split_1.mp4
problem_id: 63634
prompt_length: 1049
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/4293.png
problem_id: 63902
prompt_length: 326
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/Chart2Text/cauldron/chart2text/images/chart2text_00008022.png
problem_id: 44633
prompt_length: 295
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_RYJ3yzxZB8k.mp4
problem_id: 77849
prompt_length: 1066
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00005153.png
problem_id: 137282
prompt_length: 287
tensor([0.2185, 0.2761, 0.2588, 0.2418, 0.2224, 0.2043, 0.1820, 0.1637],
       device='cuda:4')
tensor([0.9775, 1.0253, 1.0107, 0.9965, 0.9188, 0.9586, 0.9430, 0.9217],
       device='cuda:6')
tensor([0.3574, 0.4866, 0.2519, 0.4383, 0.4971, 0.3769, 0.4234, 0.3317],
       device='cuda:3')
tensor([1.1989, 1.2728, 1.2589, 1.2462, 1.2286, 1.2111, 1.1907, 1.1719],
       device='cuda:2')
tensor([0.3327, 0.3916, 0.3724, 0.3514, 0.3289, 0.3021, 1.2763, 0.2467],
       device='cuda:5')
tensor([1.2547, 1.3491, 1.3307, 1.3115, 1.2925, 1.2737, 1.2517, 1.2249],
       device='cuda:1')
tensor([0.3955, 0.4225, 0.9778, 0.9601, 0.9343, 0.3440, 0.3288, 0.2948],
       device='cuda:7')
tensor([1.3098, 1.4082, 1.3876, 1.3659, 1.3423, 1.3193, 1.2930, 1.2679],
       device='cuda:0')
{'completion_length': 19.8125, 'rewards/accuracy_reward': 0.5448446869850159, 'rewards/efficiency reward': 0.2630422115325928, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.8078869581222534, 'reward_std': 0.11706554889678955, 'kl': 0.66796875, 'epoch': 0.0}
{'loss': 0.0267, 'grad_norm': 29.01810852082699, 'learning_rate': 9.99996705138291e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/GEOS/images/044.png
problem_id: 105395
prompt_length: 148
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_SLdf2ZUdgEQ.mp4
problem_id: 51045
prompt_length: 1138
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_66935.png
problem_id: 69760
prompt_length: 65
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_kjd9emi-hR4.mp4
problem_id: 82379
prompt_length: 1043
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/9e6dfe13210d2dec87a18d5524b3bf47.pngpath: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_o3cR7VnixIQ.mp4
problem_id: 88994
prompt_length: 1072

problem_id: 130134
prompt_length: 110
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/27617.png
problem_id: 159386
prompt_length: 120
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_08439.mp4
problem_id: 116766
prompt_length: 730
Invalidate trace cache @ step 0 and module 31201: cache has only 0 modules
Error converting 'C' to float: could not convert string to float: 'C'
Error converting '<answer>101' to float: could not convert string to float: '<answer>101'
Error converting '1<' to float: could not convert string to float: '1<'
tensor([0.2207, 0.1994, 0.1872, 0.1702, 0.1589, 0.1467, 0.1283, 0.1113],
       device='cuda:1')
tensor([0.2206, 0.2516, 0.2352, 0.2181, 0.2022, 0.1824, 0.1621, 0.1434],
       device='cuda:2')
tensor([0.2400, 0.1929, 0.1841, 0.1692, 0.1549, 0.1399, 1.1250, 0.1081],
       device='cuda:0')
tensor([0.1983, 0.2116, 0.2022, 0.1873, 0.1730, 0.1581, 0.1402, 0.1269],
       device='cuda:4')
tensor([0.2990, 0.3998, 0.3830, 0.3614, 0.3451, 0.3175, 0.2947, 0.2693],
       device='cuda:6')
tensor([1.2834, 1.3726, 1.3535, 1.3366, 1.3166, 1.2887, 1.2641, 1.2372],
       device='cuda:3')
tensor([1.3367, 1.3994, 1.3854, 1.3622, 1.3393, 1.3150, 1.2857, 1.2573],
       device='cuda:5')
tensor([1.3262, 1.4028, 1.3874, 1.3609, 1.3353, 1.3129, 1.2870, 1.2610],
       device='cuda:7')
{'completion_length': 10.0625, 'rewards/accuracy_reward': 0.390625, 'rewards/efficiency reward': 0.25198131799697876, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.6426063179969788, 'reward_std': 0.07915344089269638, 'kl': 0.80859375, 'epoch': 0.0}
{'loss': 0.0323, 'grad_norm': 8.689919219015405, 'learning_rate': 9.9999652944296e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/19306e847f2dbe79.jpg
problem_id: 244521
prompt_length: 302
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_57084.png
problem_id: 229870
prompt_length: 148
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/63b42f4f03d3cd826578680928096c06.png
problem_id: 222259
prompt_length: 117
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_1688.png
problem_id: 96433
prompt_length: 291
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/PlotQA/images/96001.png
problem_id: 97093
prompt_length: 296
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00004548.png
problem_id: 220595
prompt_length: 299path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_4458.png
problem_id: 166900
prompt_length:
 70
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_yrf93aLQXBE.mp4
problem_id: 14347
prompt_length: 1067
Invalidate trace cache @ step 0 and module 31962: cache has only 0 modules
tensor([0.9866, 1.1803, 0.2770, 0.9250, 1.1315, 1.1146, 1.0945, 1.0676],
       device='cuda:0')
tensor([0.2756, 0.2814, 0.2656, 0.2522, 0.2303, 0.2169, 0.1949, 0.1750],
       device='cuda:5')
tensor([0.2045, 0.2638, 0.2501, 1.2340, 0.2133, 0.1991, 0.1779, 0.1583],
       device='cuda:4')
tensor([0.1591, 0.2076, 0.1926, 0.1795, 0.1621, 0.1529, 0.1334, 1.1160],
       device='cuda:2')
tensor([0.2929, 0.3288, 0.3132, 0.2931, 0.3513, 0.3658, 0.2264, 0.2005],
       device='cuda:3')
tensor([0.4557, 0.2179, 0.2020, 0.4730, 0.1735, 0.1546, 0.5689, 0.1189],
       device='cuda:1')
tensor([0.4782, 0.4510, 0.4344, 0.4160, 0.4356, 0.3979, 0.4365, 0.3412],
       device='cuda:7')
tensor([0.3441, 0.4071, 0.3848, 0.3707, 0.3447, 0.3215, 0.2953, 0.2650],
       device='cuda:6')
{'completion_length': 37.0, 'rewards/accuracy_reward': 0.16305407881736755, 'rewards/efficiency reward': 0.2359073907136917, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.39896148443222046, 'reward_std': 0.16878637671470642, 'kl': 76.5, 'epoch': 0.0}
{'loss': 3.0556, 'grad_norm': 361.68873005563904, 'learning_rate': 9.99996349184135e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/1fd447aaf2f71a4566264fae6ae7b830.png
problem_id: 199655
prompt_length: 86
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/127/EjHxMfOdl1k/split_0.mp4
problem_id: 134335
prompt_length: 1068
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC4024300_RSPB20140432F2_289625.jpg
problem_id: 132914
prompt_length: 176
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_p-l6as8o1f4.mp4
problem_id: 198632
prompt_length: 1069
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/Q7IQI.mp4
problem_id: 131613
prompt_length: 1009
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_7302.png
problem_id: 246003
prompt_length: 320
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/RoBUT_SQA/cauldron/robut_sqa/images/robut_sqa_00007051.png
problem_id: 83421
prompt_length: 283
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/MOQIU.mp4
problem_id: 6107
prompt_length: 1054
Invalidate trace cache @ step 0 and module 32723: cache has only 0 modules
tensor([1.2287, 1.2571, 1.2411, 0.2283, 1.2072, 1.1893, 1.1745, 1.1526],
       device='cuda:5')
tensor([0.2718, 0.3278, 0.3134, 0.2901, 0.2721, 0.2507, 0.2290, 0.2028],
       device='cuda:0')
Error converting '31.4<' to float: could not convert string to float: '31.4<'
tensor([0.2015, 0.2213, 0.2085, 0.1944, 0.1790, 0.1603, 0.1424, 0.1254],
       device='cuda:4')
tensor([0.3839, 0.2337, 0.2942, 0.3717, 0.3532, 0.2944, 0.1571, 0.1384],
       device='cuda:1')
tensor([0.3268, 0.4074, 0.3877, 0.3620, 0.3429, 0.3201, 1.2942, 1.2615],
       device='cuda:3')
tensor([0.4025, 0.4948, 0.3176, 0.3409, 0.2776, 0.5466, 0.3845, 0.5136],
       device='cuda:2')
tensor([1.2974, 1.3952, 1.3773, 1.3559, 1.3362, 1.3094, 1.2870, 1.2575],
       device='cuda:7')
tensor([0.3435, 0.4028, 0.3795, 0.3603, 0.3428, 0.3181, 0.2895, 0.2606],
       device='cuda:6')
{'completion_length': 68.96875, 'rewards/accuracy_reward': 0.2940327525138855, 'rewards/efficiency reward': 0.2651807367801666, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.5592135190963745, 'reward_std': 0.14215409755706787, 'kl': 45.0, 'epoch': 0.0}
{'loss': 1.7952, 'grad_norm': 152.19632745844953, 'learning_rate': 9.99996164361818e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/Rendered_Text/cauldron/rendered_text/images/rendered_text_00008860.png
problem_id: 108579
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/225/0rkJ9RFLaIE/split_4.mp4
problem_id: 151337
prompt_length: 1078
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/ChartQA/images/8310.png
problem_id: 128167
prompt_length: 323
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_26714.png
problem_id: 100441
prompt_length: 320
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_48324.png
problem_id: 146727
prompt_length: 68
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/28146.png
problem_id: 44962
prompt_length: 248
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/YE47Y.mp4
problem_id: 202562
prompt_length: 1128
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/EXAMS-V/1c9f7c96-2ce3-4032-a99e-50b02ad5ed6b.png
problem_id: 147087
prompt_length: 420
Invalidate trace cache @ step 0 and module 33484: cache has only 0 modules
tensor([0.2550, 0.2146, 1.2020, 1.1861, 1.1751, 1.1622, 1.1447, 1.1247],
       device='cuda:6')
tensor([1.2652, 1.2606, 1.2402, 0.2240, 1.2129, 1.1943, 1.1760, 1.1591],
       device='cuda:3')
tensor([0.2551, 1.2362, 0.2282, 1.2136, 1.1979, 1.1826, 1.1663, 0.1518],
       device='cuda:2')
tensor([1.2908, 1.3493, 1.3302, 1.3108, 1.2908, 1.2652, 1.2420, 1.2181],
       device='cuda:4')
tensor([0.2224, 0.2285, 0.2122, 0.3991, 0.3807, 0.1633, 0.5429, 0.9245],
       device='cuda:0')
tensor([1.2603, 1.3475, 1.3323, 1.3118, 1.2908, 1.2700, 0.2453, 1.2204],
       device='cuda:5')
tensor([1.3365, 1.3970, 1.3769, 1.3589, 1.3377, 1.3141, 1.2907, 1.2622],
       device='cuda:7')
tensor([0.3323, 0.3989, 0.3788, 1.3622, 1.3392, 1.3152, 1.2884, 1.2621],
       device='cuda:1')
{'completion_length': 101.265625, 'rewards/accuracy_reward': 0.7437499761581421, 'rewards/efficiency reward': 0.2535684108734131, 'all_wrong': 0.125, 'all_correct': 0.0, 'reward': 0.9973183870315552, 'reward_std': 0.3123862147331238, 'kl': 9.0, 'epoch': 0.0}
{'loss': 0.361, 'grad_norm': 172.94498651787353, 'learning_rate': 9.999959749760109e-06, 'epoch': 0.0}
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Nj_rPQwzllA.mp4 
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/cde6e12a31c375d744e08a66ebb159c8.pngproblem_id: 
56931
problem_id: prompt_length: 1970001068

prompt_length: 177
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00005535.png
problem_id: 76508
prompt_length: 287
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_1499.png
problem_id: 245283
prompt_length: 232
 path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_43j1BAyP6Q0.mp4
problem_id: 184351
prompt_length: 1128
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_tA2G2NZ_0vw.mp4
problem_id: 135542
prompt_length: 1064
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/IconQA/images/49143.png
problem_id: 127730
prompt_length: 125
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_wcAVS9bI6kM.mp4
problem_id: 143600
prompt_length: 1054
Invalidate trace cache @ step 0 and module 34245: cache has only 0 modules
Error converting 'C' to float: could not convert string to float: 'C'
Error converting 'C' to float: could not convert string to float: 'C'
Error converting 'C' to float: could not convert string to float: 'C'
tensor([0.2443, 0.7603, 0.7469, 0.2316, 0.7091, 0.1928, 0.6706, 0.6500],
       device='cuda:5')
Error converting 'Certain' to float: could not convert string to float: 'Certain'
Error converting 'C' to float: could not convert string to float: 'C'
tensor([1.2351, 1.2014, 1.1891, 1.1803, 1.1614, 1.1494, 0.1363, 1.1194],
       device='cuda:1')
tensor([0.2343, 0.1949, 1.1826, 0.1689, 0.1554, 0.1416, 0.1279, 0.1144],
       device='cuda:7')
tensor([0.2449, 0.3103, 0.2876, 0.2716, 0.2468, 0.2274, 0.2041, 0.1809],
       device='cuda:3')
tensor([1.3340, 1.4118, 1.3947, 1.3745, 1.3482, 1.3227, 1.2979, 0.2685],
       device='cuda:4')
tensor([1.2750, 1.3786, 1.3595, 1.3376, 1.3152, 1.2978, 1.2682, 1.2440],
       device='cuda:0')
tensor([1.2584, 0.3722, 0.3520, 0.3332, 0.3102, 0.2885, 0.2647, 0.2361],
       device='cuda:6')
tensor([0.2470, 0.3172, 0.3030, 0.2898, 0.2706, 0.2492, 0.2301, 0.2071],
       device='cuda:2')
{'completion_length': 310.625, 'rewards/accuracy_reward': 0.4140625, 'rewards/efficiency reward': 0.2520151734352112, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.6660776734352112, 'reward_std': 0.2297777533531189, 'kl': 15.25, 'epoch': 0.0}
{'loss': 0.6114, 'grad_norm': 255.62411250314216, 'learning_rate': 9.999957810267152e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/abc_question_images/human_system_digestive_13720.png
problem_id: 150562
prompt_length: 332
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/261dfb69-9db2-4e97-829a-eebb285c5f77.png
problem_id: 97226
prompt_length: 171
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_6oWl1IpbX8c.mp4
problem_id: 119160
prompt_length: 1067
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/780dca1f299b96ed3f0237919747c3da.png
problem_id: 69639
prompt_length: 106
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/EXAMS-V/37d486ec-9685-453f-bda9-0d010acb8e1c.png
problem_id: 144224
prompt_length: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Hi0L9rcsXUI.mp4
problem_id: 154895
prompt_length: 1088
415
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_0N8iIUS660o.mkv
problem_id: 69471
prompt_length: 1058
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00001031.png
problem_id: 7504
prompt_length: 301
Invalidate trace cache @ step 0 and module 35006: cache has only 0 modules
Error converting 'R =42' to float: could not convert string to float: 'R =42'
Error converting 'C^{2}' to float: could not convert string to float: 'C^{2}'
tensor([1.2392, 1.2568, 1.2412, 1.2242, 1.2133, 1.1950, 1.1750, 1.1578],
       device='cuda:0')
tensor([0.2395, 0.2162, 0.2007, 0.1850, 0.1756, 0.1606, 0.1433, 0.1259],
       device='cuda:3')
Error converting 'C*0' to float: could not convert string to float: 'C*0'
Error converting 'C' to float: could not convert string to float: 'C'
Error converting '' to float: could not convert string to float: ''
tensor([0.1928, 0.2045, 0.1934, 0.1779, 0.1634, 0.1476, 0.1324, 0.1112],
       device='cuda:5')
tensor([0.8547, 1.1054, 1.1133, 1.0754, 1.0582, 0.9508, 1.0238, 1.0289],
       device='cuda:4')
tensor([0.2050, 0.2664, 0.2528, 0.2387, 0.2176, 1.2069, 1.1806, 0.1630],
       device='cuda:7')
tensor([0.2582, 0.6441, 0.3108, 0.6081, 0.6362, 0.2502, 0.2306, 0.5192],
       device='cuda:1')
tensor([1.2734, 0.3532, 0.3319, 0.3145, 0.2939, 1.2762, 0.2500, 0.2233],
       device='cuda:2')
tensor([0.3113, 0.4038, 0.3890, 0.3651, 1.3423, 1.3136, 1.2911, 1.2559],
       device='cuda:6')
{'completion_length': 122.40625, 'rewards/accuracy_reward': 0.37383633852005005, 'rewards/efficiency reward': 0.2333534210920334, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.6071897745132446, 'reward_std': 0.22304406762123108, 'kl': 53.0, 'epoch': 0.0}
{'loss': 2.1245, 'grad_norm': 271.2590654099615, 'learning_rate': 9.999955825139322e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1008/8714747002.mp4
problem_id: 113609
prompt_length: 1058
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_19879.png
problem_id: 20181
prompt_length: 330
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/2838ea725835d5a01f456bea47decc32.png
problem_id: 40248
prompt_length: 129
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/RUKXT.mp4
problem_id: 28196
prompt_length: 1057
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/fb39aa40-08dd-4858-a51e-2e3b139db304.png
problem_id: 255363
prompt_length: 296path:
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1009/4507436121.mp4
problem_id: 218080
prompt_length: 1097
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/RoBUT_SQA/cauldron/robut_sqa/images/robut_sqa_00002749.png
problem_id: 166389
prompt_length: 283
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_kbe4iowYMqM.mp4
problem_id: 187500
prompt_length: 1047
Invalidate trace cache @ step 0 and module 35767: cache has only 0 modules
Error converting 'C90.00' to float: could not convert string to float: 'C90.00'
Error converting 'C2.54' to float: could not convert string to float: 'C2.54'
Error converting 'C' to float: could not convert string to float: 'C'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.1971, 0.5036, 0.2060, 0.1928, 0.1783, 0.3795, 0.3562, 0.3529],
       device='cuda:0')tensor([0.2061, 0.1919, 0.1827, 0.1645, 0.1531, 0.1340, 0.1218, 0.1096],
       device='cuda:2')

tensor([0.5562, 1.1009, 0.5144, 0.3377, 0.7813, 0.2654, 0.4236, 0.9696],
       device='cuda:3')
tensor([0.3107, 0.2214, 0.2070, 0.1926, 0.2430, 0.1862, 0.1668, 0.8711],
       device='cuda:6')
tensor([0.3320, 0.3860, 0.3657, 1.3475, 1.3224, 0.2984, 0.2721, 1.2451],
       device='cuda:4')
tensor([1.3342, 1.4073, 1.3874, 1.3651, 0.3460, 1.3178, 1.2931, 1.2658],
       device='cuda:7')
tensor([1.2894, 0.3971, 0.3783, 1.3596, 1.3407, 0.3131, 1.2861, 1.2585],
       device='cuda:5')
tensor([0.3300, 0.4109, 0.3896, 0.3656, 0.3454, 0.3157, 0.2947, 0.2592],
       device='cuda:1')
{'completion_length': 55.59375, 'rewards/accuracy_reward': 0.3057153522968292, 'rewards/efficiency reward': 0.2661168873310089, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.5718322396278381, 'reward_std': 0.2613951563835144, 'kl': 3.421875, 'epoch': 0.0}
{'loss': 0.1368, 'grad_norm': 68.04539089748025, 'learning_rate': 9.999953794376645e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC5833877_fig2_282964.jpg
problem_id: 204219
prompt_length: 191
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/6034.png
problem_id: 104051
prompt_length: 107
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1907.09831_0.jpg
problem_id: 66496
prompt_length: 367
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1122/2573200479.mp4
problem_id: 239499
prompt_length: 1089
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/fe7ba88b-db78-4f89-8d19-62f52df888ef.png
problem_id: 32668
prompt_length: 290
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/cycle_rock_6728.png
problem_id: 160552
prompt_length: 341
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/FigureQA/images/36147.png
problem_id: 21651
prompt_length: 312
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_yMemK20wae4.mp4
problem_id: 69042
prompt_length: 1118
Invalidate trace cache @ step 0 and module 36528: cache has only 0 modules
tensor([1.2320, 0.2360, 1.2193, 1.2076, 1.1886, 1.1706, 1.1552, 1.1379],
       device='cuda:3')
tensor([0.2848, 0.3009, 0.2804, 0.2600, 0.2420, 0.2190, 0.1989, 1.1783],
       device='cuda:4')
tensor([1.3076, 0.2711, 0.2551, 1.2389, 1.2214, 0.2081, 1.1862, 0.1656],
       device='cuda:1')
tensor([0.2802, 0.2792, 0.2586, 0.2435, 0.2248, 0.2059, 1.1900, 1.1692],
       device='cuda:7')
tensor([0.1758, 0.2276, 0.2164, 0.2021, 0.1852, 0.1688, 0.1557, 0.1388],
       device='cuda:2')
tensor([0.2183, 0.2768, 0.5592, 0.2459, 0.2280, 0.2094, 0.3371, 0.1680],
       device='cuda:0')
tensor([0.2820, 0.3914, 0.3671, 0.3516, 0.3298, 0.3034, 0.2768, 0.2541],
       device='cuda:6')
tensor([0.3076, 0.3311, 0.3160, 0.2935, 0.4925, 0.4799, 0.2486, 0.3691],
       device='cuda:5')
{'completion_length': 42.78125, 'rewards/accuracy_reward': 0.23572419583797455, 'rewards/efficiency reward': 0.23809762299060822, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.47382181882858276, 'reward_std': 0.24142012000083923, 'kl': 3.578125, 'epoch': 0.0}
{'loss': 0.1427, 'grad_norm': 51.18010850595122, 'learning_rate': 9.999951717979135e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/0601195b8bb8557acc887fa0e8b6d7e4.png
problem_id: 36420
prompt_length: 148
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_8qzR38WgsS8.mp4
problem_id: 145963
prompt_length: 1052
path: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/1462.png
problem_id: 150140
prompt_length: 335
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1507.07843_1.jpg
problem_id: 156303
prompt_length:path:  /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_07794.mp4
problem_id: 59906
prompt_length: 761
356
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/3918f4d0-63f4-471e-bdeb-84529d9b777b.png
problem_id: 125946
prompt_length: 249
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/a3507cf5-14b4-43cd-989d-9ffd2b856497.png
problem_id: 127773
prompt_length: 63
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/Chart2Text/cauldron/chart2text/images/chart2text_00001613.png
problem_id: 58006
prompt_length: 297
Invalidate trace cache @ step 0 and module 37289: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([0.2370, 0.4605, 0.3216, 0.3285, 0.1644, 0.1502, 0.1315, 0.3266],
       device='cuda:6')
Error converting '' to float: could not convert string to float: ''
tensor([1.2656, 1.2649, 0.2524, 1.2370, 1.2185, 1.2026, 1.1821, 1.1628],
       device='cuda:0')
tensor([0.2233, 0.2675, 0.4080, 0.2083, 0.2578, 0.2952, 0.1718, 0.1384],
       device='cuda:1')
tensor([1.2169, 1.2830, 1.2693, 1.2496, 1.2324, 0.2120, 1.1907, 1.1736],
       device='cuda:3')
tensor([0.2280, 0.4209, 0.2436, 0.3139, 0.2789, 0.3021, 0.2786, 0.2547],
       device='cuda:4')
Error converting '' to float: could not convert string to float: ''
tensor([0.1900, 0.1969, 0.1844, 0.1694, 0.1595, 0.1438, 0.1272, 0.1132],
       device='cuda:7')
tensor([0.2626, 0.3698, 0.3547, 0.3294, 0.3118, 0.2918, 0.2668, 0.2396],
       device='cuda:2')
tensor([0.3447, 0.4066, 1.3867, 1.3644, 0.3436, 0.3211, 1.2904, 1.2610],
       device='cuda:5')
{'completion_length': 32.609375, 'rewards/accuracy_reward': 0.3093910813331604, 'rewards/efficiency reward': 0.22896146774291992, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.5383525490760803, 'reward_std': 0.19515541195869446, 'kl': 2.28125, 'epoch': 0.0}
{'loss': 0.0914, 'grad_norm': 13.150566934203942, 'learning_rate': 9.999949595946811e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/319/_XxXWiOoyhY/split_10.mp4
problem_id: 65682
prompt_length: 1006
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_5137.png
problem_id: 111118
prompt_length: 330
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/GVLQA/data/matching/graph_visual/layout/sfdp/easy/graph86.png
problem_id: 98026
prompt_length: 409
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_5kWQSLwDYwM.mp4path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_vpu4Bwq-xHI.mp4
problem_id: 176488
prompt_length: 1135

problem_id: 251142
prompt_length: 1043
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/MWUIN.mp4
problem_id: 72959
prompt_length: 1047
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/AT9UV.mp4
problem_id: 70384
prompt_length: 1116
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_g9bzrGBzSC4.mp4
problem_id: 198244
prompt_length: 1073
Invalidate trace cache @ step 0 and module 38050: cache has only 0 modules
tensor([0.2634, 0.2758, 0.2611, 0.2441, 0.2251, 0.2109, 0.1921, 0.1714],
       device='cuda:2')
Error converting '' to float: could not convert string to float: ''
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2816, 0.3752, 0.3597, 0.3392, 0.3175, 0.2947, 0.5213, 0.2405],
       device='cuda:3')
tensor([1.2837, 1.3822, 0.3633, 0.3421, 1.3233, 1.3033, 1.2810, 1.2513],
       device='cuda:0')
tensor([1.2866, 1.3954, 1.3799, 0.3639, 0.3357, 1.3124, 1.2914, 0.2608],
       device='cuda:1')
tensor([0.3435, 0.4045, 0.3927, 0.3664, 0.3464, 0.3227, 0.2991, 0.2638],
       device='cuda:5')
tensor([1.2564, 1.3958, 1.3794, 1.3592, 1.3353, 1.3117, 1.2879, 1.2624],
       device='cuda:6')
tensor([0.2894, 0.3943, 0.3740, 0.3566, 0.3329, 0.3050, 0.2813, 0.2537],
       device='cuda:4')
tensor([0.5608, 0.3201, 0.5244, 0.2811, 0.5140, 0.4742, 0.2183, 0.4052],
       device='cuda:7')
{'completion_length': 31.078125, 'rewards/accuracy_reward': 0.31955310702323914, 'rewards/efficiency reward': 0.30455225706100464, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6241053938865662, 'reward_std': 0.17072314023971558, 'kl': 2.859375, 'epoch': 0.0}
{'loss': 0.1144, 'grad_norm': 27.994659448002942, 'learning_rate': 9.999947428279693e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_MpqXCbsqVNQ.mp4
problem_id: 259840
prompt_length: 1042
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/303/LaouC0RQoDg/split_6.mp4
problem_id: 113361
prompt_length: 887
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_HgOHqD0lWTE.mp4
problem_id: 219229
prompt_length: 1134path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_10045.mp4
problem_id: 169388
prompt_length: 1067

Invalidate trace cache @ step 0 and module 38811: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/cc9b88920d2a50fb.jpg
problem_id: 158539
prompt_length: 296
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/10341.png
problem_id: 130743
prompt_length: 324
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/5766.png
problem_id: 196935
prompt_length: 181
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/392bc43bcbe29356.png
problem_id: 23735
prompt_length: 303
tensor([1.2448, 1.2119, 1.1985, 1.1855, 1.1699, 1.1537, 1.1411, 1.1247],
       device='cuda:6')
tensor([1.3068, 1.3359, 1.3170, 1.2996, 1.2816, 1.2606, 1.2395, 1.2084],
       device='cuda:1')
tensor([0.3860, 0.4845, 0.3909, 0.2968, 0.4350, 0.4135, 0.2355, 0.2140],
       device='cuda:5')
tensor([0.3071, 0.3172, 0.3018, 0.2787, 0.2552, 0.2373, 0.2134, 0.1886],
       device='cuda:3')
tensor([0.4298, 0.5790, 0.4555, 0.2839, 0.3881, 0.3837, 0.6161, 0.2939],
       device='cuda:2')
tensor([0.3373, 0.4032, 0.3851, 0.3646, 0.3408, 1.3184, 0.2963, 1.2649],
       device='cuda:4')
tensor([0.3342, 0.3927, 0.3732, 0.3492, 0.3276, 0.3044, 0.2756, 0.2495],
       device='cuda:7')
tensor([1.3414, 1.4028, 1.3812, 1.3593, 1.3381, 1.3093, 1.2817, 0.2537],
       device='cuda:0')
{'completion_length': 11.9375, 'rewards/accuracy_reward': 0.42099177837371826, 'rewards/efficiency reward': 0.2829124331474304, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.7039042711257935, 'reward_std': 0.1521884799003601, 'kl': 2.0, 'epoch': 0.0}
{'loss': 0.08, 'grad_norm': 34.03998129323202, 'learning_rate': 9.999945214977803e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_UFVeN-ThOwU.mp4
problem_id: 1575
prompt_length: 1091
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_fs8yU4pBNm4.mp4
problem_id: 227052
prompt_length: 1105
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_2250.mp4
problem_id: 208906
prompt_length: 1056
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1411.5173_2.jpg
problem_id: 118687
prompt_length: 365
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_bFRqNqyVznA.mp4
problem_id: 247833
prompt_length: 1051
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/3b237e2a-bdda-4e65-b001-332637bff0d1.png
problem_id: 187064
prompt_length: 139
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_18498.png
problem_id: 224214
prompt_length: 308
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/dd39725db6e92fc7.jpg
problem_id: 182567
prompt_length: 302
Invalidate trace cache @ step 0 and module 39572: cache has only 0 modules
tensor([0.2653, 0.3018, 1.2855, 0.2632, 0.2460, 0.2247, 0.2007, 0.1753],
       device='cuda:1')
tensor([0.2156, 0.2610, 0.2469, 0.2278, 0.2140, 0.1701, 0.1798, 0.1613],
       device='cuda:6')
tensor([1.2495, 1.2581, 1.2435, 1.2309, 1.2112, 1.1969, 1.1756, 1.1566],
       device='cuda:5')
tensor([1.3187, 1.3721, 1.3531, 0.3370, 1.3188, 1.2944, 1.2679, 1.2454],
       device='cuda:2')
tensor([1.3488, 1.4214, 1.4019, 1.3787, 1.3545, 1.3336, 1.3014, 1.2728],
       device='cuda:0')
tensor([1.3167, 1.4143, 1.3965, 1.3710, 1.3488, 1.3255, 1.2915, 1.2622],
       device='cuda:3')
tensor([0.3374, 0.4064, 0.3873, 0.3634, 0.3433, 0.3171, 0.2905, 0.2630],
       device='cuda:7')
tensor([0.6126, 0.5293, 0.6124, 0.5739, 0.2812, 0.5139, 0.4546, 0.2094],
       device='cuda:4')
{'completion_length': 56.96875, 'rewards/accuracy_reward': 0.5268242955207825, 'rewards/efficiency reward': 0.2841717302799225, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.8109960556030273, 'reward_std': 0.13654103875160217, 'kl': 9.375, 'epoch': 0.0}
{'loss': 0.3735, 'grad_norm': 39.90216612528558, 'learning_rate': 9.999942956041158e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/a118e3c6-27cb-4dd2-b3f2-61b0fc6045c0.png
problem_id: 108600
prompt_length: 197
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/613ccdb6-b994-4b8c-af76-e61521bb4947.png
problem_id: 131311
prompt_length: 297
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_02984.mp4
problem_id: 102287
prompt_length: 693
 path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1904.11963_2.jpg
problem_id: 1372
prompt_length: 367
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_10643.png
problem_id: 27080
prompt_length: 314
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1023/3438416167.mp4
problem_id: 152992
prompt_length: 1145
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_10647.png
problem_id: 66089
prompt_length: 293
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/Chart2Text/cauldron/chart2text/images/chart2text_00005006.png
problem_id: 54797
prompt_length: 295
Invalidate trace cache @ step 0 and module 40333: cache has only 0 modules
tensor([0.6904, 0.8837, 0.8638, 0.8854, 0.2555, 0.8419, 0.2746, 0.4344],
       device='cuda:0')
tensor([0.2110, 0.2530, 0.2399, 0.2234, 0.2077, 0.1935, 0.1724, 0.1531],
       device='cuda:5')
tensor([0.2606, 0.2568, 0.2555, 0.2043, 0.1930, 0.1717, 0.1598, 0.1358],
       device='cuda:7')
tensor([0.2855, 0.3669, 0.3498, 0.3256, 0.3095, 0.2879, 0.2576, 0.2243],
       device='cuda:4')
tensor([0.4668, 0.5480, 0.4365, 0.5369, 0.3910, 0.4031, 0.4643, 0.4475],
       device='cuda:1')
tensor([0.3792, 0.5306, 0.5567, 0.5502, 0.4427, 0.1944, 0.3414, 0.4390],
       device='cuda:3')
tensor([1.3215, 1.3768, 1.3595, 1.3418, 1.3168, 1.2954, 1.2712, 1.2464],
       device='cuda:2')
tensor([1.2891, 1.4195, 1.3983, 0.3768, 1.3514, 1.3275, 1.2959, 1.2690],
       device='cuda:6')
{'completion_length': 50.265625, 'rewards/accuracy_reward': 0.33865809440612793, 'rewards/efficiency reward': 0.2584218978881836, 'all_wrong': 0.75, 'all_correct': 0.0, 'reward': 0.5970799922943115, 'reward_std': 0.12163177132606506, 'kl': 1.640625, 'epoch': 0.0}
{'loss': 0.0655, 'grad_norm': 63.739177190367016, 'learning_rate': 9.999940651469779e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/PDA4B.mp4
problem_id: 56890
prompt_length: 1078
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_5168.mp4
problem_id: 232298
prompt_length: 1097
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2007.04885_0.jpg
problem_id: 254591
prompt_length: 395
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/4873.png
problem_id: 106773
prompt_length: 324
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/352.png
problem_id: 151025
prompt_length: 108
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_z8QpIwv0PqI.mp4
problem_id: 202063
prompt_length: 1066path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_qlHpGwHLqXA.mp4
path:problem_id: 26679
 prompt_length:/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/XNGAV.mp4 
1058
problem_id: 191863
prompt_length: 1055

Invalidate trace cache @ step 0 and module 41094: cache has only 0 modules
tensor([1.2124, 0.2533, 1.2418, 1.2239, 1.2103, 1.1909, 1.1781, 1.1589],
       device='cuda:5')
tensor([0.2568, 0.2341, 1.2176, 0.2027, 1.1882, 1.1700, 1.1564, 1.1356],
       device='cuda:2')
tensor([1.2164, 0.2610, 1.2494, 1.2341, 1.2184, 1.1996, 1.1828, 1.1623],
       device='cuda:3')
tensor([1.3295, 1.4121, 1.3907, 1.3738, 1.3517, 1.3283, 1.3049, 1.2751],
       device='cuda:4')
tensor([1.3086, 1.4148, 1.4004, 1.3756, 1.3546, 1.3307, 1.3053, 1.2730],
       device='cuda:1')
tensor([1.2773, 1.3913, 1.3733, 1.3508, 1.3345, 1.3080, 1.2858, 1.2585],
       device='cuda:7')
tensor([1.3008, 0.3767, 0.3559, 0.3355, 0.3195, 0.2964, 0.2718, 0.2425],
       device='cuda:0')
tensor([1.3479, 1.4118, 1.3933, 1.3756, 1.3556, 1.3313, 1.3060, 0.2779],
       device='cuda:6')
{'completion_length': 7.765625, 'rewards/accuracy_reward': 0.796875, 'rewards/efficiency reward': 0.28690052032470703, 'all_wrong': 0.0, 'all_correct': 0.0, 'reward': 1.083775520324707, 'reward_std': 0.25499677658081055, 'kl': 2.265625, 'epoch': 0.0}
{'loss': 0.0907, 'grad_norm': 20.971239083727088, 'learning_rate': 9.99993830126369e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1703.00168_0.jpg
problem_id: 193048
prompt_length: 375
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/53230dc1ef1df3b0.jpg
problem_id: 177149
prompt_length: 317
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/NextQA/1016/11049178966.mp4
problem_id: 110564
prompt_length: 1063
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/W7EHF.mp4
path:problem_id:  229471/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1203/5705939237.mp4

prompt_length: problem_id:1052 
154131
prompt_length: 1124
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ScienceQA/images/4593.png
problem_id: 73029
prompt_length: 276
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_5d08rS9wOeI.mp4
problem_id: 46515
prompt_length: 1073
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/RoBUT_SQA/cauldron/robut_sqa/images/robut_sqa_00003648.png
problem_id: 13474
prompt_length: 291
Invalidate trace cache @ step 0 and module 41855: cache has only 0 modules
tensor([1.2554, 0.2500, 0.2371, 0.2246, 0.2056, 0.1915, 0.1744, 0.1554],
       device='cuda:0')
tensor([0.3283, 0.3813, 0.3606, 1.3432, 0.3255, 0.3001, 0.2763, 1.2480],
       device='cuda:7')
tensor([1.2744, 1.2567, 1.2419, 1.2237, 1.2110, 0.1938, 1.1770, 1.1552],
       device='cuda:6')
tensor([0.7217, 0.7829, 0.5591, 0.4910, 0.9761, 0.3821, 0.3246, 0.6425],
       device='cuda:1')
tensor([1.3375, 1.3769, 1.3622, 1.3374, 1.3214, 1.2980, 0.2727, 0.2465],
       device='cuda:4')
tensor([0.2939, 0.4045, 0.3827, 0.3606, 0.3364, 0.3153, 0.2915, 0.2611],
       device='cuda:3')
tensor([1.2833, 1.4092, 1.3876, 1.3701, 1.3477, 1.3241, 1.2938, 1.2636],
       device='cuda:2')
tensor([1.2859, 1.3966, 1.3771, 1.3555, 1.3354, 1.3091, 1.2816, 0.2544],
       device='cuda:5')
{'completion_length': 42.734375, 'rewards/accuracy_reward': 0.5356736779212952, 'rewards/efficiency reward': 0.2822122573852539, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.8178859949111938, 'reward_std': 0.29867786169052124, 'kl': 8.75, 'epoch': 0.0}
{'loss': 0.3496, 'grad_norm': 323.8380271442433, 'learning_rate': 9.999935905422909e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/419/Kbrkj7wZ4UA/split_0.mp4
problem_id: 202726
prompt_length: 1049
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC6627094_ijms-20-03047-f007_7646.jpg
problem_id: 245170
prompt_length: 299
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_06038.mp4
problem_id: 256971
prompt_length: 733
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_06r6DtoTtSQ.mp4
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/19bf0fc9d801f6c215b2b9b28c6c1502.png
problem_id: 146755
prompt_length: 130
problem_id: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_6599.png
problem_id: 195331
prompt_length: 303
79390
prompt_length: 1056
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/3d17f0bc83b39d91.png
problem_id: 84601
prompt_length: 312
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D-gpt4v/ai2d_azuregpt_2719.png
problem_id: 180895
prompt_length: 354
Invalidate trace cache @ step 0 and module 42616: cache has only 0 modules
tensor([0.2741, 0.2992, 0.2863, 0.2667, 0.2466, 0.2283, 0.2065, 0.1869],
       device='cuda:1')
tensor([1.2850, 1.3059, 1.2919, 1.2707, 1.2536, 1.2336, 0.2131, 1.1914],
       device='cuda:7')
tensor([0.3106, 0.3052, 0.2889, 0.2715, 0.2525, 0.2268, 0.2079, 0.1851],
       device='cuda:5')
tensor([0.2252, 0.2033, 0.1918, 0.1742, 0.1624, 0.1484, 0.1332, 0.1203],
       device='cuda:2')
tensor([0.5810, 0.5569, 0.5820, 0.4339, 0.5601, 0.2700, 0.5034, 0.5114],
       device='cuda:4')
tensor([1.3297, 1.3776, 1.3590, 1.3426, 1.3160, 1.2976, 1.2721, 1.2454],
       device='cuda:3')
tensor([1.2914, 1.4077, 0.3897, 1.3699, 1.3473, 1.3247, 0.2989, 1.2725],
       device='cuda:6')
tensor([1.2912, 1.4010, 1.3824, 1.3628, 1.3390, 1.3169, 1.2892, 1.2584],
       device='cuda:0')
{'completion_length': 35.515625, 'rewards/accuracy_reward': 0.4865707755088806, 'rewards/efficiency reward': 0.26856446266174316, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.755135178565979, 'reward_std': 0.14439022541046143, 'kl': 1.6328125, 'epoch': 0.0}
{'loss': 0.0654, 'grad_norm': 9.939699417098142, 'learning_rate': 9.99993346394746e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_xtY-SzWbCvE.mp4
problem_id: 163668
prompt_length: 1080
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/1509.09190_0.jpg
problem_id: 162901
prompt_length: 333
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/402cf59890cc3fce528959ddfdc3a7cd.png
problem_id: 255994
prompt_length: 101
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_kuPWb9E4aUQ.mp4
problem_id: 29334
prompt_length: 1065
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_xE9anOqq2XQ.mp4
problem_id: 157387
prompt_length: 1072
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/JXKPA.mp4
problem_id:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_bnQVFmXUx_U.mp4
problem_id: 248567
prompt_length: 1058
 92539
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_gCxLSh-cgng.mp4
problem_id: 244171
prompt_length: 1070
prompt_length: 1127
Invalidate trace cache @ step 0 and module 43377: cache has only 0 modules
tensor([0.2422, 0.2772, 0.2636, 0.2472, 0.2324, 0.2173, 0.1920, 0.1703],
       device='cuda:7')
tensor([0.1724, 0.2001, 0.1874, 0.1715, 0.1602, 0.1432, 0.1259, 0.1096],
       device='cuda:2')
tensor([1.2798, 1.3826, 1.3634, 1.3483, 1.3245, 1.3028, 1.2796, 1.2565],
       device='cuda:4')
tensor([1.3075, 1.3947, 1.3707, 1.3532, 1.3363, 1.3044, 1.2820, 1.2522],
       device='cuda:5')
tensor([1.3282, 1.3602, 1.3410, 1.3278, 0.3079, 1.2893, 1.2663, 1.2417],
       device='cuda:6')
tensor([1.3324, 1.3998, 1.3807, 1.3569, 1.3367, 1.3166, 1.2904, 1.2616],
       device='cuda:3')
tensor([1.2987, 1.3758, 1.3577, 1.3362, 1.3179, 1.2949, 1.2706, 1.2474],
       device='cuda:1')
tensor([1.3397, 1.3928, 1.3751, 0.3570, 0.3362, 0.3125, 0.2890, 0.2597],
       device='cuda:0')
{'completion_length': 31.296875, 'rewards/accuracy_reward': 0.65625, 'rewards/efficiency reward': 0.28983908891677856, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.9460891485214233, 'reward_std': 0.1440202295780182, 'kl': 2.296875, 'epoch': 0.0}
{'loss': 0.0917, 'grad_norm': 66.54645682458965, 'learning_rate': 9.999930976837363e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2011.14389_2.jpg
problem_id: 240247
prompt_length: 346
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/f33f9647efaee1169fc59d84bb6adc89.png
problem_id: 118714
prompt_length: 206
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/12dede2dfd0464f074dcc20e6420c521.png
problem_id: 243929
prompt_length: 105
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/31009.pngpath: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v1-3/train_val/v_0bzSBV3jHIY.mp4
problem_id: 70194
prompt_length: 1054

problem_id:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_9428.png
problem_id: 61697
prompt_length: 315
 40008
prompt_length: 86
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/346d22122174e04c6936ea4dcba4017c.png
problem_id: 26071
prompt_length: 161
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_8478.png
problem_id: 255410
prompt_length: 317
Invalidate trace cache @ step 0 and module 44138: cache has only 0 modules
tensor([0.2251, 0.1879, 1.1768, 1.1668, 1.1528, 1.1402, 1.1238, 1.1080],
       device='cuda:1')
tensor([0.2056, 0.2282, 0.2162, 0.2028, 0.1871, 0.1697, 0.1546, 0.1331],
       device='cuda:7')
tensor([1.2715, 1.2855, 1.2673, 1.2511, 1.2338, 1.2174, 1.1997, 1.1793],
       device='cuda:6')
tensor([1.2375, 1.1985, 1.1894, 1.1752, 1.1620, 1.1474, 1.1296, 1.1109],
       device='cuda:0')
tensor([0.3453, 0.4218, 0.4063, 0.3867, 0.3647, 0.3415, 0.3195, 0.2865],
       device='cuda:4')
Error converting '40°' to float: could not convert string to float: '40°'
Error converting '' to float: could not convert string to float: ''
Error converting '1.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)0)0000001m' to float: could not convert string to float: '1.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)0)0000001m'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2356, 0.1997, 0.1864, 0.1752, 0.1595, 0.1435, 0.1303, 0.1099],
       device='cuda:3')
tensor([0.4992, 0.4261, 0.3173, 0.5870, 0.3161, 0.7437, 0.2223, 1.1954],
       device='cuda:5')
tensor([1.2721, 1.3886, 1.3718, 1.3532, 1.3285, 1.3073, 1.2766, 1.2500],
       device='cuda:2')
{'completion_length': 42.484375, 'rewards/accuracy_reward': 0.5024999976158142, 'rewards/efficiency reward': 0.23346908390522003, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.735969066619873, 'reward_std': 0.1243627667427063, 'kl': 35.5, 'epoch': 0.0}
{'loss': 1.4188, 'grad_norm': 215.08646814775014, 'learning_rate': 9.999928444092641e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/GVLQA/data/flow/graph_visual/layout/fdp/hard/graph522.png
problem_id: 183888
prompt_length: 362
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_Lv4HkRzwTrs.mp4
problem_id: 114406
prompt_length: 1064
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2302.03026_2.jpg
problem_id: 28843
prompt_length: 357
path: path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_aoY0XhAXm7M.mp4
problem_id: 133528
prompt_length: 1153
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/ChromeWriting/identity_8157.png
problem_id: 20995
prompt_length: 88
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_21931.png
problem_id: 250853
prompt_length: 301
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/108/bjjLIgdnB1Y/split_5.mp4
problem_id: 164368
prompt_length: 1009
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_o7O6XJYVDRE.mp4
problem_id: 8617
prompt_length: 1069
Invalidate trace cache @ step 0 and module 44899: cache has only 0 modules
tensor([1.2648, 1.2585, 1.2426, 1.2296, 1.2102, 1.1928, 1.1743, 1.1574],
       device='cuda:2')
tensor([1.2291, 1.3274, 1.3083, 1.2926, 1.2721, 1.2485, 1.2277, 1.2009],
       device='cuda:1')
tensor([0.2166, 0.2774, 0.2631, 0.2483, 0.2274, 0.2103, 0.1930, 0.1703],
       device='cuda:7')
tensor([0.1942, 0.1956, 0.1847, 0.2014, 0.1522, 0.1364, 0.1222, 0.1710],
       device='cuda:0')
tensor([0.3526, 0.3977, 0.3750, 0.3576, 0.3319, 0.3070, 0.2821, 0.2498],
       device='cuda:4')
tensor([0.3468, 0.4159, 0.3935, 0.3765, 0.3531, 0.3249, 0.3003, 0.2677],
       device='cuda:5')
tensor([1.3040, 1.3885, 1.3706, 1.3492, 1.3263, 1.3026, 1.2773, 1.2477],
       device='cuda:3')
tensor([0.4237, 0.5487, 0.5816, 0.4091, 0.5581, 0.5296, 0.2422, 0.3961],
       device='cuda:6')
{'completion_length': 86.03125, 'rewards/accuracy_reward': 0.399394690990448, 'rewards/efficiency reward': 0.26760804653167725, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6670026779174805, 'reward_std': 0.0512375645339489, 'kl': 14.875, 'epoch': 0.0}
{'loss': 0.5924, 'grad_norm': 201.31844957830802, 'learning_rate': 9.999925865713321e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/TQA/images/question_images/human_system_eye_6146.png
problem_id: 153319
prompt_length: 337
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/2a9fe25f43f43c05026d1fcc758a880e.png
problem_id: 189560
prompt_length: 113
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/56428db3-a208-4262-b135-575f3add3ff5.png
problem_id: 64583
prompt_length: 284
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/ddfb845e668255a559e31b63aaea4fba.png
problem_id: 204745
prompt_length: 105
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_7026.png
problem_id: 86743
prompt_length: 326
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Gxjfc5O6AAc.mp4
problem_id: 97784
prompt_length: 1130
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/astro-ph0511070_0.jpg
problem_id: 15275
prompt_length: 324
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/A-OKVQA/images/268306.jpg
problem_id: 71334
prompt_length: 332
Invalidate trace cache @ step 0 and module 45660: cache has only 0 modules
Error converting 'The distance between the projector screen and the vase of red and white flowers with the table is approximately 42 inches.' to float: could not convert string to float: 'The distance between the projector screen and the vase of red and white flowers with the table is approximately 42 inches.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([1.3086, 1.3433, 1.3252, 1.3076, 1.2865, 1.2626, 1.2413, 1.2108],
       device='cuda:4')
Error converting '2051200</answer}>' to float: could not convert string to float: '2051200</answer}>'
tensor([1.2764, 1.2755, 1.2614, 1.2416, 1.2263, 1.2069, 1.1915, 1.1616],
       device='cuda:6')
tensor([0.2377, 0.8102, 0.3310, 0.3123, 0.2921, 0.3625, 0.4351, 0.2157],
       device='cuda:0')
tensor([1.2313, 1.2735, 1.2623, 1.2479, 1.2291, 1.2115, 1.1909, 0.1704],
       device='cuda:7')
Error converting 'The sum of the first 10 terms of the sequence is:103221' to float: could not convert string to float: 'The sum of the first 10 terms of the sequence is:103221'
Error converting '' to float: could not convert string to float: ''
Error converting '' to float: could not convert string to float: ''
tensor([0.2366, 0.1901, 0.1782, 0.1619, 0.1500, 0.1381, 0.1221, 0.1048],
       device='cuda:5')
tensor([0.2325, 0.2010, 0.1846, 0.1713, 0.1542, 0.1391, 0.1252, 0.1087],
       device='cuda:3')
tensor([0.5487, 0.2942, 0.6511, 0.6368, 0.1992, 0.5855, 0.5580, 0.4574],
       device='cuda:2')
tensor([1.3227, 1.4195, 1.4043, 1.3822, 1.3548, 1.3364, 1.3053, 1.2723],
       device='cuda:1')
{'completion_length': 100.953125, 'rewards/accuracy_reward': 0.5323783159255981, 'rewards/efficiency reward': 0.2374236285686493, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.769801914691925, 'reward_std': 0.11875341832637787, 'kl': 2.328125, 'epoch': 0.0}
{'loss': 0.0933, 'grad_norm': 14.486940432454581, 'learning_rate': 9.999923241699424e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextVQA/images/e762b229f7140d59.jpg
problem_id: 240907
prompt_length: 307
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_295.png
problem_id: 83123
prompt_length: 327
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/422/POgIE_9QuYA/split_6.mp4
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/Rendered_Text/cauldron/rendered_text/images/rendered_text_00008457.png
problem_id: 47755
prompt_length: 309
problem_id: 53979
prompt_length: 1057
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_j2lcGA5X7xQ.mp4
problem_id: 135515
prompt_length: 1059
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_42022.png
problem_id: 215711
prompt_length: 68
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/124/qOB709GWcE8/split_5.mp4
problem_id: 67478
prompt_length: 1012
Invalidate trace cache @ step 0 and module 46421: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_02744.mp4
problem_id: 185724
prompt_length: 756
tensor([0.2602, 0.3773, 0.3660, 0.3397, 0.3173, 0.2929, 0.2634, 0.2361],
       device='cuda:3')
tensor([0.2499, 0.3141, 0.2963, 0.2819, 0.2614, 0.2421, 0.2193, 0.1953],
       device='cuda:7')
tensor([1.2248, 1.3200, 1.1901, 1.0606, 1.2655, 1.2456, 0.5554, 0.8649],
       device='cuda:5')
tensor([0.2052, 0.2190, 0.2069, 0.1896, 0.1738, 0.1565, 0.1387, 0.1213],
       device='cuda:6')
tensor([0.2796, 0.3738, 0.3544, 0.3384, 0.3113, 0.2934, 0.2731, 0.2434],
       device='cuda:2')
tensor([1.3314, 1.3710, 1.3499, 1.3315, 1.3080, 1.2852, 1.2591, 1.2317],
       device='cuda:1')
tensor([1.3206, 1.4078, 1.3848, 1.3626, 1.3454, 1.3214, 1.2957, 1.2682],
       device='cuda:0')
tensor([0.5470, 0.5950, 0.5744, 0.5600, 0.4965, 0.4390, 0.4736, 0.4037],
       device='cuda:4')
{'completion_length': 132.171875, 'rewards/accuracy_reward': 0.3852905333042145, 'rewards/efficiency reward': 0.27693095803260803, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6622214317321777, 'reward_std': 0.0741487666964531, 'kl': 2.265625, 'epoch': 0.0}
{'loss': 0.0907, 'grad_norm': 8.120567292280791, 'learning_rate': 9.999920572050972e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/FigureQA/images/47544.png
problem_id: 25875
prompt_length: 326
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC6776557_Fig6_45626.jpg
problem_id: 178965
path:prompt_length: 166
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/1673.png
problem_id: 64594
prompt_length: 301
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2107.02250_3.jpg
problem_id: 130059
prompt_length: 398
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_8504.mp4
problem_id: 223832
prompt_length: 1046
Invalidate trace cache @ step 0 and module 47182: cache has only 0 modules
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/HME100k/identity_71173.png
problem_id: 150657
prompt_length: 68
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_7388.png
problem_id: 130387
prompt_length: 317
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1014/2786014123.mp4
problem_id: 210711
prompt_length: 1147
Error converting 'The width of the white clouds in the sky is 254.65 meters.' to float: could not convert string to float: 'The width of the white clouds in the sky is 254.65 meters.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting 'The width of the white clouds in the sky is approximately 5000 meters.' to float: could not convert string to float: 'The width of the white clouds in the sky is approximately 5000 meters.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting 'In this image the size of the white clouds and the width of the sky is considered to be 20 meters in total.' to float: could not convert string to float: 'In this image the size of the white clouds and the width of the sky is considered to be 20 meters in total.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting 'The white clouds with the sky in the image could be measured in a wide range of meters as it's difficult to accurately gauge the width of clouds from a photograph. Without a specific reference point such as a building or landscape feature of known measurement a numerical value cannot be provided.' to float: could not convert string to float: "The white clouds with the sky in the image could be measured in a wide range of meters as it's difficult to accurately gauge the width of clouds from a photograph. Without a specific reference point such as a building or landscape feature of known measurement a numerical value cannot be provided."
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([1.2569, 1.2584, 1.2448, 1.2298, 1.2089, 1.1926, 1.1723, 1.1523],
       device='cuda:2')
tensor([1.2764, 0.2699, 0.2557, 0.2352, 1.2231, 1.2048, 0.1865, 0.1639],
       device='cuda:0')
tensor([0.2527, 0.2532, 1.2355, 0.2271, 1.2067, 1.1878, 0.1741, 1.1514],
       device='cuda:1')
Error converting 'The width of the white clouds in terms of the sky is approximately 0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' to float: could not convert string to float: 'The width of the white clouds in terms of the sky is approximately 0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2638, 0.2989, 0.2849, 0.2696, 0.2514, 0.2327, 0.2123, 0.1924],
       device='cuda:7')
tensor([0.2450, 0.3695, 0.3499, 0.3321, 0.3107, 0.2856, 1.0099, 0.2299],
       device='cuda:6')
tensor([0.2382, 0.2264, 0.2075, 0.1933, 0.1770, 0.1618, 0.1429, 0.1239],
       device='cuda:3')
tensor([1.3377, 1.4133, 1.3920, 1.3756, 1.3512, 1.3262, 1.2930, 1.2686],
       device='cuda:4')
tensor([1.3288, 1.4142, 1.3905, 1.3696, 1.3426, 1.3212, 1.2944, 1.2587],
       device='cuda:5')
{'completion_length': 64.796875, 'rewards/accuracy_reward': 0.49609375, 'rewards/efficiency reward': 0.2587069272994995, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 0.7548006772994995, 'reward_std': 0.18933987617492676, 'kl': 5.75, 'epoch': 0.0}
{'loss': 0.2305, 'grad_norm': 1909.9309837650158, 'learning_rate': 9.99991785676799e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_05704.mp4
problem_id: 3998
prompt_length: 753
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_e8w3gmiv-mM.mp4
problem_id: 4448
prompt_length: 1053
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/0017/4565306431.mp4
problem_id: 140233
prompt_length: 1058
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_08219.mp4
problem_id: 68466
prompt_length: 732
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/c0a4cb58-6523-445c-8808-c930c724bda4.png
problem_id: 215213
prompt_length: 166
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_5221.mp4
problem_id: 22080
prompt_length: 1044
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_7123.mp4
problem_id: 146384
prompt_length: 1038
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_09253.mp4
problem_id: 196668
prompt_length: 731
Invalidate trace cache @ step 0 and module 47943: cache has only 0 modules
tensor([0.4072, 0.4323, 0.4430, 0.4517, 0.3859, 0.4521, 0.4431, 0.3881],
       device='cuda:0')
tensor([0.3056, 0.4363, 0.4118, 0.4000, 0.3747, 0.3525, 0.3275, 0.2975],
       device='cuda:2')
tensor([0.3428, 0.3951, 0.3762, 0.3581, 0.3396, 0.3156, 0.2885, 0.2644],
       device='cuda:4')
tensor([1.3226, 1.4309, 1.4079, 1.3902, 1.3693, 1.3364, 1.3119, 1.2821],
       device='cuda:7')
tensor([1.2984, 1.3786, 1.3617, 1.3432, 1.3222, 1.2968, 1.2755, 1.2475],
       device='cuda:6')
tensor([1.3437, 1.3991, 1.3782, 1.3614, 1.3366, 1.3173, 1.2881, 1.2606],
       device='cuda:5')
tensor([0.2942, 0.4049, 0.3860, 0.3630, 0.3406, 0.3208, 0.2920, 0.2651],
       device='cuda:1')
tensor([1.2626, 1.4066, 1.3872, 1.3637, 1.3419, 1.3150, 1.2840, 1.2551],
       device='cuda:3')
{'completion_length': 14.34375, 'rewards/accuracy_reward': 0.5297961235046387, 'rewards/efficiency reward': 0.3191491365432739, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.8489452004432678, 'reward_std': 0.046326376497745514, 'kl': 1.96875, 'epoch': 0.0}
{'loss': 0.0785, 'grad_norm': 13.566830279036013, 'learning_rate': 9.999915095850507e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_RhH19y0V6-0.mp4
problem_id: 106947
prompt_length: 1065
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/CLEVRER/train_videos/video_01032.mp4
problem_id: 219847
prompt_length: 731
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/3c90e4f60dbf97d21bc3c1bed0463748.png
problem_id: 121511
prompt_length: 88
path:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_j56eH9M0ObY.mp4
problem_id: 229715
prompt_length: 1129
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC6525736_Fig3_482783.jpg
problem_id: 109218
prompt_length: 307
 /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/KCRQU.mp4
problem_id: 9605
prompt_length: 1056
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_hBT_SN8IGsQ.mp4
problem_id: 60536
prompt_length: 1053
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/316/FiE1KczH4pc/split_2.mp4
problem_id: 118499
prompt_length: 1147
Invalidate trace cache @ step 0 and module 48704: cache has only 0 modules
tensor([1.2921, 1.3001, 1.2837, 1.2670, 1.2466, 1.2266, 1.2065, 1.1868],
       device='cuda:5')
tensor([0.2444, 0.2001, 0.1924, 0.1757, 0.1627, 0.1481, 0.1307, 0.1136],
       device='cuda:3')
tensor([1.3515, 1.4061, 1.3836, 1.3642, 1.3447, 1.3220, 1.2948, 1.2698],
       device='cuda:7')
tensor([0.3276, 0.3806, 0.3606, 0.3360, 0.3179, 0.2888, 0.2654, 0.2419],
       device='cuda:6')
tensor([1.3648, 1.4056, 1.3891, 1.3638, 1.3458, 1.3198, 1.2945, 1.2623],
       device='cuda:2')
tensor([1.3470, 1.4068, 1.3862, 1.3721, 1.3462, 1.3203, 1.2958, 1.2687],
       device='cuda:1')
tensor([1.3256, 1.3706, 1.3501, 1.3338, 1.3137, 1.2892, 1.2658, 1.2368],
       device='cuda:4')
tensor([1.2867, 1.4061, 1.3843, 1.3652, 1.3427, 1.3193, 1.2945, 1.2593],
       device='cuda:0')
{'completion_length': 59.546875, 'rewards/accuracy_reward': 0.75, 'rewards/efficiency reward': 0.3010188937187195, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 1.0510189533233643, 'reward_std': 0.045755624771118164, 'kl': 4.125, 'epoch': 0.0}
{'loss': 0.1651, 'grad_norm': 12.813643328528025, 'learning_rate': 9.999912289298545e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_LgZexWYPfUw.mp4
problem_id: 141696
prompt_length: 1049
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_7476.png
problem_id: 100073
prompt_length: 294
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/d05e8c4fa42f9d28.png
problem_id: 28662
prompt_length: 303
path:path:  /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_25487.png
problem_id: 33210
prompt_length: 319
/mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_pwKZRo19Vf0.mp4
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_vCeaFAiokrU.mp4
problem_id: 100463
prompt_length: 1028
problem_id: 146913
prompt_length: 1068
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/3e7166cc7dde9148872fea6bb3b5fdf3.png
problem_id: 163920
prompt_length: 101
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/2276.png
problem_id: 38147
prompt_length: 192
Invalidate trace cache @ step 0 and module 49465: cache has only 0 modules
Error converting '45°' to float: could not convert string to float: '45°'
Error converting '30°' to float: could not convert string to float: '30°'
Error converting '{' to float: could not convert string to float: '{'
tensor([0.2398, 0.1948, 0.1864, 0.1690, 0.1559, 1.1400, 1.1262, 0.1089],
       device='cuda:1')
tensor([0.2414, 0.2156, 0.1979, 0.1886, 0.1735, 0.1551, 0.1403, 0.1246],
       device='cuda:6')
tensor([1.3117, 1.3372, 1.3220, 1.3024, 1.2783, 1.2559, 1.2332, 1.2085],
       device='cuda:2')
tensor([0.2251, 0.3115, 0.2936, 0.2758, 0.2612, 0.2384, 0.2176, 0.1914],
       device='cuda:3')
tensor([0.6586, 0.6314, 0.7489, 0.7305, 0.6809, 0.7288, 0.7481, 0.4434],
       device='cuda:7')
tensor([1.2950, 1.3848, 1.3683, 1.3477, 1.3269, 1.3066, 1.2786, 1.2516],
       device='cuda:5')
tensor([1.3150, 1.3997, 1.3817, 1.3585, 1.3382, 1.3121, 1.2886, 1.2539],
       device='cuda:4')
tensor([1.3670, 1.4008, 1.3792, 1.3602, 1.3403, 1.3186, 1.2895, 1.2603],
       device='cuda:0')
{'completion_length': 28.546875, 'rewards/accuracy_reward': 0.5830106735229492, 'rewards/efficiency reward': 0.265680730342865, 'all_wrong': 0.375, 'all_correct': 0.0, 'reward': 0.848691463470459, 'reward_std': 0.10166902840137482, 'kl': 1.609375, 'epoch': 0.0}
[2025-04-22 06:37:44,544] [WARNING] [stage3.py:2114:step] 1 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time
{'loss': 0.0643, 'grad_norm': 6.515358915851339, 'learning_rate': 9.999909437112127e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/PerceptionTest/video_11019.mp4
problem_id: 159393
prompt_length: 1051
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_ahq-qmtrpMY.mp4
problem_id: 69229
prompt_length: 1069
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/TabMWP/images/7422.png
problem_id: 127962
prompt_length: 187
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/b9e3dd6729bb518265e51479f31f67fd.png
problem_id:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/bfacf1ce2ec0803a4b70eaf3055bba06.png
problem_id: 214279
prompt_length: 142
 188688
prompt_length: 120
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/cd02ec64676bb655dabd59d7a5f8fb76.png
problem_id: 204921
prompt_length: 107
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/86E2E.mp4
problem_id: 173681
prompt_length: 1102
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4V/claude-monet_the-seine-near-giverny-1.png
problem_id: 235853
prompt_length: 290
Invalidate trace cache @ step 0 and module 50226: cache has only 0 modules
Error converting 'A: Xiao Ming needs to walk a minimum of <answer>A is 523米' to float: could not convert string to float: 'A: Xiao Ming needs to walk a minimum of <answer>A is 523米'
Error converting 'The shortest distance from the gate to the panda pavilion is 902' to float: could not convert string to float: 'The shortest distance from the gate to the panda pavilion is 902'
Error converting '<sum>0' to float: could not convert string to float: '<sum>0'
tensor([1.2192, 0.2105, 0.1977, 0.1845, 0.1715, 0.1555, 0.1400, 0.1215],
       device='cuda:3')
tensor([0.2187, 0.2057, 0.1902, 0.1741, 0.1604, 0.1452, 0.1312, 0.1143],
       device='cuda:0')
tensor([0.2141, 0.2002, 0.1836, 0.1723, 0.1590, 0.1461, 0.1310, 0.1139],
       device='cuda:2')
tensor([0.1628, 0.2019, 0.1903, 0.1746, 0.1602, 0.1475, 0.1268, 0.1121],
       device='cuda:1')
tensor([0.5332, 0.5371, 0.5954, 0.5962, 0.4979, 0.5580, 0.5294, 0.4625],
       device='cuda:5')
tensor([1.3163, 1.4082, 1.3903, 1.3656, 1.3493, 1.3171, 1.2989, 1.2703],
       device='cuda:7')
tensor([1.3338, 1.3886, 1.3691, 1.3498, 1.3245, 1.3076, 1.2829, 1.2548],
       device='cuda:4')
tensor([1.2738, 1.4037, 1.3836, 1.3615, 1.3381, 1.3161, 1.2900, 1.2605],
       device='cuda:6')
{'completion_length': 44.65625, 'rewards/accuracy_reward': 0.4236566722393036, 'rewards/efficiency reward': 0.24198031425476074, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.6656369566917419, 'reward_std': 0.08284100890159607, 'kl': 1.6015625, 'epoch': 0.0}
{'loss': 0.0639, 'grad_norm': 7.151672534889523, 'learning_rate': 9.999906539291283e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC8029940_f10_412209.jpg
problem_id: 176409
prompt_length: 252
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/TextCaps/77c6dd0a9a994e28.png
problem_id: 161605
prompt_length: 303
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/IAM/cauldron/iam/images/iam_00003159.png
problem_id: 5836
prompt_length: 282
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/d7c8b9ee32ef644e9740f11e0c8d72b5.png
problem_id:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_TVmuh_sR1KI.mp4
problem_id: 221378
prompt_length: 1132
 173396path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1026/6571774901.mp4
problem_id: 7622
prompt_length: 1054

prompt_length: 154
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/f4fc8be5-2c32-4822-b567-2debfd98cc15.png
problem_id: 3510
prompt_length: 211
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_22321.png
problem_id: 90394
prompt_length: 303
Invalidate trace cache @ step 0 and module 50987: cache has only 0 modules
Error converting 'The output number S is' to float: could not convert string to float: 'The output number S is'
Error converting '2}\sum\frac{x^{i^{2}+i+1}}} }{2}}\frac{/answer>458' to float: could not convert string to float: '2}\\sum\\frac{x^{i^{2}+i+1}}} }{2}}\\frac{/answer>458'
tensor([0.1645, 0.2008, 0.1867, 1.1732, 0.1591, 0.1436, 0.1317, 0.1122],
       device='cuda:0')
tensor([0.2259, 0.3031, 1.1962, 0.2708, 0.5268, 0.3263, 0.2142, 0.1914],
       device='cuda:2')
tensor([0.2961, 0.3029, 0.2851, 0.2711, 0.2494, 0.2287, 0.2078, 0.1866],
       device='cuda:7')
tensor([0.3728, 0.3131, 0.2974, 0.2752, 0.2554, 0.2384, 0.2143, 0.1921],
       device='cuda:4')
tensor([0.5633, 0.5248, 0.6125, 0.4988, 0.5682, 0.5347, 0.5615, 0.4353],
       device='cuda:1')
tensor([1.2821, 1.4141, 1.3949, 1.3713, 1.3512, 1.3306, 1.3013, 0.2697],
       device='cuda:3')
tensor([1.3279, 1.4121, 1.3944, 1.3711, 1.3498, 1.3254, 1.3032, 1.2720],
       device='cuda:5')
tensor([0.4110, 0.4511, 0.3908, 0.2161, 0.3939, 0.3568, 0.3477, 0.2204],
       device='cuda:6')
{'completion_length': 80.171875, 'rewards/accuracy_reward': 0.333169162273407, 'rewards/efficiency reward': 0.24918696284294128, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.5823560953140259, 'reward_std': 0.17100316286087036, 'kl': 1.34375, 'epoch': 0.0}
{'loss': 0.0539, 'grad_norm': 5.699262415764857, 'learning_rate': 9.99990359583604e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/ArxivQA/images/2201.00047_2.jpg
problem_id: 120101
prompt_length: 355
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_y7tActqdm9A.mp4
problem_id: 190663
prompt_length: 1131
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_2043.png
problem_id: 166711
prompt_length: 319
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/General/ShareGPT4o/sharegpt4o_21559.png
problem_id:path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/2fd1d9c5efb059aa7f963549b183c601.png
 problem_id: 105669
prompt_length: 109
9491
prompt_length: 280
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_kkIClKG5xY8.mp4
problem_id: 12240
prompt_length: 1126
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/bc93d360262cb18ccd7d144bd5a2521f.png
problem_id: 71485
prompt_length: 119
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Geometry3K/images/79.png
problem_id: 173923
prompt_length: 188
Invalidate trace cache @ step 0 and module 51748: cache has only 0 modules
Error converting 'The horizontal size of the background brick wall potted plant is approximately 4.' to float: could not convert string to float: 'The horizontal size of the background brick wall potted plant is approximately 4.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting '(5× 121}+(5× 143} = 1650' to float: could not convert string to float: '(5× 121}+(5× 143} = 1650'
Error converting 'The width of the background brick wall and potted plant together appears to be approximately 6' to float: could not convert string to float: 'The width of the background brick wall and potted plant together appears to be approximately 6'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting 'The width of the background brick wall and potted plant is approximately 15.00' to float: could not convert string to float: 'The width of the background brick wall and potted plant is approximately 15.00'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
Error converting 'The horizontal size of the background brick wall potted plant is approximately 2.00 feet.' to float: could not convert string to float: 'The horizontal size of the background brick wall potted plant is approximately 2.00 feet.'
Error in reward_fn for question_type 'regression': unsupported operand type(s) for -: 'NoneType' and 'float'
tensor([0.2157, 0.2077, 0.1995, 0.1818, 0.1645, 0.1494, 0.1350, 0.1173],
       device='cuda:2')
tensor([0.1751, 0.1981, 0.1870, 0.1735, 0.1585, 0.1444, 0.1252, 0.1068],
       device='cuda:6')
tensor([0.3181, 0.8472, 1.0752, 0.8083, 0.2852, 0.2634, 0.2385, 0.2109],
       device='cuda:3')
tensor([1.2323, 1.2638, 1.2484, 1.2356, 1.2177, 1.2001, 1.1837, 1.1616],
       device='cuda:7')
tensor([1.1869, 1.2238, 1.2071, 1.1953, 1.1816, 1.1636, 1.1474, 1.1284],
       device='cuda:1')
tensor([0.3225, 0.4430, 0.4111, 0.3790, 0.4184, 0.4341, 0.2959, 0.3190],
       device='cuda:0')
tensor([1.3447, 1.4006, 1.3788, 1.3613, 1.3385, 1.3156, 1.2864, 1.2577],
       device='cuda:5')
tensor([0.3225, 0.4128, 0.3891, 0.3683, 0.3460, 0.3205, 0.2940, 0.2659],
       device='cuda:4')
{'completion_length': 34.0625, 'rewards/accuracy_reward': 0.41609612107276917, 'rewards/efficiency reward': 0.2446744590997696, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.6607705950737, 'reward_std': 0.07901610434055328, 'kl': 1.5390625, 'epoch': 0.0}
{'loss': 0.0616, 'grad_norm': 7.4777091840735235, 'learning_rate': 9.999900606746421e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/OpenSpaces/image_193.png
problem_id: 146129
prompt_length: 287
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/00T1E.mp4
problem_id: 7272
prompt_length: 1052
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/OCR/ChromeWriting/identity_8366.png
problem_id: 221693
prompt_length: 105
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/c16064d326aba015b96ed1054b9e51df.png
problem_id: 21928
prompt_length: 101
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/AI2D/images/1681.png
problem_id: 152081
prompt_length: 248
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/liwei_youtube_videos/videos/youtube_video_2024/ytb_udbdkinWoM0.mp4
problem_id: 34937
prompt_length: 1083
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/c328943d1af540bee72d1f489e5ef7db.png
problem_id: 114236
prompt_length: 94
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/TXNDA.mp4
problem_id: 249294
prompt_length: 1008
Invalidate trace cache @ step 0 and module 52509: cache has only 0 modules
Error converting '' to float: could not convert string to float: ''
tensor([1.2719, 1.2637, 1.2464, 1.2296, 1.2131, 1.1999, 1.1816, 1.1583],
       device='cuda:5')
tensor([0.1606, 0.1987, 0.1854, 0.1717, 0.1558, 0.1436, 0.1265, 0.1075],
       device='cuda:6')
tensor([0.1711, 0.2016, 1.1849, 0.1690, 0.1598, 0.1439, 0.1259, 0.1110],
       device='cuda:2')
tensor([0.5452, 0.8139, 0.5032, 0.6345, 0.6638, 0.6531, 0.4635, 0.4102],
       device='cuda:7')
tensor([0.2132, 0.2833, 0.2724, 0.3054, 0.1497, 0.2286, 0.3075, 0.0990],
       device='cuda:3')
tensor([1.3412, 1.3918, 1.3705, 1.3486, 1.3273, 1.3023, 1.2784, 1.2502],
       device='cuda:4')
tensor([0.3187, 1.3753, 1.3573, 1.3394, 0.3179, 1.2982, 1.2770, 1.2458],
       device='cuda:0')
tensor([0.5546, 0.6323, 0.5843, 0.6013, 0.5358, 0.5687, 0.5706, 0.5211],
       device='cuda:1')
{'completion_length': 85.46875, 'rewards/accuracy_reward': 0.44729453325271606, 'rewards/efficiency reward': 0.23296818137168884, 'all_wrong': 0.5, 'all_correct': 0.0, 'reward': 0.6802627444267273, 'reward_std': 0.14850084483623505, 'kl': 5.0625, 'epoch': 0.0}
{'loss': 0.2017, 'grad_norm': 27.453683719104642, 'learning_rate': 9.999897572022456e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/215/BPgpTaUH6t0/split_2.mp4
problem_id: 194559
prompt_length: 1153
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_Lg-mJS-BZZo.mp4
problem_id: 181067
prompt_length: 1005
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/932d4b7b-64d8-43c5-b776-5c85348b43ca.png
problem_id: 237879
prompt_length: 220
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Knowledge/PMC-VQA/images/PMC4355500_Fig2_365821.jpg
problem_id: 215363
prompt_length: 206
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/youcook2/121/nfO0YS2dSdw/split_5.mp4
problem_id: 217508
prompt_length: 888
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Math/Multimath-300k/3a4d058ec4949beda69282fdb2fe0f45.png
problem_id: 102306
prompt_length: 122
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_1150.png
problem_id: 121825
prompt_length: 321
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/SMLI0.mp4
problem_id: 11834
prompt_length: 1051
Invalidate trace cache @ step 0 and module 53270: cache has only 0 modules
Error converting '{<50><<18' to float: could not convert string to float: '{<50><<18'
Error converting '(<answer>A)' to float: could not convert string to float: '(<answer>A)'
tensor([0.2098, 0.1917, 0.1792, 0.1663, 0.1537, 0.1367, 0.1238, 0.1069],
       device='cuda:0')
tensor([0.2347, 0.2276, 0.2141, 0.1883, 0.1870, 0.1708, 0.1581, 0.1585],
       device='cuda:3')
tensor([1.2043, 1.2803, 1.2618, 1.2514, 1.2323, 1.2152, 1.1938, 1.1777],
       device='cuda:2')
tensor([0.2973, 0.3186, 0.3021, 0.2850, 0.2701, 0.2468, 0.2250, 0.1965],
       device='cuda:1')
tensor([0.4904, 0.5060, 0.4988, 0.5338, 0.5053, 0.5391, 0.3878, 0.4219],
       device='cuda:7')
tensor([0.5409, 0.5702, 0.6188, 0.6258, 0.2841, 0.5525, 0.5552, 0.5054],
       device='cuda:5')
tensor([1.2679, 1.4065, 1.3793, 1.3647, 1.3395, 1.3142, 1.2899, 1.2565],
       device='cuda:6')
tensor([1.3422, 1.4131, 1.3918, 1.3707, 1.3481, 1.3206, 1.2974, 1.2608],
       device='cuda:4')
{'completion_length': 223.96875, 'rewards/accuracy_reward': 0.43474096059799194, 'rewards/efficiency reward': 0.2568950057029724, 'all_wrong': 0.625, 'all_correct': 0.0, 'reward': 0.6916359663009644, 'reward_std': 0.05094529688358307, 'kl': 1.28125, 'epoch': 0.0}
{'loss': 0.0512, 'grad_norm': 9.067655999337287, 'learning_rate': 9.999894491664173e-06, 'epoch': 0.0}
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Chart/VisualWebInstruct/59c30d0d-0764-4ab8-bf8b-e893a2d7f86d.png
problem_id: 164765
prompt_length: 199
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/STAR/PM1VD.mp4
problem_id: 118625
prompt_length: 1052
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/NeXT-QA/NextQA/NExTVideo/1014/4715646346.mp4
problem_id: 170899
prompt_length: 1108
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/Spatial/Spacellava/image_15539.png
problem_id: 231118
prompt_length: 317
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/Charades/IA7X9.mp4
problem_id: 154105
prompt_length: 1135
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_smh90DBXsBg.mp4
problem_id: 203096
prompt_length: 1126
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_hPQEkPZmNsU.mp4
problem_id: 247315
prompt_length: 1052
path: /mnt/bn/tns-algo-live-my/chenfeng/longvlm/zipgrpo/Video-R1/src/r1-v/Video-R1-data/LLaVA-Video-178K/academic_source/activitynet/v_XFlKGUFgBnc.mp4
problem_id: 195733
prompt_length: 1071
Invalidate trace cache @ step 0 and module 54031: cache has only 0 modules
tensor([0.4975, 0.4808, 0.5075, 0.4796, 0.3976, 0.4031, 0.3730, 0.1258],
       device='cuda:7')
tensor([0.2995, 0.3345, 0.3151, 0.2948, 0.2773, 0.2553, 0.2305, 0.2076],
       device='cuda:2')
tensor([1.3218, 1.4118, 1.3852, 1.3620, 1.3409, 1.3176, 1.2873, 1.2597],
       device='cuda:6')
tensor([1.3252, 1.3951, 1.3798, 1.3557, 1.3329, 1.3065, 1.2771, 1.2463],
       device='cuda:4')
tensor([1.2857, 1.4109, 1.3931, 1.3748, 1.3481, 1.3280, 1.2989, 1.2646],
       device='cuda:3')
tensor([1.3452, 1.4067, 1.3892, 1.3628, 1.3419, 1.3162, 1.2875, 1.2565],
       device='cuda:5')
tensor([1.3065, 1.3976, 1.3772, 1.3605, 1.3347, 1.3099, 1.2843, 1.2556],
       device='cuda:1')
tensor([1.2654, 1.4013, 1.3831, 1.3599, 1.3418, 1.3161, 1.2895, 1.2618],
       device='cuda:0')
{'completion_length': 141.3125, 'rewards/accuracy_reward': 0.7777961492538452, 'rewards/efficiency reward': 0.3072006106376648, 'all_wrong': 0.25, 'all_correct': 0.0, 'reward': 1.0849967002868652, 'reward_std': 0.05909639969468117, 'kl': 2.03125, 'epoch': 0.0}
{'loss': 0.081, 'grad_norm': 89.21762857726239, 'learning_rate': 9.999891365671597e-06, 'epoch': 0.0}
2025-04-23 01:06:18.010 n116-043-044:107011:108482 [1] NCCL INFO [Service thread] Connection closed by localRank 1
2025-04-23 01:06:18.010 n116-043-044:107016:108472 [6] NCCL INFO [Service thread] Connection closed by localRank 6
2025-04-23 01:06:18.010 n116-043-044:107010:108485 [0] NCCL INFO [Service thread] Connection closed by localRank 0
2025-04-23 01:06:18.010 n116-043-044:107015:108473 [5] NCCL INFO [Service thread] Connection closed by localRank 5
2025-04-23 01:06:18.010 n116-043-044:107014:108479 [4] NCCL INFO [Service thread] Connection closed by localRank 4
2025-04-23 01:06:18.010 n116-043-044:107013:108484 [3] NCCL INFO [Service thread] Connection closed by localRank 3
2025-04-23 01:06:18.010 n116-043-044:107017:108476 [7] NCCL INFO [Service thread] Connection closed by localRank 7
2025-04-23 01:06:18.040 n116-043-044:107013:107930 [3] NCCL INFO comm 0x10ec6240 rank 3 nranks 8 cudaDev 3 busId c000 - Abort COMPLETE
2025-04-23 01:06:18.042 n116-043-044:107014:107933 [4] NCCL INFO comm 0xfdc8140 rank 4 nranks 8 cudaDev 4 busId 84000 - Abort COMPLETE
2025-04-23 01:06:18.560 n116-043-044:107017:107731 [7] NCCL INFO comm 0xf646140 rank 7 nranks 8 cudaDev 7 busId 8c000 - Abort COMPLETE
