Unable to load model in 8bit model or contains error when using 4bit
Hi, when I tried to load model in 8bit for fine-tuning, it gives me "RuntimeError: The size of tensor a (32) must match the size of tensor b (8) at non-singleton dimension 0" errors. And we I load model in 4bit, the shape of each expert's weight becomes really large:
Layer 8, Expert 0 - w1 Shape: (29360128, 1)
Layer 8, Expert 1 - w1 Shape: (29360128, 1)
Layer 8, Expert 2 - w1 Shape: (29360128, 1)
Layer 8, Expert 3 - w1 Shape: (29360128, 1)
Layer 8, Expert 4 - w1 Shape: (29360128, 1)
Layer 8, Expert 5 - w1 Shape: (29360128, 1)
Layer 8, Expert 6 - w1 Shape: (29360128, 1)
Layer 8, Expert 7 - w1 Shape: (29360128, 1)
which should be originally if I load in 16bit:
Layer 8, Expert 0 - w1 Shape: (14336, 4096)
Layer 8, Expert 1 - w1 Shape: (14336, 4096)
Layer 8, Expert 2 - w1 Shape: (14336, 4096)
Layer 8, Expert 3 - w1 Shape: (14336, 4096)
Layer 8, Expert 4 - w1 Shape: (14336, 4096)
Layer 8, Expert 5 - w1 Shape: (14336, 4096)
Layer 8, Expert 6 - w1 Shape: (14336, 4096)
Layer 8, Expert 7 - w1 Shape: (14336, 4096)
Just want to get some helps on this, thanks.