This Model Sucks
It doesn't listen to what I want it to do even with the correct prompt. It also wants to add its own opinions to it too.
@ReXommendation I did highlight this model is experimental, this model is just an upgrade to the previous one. The whole point of these experiments was to learn about MOE compression, and more about this specific MOE model. I learned this model was made from a single 22b base, not from scratch. Through testing the perplexity across 2 trained models, one with all experts compressed, and one with just a single expert. And while I did ask for feedback, nothing you said was constructive. Providing some examples that could’ve been helpful. Either way, as stated before this model is still experimental, due to the fact it hasn’t had additional pre-training.