Not an expert, but I think you should create your negative examples in a way where the first few tokens are not enough to differentiate between good and bad.

One easy way to do this would be to first sample the GPT-4 examples, then keep "n" tokens (sampled from 0 to the length of the answer) then generate the rest of the answer with the other (worse) model.

That way, the DPO model cannot just ignore every tokens after a few, because the branching can happen at any point.

(my other thought is that you should increase the KL divergence penalty, if your DPO models diverges too much from your initial model, but I think making the negative examples better is a stronger first step to take)

I think this happened to my model: CultriX/MistralTrix-v1 as well. It sometimes starts to randomly output sequences of exactly that and I am not sure why. @mlabonne maybe you have any input on this?

What model and hyperparameters are you using for your DPO training?

I have seen similar behaviour with Mistral 7B when beta was large (ie >0.5) and also when training for too many epochs. In both cases I think it’s a form of “reward hacking” where the model learns that some tokens have a spurious high reward.

In general, DPO takes a bit of tweaking to find good hyperparamters and the main ones I’d look at are beta, batch size, and number of epochs (assuming you are using a small LR like 5e-7).

The other ablation would be to run your model through something like dpo_orca_pairs and see if you observe the same effect - if you do then you can eliminate the dataset as the source of the problem.

Another perspective to consider:

1. While you've utilized reward accuracies to validate the effectiveness of your model, there are additional aspects to examine. It's advisable to ensure that the responses selected by your model are associated with higher probabilities (logits) within your policy model.
2. However, it's important to note that even if a chosen response has higher probabilities compared to those that were rejected, this does not necessarily mean it is the optimal response with the highest probability for the given prompt. The DPO loss primarily focuses on enhancing the chosen responses over the rejected ones, rather than identifying the absolute best response.
   

Just guessing here but perhaps part of the syntax for the positives is not parsed correctly and ends up in the tokens used for DPO. If that is always present in the positives, those sequences drive the DPO and end up as the only thing being generated.

So I tweaked the learning rate to 1e-7 (was 5e-7), removed portion of the dataset (I used two datasets, one based on Orca Pairs, the other based on UltraFeedback, I kept the latter), and I decreased the max length to 2048 max_length 1152 prompt length. EDIT: I thought this had worked, but apparently I am wrong. For some prompts, the same result occurs with repetitions of /******/ everywhere. I am very confused about this and should find time to dig deeper but it is a tedious trial and error process of training and testing that east up a lot of my time. If anyone wants to have a look, I can provide gated access to the model.

Mostly cc @lewtun but also a shout out to everyone in this thread for brainstorming along!