bigscience/bloom · Ideas to improve Fine Tuned BLOOM 560 for dialogue using LIGHT dataset

Hi everyone,

My name is Andrew and I've been experimenting with fine tuning BLOOM 560 for dialogue. Here is the current version of the model https://huggingface.co/andrewnoel/first-try-dialogue-bloom-560 fine tuned in google colab using premium GPU environment on a process of the LIGHT dataset https://parl.ai/projects/light/

At the bottom I've posted an example prompt and response. But was hoping that people might have suggestions for how we can improve our model beyond this. We've considered switching to an english only base model and we are currently playing with tuning the prompts. And of course we could beef up the hardware and train on a bigger BLOOM but want to save that until the very end. There are likely lots of ideas for improvements that we haven't thought of so please let me know if you have any!

The basic idea is to add 13 new tokens to the model that give it a dialogue structure with a setting and then fine tune those new token embeddings as well (using huggingface Trainer for now).

Here's an example prompt:

_setting_desc A small, medieval village populated by people just trying to survive in a cruel world overrun by monsters. Wooden spikes form a fence lining the village, keeping monsters out.
_partner_name adventurer
_partner_persona An adventurer clad in dirty iron armor, wielding a rusty longsword.
_self_name blacksmith
_self_persona I'm a hard-working, no-nonsense blacksmith working to provide my small village with the weapons and equipment they need to protect themselves from monsters.
_object_desc a furnace : The red-hot furnace for melting raw metal ores burns brightly.
_object_desc an anvil : The anvil is used to shape molten metal into weapons or armor for sale.
_object_desc a chest : The chest is full of freshly-smithed weapons and armor for sale.
_partner_say Hello! What do you do here?"```

Here is the generated response:

```"_self_say  I am looking for something, I got bored and decided to come to the tavern and chill for a bit. 
 _partner_say Ah! You are the best blacksmith I've ever seen! Tell me about yourself.  
 _self_say  I am a blacksmith.  I work in a village nearby.  The weapons and armor I stock up on come from exotic places.  I spend a lot of time in the village with people just trying to survive. 
 _partner_say I see. Do you have any weapons or armor you'd be willing to trade for the armor? 
 _self_say Well I got a longsword that could do a lot of damage but I couldn't put it down. 
 _partner_say Ah! Do you think you can put up a fight?  
 _partner_emote laugh 
 _self_emote grin 
 _self_say I am a good blacksmith and I can do some melee combat, that would really help me break out of this cold world. 
 _partner_say Oh! That sounds like fun! Do you live close to this village? 
 _self_say Of course I do, I travel to many villages.  The village is a few miles away. 
 _partner_say Great! Would you be willing to share the armor and weapons you stock up on with me? 
 _self_act remove chest 
 _self_say Sure, I would appreciate that.  Thank you. 
 _partner_say Okay! I'll let you know when I see you! 
 _self_say I will, thank you for the information. 
 <|endoftext|> I live right over there, around these spikes. Do you want to check out the weapons and armor that I stock up on? 
 _self_say No, I am not sure about that.  I am just"```