Still active?
Are you still working on this?
I have some plans to finetune Yi 1.5 32B 32K on Magpie Ultra or something similar, with this or just 01's base model used as a base for the finetune.
Nice, I look forward to seeing it.
Will have to give this a go tonight! Has it been tuned with your AEZAKMI dataset as well?
No, it's just Magpie Ultra. AEZAKMI dataset is in a bit of an existential crisis, I like v2 more then newer versions. I'll try to make a new version of the AEZAKMI that I'll like and then train Yi 34B 200K/Yi 1.5 34B / Yi 1.5 9B on it.
I'm open to ideas when it comes to the direction of the AEZAKMI dataset, I'm thinking about adding some distilled multi-turn conversations from Hermes 3 70B to it and some non-synthetic reddit&4chan data
I think that would be cool and maybe some Gutenberg and LimaRP stuff?
https://huggingface.co/datasets/Dampfinchen/Creative_Writing_Multiturn
I'll look into it. I know LimaRP has some ERP samples with small kids so I am hesitant to use it anywhere, unless it would be filtered out properly.
I had no idea! Jesus!
I remember using a model for creative writing once and it suggested I include some paedophilia in it. Never deleted a model so fast in my life!