ITS NOT REAL
I hope you know, I only criticize you because I know you can do better. Prove the model can remember every single token and explain all of them at once of the entire context and then the extended context is worth it. Otherwise its just useless.
If the model can summarize an entire book thats worth 1048k tokens an rewrite it into an essay, then you did your job right. If it doesnt remember anything but 1 idea then the model is a failure and needs further work
Thanks for the meme, and the motivating words.
- Just published a blog: https://gradient.ai/blog/the-haystack-matters-for-niah-evals
- We are looking into ways on improved alignment for long context w.r.t. special tokens.
entire book thats worth 1048k tokens- there might be only few books of that size. That's around the size of the bible including old + new testament or other books of 1.5k pages. What we would need is a copyleft book of 1500 pages, published after the llama3 training data, in plain.txt. As you are asking for it, do you happen to have such data @rombodawg ?
@michaelfeil You can also copy multiple books into an llm, a series of the same novels, or diffrent books, and ask the model to summarize all of the books in seperate paragraphs, or while an essay describing all the books in detail and what impact they made. You can easily think of multiple benchmarks you can do from an llm reading through a large amount of books and writing something detailed about them, just by doing some critical thinking. 😉
I create my own benchmarks for llms all the time, its very fun
@rombodawg We took your request seriously, and massivley improved chat quality. https://huggingface.co/gradientai/Llama-3-70B-Instruct-Gradient-262k
Simply amazing, if i ever traing codellama-70b, im deffinetly using your models
Getting a dual 4090 setup for work and hoping that creates a smooth experience for 70b and cant wait to try this. (fingers crossed)
@jmachere some of the quants are regenerating now!