Thank you Richard for making this available
I been wanting to test the IBM Granite model(s) in a RAG use case scenario and compare it with other open source options. I run private systems and wanted to use the IBM model in gguf versions as my LLM's ands with chat retrieval agents. Much appreciated you making this available to the community. Worked flawlessly to run the queries that I have against a vector DB of PDF's. Still a question of precision with these models for RAG Q&A.
Regards,
Frank R.
Hello! Thank you for your kind words! If you need anything else, please let me know!
Hi Richard,
We hope all is well.
I left you a quick message on Discord as a suggestion. The New IBM Granite models are out as they start their TechXChange Conference kickoff in Las Vegas this week. - https://huggingface.co/collections/ibm-granite/granite-30-models-66fdb59bbb54785c3512114f . Wondering if you could convert these into GGUF format soon. :-) .
Much appreciated,
Madison Digital Service
not to take away from what our host is doing as its appreciated, but if you need it today its not hard to convert them yourself, and you dont need high end hardware to do it. its not like training or anything.
No need today. Would like to have it in my collection of models for our UI and system in the near term.
@madisondigitalservice I put it in queue, hopefully server will survive and quant it soon, have fun using the models !
Thank you Richard!!
@madisondigitalservice server is finally half-alive and biting on granite, hopefully will upload it fully before dying again ...
lol
Thank you. I downloaded a few of them to test out on variety of RAG use cases. I will let the community know based on some test on four of these Granite Models. Also, one of the better LLM's I have evaluated are the Llama Claude 3 its called. I believe its a derivative of Claude 3 LLM by Anthropic.. https://www.anthropic.com/news/claude-3-family
If you come across any of these as updates I would highly recommend for their accuracy and speed. Zips on NVIDIA!
Regards, MDS
thanks, if you need anything else let me know
model fails to load. Errors in reading file format.
dont forget to update your llama-cpp-python, I had the same until I updated the server