BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials

Reference: R. Luu and M.J. Buehler, "BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials," Adv. Science, 2023, DOI: https://doi.org/10.1002/advs.202306724

Abstract: The study of biological materials and bio-inspired materials science is well established; however, surprisingly little knowledge is systematically translated to engineering solutions. To accelerate discovery and guide insights, an open-source autoregressive transformer large language model (LLM), BioinspiredLLM, is reported. The model is finetuned with a corpus of over a thousand peer-reviewed articles in the field of structural biological and bio-inspired materials and can be prompted to recall information, assist with research tasks, and function as an engine for creativity. The model has proven that it is able to accurately recall information about biological materials and is further strengthened with enhanced reasoning ability, as well as with Retrieval-Augmented Generation (RAG) to incorporate new data during generation that can also help to traceback sources, update the knowledge base, and connect knowledge domains. BioinspiredLLM also has shown to develop sound hypotheses regarding biological materials design and remarkably so for materials that have never been explicitly studied before. Lastly, the model shows impressive promise in collaborating with other generative artificial intelligence models in a workflow that can reshape the traditional materials design process. This collaborative generative artificial intelligence method can stimulate and enhance bio-inspired materials design workflows. Biological materials are at a critical intersection of multiple scientific fields and models like BioinspiredLLM help to connect knowledge domains.

from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import infer_auto_device_map

model = AutoModelForCausalLM.from_pretrained('lamm-mit/BioinspiredLLM')
tokenizer = AutoTokenizer.from_pretrained('lamm-mit/BioinspiredLLM')

Variants of the model are included, featuring various GGUF versions for use withm llama.cpp, for instance. In LM Studio, choose the "ChatML" prompt template.

Generate:

device='cuda'
def generate_response (text_input="Biological materials offer amazing",
                      num_return_sequences=1,
                      temperature=1.,  
                      max_new_tokens=127,
                      num_beams=1,
                      top_k = 50,
                      top_p =0.9,repetition_penalty=1.,eos_token_id=2,verbatim=False,
                      exponential_decay_length_penalty_fac=None,
                      ):

    inputs = tokenizer.encode(text_input,  add_special_tokens  =False,  return_tensors ='pt')
    if verbatim:
        print ("Length of input, tokenized: ", inputs.shape, inputs)
    with torch.no_grad():
          outputs = model.generate(input_ids=inputs.to(device), 
                                   max_new_tokens=max_new_tokens,
                                   temperature=temperature, #value used to modulate the next token probabilities.
                                   num_beams=num_beams,
                                   top_k = top_k,
                                   top_p =top_p,
                                   num_return_sequences = num_return_sequences, eos_token_id=eos_token_id,
                                   do_sample =True, 
                                   repetition_penalty=repetition_penalty,
                                  )
    return tokenizer.batch_decode(outputs[:,inputs.shape[1]:].detach().cpu().numpy(), skip_special_tokens=True)

Generation example and prompt template

Prompt template:

<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_message}<|im_end|>\n<|im_start|>assistant....

Sample code:

system_prompt = "You are BioinspiredLLM. You are knowledgeable in biological and bio-inspired materials and provide accurate and qualitative insights about biological materials found in Nature. You are a cautious assistant. You think step by step. You carefully follow instructions."
user_message = "What are hierarchical, biological materials?"

txt =  f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_message}<|im_end|>\n<|im_start|>assistant"

# modulate temperature(0.1-1.0) to adjust 'creativity'
# modulate max_new_tokens to change length of generated response 
output_text=generate_response ( text_input=txt,eos_token_id=2,
                                num_return_sequences=1,
                                repetition_penalty=1.1,
                                top_p=0.95,
                                top_k=50, 
                                temperature=0.1,
                                max_new_tokens=512,
                                verbatim=False, 
                              )

print(output_text)

Dataset: https://onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1002%2Fadvs.202306724&file=advs7235-sup-0002-SuppMat.csv

Performance:

The figure below shows results from knowledge recall evaluation experiments of BioinspiredLLM a) Total scores of each model, Llama 13b-chat (grey), Orca-2 13b (blue), Llama-BioLLM (orange), BioinspiredLLM (light green) and BioinspiredLLM with Retrieval-Augmented Generation (RAG) (dark green) on the 100-question biological materials exam b) Scores on the exam separated by question category: general, specific, numerical, and non-biological. c) Retrieval-Augmented Generation(RAG) method framework and two examples of BioinspiredLLM's response when supplemented using RAG, additionally showing the source the retrieved content traces back to. This method allows tracing the origin of certain knowledge, ideas, or data used as BioinspiredLLM formulates its response.

Retrieval Augmented Generation (RAG)

Example based on Llama Index.

First, set up BioinspiredLMM as custom LLM:

from llama_index.prompts.prompts import SimpleInputPrompt
from llama_index import (
    VectorStoreIndex,
    get_response_synthesizer,)
from llama_index.retrievers import VectorIndexRetriever
from llama_index.query_engine import RetrieverQueryEngine
eos_token=32000

#system_prompt = ""
system_prompt = "You are BioinspiredLLM. You are knowledgeable in biological and bio-inspired materials and provide accurate and qualitative insights about biological materials found in Nature. You are a cautious assistant. You think step by step. You carefully follow instructions."

query_wrapper_prompt = SimpleInputPrompt( "<|im_start|>system\n"+system_prompt+"<|im_end|>\n<|im_start|>user\n{query_str}<|im_end|>\n<|im_start|>assistant")

from llama_index.llms import HuggingFaceLLM
llm_custom = HuggingFaceLLM(context_window=2048,
                    max_new_tokens=300,
                    query_wrapper_prompt=query_wrapper_prompt,
                    stopping_ids=[eos_token, 2],
                    model=model,
                    generate_kwargs={"temperature": 0.1, "do_sample": True,
                    "repetition_penalty":1.1, "top_p":0.95, "top_k":50,  "eos_token_id": [eos_token, 2] , #"max_new_tokens": 1024,
                                    },
                    tokenizer=tokenizer)
llm_custom.model_name='BioinspiredLLM'

Use Chroma database collection (for the purpose of this example it has already been created, load here):

import chromadb
from llama_index import VectorStoreIndex, SimpleDirectoryReader
from chromadb.config import Settings
from llama_index.vector_stores import ChromaVectorStore
from llama_index.storage.storage_context import StorageContext

coll_name="Bioinspired"
coll_path='./Bioinspired_Chroma'    ## PATH TO CHROMA DATABASE

client = chromadb.PersistentClient(path=coll_path)
collection = client.get_collection (name=coll_name,)

db2 = chromadb.PersistentClient(path=coll_path)
chroma_collection = db2.get_or_create_collection(coll_name)
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

chroma_collection.count()

Set up custom LLM service context and vector store indedx:

from llama_index.llms import LlamaCPP
from llama_index import ServiceContext
from llama_index.llms.llama_utils import (
        messages_to_prompt,
        completion_to_prompt,
    )

service_context = ServiceContext.from_defaults(
    llm=llm_custom,
    chunk_size=1024,
    embed_model="local:BAAI/bge-large-en"
)
index = VectorStoreIndex.from_vector_store(
     vector_store,
     service_context=service_context,
)

Set up query engine:

from IPython.display import Markdown, display
query_engine = index.as_query_engine( 
    #response_mode="tree_summarize",
    #response_mode='compact',
    #response_mode='accumulate',
    #streaming=True,
    similarity_top_k=5,
                                    )

question = "Which horn does not have tubules? A) big horn sheep B) pronghorn C) mountain goat"
response = query_engine.query(question)
display(Markdown(f"<b>{response}</b>"))

Alternatively, load new documents, here with all-mpnet-base-v2 embeddings:

from langchain.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-mpnet-base-v2",
)
documents_graph = SimpleDirectoryReader(
            input_files=[
            "./XXXXXXXXXX/XXXXX.pdf", 
                    ]
    ).load_data()
index_doc = VectorStoreIndex.from_documents(documents_graph, service_context=
                                            service_context,
                                            show_progress=True,
                                            embeddings=embeddings,
                                           )

Query:

question="Which rapid prototyping techniques would be useful for creating hierarchical, bio-inspired materials?"

response = index_doc.as_query_engine(service_context=service_context,
    response_mode="tree_summarize",
    similarity_top_k=5,      
    ).query(question, 
    )
print(response)

Reference

Rachel K. Luu, Markus J. Buehler, BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials arxiv.org/abs/2309.08788 and Adv. Science, https://doi.org/10.1002/advs.202306724, 2023

Notes and licenses

BioinspiredLLM is a 13b parameter model. It was fine-tuned based on the Orca-2 model (https://huggingface.co/microsoft/Orca-2-13b), using the LLaMA-2 13b base model. Please refer to LLaMA-2 technical report for details on the model architecture. Details see: https://onlinelibrary.wiley.com/doi/full/10.1002/advs.202306724

Orca 2 is licensed under the Microsoft Research License (https://huggingface.co/microsoft/Orca-2-13b/blob/main/LICENSE).

Llama 2 is licensed under the LLAMA 2 Community License (https://ai.meta.com/llama/license/).

Bias, Risks, and Limitations

Like in all techniques of modeling, there are possibilities of errors. The base models Llama 2 and Orca 2 models were aligned to not spread misinformation and produce safer responses. As a result, BioinspiredLLM has inherited these traits and performs reasonably well in these dimensions. However, it is still of utmost importance for researchers to also verify responses and avoid propagating errors. To minimize risk of mistakes, employing chain-of-thought prompting and RAG methods, as introduced, proves beneficial. Additionally, the system prompt of BioinspiredLLM can be edited to guide context. Further details see the main paper.

BioinspiredLLM, built upon the LLaMA 2 and Orca-2 model family, retains many of its limitations, as well as the common limitations of other large language models or limitation caused by its training process, including:

Data Biases: Large language models, trained on extensive data, can inadvertently carry biases present in the source data. Consequently, the models may generate outputs that could be potentially biased or unfair.

Lack of Contextual Understanding: Despite their impressive capabilities in language understanding and generation, these models exhibit limited real-world understanding, resulting in potential inaccuracies or nonsensical responses.

Lack of Transparency: Due to the complexity and size, large language models can act as “black boxes”, making it difficult to comprehend the rationale behind specific outputs or decisions. We recommend reviewing transparency notes from Azure for more information.

Content Harms: There are various types of content harms that large language models can cause. It is important to be aware of them when using these models, and to take actions to prevent them. It is recommended to leverage various content moderation services provided by different companies and institutions. On an important note, we hope for better regulations and standards from government and technology leaders around content harms for AI technologies in future. We value and acknowledge the important role that research and open source community can play in this direction.

Hallucination: It is important to be aware and cautious not to entirely rely on a given language model for critical decisions or information that might have deep impact as it is not obvious how to prevent these models from fabricating content. Moreover, it is not clear whether small models may be more susceptible to hallucination in ungrounded generation use cases due to their smaller sizes and hence reduced memorization capacities. This is an active research topic and we hope there will be more rigorous measurement, understanding and mitigations around this topic.

Potential for Misuse: Without suitable safeguards, there is a risk that these models could be maliciously used for generating disinformation or harmful content.

This model is solely designed for research settings, and its testing has only been carried out in such environments. It should not be used in downstream applications, as additional analysis is needed to assess potential harm or bias in the proposed application.