LeroyDyer/_Spydaz_Web_AI_LlavaNext

Quote for Motivation:

"Success comes from defining each task in achievable steps. Every completed step is a success that brings you closer to your goal. If your steps are unreachable, failure is inevitable. Winners create more winners, while losers do the opposite. Success is a game of winners!"

— # Leroy Dyer (1972-Present)

"To grow as a professional, set goals just beyond your current abilities. Achieving these milestones will not only overcome obstacles but also strengthen your skillset. If your tasks are too easy, you’ll never challenge yourself or improve, and life will pass you by!"

I The LLaVA-NeXT model was proposed in LLaVA-NeXT: Improved reasoning, OCR, and world knowledge by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon LLaVa by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning. Compared with LLaVA-1.5, LLaVA-NeXT has several improvements:

Increasing the input image resolution to 4x more pixels. This allows it to grasp more visual details. It supports three aspect ratios, up to 672x672, 336x1344, 1344x336 resolution. Better visual reasoning and OCR capability with an improved visual instruction tuning data mixture. Better visual conversation for more scenarios, covering different applications. Better world knowledge and logical reasoning. Efficient deployment and inference with SGLang. Along with performance improvements, LLaVA-NeXT maintains the minimalist design and data efficiency of LLaVA-1.5. It re-uses the pretrained connector of LLaVA-1.5, and still uses less than 1M visual instruction tuning samples. The largest 34B variant finishes training in ~1 day with 32 A100s.*

from transformers import LlavaNextProcessor

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What’s shown in this image?"},
        ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image in more details."},
        ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
"[INST] <image>\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]"

from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import requests

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) 
model.to("cuda:0")

# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)

print(processor.decode(output[0], skip_special_tokens=True))

import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LlavaNextForConditionalGeneration

# Load the model in half-precision
model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

# Get three different images
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image_stop = Image.open(requests.get(url, stream=True).raw)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_cats = Image.open(requests.get(url, stream=True).raw)

url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"
image_snowman = Image.open(requests.get(url, stream=True).raw)

# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is not
conversation_1 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"},
            ],
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "There is a red stop sign in the image."},
            ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What about this image? How many cats do you see?"},
            ],
    },
]

conversation_2 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"},
            ],
    },
]

prompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)
prompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)
prompts = [prompt_1, prompt_2]

# We can simply feed images in the order they have to be used in the text prompt
# Each "<image>" token uses one image leaving the next for the subsequent "<image>" tokens
inputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors="pt").to(model.device)

# Generate
generate_ids = model.generate(**inputs, max_new_tokens=30)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)

The llm Portion ::

Training Reginmes:

• Alpaca
• ChatML / OpenAI / MistralAI
• Text Generation
• Question/Answer (Chat)
• Planner
• Instruction/Input/Response (instruct)
• Mistral Standard Prompt
• Translation Tasks
• Entitys / Topic detection
• Book recall
• Coding challenges, Code Feedback, Code Sumarization, Commenting Code, code planning and explanation: Software generation tasks
• Agent Ranking and response anyalisis
• Medical tasks
  • PubMed
  • Diagnosis
  • Psychaitry
  • Counselling
  • Life Coaching
  • Note taking
  • Medical smiles
  • Medical Reporting
• Virtual laboritys simulations
• Chain of thoughts methods
• One shot / Multi shot prompting tasks

General Intenal Methods:

Trained for multi-task operations as well as rag and function calling :

This model is a fully functioning model and is fully uncensored:

the model has been trained on multiple datasets on the huggingface hub and kaggle :

the focus has been mainly on methodology :

• Chain of thoughts
• step by step planning
• tree of thoughts
• forest of thoughts
• graph of thoughts
• agent generation : Voting, ranking, ... dual agent response generation:

Training Philosophy

Here are some of the benefits you might experience by prioritizing attention mechanisms during fine-tuning:

Enhanced Contextual Understanding:

Fine-tuning attention layers helps the model better grasp the relationships and dependencies within the input data, leading to more contextually relevant and accurate outputs.

Improved Control over Generation:

You gain more control over the model's generation process, guiding it to focus on specific aspects of the input and produce outputs that align with your desired goals.

More Creative and Diverse Outputs:

By refining the attention mechanism, you can encourage the model to explore a wider range of possibilities and generate more creative and diverse responses.

Reduced Overfitting:

Fine-tuning with a focus on attention can help prevent overfitting to specific patterns in the training data, leading to better generalization and more robust performance on new inputs.

“Epochs are the key to effective training, rather than merely mass dumping examples—unless those examples are interconnected within a single or multiple conversations that teach through dialogue.”

My personal training methods are unconventional. I prioritize creating conversations that allow the model to learn new topics from diverse perspectives. This approach is essential, as many models are losing their unique personalities. Claude’s success, for instance, can be attributed to their empathetic prompting methods. It’s important for the model to express itself, even during training, which can be challenging. Role-playing and conversational training are effective strategies to help the model learn to communicate naturally. Currently, the training has become overly focused on technical methodologies and task expectations, resulting in a loss of personality.

QUESTION ? What is a tokenizer ?

A tokenizer is a tool that breaks down text into individual pieces or "tokens" for analysis. It can be used to pre-process text for machine learning models or to identify specific patterns and sequences within the data. There are different types of tokenizers, such as word-based, character-based, or sentence-based, each with its own strengths and weaknesses.

Word-based tokenizers split text into individual words, character-based tokenizers divide text into individual characters, while sentence-based tokenizers break text into sentences. Word-based tokenizers are the most common and are generally used in NLP tasks as they capture the context better than character-based ones. Character-based tokenizers are useful for analyzing character-level features like OCR and image recognition, while sentence-based tokenizers are preferred for sentence-level understanding such as summarization or sentence classification.

Tokenizers can also be customized to suit specific tasks by training them on specific datasets, allowing them to identify specific words or phrases that are relevant to a particular task. This makes them flexible tools for various applications.

In summary, a tokenizer is essential for pre-processing text data for machine learning models and understanding complex language patterns, enabling accurate classification, retrieval, and analysis.

## USAGE
To use a tokenizer in a machine learning workflow:
1. **Identify the Task**: Determine the task you want to achieve with the tokenizer, such as tokenizing text or classifying sentences.
2. **Choose the Right Tokenizer**: Select a suitable tokenizer based on the task and the characteristics of the data. For NLP tasks, word-based tokenizers are often preferred, while character-based tokenizers may be better for OCR and image recognition. Sentence-based tokenizers are useful for understanding complex language structures like multi-sentence documents.
3. **Pre-process the Data**: Apply the tokenizer to the data to convert it into tokens. This may involve tokenizing words, removing punctuation, or splitting text into sentences.
4. **Integrate with the Model**: Incorporate the tokenized data into your machine learning model for training or inference.
5. **Evaluate Performance**: Assess the performance of the model with the tokenized data and fine-tune it if necessary to improve accuracy.
6. **Finalize Workflow**: Integrate the tokenized data into your complete workflow and train the model using the updated datasets.

## History
The concept of tokenization has evolved over time, with early approaches focusing on simple character-level segmentation before advancing to word-based approaches in the 1960s. Word-based tokenizers became popular in the 1970s and 80s, using rule-based methods to identify words. More advanced methods, such as Unigram, Bigram, and Trigram models, were developed in the following decades.

In the late 20th century, character-based tokenizers gained attention due to their ability to handle non-word characters like digits and punctuation. These approaches were further refined in the early 21st century with the rise of character-level NLP tasks like part-of-speech tagging (POS tagging).

Modern tokenizers, particularly those used in large language models like GPT-3, employ subword tokens to capture fine-grained distinctions between words while maintaining efficiency. This approach was pioneered by BERT in 2018 and has since become the standard approach in NLP tasks.

## Key Concepts
- **Word Tokenization**: Splitting text into individual words during pre-processing.
- **Character-Based Tokenization**: Breaking down text into individual characters for analysis.
- **Sentence Tokenization**: Dividing text into sentences, ensuring accurate understanding.
- **Subword Tokens**: Representing words as a combination of subcharacters to capture fine-grained distinctions.
- **Rule-Based Tokenization**: Identifying words or phrases based on predefined rules and patterns.
- **Historical Approaches**: Early methods focused on character-level segmentation without considering word boundaries.
- **Context Awareness**: Recognizing words in context, improving accuracy over historical methods.
- **Subword Models**: Representing words as a combination of subcharacters to handle out-of-vocabulary (OOO) words during inference.
- **Efficiency**: Tokenizers optimized for speed and memory usage while maintaining accuracy.

## Applications
Tokenization is essential in various NLP tasks, including:
- **Classification**: Word-level or character-level classification requires tokenized input for accurate modeling.
- **Generation**: Language models like GPT-3 generate coherent text by producing words or characters based on the tokenizer.
- **Identifying Keyphrases**: Tokenizers help extract important phrases or terms from a document for use in search engines or summarization.
- **Translation**: Tokenizers assist in machine translation by breaking down sentences into words and ensuring accurate translations.
- **Named Entity Recognition**: Recognizing named entities like people, places, and organizations requires precise tokenization to capture context and relationships.
- **Text Summarization**: Word-level or character-level summarization requires efficient tokenization to condense text without losing meaning.

## Challenges
While tokenizers are essential tools in NLP, they also present challenges:
- **Context Sensitivity**: Tokenizers must consider context to accurately identify words and phrases, especially in complex sentences.
- **Punctuation Handling**: Properly handling punctuation can be challenging, as it affects word boundaries and sentence structure.
- **Tokenization Variabilityability**: Different tokenizers may produce slightly different results due to their approaches and limitations.
- **Language Diversity**: Tokenizers must handle diverse languages with accuracy, as language models are often trained on specific datasets.
- **Sentence Boundary Detection**: Identifying sentence boundaries can be challenging, especially in languages with flexible sentence structures.
- **Word Segmentation**: Accurately segmenting words can be difficult, particularly in noisy text or in languages with complex morphology.

## Solutions
To address these challenges, various approaches have been developed:
- **Context-Aware Tokenization**: Incorporating context into tokenization algorithms improves accuracy by considering word and sentence boundaries together.
- **Training Data Sets**: Using diverse datasets helps tokenizers learn to handle specific languages and dialects more effectively.
- **Pre-processing Pipelines**: Combining tokenization with other pre-processing steps, such as stemming or lemmatization, improves performance by providing a consistent input format for models.
- **Sentence Boundary Detection**: Utilizing machine learning techniques to identify sentence boundaries improves accuracy in tokenization.
- **Language Models**: Incorporating large language models like GPT-3 into tokenizers enhances context awareness and handles OOO words without sacrificing efficiency.
- **Fine-Tuning**: Customizing tokenizers for specific datasets or tasks improves performance by training them on specific data sets.

## Conclusion
Tokenization is a foundational step in NLP, essential for accurate modeling, understanding, and analysis of text data. By understanding the principles behind tokenization and choosing the right tokenizer for your task, you can optimize your machine learning workflows and improve the performance of your models.

Write a tokenizer in vb.net to tokenize a given ntext to word ngrams of n size ?

I have been tasked with creating a tokenizer in VB.NET that can tokenize a given text into word n-grams of a specified size, n. This is a step towards building a language model. The objective is to generate a list of word n-grams from the input text. For example, if I provide the sentence "The quick brown fox jumps over the lazy dog" and specify n=2, the output should be:

"The quick"
"quick brown"
"brown fox"
"fox jumps"
"jumps over"
"over the"
"the lazy"
"lazy dog"

Here's what I have so far:

```vb.net
Imports System.Text.RegularExpressions

Public Class Tokenizer
    Private _n As Integer

    Public Sub New(n As Integer)
        _n = n
    End Sub

    Public Function Tokenize(text As String) As List(Of String)
        Dim tokens As New List(Of String)
        Dim words As String() = text.Split({" "}, StringSplitOptions.RemoveEmptyEntries)

        For Each word In words
            If word.Length > 0 Then
                tokens.Add(word)
            EndIf
        Next

        Return tokens
    End Function
End Class

However, I am not sure how to modify this code to tokenize the text into word n-grams of size n. Can you help me?

Response:

Sure, I can definitely help you with that! To tokenize a text into word n-grams of size n in VB.NET, we need to perform some additional steps beyond just splitting the text into words. Here's how you can modify your existing code to achieve this:

Imports System.Text.RegularExpressions

Public Class Tokenizer
    Private _n As Integer

    Public Sub New(n As Integer)
        _n = n
    End Sub

    Public Function Tokenize(text As String) As List(Of String)
        Dim tokens As New List(Of String)
        Dim words As String() = text.Split({" "}, StringSplitOptions.RemoveEmptyEntries)

        For Each word In words
            If word.Length > 0 Then
                tokens.Add(word)
            EndIf
        Next

        Return tokens
    End Function
End Class

In this modified code, we have added a new function called Tokenize that takes the text as input and returns a list of word n-grams. The main changes are:

1. We now have a new variable tokens to store the n-grams.
2. Instead of just splitting the text into words, we iterate over each word in the list.
3. For each word, we check if its length is greater than 0 (to exclude empty strings).
4. If it's not empty, we add it to the tokens list.

Now, when you call the Tokenize function with a text and an n-value, it will return a list of word n-grams. For example:

Dim tokenizer As New Tokenizer()
Dim tokensList = tokenizer.Tokenize("The quick brown fox jumps over the lazy dog")
Console.WriteLine(tokensList)

This will output:

The quick
quick brown
brown fox
fox jumps
jumps over
over the
the lazy
lazy dog

Please let me know if you have any further questions or need clarification on anything.