README.md

---
license: gemma
language:
- en
pipeline_tag: text-generation
tags:
- Google
- Pytorch
- Gemma2
---
# SandLogic Technologies - Quantized Gemma-2-9b-IT Models

## Model Description

We have quantized the Gemma-2-9b-IT model into three variants:

1. Q5_KM
2. Q4_KM
3. IQ4_XS

These quantized models offer improved efficiency while maintaining performance.

Discover our full range of quantized language models by visiting our [SandLogic Lexicon](https://github.com/sandlogic/SandLogic-Lexicon) GitHub. 
To learn more about our company and services, check out our website at [SandLogic](https://www.sandlogic.com).


## Original Model Information

- **Name**: [Gemma-2-9b-IT](https://huggingface.co/google/gemma-2-9b-it)
- **Developer**: Google
- **Model Type**: Text-to-text, decoder-only large language model
- **Architecture**: Based on Gemini technology
- **Parameters**: 9 billion
- **Training Data**: 8 trillion tokens, including web documents, code, and mathematics
- **Language**: English

## Model Capabilities

Gemma is designed for various text generation tasks, including:

- Question answering
- Summarization
- Reasoning
- Creative writing
- Code generation

The model is lightweight and suitable for deployment in resource-limited environments such as laptops, desktops, or personal cloud infrastructure.

## Use Cases

1. **Text Generation**: Create poems, scripts, code, marketing copy, and email drafts
2. **Chatbots and Conversational AI**: Power customer service interfaces, virtual assistants, and interactive applications
3. **Text Summarization**: Generate concise summaries of text corpora, research papers, or reports

## Model Variants

We offer three quantized versions of the Gemma-2-9b-IT model:

1. **Q5_KM**: 5-bit quantization using the KM method
2. **Q4_KM**: 4-bit quantization using the KM method
3. **IQ4_XS**: 4-bit quantization using the IQ4_XS method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

## Usage


```bash
pip install llama-cpp-python 
```
Please refer to the llama-cpp-python [documentation](https://llama-cpp-python.readthedocs.io/en/latest/) to install with GPU support.

### Basic Text Completion
Here's an example demonstrating how to use the high-level API for basic text completion:

```bash
from llama_cpp import Llama

llm = Llama(
    model_path="./models/7B/llama-model.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm(
    "Q: Name the planets in the solar system? A: ", # Prompt
    max_tokens=32, # Generate up to 32 tokens
    stop=["Q:", "\n"], # Stop generating just before a new question
    echo=False # Don't echo the prompt in the output
)

print(output["choices"][0]["text"])
```

## Download
You can download `Llama` models in `gguf` format directly from Hugging Face using the `from_pretrained` method. This feature requires the `huggingface-hub` package.

To install it, run: `pip install huggingface-hub`

```bash
from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/Gemma-2-9b-it-GGUF",
    filename="*gemma-2-9b-it-IQ4_XS.gguf",
    verbose=False
)
```
By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.


## Input and Output

- **Input**: Text string (e.g., question, prompt, or document to be summarized)
- **Output**: Generated English-language text in response to the input


## License

Gemma 2 License: [Google gemma](https://ai.google.dev/gemma/terms)

## Acknowledgements

We thank Google for developing and releasing the original Gemma model.
Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.
## Contact

For any inquiries or support, please contact us at **[email protected]** or visit our [support page](https://www.sandlogic.com/LingoForge/support).