README.md

---
license: apache-2.0
title: Tokenizer Compresor
---

# NOS TOKENIZER COMPRESSOR

The NOS (Num Optimized Split) algorithm and the LZ (Lempel-Ziv) algorithm are both data compression algorithms, but they differ in their approach and operation.

### Approach:

NOS: This algorithm focuses on finding the best way to split a text string into smaller substrings, so that the resulting split allows for efficient compression.
LZ: The LZ algorithm is based on the identification and replacement of repetitive sequences in the original text with references to previous occurrences of those sequences.
Chain Division:

NOS: Uses a splitting approach based on finding common substrings between different parts of the text string.
LZ: Looks for repetitive sequences within the original text and replaces them with references to previous occurrences.
### Compression:

NOS: After finding the optimal split, use Huffman encoding to compress the information.
LZ: Identifies and replaces repetitive sequences with references, reducing redundancy in the text and therefore compressing it.

### Coding:

NOS: Encodes information using the Huffman algorithm, which assigns variable length codes to symbols, where more frequent symbols have shorter codes.
LZ: It does not use Huffman coding as part of its main algorithm; however, additional coding techniques can be applied to improve compression efficiency.

### Efficiency:

NOS: The efficiency of NOS largely depends on its ability to split the text string into meaningful parts. The optimal split quality will directly influence the final compression.
LZ: LZ's efficiency is determined by its ability to identify and exploit repetitions in the text. The more redundant the text, the greater the compression you can achieve.
In summary, although both NOS and LZ are compression algorithms, they differ in their approach to achieving compression. While NOS focuses on optimal string splitting and uses Huffman coding, LZ relies on the identification of repetitive sequences to reduce redundancy in the original text.


Determining which algorithm is better, NOS or LZ, depends largely on the type of data being compressed, the compression and decompression requirements, as well as the specific characteristics of the text or data in question. Here are some considerations to help you evaluate which algorithm might be best suited in different contexts:

### Type of data:

NOS: Works well with data that has repetitive patterns but not necessarily repeated identical sequences. It may be more effective with data where redundancy is more related to the structure or distribution of characters than with exact sequences.
LZ: Excels on data with exact or approximate repetitive sequences, such as natural text, text files, and compressed data.
Compression and Decompression Requirements:

US: May be faster to compress due to its direct focus on splitting and Huffman encoding. The speed of decompression will depend on the complexity of the split and the length of the common substrings.
LZ: Compression speed may be slower as it involves searching for repetitive sequences. However, decompression is usually fast, especially if an index table is used for repetitive sequences.

### Compression Efficiency:

NOS: May be more efficient on data with non-trivial patterns that are not easily identifiable by LZ, such as certain types of scientific or structured data.
LZ: Tends to achieve high compression on data with exact repetitions, such as natural text and text files.

### Flexibility and Adaptability:

NOS: May be easier to adapt and customize for specific applications due to its modular approach and reliance on Huffman coding, which can be adjusted based on requirements.
LZ: Although the basic algorithm is robust, it may require additional adjustments to effectively handle different types of data or repeat patterns.
In short, there is no "best" compression algorithm in general; The choice between NOS and LZ depends on several factors, including the specific data being compressed, performance requirements, and system needs. It is important to evaluate each algorithm based on these factors and perform comparative tests to determine which is best suited for a particular application.