Aligner

README.md

@@ -1,199 +1,525 @@
 ---
 library_name: transformers
-tags: []
+license: apache-2.0
+pipeline_tag: image-text-to-text
+language:
+  - en
+  - de
+  - fr
+  - es
+  - it
+  - pt
+  - hi
+  - ja
+  - ko
+  - zh
+  - ar
+tags:
+  - vision
+  - multimodal
+  - conversational
+  - multilingual
+  - native-resolution
 ---
 
-# Model Card for Model ID
+# North Micro Vision Instruct
+![North-Micro-Vision_Hero](https://cdn-uploads.huggingface.co/production/uploads/66d732effe6684fc16b12c28/qQQSd5Pldz30hvGMhNMq_.png)
 
-<!-- Provide a quick summary of what the model is/does. -->
+North Micro Vision Instruct is a 2.4B-parameter open-weight vision-language model with native-resolution image support, released under the Apache 2.0 license. It is designed as a compact foundation for prototyping, task-specific fine-tuning, and specialized multimodal applications.
 
+Developed by [Cohere](https://cohere.com/).
 
+> **Technical deep dive:** Read the [North Micro Vision technical blog post](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) for architecture, training, and evaluation details.
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
+## Highlights
 
-#### Testing Data
+- Native-resolution image processing that preserves aspect ratios and fine visual detail.
+- Broad image-understanding capabilities across VQA, captioning, grounding, OCR, charts, and documents.
+- Multilingual and multi-image support.
+- Compact 2.4B-parameter scale suited to customization and deployment experimentation.
+- Apache 2.0-licensed model weights.
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+## Model Details
+| Property | Value |
+| --- | --- |
+| Model ID | `CohereLabs/North-Micro-Vision-Instruct` |
+| Total parameters | 2.4B |
+| Language model | 2B parameters |
+| Vision encoder | 400M parameters; custom-trained starting from [SigLIP 2 SO400M](https://huggingface.co/google/siglip2-so400m-patch16-384) |
+| Inputs | Interleaved text and images |
+| Output | Text |
+| Languages | English, German, French, Spanish, Italian, Portuguese, Hindi, Japanese, Korean, Chinese, Arabic, and more |
+| Tokenizer vocabulary size | 262,144 |
+| LM Backbone context window | 128K tokens |
+| Multimodal training context | 8K tokens |
+| Checkpoint precision | bfloat16 |
+| License | Apache 2.0 |
+
+The language backbone supports a 128K-token context window, but the validated operating range for multimodal prompts is up to 8K tokens. Longer multimodal contexts may rely on extrapolation and have not been benchmarked.
+
+## Quickstart
+
+### Installation
+
+Install [PyTorch](https://pytorch.org/get-started/locally/) for your platform first. North Micro Vision requires Transformers 5.16.0, together with `accelerate` for automatic device placement and Pillow for image loading. Until Transformers 5.16.0 is released, install the runtime dependencies and Transformers from source:
+
+```bash
+uv pip install accelerate pillow
+uv pip install "git+https://github.com/huggingface/transformers.git"
+```
+
+Once Transformers 5.16.0 is available on PyPI, install the released package with:
+
+```bash
+uv pip install accelerate pillow "transformers==5.16.0"
+```
+
+Flash Attention 2 is optional. On supported CUDA systems, install it with:
+
+```bash
+uv pip install flash-attn --no-build-isolation
+```
+
+If you do not use `uv`, replace `uv pip` with `pip` in the commands above.
+
+### Transformers
+
+The following example loads an image from a URL and asks the model to describe it. Prompts can interleave text with one or more images; for text-only prompts, omit the image entries.
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "CohereLabs/North-Micro-Vision-Instruct"
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+)
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    dtype="auto",
+    device_map="auto",
+)
+
+# To enable Flash Attention 2, load the model with the following settings:
+# model = AutoModelForImageTextToText.from_pretrained(
+#     model_id,
+#     dtype=torch.bfloat16,
+#     attn_implementation="flash_attention_2",
+#     device_map="auto",
+# )
+
+image_url = "https://cdn-uploads.huggingface.co/production/uploads/66d732effe6684fc16b12c28/Io_5OCmftsmH-n158ZtPs.png"
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": image_url},
+            {"type": "text", "text": "What do you see?"},
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=128,
+    do_sample=True,
+    temperature=0.7,
+    top_p=0.8,
+    top_k=20,
+)
+
+generated_ids = [
+    output_ids[len(input_ids) :]
+    for input_ids, output_ids in zip(inputs.input_ids, outputs)
+]
+response = processor.batch_decode(
+    generated_ids,
+    skip_special_tokens=True,
+    clean_up_tokenization_spaces=False,
+)[0]
+print(response)
+```
+
+The example uses the recommended Transformers sampling settings. For deterministic output, set `do_sample=False` and omit `temperature`, `top_p`, and `top_k`.
+
+### Grounding Coordinates
+
+Bounding boxes are returned as `[x1, y1, x2, y2]` on a normalized 0–1000 scale. Map them back to the original image by scaling each axis:
+
+```python
+x1_px = x1 / 1000 * image_width
+y1_px = y1 / 1000 * image_height
+x2_px = x2 / 1000 * image_width
+y2_px = y2 / 1000 * image_height
+```
+
+### vLLM
+
+Public vLLM support is coming soon. Until it is available, use Transformers as shown above. The recommended vLLM settings will be:
+
+```python
+temperature = 0.7
+top_p = 0.8
+top_k = 20
+min_p = 0.0
+presence_penalty = 1.5
+repetition_penalty = 1.0
+```
+
+## Intended Use
+
+North Micro Vision Instruct is intended for research and development use cases such as:
+
+- Prototyping and task-specific fine-tuning.
+- General visual question answering and image captioning.
+- Multilingual and multi-image understanding.
+- Visual grounding and spatial understanding.
+- OCR, chart and document understanding, and structured information extraction.
+
+## Limitations
+
+- The model is intended as a compact foundation for customization rather than a replacement for larger general-purpose chat assistants.
+- It is not a reasoning model and has limited math and code-generation capabilities.
+- Tool calling and agentic workflows are not supported.
+- System prompts are not recommended because the model was not trained with them, although the chat template accepts the `system` role.
+- Multimodal training used an 8K-token context; longer contexts have not been validated.
+- Native-resolution inputs can increase memory use and latency as image dimensions grow.
+
+## Benchmark Results
+
+The complete comparison is provided below. We ran vision-language and text-only evaluations with [VLMEvalKit](https://github.com/open-compass/vlmevalkit), capping generation at 1,024 tokens; see the [technical blog post](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) for the full methodology.
+
+<table>
+    <thead>
+        <tr>
+            <th></th>
+            <th style="font-weight: bold; background-color: rgba(127, 127, 127, 0.08);">North-Micro-Vision-Instruct</th>
+            <th>Ministral-3-3B-Instruct</th>
+            <th>LFM2.5-VL-1.6B</th>
+            <th>Phi-3.5-vision-instruct</th>
+            <th>Gemma-4-E2B-it</th>
+            <th>Qwen3-VL-2B-Instruct</th>
+            <th>Qwen3.5-2B-Instruct</th>
+            <th>SmolVLM2.2B</th>
+        </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">Size</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">2.4B</td>
+            <td>3.8B</td>
+            <td>1.6B</td>
+            <td>4.2B</td>
+            <td>5.1B</td>
+            <td>2.2B</td>
+            <td>2.1B</td>
+            <td>2.2B</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">License</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">Apache 2.0</td>
+            <td>Apache 2.0</td>
+            <td>LFM v1.0</td>
+            <td>MIT</td>
+            <td>Apache 2.0</td>
+            <td>Apache 2.0</td>
+            <td>Apache 2.0</td>
+            <td>Apache 2.0</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">General VQA</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MMBench<sub>DEV_EN_V11</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.687</td>
+            <td>0.692</td>
+            <td>0.696</td>
+            <td>0.731</td>
+            <td>0.693</td>
+            <td>0.744</td>
+            <td>0.760</td>
+            <td>0.674</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MMStar</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.518</td>
+            <td>0.531</td>
+            <td>0.508</td>
+            <td>0.495</td>
+            <td>0.529</td>
+            <td>0.506</td>
+            <td>0.614</td>
+            <td>0.460</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">RealWorldQA</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.622</td>
+            <td>0.583</td>
+            <td>0.642</td>
+            <td>0.580</td>
+            <td>0.507</td>
+            <td>0.646</td>
+            <td>0.693</td>
+            <td>0.567</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">GQA<sub>TestDev_Balanced</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.574</td>
+            <td>0.544</td>
+            <td>0.395</td>
+            <td>0.650</td>
+            <td>0.387</td>
+            <td>0.572</td>
+            <td>0.539</td>
+            <td>0.000<sup>&Dagger;</sup></td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">Multilingual</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MTL<sub>MMBench_DEV</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.636</td>
+            <td>0.674</td>
+            <td>0.623</td>
+            <td>0.619</td>
+            <td>0.648</td>
+            <td>0.664</td>
+            <td>0.669</td>
+            <td>0.454</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MMMB</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.728</td>
+            <td>0.734</td>
+            <td>0.717</td>
+            <td>0.686</td>
+            <td>0.743</td>
+            <td>0.723</td>
+            <td>0.745</td>
+            <td>0.577</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">Multi-image</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">BLINK</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.527</td>
+            <td>0.471</td>
+            <td>0.484</td>
+            <td>0.561</td>
+            <td>0.468</td>
+            <td>0.514</td>
+            <td>0.563</td>
+            <td>0.420</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">Chart / Document / OCR</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">ChartQA<sub>Test</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.808</td>
+            <td>0.791</td>
+            <td>0.739</td>
+            <td>0.821</td>
+            <td>0.422</td>
+            <td>0.693</td>
+            <td>0.775</td>
+            <td>0.682</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">DocVQA<sub>VAL</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.921</td>
+            <td>0.896</td>
+            <td>0.877</td>
+            <td>0.860</td>
+            <td>0.732</td>
+            <td>0.825</td>
+            <td>0.926</td>
+            <td>0.799</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">InfoVQA<sub>VAL</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.652</td>
+            <td>0.589</td>
+            <td>0.627</td>
+            <td>0.561</td>
+            <td>0.380</td>
+            <td>0.622</td>
+            <td>0.731</td>
+            <td>0.383</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">OCRBench<sub>v2_en</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.367</td>
+            <td>0.414</td>
+            <td>0.415</td>
+            <td>0.339</td>
+            <td>0.435</td>
+            <td>0.417</td>
+            <td>0.481</td>
+            <td>0.304</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">OCRBench</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.792</td>
+            <td>0.735</td>
+            <td>0.802</td>
+            <td>0.642</td>
+            <td>0.719</td>
+            <td>0.751</td>
+            <td>0.861</td>
+            <td>0.727</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">AI2D_TEST</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.775</td>
+            <td>0.741</td>
+            <td>0.728</td>
+            <td>0.790</td>
+            <td>0.712</td>
+            <td>0.713</td>
+            <td>0.752</td>
+            <td>0.697</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">CharXiv<sub>DQ</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.600</td>
+            <td>0.766</td>
+            <td>0.516</td>
+            <td>0.637</td>
+            <td>0.751</td>
+            <td>0.595</td>
+            <td>0.761</td>
+            <td>0.482</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">STEM</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MMMU<sub>DEV_VAL</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.329</td>
+            <td>0.508</td>
+            <td>0.380</td>
+            <td>0.432</td>
+            <td>0.477</td>
+            <td>0.379</td>
+            <td>0.474</td>
+            <td>0.399</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">Grounding / Counting</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">RefCOCO<sub>avg</sub><sup>&dagger;</sup></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.732</td>
+            <td>0.317</td>
+            <td>0.581</td>
+            <td>0.451</td>
+            <td>0.084</td>
+            <td>0.304</td>
+            <td>0.785</td>
+            <td>0.018</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">CountBench</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.725</td>
+            <td>0.737</td>
+            <td>0.910</td>
+            <td>0.645</td>
+            <td>0.534</td>
+            <td>0.848</td>
+            <td>0.805</td>
+            <td>0.764</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">Robustness / Hallucination</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">HallusionBench</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.615</td>
+            <td>0.652</td>
+            <td>0.601</td>
+            <td>0.585</td>
+            <td>0.598</td>
+            <td>0.673</td>
+            <td>0.655</td>
+            <td>0.600</td>
+        </tr>
+        <tr>
+            <th colspan="9" style="text-align: left; background-color: rgba(127, 127, 127, 0.16);">Text</th>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MMLU<sub>test</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.504</td>
+            <td>0.660</td>
+            <td>0.464</td>
+            <td>0.355</td>
+            <td>0.692</td>
+            <td>0.630</td>
+            <td>0.543</td>
+            <td>0.084</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">MMLU-Pro<sub>test</sub></th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.307</td>
+            <td>0.475</td>
+            <td>0.199</td>
+            <td>0.286</td>
+            <td>0.441</td>
+            <td>0.428</td>
+            <td>0.298</td>
+            <td>0.099</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">Multi-If</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.373</td>
+            <td>0.470</td>
+            <td>0.443</td>
+            <td>0.304</td>
+            <td>0.687</td>
+            <td>0.523</td>
+            <td>0.464</td>
+            <td>0.236</td>
+        </tr>
+        <tr>
+            <th style="text-align: left; font-weight: normal;">IFEval</th>
+            <td style="background-color: rgba(127, 127, 127, 0.08);">0.749</td>
+            <td>0.725</td>
+            <td>0.776</td>
+            <td>0.543</td>
+            <td>0.869</td>
+            <td>0.734</td>
+            <td>0.679</td>
+            <td>0.501</td>
+        </tr>
+    </tbody>
+</table>
+<p><small><sup>&dagger;</sup> Averaged over RefCOCO_val, RefCOCO_testA, RefCOCO_testB, RefCOCO+_val, RefCOCO+_testA, RefCOCO+_testB, RefCOCOg_val, RefCOCOg_test.</small></p>
+<p><small><sup>&Dagger;</sup> SmolVLM2.2B's GQA output was scored as 0.000 under VLMEvalKit's answer-extraction rules.</small></p>
+
+
+
+## Citation
+
+```bibtex
+@misc{cohere_north_micro_vision_instruct,
+    title = {{North Micro Vision}: A 2.4B Native-Resolution Vision-Language Model},
+    url = {https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct},
+    author = {{Team Cohere}},
+    month = {August},
+    year = {2026}
+}
+```
+
+## Contact
+
+For errors or questions about this model card, contact [Cohere Labs](mailto:labs@cohere.com).

config.json

@@ -1,27 +1,72 @@
 {
+  "transformers_version": "5.15.0.dev0",
   "architectures": [
     "CohereCompassForConditionalGeneration"
   ],
+  "output_hidden_states": false,
+  "return_dict": true,
   "dtype": "bfloat16",
-  "eos_token_id": 255001,
-  "fusion_config": {
-    "patch_embeddings": true
+  "chunk_size_feed_forward": 0,
+  "is_encoder_decoder": false,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
   },
-  "image_token_id": 255031,
-  "model_type": "cohere_compass",
-  "pad_token_id": 0,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "problem_type": null,
   "text_config": {
-    "attention_bias": false,
-    "attention_dropout": 0.0,
-    "bos_token_id": 2,
+    "architectures": null,
+    "output_hidden_states": false,
+    "return_dict": true,
     "dtype": "bfloat16",
-    "eos_token_id": 255001,
-    "head_dim": 128,
-    "hidden_act": "silu",
+    "chunk_size_feed_forward": 0,
+    "is_encoder_decoder": false,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "problem_type": null,
+    "vocab_size": 262144,
     "hidden_size": 2048,
-    "initializer_range": 0.02,
     "intermediate_size": 6144,
+    "logit_scale": 0.25,
+    "num_hidden_layers": 28,
+    "num_attention_heads": 16,
+    "num_key_value_heads": 8,
+    "hidden_act": "silu",
+    "max_position_embeddings": 500000,
+    "initializer_range": 0.02,
     "layer_norm_eps": 1e-05,
+    "use_cache": true,
+    "pad_token_id": 0,
+    "bos_token_id": 2,
+    "eos_token_id": 255001,
+    "tie_word_embeddings": true,
+    "rope_parameters": {
+      "sliding_attention": {
+        "mrope_interleaved": true,
+        "mrope_section": [
+          24,
+          20,
+          20
+        ],
+        "rope_type": "default",
+        "rope_theta": 50000
+      },
+      "full_attention": null,
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    },
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "sliding_window": 4096,
     "layer_types": [
       "sliding_attention",
       "sliding_attention",
@@ -52,60 +97,62 @@
       "sliding_attention",
       "full_attention"
     ],
-    "logit_scale": 0.25,
-    "max_position_embeddings": 500000,
-    "model_type": "cohere_compass_text",
-    "num_attention_heads": 16,
-    "num_hidden_layers": 28,
-    "num_key_value_heads": 8,
-    "pad_token_id": 0,
-    "pooling": null,
-    "rope_parameters": {
-      "full_attention": null,
-      "rope_theta": 10000.0,
-      "rope_type": "default",
-      "sliding_attention": {
-        "mrope_interleaved": true,
-        "mrope_section": [
-          24,
-          20,
-          20
-        ],
-        "rope_theta": 50000,
-        "rope_type": "default"
-      }
-    },
     "score_shift_a": null,
     "score_shift_b": null,
-    "sliding_window": 4096,
-    "tie_word_embeddings": true,
-    "use_cache": true,
-    "vocab_size": 262144
+    "pooling": null,
+    "head_dim": 128,
+    "_name_or_path": "",
+    "model_type": "cohere_compass_text",
+    "output_attentions": false
   },
-  "tie_word_embeddings": true,
-  "transformers_version": "5.16.0.dev0",
-  "video_token_id": 255032,
   "vision_config": {
+    "architectures": null,
+    "output_hidden_states": false,
+    "return_dict": true,
+    "dtype": "bfloat16",
+    "chunk_size_feed_forward": 0,
+    "is_encoder_decoder": false,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "problem_type": null,
+    "depth": 27,
+    "hidden_size": 1152,
+    "hidden_act": "gelu_pytorch_tanh",
+    "intermediate_size": 4304,
+    "num_heads": 16,
+    "in_channels": 3,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2,
+    "out_hidden_size": 2048,
+    "num_position_embeddings": 2304,
     "deepstack_visual_indexes": [
       8,
       16,
       24
     ],
-    "depth": 27,
-    "dtype": "bfloat16",
-    "hidden_act": "gelu_pytorch_tanh",
-    "hidden_size": 1152,
-    "in_channels": 3,
     "initializer_range": 0.02,
-    "intermediate_size": 4304,
+    "_name_or_path": "",
     "model_type": "cohere_compass_vision",
-    "num_heads": 16,
-    "num_position_embeddings": 2304,
-    "out_hidden_size": 2048,
-    "patch_size": 16,
-    "spatial_merge_size": 2,
-    "temporal_patch_size": 2
+    "output_attentions": false
+  },
+  "fusion_config": {
+    "patch_embeddings": true
   },
+  "image_token_id": 255031,
+  "video_token_id": 255032,
+  "vision_start_token_id": 255028,
   "vision_end_token_id": 255029,
-  "vision_start_token_id": 255028
-}
+  "tie_word_embeddings": true,
+  "_name_or_path": "",
+  "eos_token_id": 255001,
+  "pad_token_id": 0,
+  "model_type": "cohere_compass",
+  "output_attentions": false
+}

generation_config.json

@@ -1,12 +1,12 @@
 {
   "bos_token_id": 2,
-  "do_sample": true,
   "eos_token_id": [
     255001
   ],
   "pad_token_id": 0,
+  "do_sample": true,
   "temperature": 0.7,
-  "top_k": 20,
   "top_p": 0.8,
-  "transformers_version": "5.16.0.dev0"
+  "top_k": 20
 }
+

preprocessor_config.json

@@ -0,0 +1,37 @@
+{
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": false,
+  "device": null,
+  "disable_grouping": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_pad": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "input_data_format": null,
+  "merge_size": 2,
+  "pad_size": null,
+  "patch_size": 16,
+  "processor_class": "CohereCompassProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_tensors": null,
+  "size": {
+    "longest_edge": 16777216,
+    "shortest_edge": 65536
+  },
+  "temporal_patch_size": 2,
+  "image_processor_type": "CohereCompassImageProcessor"
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "additional_special_tokens": [
+    "<|VISION_START|>",
+    "<|IMAGE_PAD|>",
+    "<|VISION_END|>",
+    "<|VISION_PAD|>",
+    "<|VIDEO_PAD|>"
+  ],
+  "bos_token": {
+    "content": "<BOS_TOKEN>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|END_OF_TURN_TOKEN|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<PAD>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<UNK>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4594953eca27aa1122252e8a61a91a5b4941c92aba78d3811c8fe9b1b95d7f68
-size 19550475
+oid sha256:6fcc5292908e0c8ad1400c67fe9413825c656486fe4190dce7d8d62c38bfbbf6
+size 19550662

tokenizer_config.json

@@ -1,29 +1,333 @@
 {
+  "add_bos_token": false,
+  "add_eos_token": false,
   "add_prefix_space": false,
-  "backend": "tokenizers",
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<MASK_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<BOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<EOS_TOKEN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<UNK>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255000": {
+      "content": "<|START_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255001": {
+      "content": "<|END_OF_TURN_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255002": {
+      "content": "<|USER_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255003": {
+      "content": "<|CHATBOT_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255004": {
+      "content": "<|SYSTEM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255005": {
+      "content": "<|NEW_FILE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255006": {
+      "content": "<|BEGINNING_OF_PREFIX_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255007": {
+      "content": "<|BEGINNING_OF_MIDDLE_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255008": {
+      "content": "<|BEGINNING_OF_SUFFIX_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255009": {
+      "content": "<|END_OF_MIDDLE_FIM_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255010": {
+      "content": "<|START_THINKING|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255011": {
+      "content": "<|END_THINKING|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255012": {
+      "content": "<|START_RESPONSE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255013": {
+      "content": "<|END_RESPONSE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255014": {
+      "content": "<|START_ACTION|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255015": {
+      "content": "<|END_ACTION|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255016": {
+      "content": "<|START_TOOL_RESULT|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255017": {
+      "content": "<|END_TOOL_RESULT|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255018": {
+      "content": "<|USER_0_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255019": {
+      "content": "<|USER_1_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255020": {
+      "content": "<|USER_2_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255021": {
+      "content": "<|USER_3_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255022": {
+      "content": "<|USER_4_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255023": {
+      "content": "<|USER_5_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255024": {
+      "content": "<|USER_6_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255025": {
+      "content": "<|USER_7_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255026": {
+      "content": "<|USER_8_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255027": {
+      "content": "<|USER_9_TOKEN|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255028": {
+      "content": "<|VISION_START|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255029": {
+      "content": "<|VISION_END|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255030": {
+      "content": "<|VISION_PAD|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255031": {
+      "content": "<|IMAGE_PAD|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255032": {
+      "content": "<|VIDEO_PAD|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|VISION_START|>",
+    "<|IMAGE_PAD|>",
+    "<|VISION_END|>",
+    "<|VISION_PAD|>",
+    "<|VIDEO_PAD|>"
+  ],
   "bos_token": "<BOS_TOKEN>",
   "clean_up_tokenization_spaces": false,
-  "cls_token": "<CLS>",
   "eos_token": "<|END_OF_TURN_TOKEN|>",
-  "errors": "replace",
+  "extra_special_tokens": {},
   "image_token": "<|IMAGE_PAD|>",
-  "is_local": true,
   "legacy": true,
-  "local_files_only": false,
-  "mask_token": "<MASK_TOKEN>",
   "max_pixels": 3868706,
+  "merges_file": null,
   "min_pixels": 16384,
   "model_max_length": 1000000000000000019884624838656,
-  "model_specific_special_tokens": {
-    "image_token": "<|IMAGE_PAD|>",
-    "video_token": "<|VIDEO_PAD|>",
-    "vision_end_token": "<|VISION_END|>",
-    "vision_start_token": "<|VISION_START|>"
-  },
   "pad_token": "<PAD>",
   "padding_side": "right",
   "processor_class": "CohereCompassProcessor",
-  "sep_token": "<SEP>",
   "sp_model_kwargs": {},
   "spaces_between_special_tokens": false,
   "tokenizer_class": "CohereTokenizer",
@@ -31,5 +335,6 @@
   "use_default_system_prompt": false,
   "video_token": "<|VIDEO_PAD|>",
   "vision_end_token": "<|VISION_END|>",
-  "vision_start_token": "<|VISION_START|>"
+  "vision_start_token": "<|VISION_START|>",
+  "vocab_file": null
 }

video_preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+  "size": {
+    "longest_edge": 25165824,
+    "shortest_edge": 4096
+  },
+  "patch_size": 16,
+  "temporal_patch_size": 2,
+  "merge_size": 2,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "CohereCompassProcessor",
+  "video_processor_type": "CohereCompassVideoProcessor"
+}