---
library_name: peft
license: other
base_model: OpenGVLab/InternVL3_5-8B-Pretrained
tags:
- llama-factory
- lora
- generated_from_trainer
model-index:
- name: intern35_8b_lora_expert_chart-102400
  results: []
---

<h2 align="center">
Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization

<br>

[![arXiv](https://img.shields.io/badge/arXiv-2602.04937-b31b1b.svg)](https://www.arxiv.org/pdf/2602.04937)
[![🤗 Model (HuggingFace)](https://img.shields.io/badge/Models-HuggingFace-FFD21E.svg?logo=huggingface&logoColor=yellow)](https://huggingface.co/collections/daviBera/mllms-merging-4-dmo)
[![🤗 Dataset (HuggingFace)](https://img.shields.io/badge/Datasets-HuggingFace-FFD21E.svg?logo=huggingface&logoColor=yellow)](https://huggingface.co/datasets/daviBera/experts_datasets-102400)
[![github](https://img.shields.io/badge/github-repo-blue?logo=github)](https://github.com/BerasiDavide/mLLMs_merging_4_DMO)
</h2>

This is an official checkpoint from the paper: "Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
" ([link](https://www.arxiv.org/pdf/2602.04937)).
See the [official implementation](https://github.com/BerasiDavide/mLLMs_merging_4_DMO) for more information on how to use the models.

# intern35_8b_lora_expert_chart-102400

This model is a fine-tuned version of [OpenGVLab/InternVL3_5-8B-Pretrained-HF](https://huggingface.co/OpenGVLab/InternVL3_5-8B-Pretrained) on a custom dataset with Chart data (~100k samples).

It achieves the following results on the evaluation set:
- Loss: 0.5061

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- total_eval_batch_size: 4
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- training_steps: 800

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.0387        | 0.125 | 100  | 0.6001          |
| 0.9995        | 0.25  | 200  | 0.5526          |
| 1.0029        | 0.375 | 300  | 0.5336          |
| 0.9927        | 0.5   | 400  | 0.5198          |
| 1.0089        | 0.625 | 500  | 0.5116          |
| 0.9469        | 0.75  | 600  | 0.5086          |
| 0.9423        | 0.875 | 700  | 0.5073          |
| 0.9911        | 1.0   | 800  | 0.5061          |


### Framework versions

- PEFT 0.15.2
- Transformers 4.52.4
- Pytorch 2.7.1+cu126
- Datasets 3.6.0
- Tokenizers 0.21.1