metadata
library_name: peft
license: other
base_model: OpenGVLab/InternVL3_5-8B-Pretrained
tags:
- llama-factory
- lora
- generated_from_trainer
model-index:
- name: intern35_8b_lora_expert_chart-102400
results: []
Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
This is an official checkpoint from the paper: "Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization " (link). See the official implementation for more information on how to use the models.
intern35_8b_lora_expert_chart-102400
This model is a fine-tuned version of OpenGVLab/InternVL3_5-8B-Pretrained-HF on a custom dataset with Chart data (~100k samples).
It achieves the following results on the evaluation set:
- Loss: 0.5061
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- total_eval_batch_size: 4
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- training_steps: 800
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.0387 | 0.125 | 100 | 0.6001 |
| 0.9995 | 0.25 | 200 | 0.5526 |
| 1.0029 | 0.375 | 300 | 0.5336 |
| 0.9927 | 0.5 | 400 | 0.5198 |
| 1.0089 | 0.625 | 500 | 0.5116 |
| 0.9469 | 0.75 | 600 | 0.5086 |
| 0.9423 | 0.875 | 700 | 0.5073 |
| 0.9911 | 1.0 | 800 | 0.5061 |
Framework versions
- PEFT 0.15.2
- Transformers 4.52.4
- Pytorch 2.7.1+cu126
- Datasets 3.6.0
- Tokenizers 0.21.1