VideoChat3-4B

VideoChat3-4B is the 4B-parameter release of VideoChat3, a fully open, efficient, and generalist Video Multimodal Large Language Model for video understanding. The model is trained with a scalable video data synthesis and curation pipeline covering general, long-form, and streaming video scenarios. VideoChat3 achieves strong performance across mainstream video understanding benchmarks and surpasses prior open-source models at comparable or larger parameter scales, while releasing model weights, training code, training strategy, and training datasets to support reproducible open research.

Installation

pip install torch transformers accelerate qwen-vl-utils
pip install decord opencv-python-headless

For faster inference, FlashAttention 2 can be used if it is correctly installed in your environment:

pip install flash-attn --no-build-isolation

Quick Start

The following examples show how to run image and video inputs. For a complete runnable script, please refer to demo_vc3.py.

Image/Video Inference

from transformers import AutoModelForCausalLM, AutoProcessor
from qwen_vl_utils import process_vision_info

model_id = "MCG-NJU/VideoChat3-4B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

# image
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "/path/to/image.jpg"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
# video
# messages = [
#     {
#         "role": "user",
#         "content": [
#             {"type": "video", "video": "/path/to/video.mp4"},
#             {"type": "text", "text": "What happens in this video?"},
#         ],
#     }
# ]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

images, videos, video_kwargs = process_vision_info(
    messages,
    image_patch_size=14,
    return_video_kwargs=True,
    return_video_metadata=True,
)
inputs = processor(
    text=text,
    images=images,
    videos=videos,
    do_resize=False,
    return_tensors="pt",
    **(video_kwargs or {}),
)
inputs = inputs.to(model.device)
if hasattr(model, "dtype"):
    inputs = inputs.to(model.dtype)
generated_ids = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,
)
generated_ids = [
    output_ids[len(input_ids):]
    for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.tokenizer.batch_decode(
    generated_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text[0])

Online Proactive Response

VideoChat3-4B supports proactive response for streaming video understanding. The proactive response protocol uses three explicit states:

• </Silence>: the current frame is irrelevant to the user question, or no meaningful event has started yet.
• </Standby>: a relevant event is unfolding, but the model needs more future frames before giving a reliable answer. When </Standby> is predicted, the next frame is processed with a larger visual budget.
• </Response>: the model has collected enough visual evidence and provides the answer.

Example usage with the provided proactive demo:

python demo_vc3_proactive.py \
  --model MCG-NJU/VideoChat3-4B \
  --video /path/to/video.mp4 \
  --question "your question"