Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
Paper • 2602.11543 • Published • 6
Architecture and Training Paradigms
This collection showcases VCLab's investigations on new architectures of vision transformers, LLMs, VLMs and their training paradigms.
Note [arXiv 2026] Memory-efficient decentralized LLM pretraining. | Code: https://github.com/zjr2000/SPES
BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
Paper • 2603.09582 • Published • 1Note [CVPR 2026] One-bit QK-attention for ViT/DiT. | Code: https://github.com/EdwardChasel/BinaryAttention
Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion
Paper • 2410.15091 • PublishedNote [ICLR 2025] Structure-aware state fusion for visual Mamba. | Code: https://github.com/EdwardChasel/Spatial-Mamba
Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection
Paper • 2406.10700 • PublishedNote [NeurIPS 2024 Spotlight] Group-free Mamba SSM for 3D detection. | Code: https://github.com/gwenzhang/Voxel-Mamba
UniVS: Unified and Universal Video Segmentation with Prompts as Queries
Paper • 2402.18115 • PublishedNote [CVPR 2024] Unified universal video segmentation. | Code: https://github.com/MinghanLi/UniVS