FatimahEmadEldin/ytasl-t5-base

YouTube-ASL (YT-ASL) Fine-Tuned T5-Base for Sign Language Translation

This model is a T5-base sequence-to-sequence model fine-tuned on the YouTube-ASL dataset for end-to-end Sign Language Translation (SLT). It directly maps continuous 208-dimensional MediaPipe Holistic pose landmarks to English text.

Architecture

• Base Architecture: google-t5/t5-base (220M parameters)
• Input Dimension: 208 (MediaPipe Holistic keypoints: pose + hands)
• Visual Projection: Linear projection (208 -> 768) with GELU activation
• Training Strategy: Pretrained / Fine-tuned on YouTube-ASL with Adafactor optimizer

Usage

import torch
from transformers import AutoModel, AutoTokenizer

# Load model and tokenizer
repo_id = "FatimahEmadEldin/ytasl-t5-base"
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo_id)

# Example input pose sequence [batch_size, sequence_length, 208]
batch_size, seq_len, feature_dim = 1, 100, 208
dummy_features = torch.randn(batch_size, seq_len, feature_dim)
attention_mask = torch.ones(batch_size, seq_len)

# Generate English translation
model.eval()
with torch.no_grad():
    output_ids = model.generate(features=dummy_features, attention_mask=attention_mask)
    translation = tokenizer.decode(output_ids[0], skip_special_tokens=True)
    print("Translation:", translation)

Citation

If you use this model, please cite the YouTube-ASL research and the Saudi Sign Language translation benchmark.