project · 2024 · Master's Thesis · RPTU Kaiserslautern / Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)

ASL-LLM: Text to 3D Sign Language Motion

Master's thesis at RPTU Kaiserslautern: an end-to-end system that translates open-vocabulary text into expressive 3D American Sign Language motion, including body, hands, and facial expressions, using VQ-VAE motion encoding and a GPT-based autoregressive transformer.

Sign language is a primary communication medium for millions of people with hearing impairments, yet automated sign language generation remains underdeveloped. Most existing systems rely on gloss annotations as an intermediary, which constrains them to predefined vocabularies and produces static, unnatural outputs. This thesis eliminates that bottleneck entirely: ASL-LLM translates open-vocabulary natural language text directly into expressive 3D sign language motion, including coordinated body, hand, and facial expressions, with no gloss step in the pipeline.

Dataset construction

Sign language generation is bottlenecked by data. No ready-made 3D motion dataset existed for this task, so a custom one was built from scratch. Starting from How2Sign (a large-scale continuous American Sign Language video dataset), a custom extraction pipeline was designed:

• YOLOv5 detects the signer in each video frame
• SMPL-X (via the OSX framework) recovers the full 3D body mesh per frame
• A custom joint reduction schema collapses the raw 64-joint output to 52 joints: 11 body, 15 per hand, 1 jaw, and 10 facial expression parameters

The reduction was deliberate. Sign language meaning lives in the upper body and hands. Keeping leg joints inflates the feature space and degrades model focus. The final representation is 136 dimensions per frame, down from 169, retaining everything that matters for sign communication.

The resulting dataset: 31,048 training samples, 1,739 validation, 2,343 test, totalling 35,130 sentence-level clips, each frame-synchronized with an English text transcription.

Two-stage architecture

The core insight is to separate the two hard sub-problems: how to represent motion compactly, and how to generate it from language.

             Text input
                │
                ▼
┌──────────────────────────────────┐
│   CLIP Text Encoder              │
│   (semantic text embeddings)     │
└───────────────┬──────────────────┘
                │
                ▼
┌──────────────────────────────────┐
│   GPT-based Transformer          │
│   (autoregressive token pred.)   │
│   9 layers · 16 attention heads  │
│   causal + cross-conditional     │
│   attention                      │
└───────────────┬──────────────────┘
                │  motion token sequence
                ▼
┌──────────────────────────────────┐
│   VQ-VAE Decoder                 │
│   (codebook: 1024 entries x      │
│    1024 dims)                    │
└───────────────┬──────────────────┘
                │
                ▼
    3D sign language motion
    (body + hands + face)

Stage 1: VQ-VAE motion encoder

A 1D convolutional encoder-decoder with residual blocks maps continuous 3D motion sequences to a discrete codebook of 1024 tokens. Two training techniques prevent codebook collapse, which is where only a small fraction of entries are ever used:

• Exponential Moving Average (EMA) updates codebook entries smoothly, preventing abrupt jumps
• Code Reset reactivates dead entries, ensuring the full codebook stays utilized

The VQ-VAE is pretrained independently before the second stage. At inference, the decoder reconstructs full 3D motion from any sequence of tokens.

Stage 2: GPT-based transformer

A transformer with causal and cross-conditional self-attention takes CLIP text embeddings and autoregressively predicts the sequence of VQ-VAE motion tokens one step at a time. Each prediction is conditioned on both the text and all previously generated tokens. An explicit End token terminates each segment and the overall sequence, removing the need for a separate sequence-length predictor. The model was trained with corrupted sequence augmentation to improve robustness at inference.

Results

VQ-VAE:

• Motion Matching Score: Generated 24.91 vs Real 24.85 (generated motion is nearly indistinguishable from real motion on this metric)
• FID: 22.47 (low distance between generated and real motion distributions)
• Reconstruction Loss: 20.78 (L1 Smooth)
• Codebook Perplexity: 779.77 after 376,000 training iterations

GPT model:

• Token-level accuracy: approximately 97% across diverse sentence-level inputs
• Outperforms prior gloss-based approaches: NMT baseline BLEU 12.35, SignLLM BLEU 17.73

The 97% accuracy is particularly significant because it is achieved without any gloss annotations. The system operates end-to-end from raw English text to 3D motion, supporting an open vocabulary rather than a predefined gesture lexicon.

Why this matters

Traditional sign language generation systems are locked to their training vocabulary. A word or phrase not in the gloss dictionary produces nothing. ASL-LLM handles any English sentence, generating natural, contextually coherent signing that includes facial expressions and body language, the components that carry grammatical meaning and emotional nuance in sign language. The work establishes a framework for accessible, scalable communication tools for the hearing-impaired community.

← all work