Better tracking helps.
Changing the movement tracker improved the English translation.
AlphaPose4.97
DWPose9.94
+4.97 BLEU-4 points · mean of 3 runs
See the tracking comparisonUniversity of Western Australia · Honours research
From Auslan
to English speech.
Auslan is the language of the Australian Deaf community.
Our research explores translating it into English text and speech for people who do not sign.
Follow one signed sentence.
Play the clip. Choose a step to see what happens next.
WATCH / THE COMPLETE RECORDINGSCHEMATIC
SOURCE VIDEO
ILLUSTRATED MOVEMENT COORDINATES
01 / REPRESENT
Turn a frame into numbers.
Each tracked point contributes two coordinates and a detection confidence.
133 × 3 → 399
x · normalised
y · normalised
confidence
Coordinates are centred and scaled within each frame.
LEARNED FEATURE VECTOR
FEATURE CHANNELS
0 sTime →End
0
Scrub the timeline to follow the same frame through every layer.
ENGLISH DECODERAfter the utterance
A sentence, from a sequence.
REFERENCE TRANSLATION · FROM THE DATASET
Review the English before choosing to speak.
Speech becomes available with a recorded example.
BodyFaceHands
Diagram, not an Auslan recordingAn illustrated guide to the translation steps. A recording loads here when available.
Try the translator ↗Example: Auslan-Daily validation clip, via Auslan-Daily · Shen et al. (2023) · CC BY 4.0 · Source & modifications
How this example was made
The recorded example synchronises video and extracted pose, displays measured encoder features and self-attention, and compares the saved prediction with the dataset reference. A labelled schematic appears if the recording cannot be loaded.
The feature matrix shows activation values; a separate diagram shows self-attention. Neither establishes causal importance or sign-to-word alignment. Speech requires a separate, explicit action.
What we found
What helps translation?
What still holds it back?
Tested on 800 conversational video clips from Auslan-Daily, a research dataset.
Reading the scoresBLEU-4 measures overlap with reference translations. Higher is better; it is not a percentage of correct translations.
Extra examples need the right timing.
Joining recordings of individual signs helped when their speed matched conversation.
+1.48BLEU-4 points
vs real clips alone
vs real clips alone
Mean improvement across 3 paired runs
Read about the training examplesNew sentences are harder.
The same six-model system scored lower on English sentences absent from training.
All test clips11.80
New sentences6.60
BLEU-4 · 800 clips → 570-clip subset
Explore the sentence comparisonResearch in progress
A step towards translation.
Still a research prototype.
Reliable translation for new webcam users remains unproven. This system cannot replace an Auslan interpreter.
Understand the limits