The research question
How can an Auslan translator learn useful connections between visual movement and English when annotated training data is limited? This thesis tests the effects of movement tracking, training examples and translation models using the conversational part of Auslan-Daily.
Gloss-free means learning from movement sequences paired with English sentences, without requiring an intermediate sequence of written labels for individual signs. Auslan has its own grammar; translation is more than replacing signs with English words.
What the walkthrough shows
Each example follows one recorded validation clip through five stages: video, tracked movement, learned features, English output and optional speech. Pose coordinates, encoder activations, self-attention and predictions were computed in advance by a pose-only model. The page replays those saved calculations; it does not run inference on the clip while you watch.
Encoder activations describe the model’s internal representation. Self-attention shows how it combines information across frames. Neither proves causal importance or sign-to-word alignment. The English reference is a dataset translation, while the model prediction is a separate output that can be wrong.
The examples are selected illustrations, not a sample from which to estimate typical accuracy. Each clip’s “How this example was made” disclosure gives its model identity and processing details. Source credits and manifests are available separately.
Reading the scores
BLEU-4 measures overlap between generated English and reference translations using sequences of up to four words. Higher means closer wording under this metric; it is not a percentage of correct translations or a complete measure of meaning.
The headline evaluation uses 800 conversational test clips. Comparisons on selected subsets, such as new sentences or high-confidence answers, have different conditions and should not be treated as full-test scores.
Better movement tracking
Changing only the tracker from AlphaPose to DWPose raised mean test BLEU-4 from 4.966 to 9.936 across three matched training runs: a gain of 4.970 points, with a reported 95% interval of +4.414 to +5.527.
This is the landing page’s +4.97 result. It is a controlled tracking comparison, separate from the model comparison below. An earlier single-run comparison, 5.58 to 10.03, used a different experiment and is not pooled with these means.
Constructing useful training examples
Adding broadcast-news recordings reduced conversational translation performance, while pretraining on isolated-sign recognition transferred poorly. Joining isolated-sign recordings into synthetic sequences and training alongside real clips was more useful.
The mean gain over matched controls rose from +0.56 to +1.48 BLEU-4 after shortening synthetic signs to better match conversational timing, across three paired runs. These sequences link signs to English words; they do not reproduce Auslan grammar. A smaller random-sign comparison removed the average benefit, supporting the importance of the sign–word associations.
Familiar and unfamiliar sentences
Of the 800 test clips, 570 have English reference sentences absent from training. The remaining 230 share an English reference with training. This tests unfamiliar sentences, not unfamiliar signers; the subsets also differ in composition.
Translation score · BLEU-4
Does the gain hold for new sentences?
Switch to the 570 clips whose English reference sentences were absent from training.
Same models, different test subset.
Higher scores mean closer wording to the reference.
Exact values, uncertainty and comparison protocol
| System · research name | All clips | New sentences |
|---|---|---|
| Movement only · DWPose + CTC | 9.522 | 4.88 |
| Extra pretraining · VLP-40 (40 epochs) | 10.272 | 5.31 |
| Movement + images · RGB fusion | 10.335 | 5.81 |
| Six models combined · S6 ensemble | 11.802 | 6.60 |
S6 exceeds RGB fusion by 1.467 BLEU-4: paired-bootstrap 95% interval +0.396 to +2.563; P(better) = 0.994; 2,000 resamples of 800 sentences, seed 0. This measures test-sentence sampling, not training-run variation. Earlier adjacent single-model increments are not distinguishable from zero.
On unseen English targets, S6 vs RGB is +0.79 [−0.25, +1.76]; S6 vs DWPose + CTC is +1.72 [+0.43, +2.95]. The 230 clips with verbatim training targets score 35.00 for S6. They represent 28.75% of clips, or 22.7% of distinct test sentences.
The three single models use beam 5 / length penalty 0.6; S6 uses beam 10 / 0.3, selected on validation. S6 was trained without synthetic sequences. The later single-model configuration uses a different training recipe. CTC is an auxiliary English-content-word training loss, not an intermediate gloss sequence.
What remains difficult
Of 786 test clips with content words in the reference, three single models recovered none on 70–73%; the six-model S6 ensemble recovered none on 67%. Signers beyond the two recurring performers scored about 3 BLEU-4 even with S6.
Confidence is not calibrated as a probability of correctness and cannot establish whether someone is signing. An earlier VLP-40 model answered nonsigners about as often as signers (54% versus 53.5%). High-confidence answers also favour short, familiar expressions.
Reliable translation for new webcam users remains unproven. Evaluation still needs new signers, translation quality and coverage measurements, capture timing, response delay and signer feedback. The prototype cannot replace a qualified Auslan interpreter.
Models and operating points
The research compares a pose-only operating point and the six-model S6 ensemble. Both decode complete utterances; S6 also uses DINOv2 image features from signer and hand crops. Each model’s confidence threshold is fitted separately on validation.
Inspect the current model snapshot for checkpoint identities, validation selection, test coverage, quality and timing measurements.
These are measured research configurations, not a promise that a particular model is running on the public site. Check the translator’s current service status before attempting capture.
Evidence and further reading
- Machine-readable evidence snapshot: checkpoint identities, validation selection, test results and caveats.
- Research repository: methods and experiment records; access may be required.
- Shen, X., Yuan, S., Sheng, H., Du, H., & Yu, X. (2023). Auslan-Daily: Australian Sign Language Translation for Daily Communication and News. NeurIPS 36, Datasets and Benchmarks.