WMT19 News Translation Task

What is the WMT News Translation Task?

The Workshop on Machine Translation (WMT) organizes an annual shared task that focuses on translating news articles. The 2019 edition, often referred to as **WMT19**, continued the tradition of providing a realistic, largescale benchmark for neural machine translation (NMT) systems. Participants are given parallel news data in several language pairs, asked to train models, and finally evaluated on a blind test set that mimics realworld news streams.

Key Objectives

• Progress Measurement: Offer a common platform to assess the stateoftheart in NMT.
• Reproducibility: Provide openly available training data and evaluation scripts.
• Industry Relevance: News articles pose challenges such as timesensitive terminology, varied styles, and proper names.
• Multilingual Exploration: Encourage work on both highresource (e.g., EnglishGerman) and lowresource (e.g., EnglishLatvian) pairs.

Data Used in 2019

The organizers supplied several corpora, all preprocessed in a consistent way (sentencepiece tokenisation, truecasing, etc.). The main sources were:

For lowresource pairs, the organizers added backtranslated monolingual news data to increase the effective training size.

Evaluation Metrics

WMT19 continued to rely primarily on BLEU (Papineni et al., 2002) as the official metric. In addition, the following were reported for a more nuanced view:

• chrF++ characterlevel Fscore that handles morphology better.
• COMET a neural quality estimation metric trained on human judgments.
• Human direct assessment (DA) on a subset of the test set, using a 5point adequacyfluency scale.

Scores were computed after detokenising and applying mteval-v13a.pl for BLEU, while chrF++ and COMET used the standard scripts provided by the WMT organisers.

Top Performing Approaches

Several research groups and companies achieved stateoftheart results. The most common patterns among the winners were:

1. Transformerbased Architectures

All top submissions used the Transformer (Vaswani et al., 2017) as the backbone, often with the big configuration (larger feedforward dimension, more attention heads, and higher dropout). Some teams experimented with TransformerXL or Dynamic Conv variations, but gains were modest compared to careful training.

2. LargeScale BackTranslation

For each language direction, monolingual news articles were translated with a reverse model and added to the parallel training set. This approach boosted performance especially on lowresource directions (e.g., ENLV), yielding 13 BLEU points over pure parallel training.

3. Data Filtering and Weighting

Noise in harvested web corpora such as Common Crawl can be severe. Winners applied languageidentification filters, duplicate removal, and sentencelevel quality scoring (e.g., with a multilingual BERT classifier). Weighted sampling gave higher probability to cleaner, indomain data.

4. FineTuning on InDomain News

After pretraining on the full data, a short finetuning phase on the most recent news crawl (the same year as the test set) helped adapt models to current terminology and named entities.

5. Ensemble and Reranking

Most top runs used a shallow ensemble of 46 independently trained checkpoints. Some teams added a language modelbased rescoring step, using a separately trained Transformer LM on monolingual news.

6. MixedPrecision Training

To handle the large data volume, participants employed FP16 training with lossscaling, reducing GPU memory use and enabling larger batch sizes, which correlated with higher BLEU.

Notable System Descriptions

Below is a concise summary of three highlighted submissions.

Microsoft (ENDE, ENFR)

• Architecture: Dualsource Transformerbig with shared embeddings.
• Training: 12GPU days, batch size 4096 tokens, Adam optimizer with warmup.
• Backtranslation: 20M synthetic sentences per direction, filtered by language model score.
• Result: BLEU 44.7 (ENDE) and 42.3 (ENFR), highest among all submissions.

Unbabel (ENRU)

• Architecture: Transformerbig with relative positional encoding.
• Data: Combined all parallel corpora + 50M backtranslated ru en sentences.
• Special technique: Tagbased domain control added a token indicating the source of each sentence (news, web, synthetic).
• Result: BLEU 32.9, chrF++ 59.2, COMET 0.75.

LMU (ENLV, ENCS)

• Architecture: Transformerbase with a shallow 2layer LSTM decoder for lowresource pairs.
• Data augmentation: Iterative backtranslation (three rounds) combined with forwardtranslation of monolingual news.
• Regularisation: Switchout and labelsmoothed loss.
• Result: BLEU 27.5 (ENLV) and 30.1 (ENCS), notable because of the limited parallel data.

Why the News Domain?

News articles are a natural testbed for machine translation for several reasons:

• Temporal Dynamics: New events introduce previously unseen entities and terminology.
• Diverse Styles: From hardnews factual reporting to editorial opinion pieces.
• Public Availability: Large volumes of multilingual news streams are freely available, facilitating reproducibility.

Consequently, progress on the News Translation Task usually translates into better performance on downstream realworld applications such as crossborder journalism and multilingual content recommendation.

Lessons Learned from WMT19

1. Scale still matters. Even with powerful architectures, the sheer amount of clean parallel data yields the biggest jumps.
2. Quality of synthetic data is crucial. Simple backtranslation works, but filtering synthetic sentences with a strong language model dramatically improves the signaltonoise ratio.
3. Domain adaptation is not optional. Finetuning on inyear news made a measurable difference, especially for highvariance terms.
4. Ensembling remains a strong baseline. The marginal cost of adding a few checkpoints at inference time is often outweighed by the BLEU gain.
5. Evaluation beyond BLEU. Human DA and neural metrics like COMET revealed quality differences that BLEU alone missed, encouraging the community to adopt multimetric reporting.

Future Directions

Building on the momentum of WMT19, the next editions are likely to explore:

• Multilingual models that share parameters across many language pairs, reducing the need for pairspecific data.
• Continual learning where models are updated daily with fresh news, testing robustness against catastrophic forgetting.
• Explainability tools that highlight which source words contribute to a particular translation, supporting journalistic verification.
• More diverse evaluation including adequacyfocused test sets, bias analysis, and realtime latency measurement.

Resources

For readers interested in reproducing or extending the WMT19 experiments, the following links are useful:

• Official WMT19 News Translation Task page
• Moses preprocessing scripts used by the organisers
• fastBPE tokenisation tool employed by many participants
• Fairseq reference implementation of the Transformer model
• BLEU computation scripts