Knowing in advance which parts of machine output can be trusted directs the post-editing effort to the right place. Quality estimation tries to predict this automatically. However, it is an estimate, and even the sentence with the highest score may be wrong.
Definition
Quality estimation tries to predict the reliability of machine output without a reference text. The result is a score. The score expresses probability.
How it differs from assessment
Quality assessment rests on human judgement, estimation is automatic. The two serve different purposes. They are not confused.
Working without a reference
The estimate is made without the correct translation to hand. This is its practical value. It is also its limit.
What the score means
The score shows the probability that the sentence is reliable. It does not express certainty. It is interpreted accordingly.
Prioritising post-editing
Priority is given to the sentences with low scores. The effort is directed to the right place. This is where the gain lies.
The risk of a high score
A sentence that receives a high score may carry a factual error. The score is sensitive to linguistic fluency. The limit is known.
It cannot see factual errors
A wrong number may sit in a fluent sentence and the score comes out high. The estimate cannot see this. Numbers are checked separately.
Negation errors
When the negative suffix drops out, the sentence stays fluent. The score may not fall. This error is looked for by hand.
Setting the threshold
It is decided below which score sentences will be examined. The threshold is adjusted to the text. The setting is found by trial.
Reading the whole text
Estimation does not remove the obligation to read the whole text. It only sets priorities. The scope does not narrow.
Model dependency
The estimation model has been trained on particular data and does not show the same success in every field. The difference between fields is known. The result is read with care.
Variation by language pair
For some language pairs the estimate is more reliable. For common pairs there is more data. Expectations are set accordingly.
Its value at large volumes
On jobs of thousands of sentences, prioritisation brings a serious gain. On small jobs its contribution is limited. Scale is decisive.
Use in reporting
The average score can be tracked as an indicator. On its own it is not proof of quality. It is read together with other criteria.
Contribution to the choice of engine
The outputs of different engines can be compared. The choice rests on data. The decision becomes objective.
Presenting it to the client
Scores are not presented to the client as quality assurance. This creates a false expectation. The use is for internal purposes.
It does not replace human review
Estimation does not take the place of human review under any circumstances. The tool prioritises, it does not decide. The limit is preserved.
Recording and monitoring
The estimate can be compared with the actual need for post-editing. This data shows the reliability of the model. Measurement is carried out.
The risk of false confidence
Trusting high scores and relaxing the review is the greatest risk. The tool gives a feeling of comfort. Discipline is maintained.
What it is confused with
Quality estimation is often taken to be quality assurance. Yet it is only a prediction of probability; it does not show where the error is, and even the sentence with the highest score may carry a wrong number.