The least discussed problem of quality measurement is this: two experts evaluate the same text differently. If the difference is large, the measurement is measuring not the quality of the text but who the evaluator is. That is why the reliability of the system is tested by consistency between evaluators.
Definition of the problem
The same text may be evaluated differently by different people. The difference can be measured. The problem is real.
Why it arises
Category and severity decisions involve interpretation. Interpretation varies from person to person. The source is human.
Effect on the reliability of measurement
A large difference shows that the measurement measures the evaluator, not the text. The result cannot be used. The effect is serious.
Measuring the difference
The same text is given to two people and the results are compared. The difference is seen in numbers. The method is simple.
Regular testing
This comparison is repeated periodically. A single measurement is not enough. A routine is set up.
Category differences
Placing the same finding in different categories is the most frequent difference. The definitions are clarified. Examples are added.
Severity differences
Giving the same finding a different severity is the second most frequent difference. Level definitions are illustrated with examples. The difference narrows.
Difference in the number of findings
One evaluator may see more findings. A difference in thoroughness is normal. The trend is monitored.
The role of written definitions
Every category and level is defined with examples. Definition narrows the difference. A document is prepared.
Shared example bank
Examples taken from real texts are collected in a shared resource. Debatable cases are added here. The resource grows.
Calibration sessions
Evaluators examine the same text together and discuss the differences. This is how a common understanding is built. Sessions are held regularly.
Resolving debatable findings
The disagreement is entered in the example bank and turned into a rule. The same debate is not repeated. The knowledge becomes permanent.
Training a new evaluator
The new person first evaluates alongside an experienced one. The results are compared. Alignment is achieved by measuring it.
Blind evaluation
The evaluator may not know who translated the text. This reduces bias. The method is preferred.
Risk of bias
A previous impression of the person may affect the evaluation. A blind arrangement reduces this risk. The practice is recommended.
The risk of a single evaluator
If all measurement depends on one person, the system is tied to that person's criterion. Rotation is applied. A balance is struck.
Rotation in practice
The evaluation duty is taken in turns. Different eyes enter the system. Consistency is monitored.
The contribution of automated checks
Tool findings are independent of the person and consistent. They reduce human decisions. A layer is added.
The acceptable level of difference
Zero difference is not expected. An acceptable range is defined. The criterion becomes realistic.
Consistency as a precondition of measurement
A quality score published without measuring the difference between evaluators is a number that measures nobody knows what; the results of a system cannot be trusted unless the system itself has been tested. That is why the consistency test is part of measurement. The step is not neglected.
Limits of measurement
No arrangement removes evaluator difference entirely. The aim is to keep the difference manageable. The limit is openly accepted.