Skip to content

Commit

Permalink
Document annotator tokens (#178)
Browse files Browse the repository at this point in the history
* fix typo
* clarify annotator tokens

---------

Co-authored-by: Roman Grundkiewicz <rgrundkiewicz@gmail.com>
  • Loading branch information
zouharvi and snukky authored Aug 19, 2024
1 parent 869669e commit 79b1ae9
Show file tree
Hide file tree
Showing 3 changed files with 16 additions and 4 deletions.
2 changes: 1 addition & 1 deletion EvalView/templates/EvalView/_instructions-esa.html
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<div class="row">
<div class="col-md-12">
<ul class="list-unstyled">
<li><strong>Higlighting errors:</strong>
<li><strong>Highlighting errors:</strong>
<ul>
<li>
Highlight the text fragment where you have identified a translation error (drag or click start & end).
Expand Down
2 changes: 1 addition & 1 deletion EvalView/templates/EvalView/_instructions-mqm.html
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<div class="row">
<div class="col-md-12">
<ul class="list-unstyled">
<li><strong>Higlighting errors:</strong>
<li><strong>Highlighting errors:</strong>
<ul>
<li>
Highlight the text fragment where you have identified a translation error (drag or click start & end).
Expand Down
16 changes: 14 additions & 2 deletions INSTALL.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Setup
# Setup

1. Basic setup:

Expand Down Expand Up @@ -39,6 +39,9 @@ python3 manage.py StartNewCampaign Examples/MQM+ESA/manifest.json \
python3 manage.py CreateInviteTokens test_group 20 --create-group test_group
```

Add `--task-confirmation-tokens` if you want to generate annotator confirmation tokens.
See [quality control](#Quality control) for more details.

5. Optionally clean up everything

```
Expand Down Expand Up @@ -122,4 +125,13 @@ For task:
- `batchNo`: task number
- `randomSeed`: number used in batch generation
- `requiredAnnotations`: how many annotations does a task need, in most cases use 1
- `source/targetLanguage`: source and target language
- `source/targetLanguage`: source and target language

## Quality control

With `--task-confirmation-tokens`, the annotators will be shown a random key/token if they fail the quality control and a correct one (matching the one in the CSV output with credentials) if they succeed.
The quality control checks if the perturbed samples (`itemType=BAD`) have statistically lower scores than the original ones (`itemType=TGT`).
Even without the switch, the campaign status page will show a p-value (last column for staff account) that corresponds to the outcome of this test.
If it's close to 1, then the annotator is annotating randomly and is of poor quality.
For values close to 0, the annotations are good.
The threshold to generate the valid token for annotators is currently p<=10%.

0 comments on commit 79b1ae9

Please sign in to comment.