Majority vote

Majority vote is a quality control method based on coinciding responses from the majority of users who are performing the same task. The response chosen by the majority is considered correct, and other responses are considered incorrect. Depending on the percentage of correct responses, you can either increase the user's skill value, or ban the user from tasks.

To set up majority vote testing, set key values in the quality_control.configs array (pool parameters).

The example below has testing set up for a pool with an overlap of “5”. The calculation counts tasks in which the responses of three or more users coincided. The percentage of the user's correct responses (coinciding with the majority) is stored as the skill value with the ID 43. You can use the skill for filtering users. The skill is first calculated after two tasks with the majority vote have been completed.

{
   "configs":[
      {
         "collector_config":{
            "type":"MAJORITY_VOTE",
            "parameters":{
               "answer_threshold":3,
               "history_size": 10
            }
         },
         "rules":[
            {
               "conditions":[
                  {
                     "key":"total_answers_count",
                     "operator":"GT",
                     "value":2
                  }
               ],
               "action":{
                  "type":"SET_SKILL_FROM_OUTPUT_FIELD",
                  "parameters":{
                     "skill_id":"43",
                     "from_field":"correct_answers_rate"
                  }
               }
            }
         ]
      }
   ]

Key

Value

Data type

Required during creation

collector_сonfig.parameters.answer_threshold

The number of users considered the majority (for example, 3 out of 5).

Integer

Yes

collector_сonfig.parameters.history_size

The maximum number of the user's recent responses in the project to use for calculating the percentage of correct responses.

If this field is omitted, the calculation is based on all the user's responses in the pool.

rules.conditions.key

Parameters that are checked in the condition:

  • total_answers_count — The number of completed tasks that had a majority vote (specify the majority in answer_threshold).

  • correct_answers_rate — The percentage of correct responses, meaning responses that coincided with the majority vote (from 0 to 100).

  • incorrect_answers_rate — The percentage of incorrect responses, meaning responses that didn't coincide with the majority vote (from 0 to 100).

String

Yes