Pools

A pool is a set of tasks that are sent out for completion at the same time. In the pool properties, you set the task price, overlap, user selection filters, quality control rules, and so on.

The pool properties should be sent in JSON format. A sample pool in JSON with key descriptions is shown below.

If the project has multiple pools, the order for completing them depends on the parameters:

  • Pools with identical filter settings and price per task are assigned to users in the order in which they were started. The pool that was started earlier will be completed sooner. You can change the order for completing the pools.

  • Pools with different filter settings and/or a different price per task are sent out for completion when the pool opens.

You can use the API for performing the following operations with pools:

Example of a pool in JSON

{
  "id": "197",
  "project_id": "1",
  "private_name": "My first pool",
  "private_comment": "This is my first pool",
  "public_description": "Pool's description ",
  "may_contain_adult_content": false,
  "will_expire": "2020-09-01T13:00",
  "auto_close_after_complete_delay_seconds": 600,
  "reward_per_assignment": 0.02,
  "dynamic_pricing_config": {
        "type": "SKILL",
        "skill_id": "1289",
        "intervals": [
            {
            "from": 0,
            "to": 60,
            "reward_per_assignment": 0.03
            },
            {
            "from": 61,
            "to": 100,
            "reward_per_assignment": 0.04
            }
        ]
    },
  "assignment_max_duration_seconds": 300,
  "auto_accept_solutions": true,
  "auto_close_after_complete_delay_seconds": 60,
  "assignments_issuing_config": {
         "issue_task_suites_in_creation_order": false
  },
  
  "filter": {<filter settings for selecting users>},
  "quality_control": {
         "training_requirement": {
                "training_pool_id": "21",
                "training_passing_skill_value": 70
         },
         "captcha_frequency": "LOW",
         "configs": {<settings for the quality control rules>},
         "checkpoints_config": {
             "real_settings": {
                 "target_overlap": 5,
                 "task_distribution_function": {
                     "scope": "PROJECT",
                     "distribution": "UNIFORM",
                     "window_days": 7,
                     "intervals":[ 
                         {
                             "from": 1,
                             "to": 100,
                             "frequency": 25
                             },
                         {
                             "from": 101,
                             "to": 1000,
                             "frequency": 5
                             }
                      ]
                   }
              }
         }
  },
  "dynamic_overlap_config": {
        "type": "BASIC",
        "max_overlap": 5,
        "min_confidence": 0.9,
        "answer_weight_skill_id": "1289",
        "fields": [
            {
                "name": "result"
            }
        ]
    },
  "defaults" : {
         "default_overlap_for_new_task_suites" : 3,
          default_overlap_for_new_tasks": 3
  },
  "mixer_config": {
         "real_tasks_count": 6,
         "golden_tasks_count": 1,
         "training_tasks_count": 1,
         "min_real_tasks_count": 2,
         "min_golden_tasks_count": 0,
         "min_training_tasks_count": 0,
         "force_last_assignment": true,
         "force_last_assignment_delay_seconds": 10,
         "mix_tasks_in_creation_order": true,
         "shuffle_tasks_in_task_suite": false,
         "golden_task_distribution_function": {
             "scope":"PROJECT",
             "distribution":"UNIFORM",
             "window_days":7 ,
             "intervals":[ 
                 {
                     "from": 1,
                     "to": 25,
                     "frequency": 5
                     },
                 {
                     "from": 26,
                     "to": 1000,
                     "frequency": 25
                     },
              ]
         },
         "training_task_distribution_function": {
             "scope": "PROJECT",
             "distribution": "UNIFORM",
             "window_days": 7,
             "intervals": [ 
                 {
                     "from": 1,
                     "to": 25,
                     "frequency": 5
                     },
                 {
                     "from": 26,
                     "to": 1000,
                     "frequency": 25
                     },
              ]
         }
  },
  "priority": 10,
  "type": "REGULAR",
  "status" : "CLOSED",
  "last_close_reason": "EXPIRED",
  "created" : "2016-04-08T14:23:04.785"
}

Keys

Key

Value

Data type

Required during creation

project_id

ID of the project that the pool was created for.

String

Yes

private_name

Name of the pool (only visible to the requester).

String

Yes

private_comment

Comments on the pool (only visible to the requester).

String

No

public_description
Description for users. If it is filled in, the text will be displayed instead of the project's public_description in the list of tasks for performers. String No
may_contain_adult_content

Whether the tasks contain adult content.

Boolean

Yes

will_expire

The date and time in UTC when the pool should be closed (even if all the task suites haven't been completed). It uses ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Yes

auto_close_after_complete_delay_seconds

Waiting time (in seconds) before automatic closure of the pool after all tasks are completed. Minimum — 0, maximum — 259200 seconds (three days).

Use it if:
  • Your data processing is close to real time.
  • You need an open pool where you upload tasks.
  • Dynamic overlap is enabled in the pool (dynamic_overlap_config).
Number No
reward_per_assignment

Payment per task suite in U.S. dollars. For cents, use the dot (".") as the separator. The minimum payment is $0.01.

Only training and control tasks can be uploaded to zero-price pools.

Number

Yes

dynamic_pricing_config

The dynamic pricing settings.

JSON

No

dynamic_pricing_config.type

Parameter type for calculating dynamic pricing. The SKILL value.

String

Yes, if dynamic pricing is used.

dynamic_pricing_config.skill_id

ID of the skill that the task price is based on

String

Yes, if dynamic pricing is used

dynamic_pricing_config.intervals

Skill level intervals. Must not overlap.

A performer with a skill level that is not included in any interval will receive the basic price for a task suite.

JSON

Yes, if dynamic pricing is used

dynamic_pricing_config.intervals.from

Lower bound of the interval. By default 0.

Number from 0 to 100

No

dynamic_pricing_config.intervals.to

Upper bound of the interval. By default 100.

Number from 0 to 100

No

dynamic_pricing_config.intervals.reward_per_assignment

The price per task page for a performer with the specified skill level.

Number

Yes, if dynamic pricing is used.

assignment_max_duration_seconds

The time allowed for completing a task suite, in seconds. Tasks not completed within this time are reassigned to other users.

We recommend allowing no more than 60 seconds per task suite (including the time for page loading and sending responses).

Number

Yes

auto_accept_solutions

Whether tasks must be checked manually:

  • true — Automatic task acceptance (manual checking isn't necessary).

  • false— The requester will check the tasks.

The default value is true.

Boolean

No

auto_close_after_complete_delay_seconds

Waiting time (in seconds) before automatic closure of the pool after all tasks are completed. Minimum — 0, maximum — 259200 seconds (three days).

By default, the value is 0.

Number

No

assignments_issuing_config

Settings for assigning tasks in the pool.

JSON

No

assignments_issuing_config.issue_task_suites_in_creation_order

For pools that don't use “smart mixing” .

Assign task suites in the order in which they were uploaded. For example, for a pool with an overlap of 5, the first task suite is assigned to five users, then the second task suite, and so on.

This parameter is available when the project has "assignments_issuing_type": "AUTOMATED".

Boolean

No

priority

The priority of the pool in relation to other pools in the project with the same task price and set of filters. Users are assigned tasks with a higher priority first.

Possible values: from -100 to 100.

By default the value is 0 .

Number

No

filter

Settings for user selection filters.

JSON

No

quality_control

Settings for quality control rules and the ID of the pool with training tasks.

JSON

No

quality_control.training_requirement

Parameters of the training pool that is linked to the main task pool.

JSON

No

quality_control.training_requirement.training_pool_id

ID of the training pool that is linked to the main task pool.

String

No

quality_control.training_requirement.training_passing_skill_value

Proportion of correct answers in training tasks (from 0 to 100) required to be admitted to the main tasks. Only first answer of the user in each task is taken into account.

String

No

quality_control.captcha_frequency

The frequency of showing captchas:

  • LOW — Show one for every 20 tasks.

  • MEDIUM, HIGH — Show one for every 10 tasks.

By default, captchas aren't displayed.

String

No

quality_control.configs

Quality control rules.

JSON

No

quality_control.checkpoints_config

Selective majority vote testing. Only some tasks are issued with higher overlap (for example, “5”) for testing. The rest of the tasks are issued with overlap set for the pool (for example, “1”) and are not tested. Selective testing allows you to save money and speed up pool completion.

You can reduce the frequency of testing tasks with time.

Example of settings: in the first 25 tasks completed by the user, issue every 5th task with an overlap of “5”, to test matching responses. In the following tasks, every 25th task is issued with an overlap of “5”.

No

quality_control.checkpoints_config.real_settings.target_overlap

Overlap in majority vote tasks.

Number

Yes, if you use selective majority vote testing
quality_control.checkpoints_config.real_settings.task_distribution_function
Distribution of majority vote tasks.

JSON

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.distribution
Distribution of majority vote tasks within an interval. Parameter has only one possible value — UNIFORM.

String

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals

Interval borders and number of majority vote tasks.

Array of JSON objects

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.from

Start of the interval (number of task completed by the user in the project or in the pool).

Number

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.to
End of the interval (number of task completed by the user in the project or in the pool).

Number

Yes, if you use selective majority vote testing

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.frequency

Frequency of majority vote tasks in an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if you use selective majority vote testing

dynamic_overlap_config

Dynamic overlap setting. Allows you to change the overlap depending on how well the performers handle the task.

Set the closing interval (auto_close_after_complete_delay_seconds). It should be enough to complete tasks with an overlap higher than the minimum.

When all pool tasks are completed, aggregate the responses.

JSON No
dynamic_overlap_config.type

The algorithm for dynamic overlap.

BASIC — Each response is assigned a weight depending on the performer's skill value. The aggregated response confidence is calculated based on the probability algorithm. The task overlap increases until it reaches max_overlap or until the confidence of the aggregated response exceeds min_confidence.

You have to specify max_overlap, min_confidence, answer_weight_skill_id and fields.

String Yes, if dynamic overlap is used
dynamic_overlap_config.max_overlap

Maximum overlap. Must be higher than the values in defaults. Minimum — 1. Maximum — 30000.

Number Yes, if dynamic overlap is used
dynamic_overlap_config.min_confidence

Minimum confidence of the aggregated response. Values from 0 to 1.

Floating-point number Yes, if dynamic overlap is used
dynamic_overlap_config.answer_weight_skill_id
A skill that determines the weight of the performer's response. For best results, use a skill calculated as percentage of correct responses in control tasks. String Yes, if dynamic overlap is used
dynamic_overlap_config.fields

Output data fields to use for aggregating responses. For best results, each of these fields must have a limited number of response options.

Don't specify several fields if their values depend on each other.

JSON Yes, if dynamic overlap is used
dynamic_overlap_config.fields.name
The output data field name. String Yes, if dynamic overlap is used
defaults

Settings that are applied by default when uploading new task suites to a pool.

JSON

Yes

defaults.default_overlap_for_new_task_suites

The overlap for task suites that are uploaded to the pool (used if the allow_defaults=true parameter is set when uploading).

Number

Yes

defaults.default_overlap_for_new_tasks

The overlap for tasks that are uploaded to the pool (used if the allow_defaults=true parameter is set when uploading).

Number

No

mixer_config

Parameters for automatically creating a task suite (“smart mixing”). For more information about creating task see Yandex.Toloka requester's guide.

Number

Yes, if “smart mixing” is used

mixer_config.real_tasks_count

The number of main tasks to put in a task suite.

The maximum number of tasks in a task suite if training_task_distribution_function or golden_task_distribution_function are used.

Number

Yes, if “smart mixing” is used

mixer_config.golden_tasks_count

The number of control ("golden set") tasks to put in a task suite.

Number

Yes, if “smart mixing” is used

mixer_config.training_tasks_count

The number of training tasks to put in a task suite.

Number

Yes, if “smart mixing” is used

mixer_config.min_real_tasks_count

Minimum number of main tasks in a task suite (if the number of assignments left is less than the one specified in mixer_config.real_tasks_count). Minimum — 0. By default, the value is the same as in mixer_config.real_tasks_count.

Number

No

mixer_config.min_golden_tasks_count

Minimum number of control tasks in a task suite (if the number of assignments left is less than the one specified in mixer_config.golden_tasks_count). Minimum — 0. By default, the value is the same as in mixer_config.golden_tasks_count.

Number

No

mixer_config.min_training_tasks_count

Minimum number of training tasks in a task suite (if the number of assignments left is less than the one specified in mixer_config.golden_tasks_count). Minimum — 0. By default, the value is the same as in mixer_config.training_tasks_count.

Number

No

force_last_assignment

Setup for the last set of tasks in the pool, if less than the minimum remaining number of tasks are not completed (mixer_config.min_real_tasks_count). Values:

  • true - issue an incomplete task set.

  • false - don't issue tasks. This option can be used if you are adding tasks after the pool is started.

The default value is true.

This parameter only applies to main tasks. The number of control and training tasks in the last set must be complete (mixer_config.golden_tasks_count, mixer_config.training_tasks_count).

Boolean

No

force_last_assignment_delay_seconds

Waiting time (in seconds) since the addition of the task, or increase in the overlap, prior to the issuance of the last set of tasks in the pool. The minimum is 0, the maximum is 86,400 seconds (one day).

This parameter can be used if the pool has force_last_assignment: true.

Number

No

mixer_config.mix_tasks_in_creation_order

The order for including tasks in suites:

  • true — Add tasks to suites in the order in which they were uploaded. For example, in a pool with an overlap of 5, the first uploaded task will be included in the first 5 task suites. They will be assigned to 5 users.

  • false — Add tasks to suites in random order.

The default value is false.

Boolean

No

mixer_config.shuffle_tasks_in_task_suite

The order of tasks within a suite:

  • true — Random.

  • false — The order in which tasks were uploaded.

The default value is true.

Boolean

No

mixer_config.golden_task_distribution_function

Issue of control tasks with uneven frequency. The option allows you to change the frequency of checking as the user completes more tasks.

If it is filled in, then in real_task_count specify the maximum number of tasks to put in a task suite. To use the option, set the golden_task_count:0 and min_golden_tasks_count:null parameter values.

The interval bound is the number of tasks in the pool. If the task suite has several intervals, the first interval will be applied to the suite.

Example:
In the first task suite (interval from 1 to 25), every fifth task will be a control, and in the following suites — every 25th task.

JSON

No

mixer_config.golden_task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.distribution

Distribution of control tasks within an interval. Parameter has only one possible value — UNIFORM.

String

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals

Interval borders and number of control tasks in an interval.

Array of JSON objects

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals.from

Start of the interval (number of task completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals.to

End of the interval (number of task completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals.frequency

Frequency of control tasks in an interval. The first task in an interval is a control task. For example, if you set frequency: 3 tasks number 1, 4, 7 and so on will be control tasks.

Number

Yes, if you use uneven distribution of control tasks

mixer_config.training_task_distribution_function

Issue of training tasks with uneven frequency. The option allows you to change the frequency of trainig tasks as the user completes more tasks.

If it is filled in, then in real_task_count specify the maximum number of tasks to put in a task suite. To use the option, set parameter values training_task_count:0 and min_training_tasks_count:null.

Example:
In the first task suite (interval from 1 to 25), every fifth task will be a control, and in the following suites — every 25th task.

JSON

No

mixer_config.training_task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.distribution

Distribution of training tasks within an interval. Parameter has only one possible value — UNIFORM.

String

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals

Interval borders and number of control tasks in an interval.

Array of JSON objects

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals.from

Start of the interval (number of task completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals.to

End of the interval (number of task completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals.frequency

Frequency of training tasks in an interval. The first task in an interval is a training task. For example, if you set frequency: 3 tasks number 1, 4, 7 and so on will be training tasks.

Number

Yes, if you use uneven distribution of training tasks

id

Pool ID.

String

Not used

status

Status of the pool:

  • OPEN

  • CLOSED

  • ARCHIVED

String

Not used

last_close_reason

The reason for closing the pool the last time:

  • MANUAL — Closed by the requester.

  • EXPIRED — Reached the date and time set in will_expire.

  • COMPLETED — Closed automatically because all the pool tasks were completed.

  • NOT_ENOUGH_BALANCE — Closed automatically because the Toloka account ran out of funds.

  • ASSIGNMENTS_LIMIT_EXCEEDED — Closed automatically because it exceeded the limit on assigned task suites (maximum of 2 million).

  • BLOCKED — Closed automatically because the requester's account was blocked by a Toloka administrator.

String

Not used

created

The UTC date and time when the pool was created, in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

last_started

The date and time when the pool was last started, in UTC in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

last_stopped

The date and time when the pool was last stopped, in UTC in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

type
Types of pool:
  • REGULAR - normal pool;

  • TRAINING - training pool.

String

Not used