Labeling Queues: Share Annotation Work Across a Team

A labeling queue is a shared pool of images that a team labels on a first-come, first-served basis. Instead of fixed portions, each annotator pulls the next unlabeled image, so fast annotators take more work and the dataset finishes sooner. This guide shows how queues work in Supervisely and how to set one up. See the collaboration features and the docs.

How do labeling queues relate to labeling jobs?

Queues are the next step after labeling jobs, which cover the roles and why team coordination matters. This guide covers sharing the work of big datasets across large teams.

Watch our 5-minute video tutorial on how to use labeling queues in Supervisely.

What is a labeling queue?

Labeling Queues are a method for distributing and managing labeling process in a team, where labeling tasks are grouped into queues and sequentially distributed among annotators. The essence of this system is that the data is not distributed between annotators, but is issued from a common queue (pool) of tasks, on a "whoever labels first" basis, until there is no more data to label. This allows to balance data annotation process within the labeling team independently of the speed of individual workers.

They help coordinate the annotation of large image datasets within teams. The walkthrough below uses images.

Labeling jobs vs labeling queues: which one to use?

	Labeling jobs	Labeling queues
How work is split	Each annotator gets a fixed portion of the data	Annotators pull the next unlabeled image from a shared pool
Faster annotators	Work only on their own portion	Take more images
Rejected items	Go to a new labeling job for re-annotation	Return to the same annotator's job
Setup	Manager or Reviewer sets roles, instructions, classes and tags	Queue creates an empty labeling job for each annotator and reviewer

How do labeling queues work in Supervisely?

In Supervisely Labeling Queues are based on Labeling Jobs and work as follows:

• When a Labeling Queue is created, an empty Labeling Job for each labeler and Reviewer is initiated.
• As the Annotators access these Jobs through the labeling interface, they retrieve the next available un-annotated image in the dataset from the queue. This selected image is then allocated as a task for the Annotator. Upon pressing the Confirm and pull next button, this process repeats. One of the key features of Labeling Queues is that the Labelers aren't required to complete the entire batch of images before sending them for review.
• Reviewers have the flexibility to access their reviewing tasks at any moment. If available, annotated images are retrieved from the queue for the review. Importantly, this reviewed images does not revert to the general pool of images for annotation. Instead, it is promptly incorporated into the Labeling Job of the Annotator who initially labeled them. This implies that the Annotator can gather several images within their assignment that have been recently selected from the shared queue.

Labeling Queue mechanism

Let's look at key benefits of using Labeling Queues:

1. Time saving and user-friendly: There is no need to create individual Labeling Jobs for each project. With Labeling Queue creation, you can save time by eliminating the need to repeatedly configure Labeling Jobs for new projects and reassign them to your team.

2. Scalable and adaptable: The Labeling Queueing system is adaptable to changing project needs and team makeup. You can easily remove unnecessary Labeling Jobs, switch team members during annotation, adjust Labeling Job's settings, and scale the annotation process as required, such as reassigning annotators to different projects.

3. Result-oriented approach: Labeling Queues system ensures even data distribution among annotators, minimizing downtime and speeding up project completion. It emphasizes results over individual team member contributions.

4. Efficient Project Management via the common queue of data: Each team member may work at their own pace, which can affect project progress. The Labeling Queues approach effectively solves this issue by creating a shared data pool. This simplifies management tasks and ensures that annotators' work speed doesn't hinder project advancement.

How to use Labeling Queues in Supervisely: Step-By-Step Guide

Before you start, make sure that you have:

• A dedicated team assigned with proper roles. To learn more about roles, refer to our post devoted to Labeling Jobs, which provides more detailed information about the functionality of each role.
• Necessary projects for annotation with relevant properties (classes, figures, or tags) defined.

The following example will show you how to use Labeling Queues:

Suppose you are the owner of a project whose goal is to train a deep learning model to recognize strawberries in images. You have 10,000 images of strawberries for annotation. In order to do that efficiently, you may rely on Labeling Queues functionality following the below steps.

Roles of team members in Labeling Queues

Step 1. Create Labeling Queues

First of all, the Manager or Reviewer creates the Labeling Queue with defined criteria and assigns Annotators to perform the labeling and Reviewers - to verify the labeling. Consequently, an empty Labeling Job is created for each labeler and reviewer.

Upon completion of the Labeling Queue setup process, creators are redirected to the Labeling Queues page, which displays a queue containing Labeling Jobs for Annotators and Reviewers. The queue includes Labeling Jobs, which are automatically created for each Labeler and Reviewer.

Step 2. Image Annotation

In each Labeling Job, Annotators are given one image at once for annotation according to the instructions provided. Labelers are not obligated to finish the whole batch of data before submitting it for review. A new image is assigned to an Annotator only after the previous image has been fully annotated and sent for review. So, Annotators capable of annotating 5 images per minute can provide support to colleagues who can only annotate 1 image per minute.

This process continues until all images are annotated by all Annotators, or until there are no more images to annotate.

Step 3. Review and Quality Control

Annotation results are sent to Reviewers for verification. Reviewers receive review-images sequentially and check the annotation for accuracy and compliance with instructions.

Reviewers can access their reviewing tasks whenever necessary.

If the annotation is correct, they approve it. If errors are found, the image is sent back for re-annotation to the same Labeler. It's crucial to note that these reviewed images does not return to the general pool of images for annotation. Instead, it is promptly integrated into the Labeling Job of the original annotator.

Once the Labeling Queue is finished, you can analyse the results in statistics, export data and create a training dataset and use it to train a model that can recognise strawberries in the images.

Labeling Queues Statistics

📊 Manager and Reviewer can view statistics for each Labeling Job in the Labeling Queue by clicking Stats button.

Labeling Job statistics

Export

You can choose to obtain activity data from either a Labeling Job or a team member in the form of a .csv file. Besides, you can easily export approved images from the dataset in Supervisely format after completing the Labeling Job review.

You can find and apply more applications on the Collaboration at Scale topic in the Collaboration section of Supervisely Ecosystem.

Additional Applications

Conclusion

Labeling queues let faster annotators take more of the work, so the dataset is finished without waiting for the slowest person. Use them for large datasets and uneven teams. Use labeling jobs when each person needs a fixed portion, and the consensus app when you need to measure agreement.