QUALITY ASSURANCE

Ensure the highest standard of labeling

Avoid mistakes and don’t lose time with multi-level quality assurance process.

Perfect training data

Large labeling workforce requires professional tools — and we have them.

Annotation at scale with labeling jobs

No large labeling could be done working alone. Go big with hundreds of people working simultaneously with Supervisely! Split large annotation work into smaller batches called labeling jobs based on multiple criteria, like presence of a specific tag, object class or divide dataset between several labelers.

Provide job description with markdown, select whether labelers can see and edit each other annotations and choose which classes and tags are available for this job.

Labelers with get notifications that a new labeling job is available and be able to check-in when ready. Track labeling progress, improve output in a multi-step review process and get valuable insights of your current status.

Multi-step reviews

When a labeling job is marked as completed by annotator, that does not necessary means it’s over. Usually, results need to be verified by either an expert, a labeling manager or other annotator.

In Supervisely you can manually or automatically assign a reviewer who will be notified each time a new labeling job is marked completed and can mark good and bad examples and leave feedback using built-in issue tracking.

When review is done, a new labeling job will be automatically created containing bad examples and the process will repeat, until you end up with perfect training data.

Labeling consensus

Improve your training data by labeling the same assets more than once by different people, who don’t see each other labels and tags.

Decide which annotation output preserve manually reviewing results side-by-side with the help of an expert or automatically compare annotations using score metrics like Intersection over Union (IoU).

Enforce labeling requirements

Avoid labeling errors before annotation even started.

Test how well your labelers understand the task with exams

Avoid unnecessary mistakes in labeling by requiring your labelers pass annotation exam first. See if annotation guides have been learned — select a perfectly labeled ground truth benchmark dataset, hidden from your labeling team and people you want to test.

After labeling an empty dataset each person will get an annotation quality score and detailed report with descriptive comparison of expected an actual output.

Additionally, put a quality score threshold and how many attempts available - and your labelers will be required to try the exam until the quality score needed is met to quarantine great labeling results.

Annotation guides

Not every annotator properly understands labeling requirements from the day one and often makes systematic errors.

Ensure your annotation policy is well presented and acknowledged with annotation guides that include rich text content via markdown, videos and pdf documents, available for the specific team or everyone.

More quality assurance

Notifications for your team

Don’t miss relevant information and stay informed of what’s going on.

Get notifications in your Supervisely dashboard, an email or web-hook event when you are assigned a new labeling job for annotation or review or when you receive a new feedback in issue tracking.

More quality assurance

Resolve issues in style

Mistakes happen, but, with the right tool, you can learn from them and prevent in the future.

Guarantee quality with issues tracking

Built in cooperation with professional labeling teams and inspired by GitHub issues, Supervisely issue tracking is specially designed for annotation at scale that requires high standards of labeling.

Create and inspect issues on invalid assets and objects, discuss and resolve them with your team in both labeling suite and overview dashboard.

Track how mistakes becomes fixes

Issue tracking in Supervisely is not just another tracking system, because it’s focused specifically on annotation. Every labeling object can be discussed and resolved separately.

Managers and reviewers can easily track changes by comparing the differences between initial and final labels.

Discuss issues and find solutions

When something needs collaborative attention of multiple people like managers, domain experts, data scientists, in house and external labelers over time, you need to keep conversation under control.

Supervisely provides a convenient way to discuss labeling issues in a single place with connection to actual labels.

More of Supervisely

And that’s not it!

importworkspacesexportbackupsstatisticsimportworkspacesexportbackupsstatistics

Data management

Easy import and export, workspaces, backups, data insights and statistics.

Learn more

issue trackingnotificationslabeling feedbackteamsissue trackingnotificationslabeling feedbackteams

User collaboration

Collaborate with your team to transform existing assets into labeled data.

Learn more

permissionsrolessslanonymizationssogdprpermissionsrolessslanonymizationssogdpr

Security & access

Protect your data and users with ACL, permissions and other security features.

Learn more

SUPERVISELY FOR ENTERPRISES

On-premise edition built for your business

A fully customizable AI infrastructure, deployed on cloud or your servers with everything you love about Supervisely, plus advanced security, control, and support.

Start 30 days free trial➔

Maximum security: hosted behind firewall on your servers with advanced governance and privacy settings
Effortless integrations: single sign-on with LDAP or OpenID, cloud storage in AWS or Azure and powerful API & SDK
Priority support: dedicated slack chat, guided onboarding and personalized training sessions with experts

supervisely install
> downloading pre-requirements...
> pulling docker images...
> installing software...
> Done! Supervisely is running on port :80

supervisely update
> checking for updates...
> Your version is up to date!

#support

Team Manager 17:33

Hello @supervisely! Is there a way to create a project via API?

Supervisely 17:35

Sure thing, check out this docs!

Here’s why our customers trust us

We use Supervisely since 2019. The key advantage of this tool is that Supervisely provides a complete data treatment pipeline. An important advantage is that a Supervisely instance can be deployed autonomously on a Client infrastructure, and distributed on different servers.

It helps to treat enterprise’s internal and often confidential data in a secured way. Together with a user-friendly interface, a clear documentation and a friendly and reactive support team it helps us to do Data Scientist work better and faster.

Dmitriy Slutskiy

PhD Research Engineer

❮

❯

BMW Group is using the Supervise.ly solution to create automated verifications for ensuring a very high product quality across the whole production chain in vehicle and vehicle component manufacturing.

BMW Group uses Supervise.ly to annotate manufacturing images from production lines in their world-wide plants for enhancing quality inspections using deep learning. The Supervise.ly tooling also supports the process for continuously updating AI models using semi-automated labeling.

Supervise.ly is integrated into the BMW Group AI Platform in order to empower computer vision based AI use cases.

❮

❯

We originally set out to look for tools that could help us with data annotation, and we discovered that Supervisely excels at that and much more. It has become an integral part of our workflow in annotation, model training, and evaluation.

We've been exceedingly impressed with the customer support, addition of new features, and the flexibility of the publicly available SDK/API. The Supervisely team has also been fast to respond to support questions, and has shown a lot of openness when given feedback on potential improvements.

Travis Prosser

Engineering Specialist

❮

❯

We have been using Supervisely for a few years now to help label and organize our data for AI training. The interface is user-friendly and the tools are intuitive to use, which has made the annotation process much more efficient for our team. We run Supervisely locally, which allows us to stay in control of our data. We also use Supervisely for annotation reviews, and the review tools have been invaluable in ensuring the quality and accuracy. The Python SDK has also been incredibly helpful in automating and streamlining our workflow. In addition, the support team on Slack has been extremely helpful and responsive. The ability to collaborate with my colleagues on the same project has also been a huge time-saver.

Overall, we have been extremely satisfied with Supervisely and would highly recommend it to anyone in need of a reliable and efficient annotation solution.