Labeling Editor Interface

Overview of the labeling interface on Ango Hub.

Ango Hub has labeling editors for each of the supported formats: audio, image, DICOM video, PDF, and text (NER).

Even though the data types supported by Ango Hub are all different, we designed our labeling editors to be as similar to one another as possible. This way, once an annotator has learned how to use one, they will be able to quickly switch to the other, without having to spend time retraining.

In this article, we will go through the features in common between all editors. At the bottom of this article are links to documentation pages for more details on each of the editors.

The following is an overview of the editor’s interface.

Overview of the Labeling Editor on Ango Hub

Left Sidebar

Tools (when available)

At the top of the left sidebar is the list of all labeling tools available in the project. Bounding boxes, polygons, points, and entities, for example, are all labeling tools.

Tools of different types can be present in the same project. To annotate, the labeler clicks on the tool from this section then clicks on the asset, where the label needs to be placed.

Questions (when available)

In the middle of the sidebar are the top-level classification tools for this project.

If created from the Label Set section of the Settings tab, classifications will be shown here. From this section, annotators can categorize the asset shown, with radio buttons, dropdowns, free text, and more.

Objects

Below the classification section is a list of all annotations (objects) placed on the current asset.

As the annotator adds labels to the asset, they will be shown in this list. The “eye” icon next to Objects allows you to hide all labels on the screen.

If the label has a nested question, by clicking on it in the list, you can expand it.

The eye icon allows you to hide the label from the asset. You can also use the shortcut H while hovering over an annotation to hide it. To unhide all annotations, press Shift + H.

The trash can deletes it. If present, nested labels will be shown below. In the example above, we ask the labeler what kind of vehicle they have just labeled with the bounding box tool.

Clicking on the three dots will open a submenu:

Change category

If in your project you have more than one of the same tool, you can quickly change the label’s category from here. If we had another bounding box tool called “tree”, for example, we would be able to change this label’s category from “vehicle” to “tree.”

It is not possible to change an annotation to a label of a different type. (for example, a bounding box into a polygon.)

Another way to quickly change a label's category is to hover over it with the mouse cursor and pressing Alt + the new category's keyboard shortcut.

Copy Object ID

Each object (annotation) has a unique ID. Clicking this button will copy the annotation's ID to the clipboard.

Lock

A locked label cannot be edited. The lock function is particularly useful when importing labels, if there are labels you'd like the annotators not to change.

Set Flag

You can flag labels. The flag does not have a particular meaning out-of-the-box. The flag function is particularly useful when performing post-labeling QA checks through our API.

For example, imagine you have a tool that automatically detects abnormal labels, connected to our platform using our API. You can have this tool flag all abnormal labels for manual review.

Quick Object Scrolling

You can quickly move between objects in the editor by hovering over an object such that it is selected, and then by pressing on Shift + Up or Shift + Down to navigate between objects. The selected object will automatically be shown in the middle of the screen and the zoom level will be accommodated to better view the object.

Reset and Submit/Save

At the bottom of the sidebar, we find the Reset and Submit/Save buttons.

Reset undoes all changes made, restoring the state found when the asset was first opened.

Save saves the annotations made on the asset. If the button shows instead the word Submit, changes will be saved and you will be brought to the next unlabeled asset.

There is no autosave.

If you try to leave the page without saving, you will be asked to save before exiting.

Top bar

On the left-hand side are links to navigate between projects and within the current project.

The Ango Logo opens your project list. The Home opens the current project’s dashboard. The Back button brings you back one page.

Asset Tasks (Reviewer and Project Owner Only)

The number of circles appearing in the top bar is the number of labeling tasks associated with the asset. In the picture above, for example, the current asset was labeled three times.

Hovering over a circle will show detailed information about each of the labeling tasks. Clicking on a circle will open the task.

A green checkmark indicates that the task was completed.

A yellow star indicates that the task has been marked as a benchmark. More on benchmarks here.

A blue label indicates that the labels in the task were created outside of Ango Hub, and have been imported. More on importing annotations here.

A grey circle indicates that the task has not been completed, thus, it will not appear in the export.

The arrows allow users to move back to the previous asset, and forward to the next asset.

The navigation arrows are only shown if the current task was opened by following an internal link from within the platform. Directly opening a task from an URL will prevent the arrows from showing.

Right-Hand Side

The Keyboard icon opens a panel showing all available keyboard shortcuts. The Bell icon shows the user’s notifications. The circle allows the user to enter their profile and log out.

Starting from the left-hand side,

Submenu (Reviewer, Manager, and Project Owner only)

The three dots open a submenu with a number of options:

Set as benchmark sets the current task as benchmark.

Requeue resets the current task and puts it back into the labeling or reviewing queue. (e.g., it will be shown to labelers and reviewers again)

Delete deletes the current task.

Change Assignee changes the assignee of the current task. Practically, it changes the name of the person who has labeled the task.

Export the current task (coming soon)
Delete all annotations in the current task
Copy all annotations from the current task
Paste all annotations to the current task
Plugins.

Mark the current task as Accepted (Project Owner, Manager, and Reviewer only)
Mark the current task as Fixed (Project Owner, Manager, and Reviewer only)
Mark the current task as Rejected (Project Owner, Manager, and Reviewer only)
Set the current task as sample (Project Owner, Manager, and Reviewer only)
Create an Issue on a specific location on the asset (Image, Video, and Text only)
Flag the current task.
Star the current task.

Spot Issues

Clicking on the bubble icon on the right side of the toolbar will turn the cursor into an issue bubble. When in this mode, click on the asset. An issue will be placed at your cursor's position.

To submit the issue, type into the box and press Enter.

Spot issues are not available for all file types. Currently, Image, Video, DICOM, and Text support Spot Issues.

Right Sidebar

Issues

From the right sidebar, open the Issues panel. This will show all issues present in the current task.

Users can type text in the + Create Issue field and press Enter to create issues, which will be sent as in-app notifications to task assignees, project owners, and project reviewers.

The user who created the issue, as well as reviewers and project owners, will be able to reply to the issue, resolve it, or delete it from the buttons on the right side of the panel.

Instructions

The Instructions button opens a drawer showing labeling instructions for the current project, if present. Project owners can upload instructions as a PDF file from the project’s Settings tab.

The buttons above the PDF allow the user to reset the zoom, zoom in and out, and to navigate between pages.

Samples

The Samples button opens a drawer showing all samples set in the project. More on samples here.

Clicking on a sample in the list will open its related asset and description. Clicking on the Arrow in box button in the sample will open an enlarged view of the sample:

Attachment

Clicking on Attachment will open a drawer showing the attachment linked to the current asset, if available. More on attachments here.

Task info

Clicking on Task info will show information related to the current task.

Assets drawer (Project Owner, Manager, and Reviewer only)

Clicking on the right-facing arrow on the left side of the screen will open the Assets drawer.

The assets drawer contains a list of all assets in the current project, together with their related information such as their status, consensus score, assignees, and more. Clicking on an item in the list will open the related asset.

From the top, clicking on will refresh the asset list. Clicking on X will close the drawer. At the bottom, arrows and page numbers allow you to navigate the asset list.

Labeling Settings

Clicking on the blue Settings button on the bottom left of the editor will open a panel with a number of quick labeling settings.

Each data type has its own set of settings. This page will only go through those that are common to all data types. For more details on each data type's settings, visit its page, for example, Image Labeling Editor.

Fast Tool Selection

By default, after placing an annotation, the current labeling tool is deselected, and the annotator needs to click on a labeling tool again to place another annotation.

When this toggle is selected, placing annotations will not deselect the currently selected labeling tool, and the annotator can immediately place another one. The annotator can deselect the currently selected labeling tool by pressing Esc.

When this setting is activated, right-clicking on an asset will automatically open the expanded version of the context menu, instead of opening the collapsed one.