Importing Private Cloud Assets from AWS S3

How to import assets in private buckets on AWS S3 for labeling on Ango Hub

Administrators and project managers can import assets to Ango Hub from AWS private S3 buckets.

The main difference between importing data from cloud storage services and using drag-and-drop is the files' location. When importing assets with drag-and-drop, the assets get copied to a folder within Ango Hub. This means that, if you are using Hub's cloud version, your assets are uploaded to Ango AI's own cloud storage. When importing assets from your own cloud storage, however, your assets are left in your own storage and are never copied anywhere else.

Overview of Importing Private S3 Assets into Ango Hub

Configure CORS

The CORS header below allows Ango Hub to send a request to your cloud storage, and allows your cloud storage to explicitly allow requests from Hub. This is a necessary step to ensure Hub can connect to your private bucket.

The steps to follow to set up CORS with Ango Hub can be found here.

Connect your Cloud Storage

Once you've set up CORS for your bucket, you will need to create a connection between Hub and the bucket itself.

To link your bucket to Hub, you will need your bucket's Access Key ID and Secret Key. To obtain them, you will need to create an IAM user on your cloud storage provider's management dashboard. More information on how to do this here.

Both your Access Key ID and your Secret Access Key will be alphanumeric strings, the former 20 and the latter 40 characters long.

Once you have obtained the two strings, go to your organization's page, then click on Integrations and Add Integration.

From the dialog that pops up, provide your connection with a unique name, pick your provider, and input the aforementioned keys.

After clicking OK, your bucket will be linked to Hub, and it will show up in your list of integrations.

Integrations belong to your organization, not your user.

If you want to be able to access your non-public cloud data in multiple organizations, you will have to integrate Hub with AWS in every one of them.

Preparing the JSON

After connecting our bucket, we will need to prepare a JSON file containing each asset’s external ID as well as the asset's full absolute path, plus optionally the image's width and height if the asset is an image.

The width and height fields are only necessary if you plan on exporting polygon annotations as image bitmasks. In most cases, width and height information will not be necessary.

Ensure your URLs are percent-encoded in the UTF-8 format.

If your filenames contain spaces, for example, ensure they are encoded as %20, and not as pluses (+).

This is what the JSON should look like:

[
  {
   "data":"https://bucket-name.s3.eu-central-1.amazonaws.com/file.jpg",
   "externalId":"cute-cat.jpg",
   "width":603,
   "height":450
  },
  {
   "data":"https://bucket-name.s3.eu-central-1.amazonaws.com/file%202.jpg",
   "externalId":"cute-dog.jpg",
   "width":1800,
   "height":1200
  }
]

There is no upper bound to the number of assets that can be imported to a project in Ango Hub this way. There is no file size restriction.

Uploading the JSON to Ango Hub

From your project’s dashboard, enter the Assets tab and click on Add Data.

A dialog will pop up. Click on “Upload Data URL” at the top.

Click on Advanced Configs and from Storage Method, pick the storage integration you created in the previous step.

Drag the JSON file you would like to upload to the box in the center. Alternatively, click on the box to open your system’s file explorer and select it from there.

Click on the Close button. Your assets will show up in the Assets tab.

It is also possible to upload assets through our API.