Multipart File Upload

In addition to the standard file upload option, BioT provides a multipart upload feature. This feature allows developers to upload a file by splitting it into smaller chunks.

Why upload a file in Multipart?

• Larger File Size Limits: Multipart uploads can handle files much larger than the typical 5GB limit, reaching capacities of up to 5TB.
• Faster Uploads for Large Files: By splitting the file into smaller chunks, multipart uploads can potentially speed up the upload process, especially for very large files.
• Resumable Uploads: In case of an upload interruption, multipart allows you to resume from the point of failure. This means you only need to re-upload the corrupted or missing chunk, rather than the entire file. This is particularly beneficial for large files or unreliable internet connections.

📘

Example

You can find an example of a Python script that uploads a file in parts here:

https://github.com/biot-med/sample-multipart-upload/

Multipart File Upload Process

Uploading a file with several parts has several stages:

1. Request Pre-Signed URLs for File Parts

2. Upload File Parts

3. Notify BioT of Upload Completion

   

Request Pre-Signed URLs for File Parts

To request pre-signed URLs for file parts, send a POST request to:

POST https://example.com/file/v1/files/upload/parts/{id}

With the following body:

{
  "name": "myFile.bin",
  "mimeType": "application/octet-stream",
  "parts": 5
}

The response will include upload URLs for all the file parts:

{
  "name": "myFile.bin",
  "mimeType": "application/octet-stream",
  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "parts": 5,
  "signedUrls": [
    {
      "signedUrl": "stringhttps://dev-file-211125.s3.eu-west-1.amazonaws.com/ba04a3ab-...",
      "partNumber": 1,
      "expirationTime": "2024-04-03T10:57:07.201851068Z"
    },
    {
      "signedUrl": "stringhttps://dev-file-211125.s3.eu-west-1.amazonaws.com/ba04a3ab-...",
      "partNumber": 2,
      "expirationTime": "2024-04-03T10:57:07.201851068Z"
    },
    {...
  ]
}

Save the file ID to be used later in other APIs.

Upload File Parts

To upload the a file part make a PUT API call to the part URL and pass the binary data as the request body.
Once uploaded the response will be empty but the header will contain an ETag header.

The ETag is a unique identifier for the part you just uploaded. You need to save this ID with the uploaded part number.

Notify BioT of Upload Completion

After uploading all file parts, notify BioT of the completion using a POST request to:

POST https://example.com/file/v1/files/upload/parts/{id}/complete

Request Body:

{
  "parts": [
    {
      "partNumber": 1,
      "etag": "70d14508610a82194e3c22df9f2a1698"
    },
    {
      "partNumber": 2,
      "etag": "jfu8608610a82194e3c22df9f75hs2dd"
    },
    ...
  ]
}

For each part set the corresponding ETag ID.
BioT will concatenate all the parts and return the ID of the file with a link to download it.

📘

Note

• Minimum Part Size: There is a minimum size requirement for each file part you upload to BioT. This minimum size is typically 5MB.
• Last Part Exception: The only exception to this rule is the last part of the file. The last part can be smaller than 5MB, as long as it encompasses the remaining data.

Error Recovery

In case a file part upload fails, you have options for recovery:

• Retry Uploading the Part: If an upload error occurs for a specific part, you can attempt to re-upload that part using the same pre-signed URL you received earlier. BioT might allow retries for failed uploads.
• Regenerating Pre-Signed URLs (if necessary): If the pre-signed URL for the failed part has expired, you'll need to generate a new set of pre-signed URLs for all parts. You can achieve this by sending a POST request to the following API:
  POST https://example.com/file/v1/files/upload/parts/{id}

Uploading files with unknown length

In some cases you might not know what is the final size of the complete file. It's perfectly acceptable to upload only a portion of the parts and then notify BioT that the file is complete.

If you need additional pre-signed URLs for more file parts during the upload process, you can request them using a POST request to the following API:

POST https://example.com/file/v1/files/upload/parts/{id}

Cancel file upload

To cancel a partially uploaded file, use the following API:

DELETE https://example.com/file/v1/files/upload/parts/{id}/cancel

Only incomplete files can be cancelled.

Progress Updates for Multipart Uploads (Optional)

You don't have to wait until the end of the upload to update BioT about an uploaded part.
You may use this API:

PUT https://example.com/file/v1/files/upload/parts/{id}

With body:

{
  "parts": [
    {
      "partNumber": 1,
      "etag": "70d14508610a82194e3c22df9f2a1698"
    },
    ...
  ]
}

BioT will keep the ETag value for each part and use them to concatenate the final file when requested to do so.