[WIP] minna-caos

Content-Addressed Object Storage server intended for usage with Minna.

minna-caos was created because the developers found themselves writing similar storage backends in every project, again and again.

It is intended as a companion to a single application (“the app”) that has full authority. You should imagine it as a library, not a stand-alone application.

Features

• resumable uploads (via draft-ietf-httpbis-resumable-upload-08)

• automatic media type (“MIME type”) detection (via Wikidata)

• multiple storage backends (via Apache OpenDAL™):

  • local filesystem
  • S3-compatible
  • FTP
  • Google Drive
  • OneDrive

• direct download from the underlying storage backend (if supported)

• named storage buckets (backed by a single storage backend each)

• deletion tombstones

• SQLite database

• CLI for administration tasks

Notes

Example file upload flow

1. The client notifies the app that it wants to upload something.

2. The app sends a POST /uploads/ request to minna-caos. It receives the following response:

   {
     "upload_id": "UPLOAD_ID"
   }
   

3. The app sends the upload ID to the client.

4. The client sends a PATCH /uploads/UPLOAD_ID request with the upload data to minna-caos.

5. The client goes offline during uploading. When it is online again, it sends a HEAD /uploads/UPLOAD_ID request to minna-caos. It receives the current upload offset 123 in the Upload-Offset header.

6. The client resumes the upload by sending another PATCH /uploads/UPLOAD_ID request with the upload data to minna-caos. The Upload-Offset request header is set to 123.

7. Meanwhile, the app continuously polls the GET /uploads/UPLOAD_ID endpoint of minna-caos until it returns a response with the state field set to "finished":

   {
     "id": "UPLOAD_ID",
     "state": "finished",
     "hash": "OBJECT_HASH",
     "size": 200,
     "media_type": "image/png",
     "bucket_ids": ["staging"]
   }
   

8. Based on the returned metadata, the app decides that it will accept the upload and place it in the local bucket. It sends a POST /uploads/UPLOAD_ID/accept request to minna-caos with the following payload:

   {
     "buckets": ["local"]
   }
   

9. minna-caos starts copying/uploading the object from the staging bucket to the local bucket.

   Even while this process is still running, the object data is already accessible at /objects/OBJECT_HASH.

Filesystem requirements

minna-caos uses the local filesystem as a staging area for uploads. (This is the special staging bucket seen in the example above.)

The filesystem containing the staging directory must…

• support hardlinks
• have enough storage space available to fit all uploads until they are uploaded to actual buckets

Roadmap

• metadata endpoints
• support non-resumable clients
• send the draft version header
• support upload cancellation
• upload expiration
• garbage-collect failed uploads
• add code comments
• graceful shutdown
• more storage backends
• CLI

See also

• minna-pima — integrates with minna-caos to provide picture management features