Use pre-signed S3 URLs for password-protected messages

[sosnovsky]

Currently password-protected messages attachments have maximum size ~5mb. To support larger attachments, we implemented new API in FES - it uses pre-signed S3 URLs, so messages are uploaded directly from browser extension.

This change requires some code changes:

• instead of using /api/v1/message/new-reply-token, we should use new POST /api/v1/messages/allocation endpoint. It returns response { "storageFileName": string, "replyToken": string, "uploadUrl": string }
• then browser extension should upload generated password-protected message content to S3 (using storageFileName as file name) by sending PUT request to uploadUrl
• after successful message upload, extension should send POST /api/v1/messages request with structure similar to current POST /api/v1/message request in webMessageUpload, but instead of including message content as encrypted.asc file, request should use storageFileName property with value received from POST /api/v1/messages/allocation request

Also, let's leave ability to use our existing api - so both these implementations will be available in browser extension. By default, new flow with pre-signed S3 URL should be used, but if client configuration has DISABLE_FES_PRESIGNED_URLS property - extension should use current flow with message upload to FES.

For now, this functionality is enabled only for ...@flowcrypt.com users, so for testing please use ...@flowcrypt.com email address. If you'll have any questions or find some issues - please mention me or @Jerbell.

[ioanatflowcrypt]

@sosnovsky

Testing the new pre-signed S3 URL upload flow and hitting a CORS error when the browser extension tries to upload directly to S3.

Error:

Access to XMLHttpRequest at 'https://flowcrypt-org-web-portal-internal.s3.eu-central-1.amazonaws.com/...' 
from origin 'chrome-extension://dmmlbphofjjnpmencolljbipabemhmoo' has been blocked by CORS policy: 
Response to preflight request doesn't pass access control check: 
No 'Access-Control-Allow-Origin' header is present on the requested resource.

What's needed:

I think we should add CORS config to the flowcrypt-org-web-portal-internal S3 bucket to allow PUT requests from browser extensions like below:

        "AllowedOrigins": ["chrome-extension://*", "moz-extension://*"],

What's your thought?

[Jerbell]

I would say - whatever works. Can you adjust the S3 CORS settings or would you like me to do it? We can at least try it out.

EDIT: I've added those 2 origins in as you've suggested.

[ioanatflowcrypt]

Yeah, now upload to pre-signed SD URL works.
Btw when I visit password protected message url, it returns 401 error.
https://fes.flowcrypt.com/messages/541bab0e-dbab-4180-a9bc-0a9396f7d3b5
Am I doing something wrong?

[ioanatflowcrypt]

AFAIK, when I visit password protected message url, it should ask for password and then open message after user enters message

[ioanatflowcrypt]

However, we may add our extension specific chrome-extension records in CORS option if we don't want to add wildcard option though

[Jerbell]

The "web page" still uses /message/ so that existing emails still work.
The API uses /messages/
so the URL you want is https://fes.flowcrypt.com/message/541bab0e-dbab-4180-a9bc-0a9396f7d3b5

[Jerbell]

However, we may add our extension specific chrome-extension records in CORS option if we don't want to add wildcard option though

I'll do wildcard for now - I don't know how often this id changes in Chrome (plugin update?), but if it's always the same then we can put the id in too.

EDIT: I'll leave this up to @sosnovsky

[sosnovsky]

id for Chrome Web Store builds is always the same - chrome-extension://bnjglocicdkmhmoohhfkfkbbkejdhdgc, so we can set it explicitly for shared-tenant bucket.
But local extension builds have dynamic ids, so for our flowcrypt bucket let's leave wildcard chrome-extension://*
Also Firefox seems to generate ids dynamically during extension installation, so we should use wildcard moz-extension://* for both our buckets.

[ioanatflowcrypt]

Thank you for your reply.
In this case you should return /message instead of messages in https://fes.flowcrypt.com/api/v1/messages api.
Currently it returns this response.

{
    "emailToExternalIdAndUrl": {
        "{email}": {
            "externalId": "1935de66-6a41-4818-a014-4cfbc82d7994",
            "url": "https://fes.flowcrypt.com/messages/1935de66-6a41-4818-a014-4cfbc82d7994"
        }
    }
}

The "web page" still uses /message/ so that existing emails still work. The API uses /messages/ so the URL you want is https://fes.flowcrypt.com/message/541bab0e-dbab-4180-a9bc-0a9396f7d3b5

[Jerbell]

That looks like a bug! I'll sort it out soon, though not immediately. (need to have dinner). Which timezone are you?

[ioanatflowcrypt]

GMT+2

[ioanatflowcrypt]

Sure, take your time. Not urgent at all