Solution Design

Resumable Upload

File Chunking

A file is essentially a stream of binary content, so it can be split according to certain rules and uploaded piece by piece, keeping track of which chunks have been uploaded successfully and which have failed or are not yet uploaded.

The general flow is as follows:

Split the file to be uploaded into chunks of equal size.
Initialize the upload task and obtain a unique task ID from the backend (it can also be generated on the frontend and sent to the backend). Initialization usually also needs to include:
- Full file hash/CRC value: used to verify whether the final merged file is intact.
- Number of chunks or the set of all chunk IDs: lets the server know which chunks will be uploaded.
Send the chunks serially or in parallel. Each chunk contains:
- Task ID: the unique ID of the task it belongs to.
- Binary content: the file's content for the current chunk.
- Sequence number/ID: marks which chunk this is.
- Hash: the hash of the current binary content, used for chunk integrity verification.
The server verifies data integrity and merges the chunks back into the original file once complete.

Resumable Upload

Resumable upload relies on file chunking; content that has not been chunked cannot be resumed.

The general flow is as follows:

Get the upload progress from the server to find out which chunks still need to be uploaded.
The frontend continues uploading the chunks that the backend reports as not yet uploaded.
Once the backend detects that all chunks are present, it reads them from temporary files, merges them, and verifies the hash.

Streaming Upload

While reading the file buffer, upload as you read, instead of uploading only after the entire file has been read.

Note that HTTP/2 streaming may suffer from head-of-line blocking; HTTP/3 is recommended.

Other Considerations

Automatic retry on upload failure, with a limit on the number of retries
Abnormal interruption caused by navigating away, network disconnection, or page refresh during upload — recovery is needed
When chunking, split by file size or by count? How should the strategy be decided?
Background upload with web worker
Instant upload
- When creating a task, the file hash is sent; the backend checks whether it already exists. If it does, there is no need to upload again — this is instant upload.
Limit concurrency to avoid the browser/server occupying too many connections
Dynamically adjusting chunk size?
Scheduled tasks to clean up file fragments

Sample Code

Chunked Upload

const SIZE = 10 * 1024 * 1024 // 10MB

class fileUpload {
  // Generate file chunks
  createFileChunk(file, size = SIZE) {
    const fileChunkList = []
    let cur = 0
    while (cur < file.size) {
      fileChunkList.push({ file: file.slice(cur, cur + size) })
      cur += size
    }
    return fileChunkList
  }

  // Upload chunks
  async uploadChunks() {
    const requestList = this.data
      .map(({ chunk, hash }) => {
        const formData = new FormData()
        formData.append('chunk', chunk)
        formData.append('hash', hash)
        formData.append('filename', this.container.file.name)
        return { formData }
      })
      .map(({ formData }) =>
        this.request({
          url: 'http://localhost:3000',
          data: formData,
        })
      )
    // Concurrent requests
    await Promise.all(requestList)
    //  Merge chunks
    await this.mergeRequest()
  }

  async mergeRequest() {
    await this.request({
      url: 'http://localhost:3000/merge',
      headers: {
        'content-type': 'application/json',
      },
      data: JSON.stringify({
        filename: this.container.file.name,
      }),
    })
  }

  async handleUpload() {
    if (!this.container.file) {
      return
    }
    const fileChunkList = this.createFileChunk(this.container.file)
    this.data = fileChunkList.map(({ file }, index) => ({
      chunk: file,
      hash: `${this.container.file.name}-${index}`,
    }))
    await this.uploadChunks()
  }
}

Computing Hash with web-worker

// web-worker code

// The spark-md5 library, which computes a file's hash from its content
globalThis.importScripts('/spark-md5.min.js')

// Generate file hash
globalThis.onmessage = (e) => {
  const { fileChunkList } = e.data
  const spark = new globalThis.SparkMD5.ArrayBuffer()
  let percentage = 0
  let count = 0

  const loadNext = (index) => {
    const reader = new FileReader()
    reader.readAsArrayBuffer(fileChunkList[index].file)
    reader.onload = (e) => {
      count++
      spark.append(e.target.result)

      if (count === fileChunkList.length) {
        globalThis.postMessage({
          percentage: 100,
          hash: spark.end()
        })
        globalThis.close()
      } else {
        percentage += 100 / fileChunkList.length
        globalThis.postMessage({
          percentage
        })
        // calculate recursively
        loadNext(count)
      }
    }
  }
  loadNext(0)
}

// Communication with the main thread
// The main thread uses postMessage to pass all chunks (fileChunkList) to the worker thread,
// and listens for postMessage events from the worker thread to get the file hash
function calculateHash(fileChunkList) {
  return new Promise((resolve) => {
    this.container.worker = new Worker('/hash.js')
    this.container.worker.postMessage({ fileChunkList })

    this.container.worker.onmessage = (e) => {
      const { percentage, hash } = e.data

      this.hashPercentage = percentage
      if (hash) {
        resolve(hash)
      }
    }
  })
}

References

Frontend Authentication

HTTP Basic Authentication (Mostly Deprecated)

Authentication Flow

---
title: HTTP Basic Authentication Flow
---
sequenceDiagram
  participant Client
  participant Server

  Client ->> Server: Request restricted data
  Server ->> Client: HTTP/1.1 401 Unauthorized<br>WWW-Authenticate: Basic realm="baidu.com"

  Client ->> Client: Prompt the user with a dialog
  Client ->> Server: Send the Base64 authorization header<br>Authorization: Basic Kitsid2FuZzp3YW5n==

  Server -->> Server: Verify identity

  alt Verification succeeded
    Server -->> Client: Verification succeeded, return the resource with status 200
  else Verification failed
    Server -->> Client: Verification failed, return 403 Forbidden
  end

Advantages

Simple, widely supported

Disadvantages

Insecure

HTTP is not encrypted, so the account and password for authentication are transmitted in plaintext.
Even if encrypted, permissions can be obtained by replaying the request.
Hard to defend against man-in-the-middle attacks.

Authentication Flow

---
title: Cookie-Session Authentication Flow
---
sequenceDiagram
  participant Browser
  participant Server
  participant User
  participant Session

  Browser ->> Server: POST account and password
  Server ->> User: Verify account and password
  User -->> Server: Verification succeeded
  Server ->> Session: Store session
  Server -->> Browser: Set-Cookie: sessionId

  Browser ->> Server: Request API with cookie
  Server ->> session: Check whether the session exists and is valid
  session -->> Server: Verification succeeded
  Server ->> Server: Handle the API request
  Server -->> Browser: API response

Storage Options

The server only gives the cookie a sessionId; the actual session content has to be stored somewhere yourself

Redis (recommended): an in-memory database that stores data as key-value pairs, which fits the sessionId-sessionData scenario perfectly; and it is fast to access.
Memory: stored directly in a variable. It is gone once the service restarts.
Database: a regular database. Performance is not great.

token

Authentication Flow

---
title: Token Authentication Flow
---
sequenceDiagram
  participant Browser
  participant Server
  participant User

  Browser ->> Server: POST account and password
  Server ->> User: Verify account and password
  User -->> Server: User data
  Server ->> Server: Generate token
  Server -->> Browser: Set-Cookie: token

  Browser ->> Server: Request API with cookie
  Server ->> Server: Verify token
  Server ->> Server: Handle the API request
  Server -->> Browser: API response

Problems

Easy to tamper with, so a token signature is needed — that is the JWT mechanism
Easy to be stolen, which is why refresh tokens exist

JWT

It consists of 3 parts: header, payload, and signature

The header looks like this:

{
  "typ": "JWT",
  "alg": "the signing algorithm used"
}

The payload is mostly custom content; some public information is stored in the payload to reduce repeated lookups

{
  "sub": "subject, usually the user ID",
  "name": "username",
  "role": "user role",
  // Expiration time
  "exp": 1627708800
}

The signature is generally just a plain string

All three parts are base64 encoded and joined with .

refresh token

The refresh token mechanism has two parts:

access token: used to access business APIs. Because its lifetime is short enough, the risk of theft is small, and the way requests are made can be more flexible
refresh token: used to obtain an access token. Its lifetime can be longer, and security is increased through a dedicated service and strict request methods; since it is not verified often, it can also be handled like the session above

The full flow is as follows:

---
title: JWT Authentication Flow Sequence Diagram
---
sequenceDiagram
  participant Browser
  participant Business Service
  participant Auth Service
  participant User

  rect rgb(222,227,232)
  Note over Browser, User: Initial login
  Browser ->> Auth Service: POST account and password
  Auth Service ->> User: Verify account and password
  User ->> Auth Service: User data
  Auth Service ->> Auth Service: Generate refresh token (long-lived)
  Auth Service ->> Auth Service: Generate access token (short-lived)
  Auth Service -->> Browser: refresh token, access token
  end

  rect rgb(222,227,232)
  Note over Browser, User: access token valid
  Browser ->> Business Service: Request API (access token)
  Business Service ->> Business Service: Verify access token (valid)
  Business Service ->> Business Service: Handle the API request
  Business Service -->> Browser: API response
  end

  rect rgb(222,227,232)
  Note over Browser, User: access token expired
  Browser ->> Business Service: Request API (access token)
  Business Service ->> Business Service: Verify access token (expired)
  Business Service -->> Browser: Reject the request
  Browser ->> Auth Service: refresh token
  Auth Service ->> Auth Service: Verify refresh token (valid)
  Auth Service -->> Browser: access token (new)
  Browser ->> Business Service: Request API (access token (new))
  Business Service ->> Business Service: Verify access token (new)
  Business Service ->> Business Service: Handle the API request
  Business Service -->> Browser: API response
  end

Single Sign-On

---
title: Single Sign-On Authentication Sequence Diagram
---
sequenceDiagram
  participant User as Browser
  participant AppA as App A
  participant AppB as App B
  participant SSO as SSO Service

  rect rgb(232,244,232)
  Note over User,SSO: Login initiated by App A

  %% Credential expired
  User ->> AppA: Visit App A
  AppA ->> AppA: Credential expired, login required
  AppA -->> SSO: Redirect to SSO

  %% SSO login
  User ->> SSO: Visit SSO, no cookie
  SSO ->> SSO: Cookie verification failed
  SSO -->> User: Login required
  User ->> SSO: Enter account and password
  SSO ->> SSO: Verify account and password
  SSO -->> User: Redirect to App A callback with code in the URL<br>Set an SSO-domain cookie

  %% SSO syncs the authorization state to AppA
  User ->> AppA: Visit App A callback with cookie
  AppA ->> SSO: Verify code
  SSO ->> SSO: Code verified successfully
  SSO -->> AppA: Exchange for a ticket
  AppA -->> User: Redirect to the original App A page, set an A-domain cookie (ticket)

  %% User revisits AppA
  User ->> AppA: Visit App A callback with cookie (ticket)
  AppA ->> SSO: Verify ticket
  SSO ->> SSO: Ticket verified successfully
  SSO -->> AppA: Credential valid
  AppA ->> AppA: Handle business logic
  AppA ->> User: Business response
  end

  rect rgb(232,232,244)
  Note over User,SSO: App B requires no login
  %% App B has not synced the ticket yet
  User ->> AppB: Visit App B
  AppB ->> AppB: Credential expired, login required
  AppB -->> User: Redirect to SSO

  %% Sync login state from SSO
  User ->> SSO: Visit SSO with SSO-domain cookie
  SSO ->> SSO: Cookie verified successfully
  SSO -->> User: Redirect to App B callback with code in the URL, set an SSO-domain cookie
  User ->> AppB: Visit App B callback with code
  AppB ->> SSO: Verify code
  SSO ->> SSO: Code verified successfully
  SSO -->> AppB: Exchange for a ticket
  AppB -->> User: Redirect to the original App B page, set a B-domain cookie (ticket)

  %% User revisits AppB
  User ->> AppB: Visit App B with B-domain cookie (ticket)
  AppB ->> SSO: Verify ticket
  SSO ->> SSO: Ticket verified successfully
  SSO -->> AppB: Credential valid
  AppB ->> AppB: Handle business logic
  AppB -->> User: Business response
  end

OAuth

---
title: OAuth Authentication Sequence Diagram
---
sequenceDiagram
  participant User as Browser
  participant Client as OAuth Client
  participant AuthServer as OAuth Server
  participant ResourceServer as Resource Server

  User ->> Client: Access a resource that requires authorization
  Client -->> User: Redirect to the authorization page

  User ->> AuthServer: Carry client_id/redirect_uri/scope
  AuthServer ->> User: Ask to log in and authorize
  User ->> AuthServer: Enter credentials and authorize
  AuthServer -->> User: Redirect to redirect_uri with code

  User ->> Client: Visit with the authorization code
  Client ->> AuthServer: Send code + client_secret to exchange for an access_token
  AuthServer -->> Client: Return the access_token

  Client ->> ResourceServer: Request the resource with the access_token
  ResourceServer ->> AuthServer: Verify the token's validity
  AuthServer -->> ResourceServer: Return the verification result

  alt Verification succeeded
    ResourceServer -->> Client: Return the requested resource
    Client -->> User: Display the authorized resource
  else Verification failed
    ResourceServer -->> Client: Return 401 Unauthorized
    Client -->> User: Display an error message
  end

  Note over Client,AuthServer: The access_token must be stored securely<br>with a reasonable expiration time

Resumable Upload

File Chunking

Resumable Upload

Streaming Upload

Other Considerations

Sample Code

Chunked Upload

Computing Hash with web-worker

References

Frontend Authentication

HTTP Basic Authentication (Mostly Deprecated)

Authentication Flow

Advantages

Disadvantages

cookie-session

Authentication Flow

Storage Options

token

Authentication Flow

Problems

JWT

refresh token

Single Sign-On

OAuth