Designing a File Storage System

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 10 of 26

← Previous: Designing a Log Aggregation System → Next: What Is an LLM Inference Server?

Layer 6 — Modern systems · Post 72 of 88

A file storage system stores blobs reliably, serves them quickly, and handles uploads, metadata, permissions, and virus-scan-sized side jobs.

What you'll learn

Why object storage plus a metadata database beats putting files on the app server disk
How presigned uploads, virus scan, and CDN download split trust and load
What to do when a multipart upload dies, or a public URL leaks

The idea in one minute

A file storage system (Dropbox-like, or "user avatars and PDFs") is two stores: blobs (immutable bytes in S3/GCS/MinIO) and metadata (owner, name, size, content-type, ACL, object key) in a database. The app never streams a 2 GB video through the API box if it can help it.

Client | 1. ask permission v [ Metadata API ] ---- row: file_id, key, status=pending ----> [ DB ] | 2. presigned PUT v [ Object store ] <--- bytes --- client | 3. complete + notify v [ Workers: virus scan, thumbnail, virus fail --> quarantine ] | v GET file --> authz on metadata --> redirect / signed GET --> [ CDN / origin ] | object store down? uploads fail; existing CDN cache may still serve hot objects

Layer 1's CDN still sits in front for public or signed download. This post is the control plane around the blob.

Why it matters

Interviewers use this to hear direct-to-object-store uploads, authorization, and async processing. If all bytes go through your web tier, you will hit timeouts, memory, and a single-region bottleneck.

Permissions on the metadata row are the product. A signed URL with a long TTL is a capability you accidentally emailed to the world.

How it works

Create. Client calls POST /files with filename, size, content-type. API checks quota, inserts status=uploading, returns a presigned PUT (or multipart upload IDs) scoped to a key like tenant/file_uuid. The client uploads directly to the object store.
Complete. Client calls POST /files/{id}/complete. You HEAD the object, verify size and checksum (MD5/ETag or CRC you required), set status=quarantined or ready.
Side jobs. A queue: antivirus, image variants, transcoding, malware quarantine. Until scan passes, do not issue a public URL. Failure: mark blocked, delete or isolate the blob.
Download. GET /files/{id} checks ACL (owner, share link, team). Then: redirect to a short-lived signed GET, or a CDN URL with a token. The API is not a reverse proxy for 4K video.
Multipart. Large files: initiate, upload parts, complete. Abandoned uploads: a sweeper aborts and deletes parts so you do not pay for garbage.
Failure. Mid-upload crash: metadata stays uploading until TTL, then GC the key. Object exists, complete never called: same GC. Metadata DB down: no new grants; signed URLs already issued still work until expiry.

Dedup (optional): hash the blob, point multiple files at one key — only after you understand that a delete must be reference-counted.

A simple example

You attach a 40 MB PDF to a ticket. The app gets a presigned URL, PUTs to S3, then hits complete. A worker ClamAV-scans, writes status=ready, and generates a preview JPEG. A teammate opens the ticket; the API checks they are on the org, returns a 5-minute signed URL. The browser fetches from CloudFront. You retry complete after a timeout: checksum matches, status stays ready, no second scan storm if the job is idempotent on file_id.

If you had PUT /upload through nginx with a 30s timeout, the 40 MB file from a slow hotel Wi-Fi would never finish.

Common mistakes

Proxying all uploads through the app. Use presigned URLs (with size and content-type constraints).

Public buckets "because the CDN is easier." Guessable keys leak. Default private; sign or token.

Long-lived signed URLs in emails. Prefer short TTL plus your own share tokens you can revoke in the DB.

Trusting Content-Type from the client without checking magic bytes for images you will serve inline (XSS).

No quota or virus step on user-generated content. You will host malware and a surprise storage bill.

How this shows up in real systems

S3 / GCS / Azure Blob + CloudFront: the default blob layer.
Dropbox, Google Drive, Notion, Slack files: metadata, sharing, and async processing on top.
Imgix / image CDNs: variants as a derived pipeline, same complete-then-process idea.

The rest of this layer shifts to AI infra. An LLM inference server is the new app server — GPUs instead of request threads.