Blob storage

Images, video, backups, documents and logs are too big and too numerous for a database row. Object storage is built for them: huge capacity, high durability, simple API, low cost per byte.

Crux: keep the file in blob storage and a pointer in your database. Let clients upload and download directly with presigned URLs, and put a CDN in front for downloads.

What it is and why it exists

Blob stands for binary large object. Object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage) stores each file as an object: the bytes, some metadata, and a key (a name such as videos/2024/abc.mp4) inside a bucket.

It is not a file system and not a database.

• No real folders. A / in a key is just a character.
• No editing in place. You replace the whole object.
• No queries by content. You find objects by key or prefix, or by metadata you keep elsewhere.

In return you get very large capacity, very high durability (cloud providers advertise many nines, for example S3 states 11 nines, by keeping copies across devices and zones), and a low price per gigabyte.

How it works, at the level you must explain

The API

A small set of HTTP calls: PUT an object, GET it (including a byte range), DELETE, and LIST by prefix. Objects are immutable from the client's point of view. Overwrite means a new object. Versioning can keep old copies. S3 now gives strong read-after-write consistency for new and overwritten objects, which older descriptions of it do not mention.

Presigned URLs: skip your servers

Never stream large files through your app servers. They would burn bandwidth and connections for nothing.

• 1. The client asks your API for permission to upload.
• 2. The server checks the user and returns a presigned URL: a URL with a signature, a limited time (minutes) and a fixed key and method.
• 3. The client uploads straight to blob storage with that URL.
• 4. The storage emits an event (a notification) when the object lands. A worker marks the record as uploaded, or starts processing.

Downloads work the same way for private files: a short-lived signed URL. For public files, serve them through a CDN.

Large files: multipart and resumable uploads

A single upload fails halfway on a bad connection, and a restart wastes everything. Multipart upload splits the file into parts (for example in S3, parts of 5 MB or more, up to 10,000 parts, with a 5 TB object limit). Parts upload in parallel, a failed part is retried alone, and a final call assembles them. For browsers and phones, the client records which parts are done so it can resume. Remember to clean up abandoned multipart uploads with a lifecycle rule, since the stored parts cost money.

Keeping the database and the store in step

Two systems means two sources of truth. Common pattern:

• Database row holds the object key, size, content type, owner and a status (pending, uploaded, processed).
• Create the row as pending, hand out the URL, and set uploaded on the storage event.
• A periodic job deletes rows stuck in pending, and objects with no row (orphans).
• Store the object under a key that does not depend on user-supplied names, such as an id.

Cost and performance knobs

• Storage classes (hot, infrequent access, archive) trade lower cost for slower or pricier retrieval. A lifecycle policy moves or deletes objects by age.
• Prefix and request rate: storage is spread by key. Requests are cheap but not free, and per-request charges matter for billions of tiny objects.
• Replication across regions for disaster recovery or nearer reads.
• Encryption at rest and in transit, and private buckets by default.

Integrity, versions and deletion

• Checksums: the client sends a checksum with the upload and the store verifies it. For large files, keep a checksum per part. This catches corruption on the way in.
• Conditional requests: an object has an entity tag (ETag), a fingerprint of its content. Clients can ask "only if unchanged", which gives cheap revalidation and safe overwrite checks.
• Versioning and soft delete: keep older versions so an accidental overwrite or delete can be undone. Costs storage, so pair it with a lifecycle rule that expires old versions.
• Deleting for real: a user asks for deletion, or retention rules require it. Delete the object, its other versions, and any copies (CDN, replicas, derived thumbnails). Backups age out on their own schedule. For legal obligations, ask Legal or Compliance; I am not an authoritative source.

Large objects you read in pieces

• Reading a byte range is a normal call. That is how video players seek and how a client resumes a download.
• Video is usually cut into short segments, stored as many small objects, and listed in a manifest file. A player fetches segments one after another, which suits both blob storage and a CDN.
• Many tiny objects are expensive and slow to list. If you have millions of 2 KB items, pack them into larger files, or keep them in a database.

Event handling

Storage events are delivered at least once and can arrive out of order. The consumer should be idempotent (mark the row uploaded if it is not already) and should confirm the object exists and matches the expected size before trusting it. A scheduled reconciliation job covers lost events.

Sizing a design

Estimate with our own numbers: 50,000 uploads a day at 4 MB is about 200 GB a day, or about 73 TB a year. That is trivial for object storage but too big for a database, which tells you where the bytes belong. Add a derived copy (thumbnails, transcodes) and the multiple replicas the provider keeps for durability, and check the price with the provider's calculator.

When to use it, trade-offs, numbers

• What: a durable, cheap, scalable store for immutable files.
• Use when: images, video, audio, documents, backups, exports, logs, ML data.
• Know: keys and buckets, presigned URLs, multipart upload, events, lifecycle rules, and keeping a pointer in the database.
• Careful: latency per request is higher than a database or a cache (tens of milliseconds, a rough guide), it is bad at many small updates, and listing is slow at large scale.

Which tool?

• S3 is the default to name. GCS and Azure Blob are equivalents.
• A database: for small structured data and metadata, with a pointer to the blob.
• A distributed file system (HDFS, EFS): when you need real file semantics such as in-place edits.
• CDN: in front, for fast downloads worldwide.

Numbers: the limits above are from S3 documentation as I know it. I am leaving out prices, request costs and per-prefix throughput, since they change. Check the current documentation before quoting them.

Common mistakes

• Using blob storage as the main database.
• Proxying uploads and downloads through the app server.
• Public buckets by accident, or presigned URLs that never expire.
• No cleanup of failed uploads, orphaned objects and stale rows.
• Keeping user-chosen file names as keys.

• No checksum, so a corrupt upload is trusted.
• Trusting storage events to arrive once and in order.
• Forgetting that deleting an object leaves CDN copies, derived files and old versions.
• Storing millions of tiny objects one by one.

Check yourself

• What goes in the database, and what goes in blob storage?
• Walk through a presigned upload. What does the app server do?
• Why is multipart upload useful?
• How would you handle an upload that never finishes?
• Why is blob storage a poor choice for frequent small edits?
• What puts the CDN in this picture?
• What does a checksum protect against?
• Why can storage events not be trusted to be exactly-once?
• Roughly how much storage do 50,000 uploads of 4 MB a day need per year?