Handling large blobs

A blob is a large chunk of bytes with no structure the database cares about: a photo, a video, a PDF, a backup. Pushing a 2 GB video through an app server ties up its memory, its network and a connection for minutes, for no benefit.

Crux: keep the bytes out of your servers. Your database holds the metadata, blob storage holds the bytes, and the client talks to storage directly using a short-lived permission you hand out.

The problem

Three things go wrong when blobs flow through the application:

• Cost and capacity. Every byte is paid for twice (client to server, server to storage), and a few big uploads can saturate the servers.
• Slow and fragile. A long transfer over a mobile network drops halfway, and the user starts over.
• Wrong place for the data. Databases are poor at holding large binary values. They bloat, slow backups and waste memory.

Ask first: how big are the files, who may read them (public or private), do they need processing after upload (resize, scan, transcode), and how long are they kept?

The shape

Uploads: presigned URLs

• 1. The client asks the API: "I want to upload cat.jpg, 3 MB, image/jpeg."
• 2. The API checks who is asking and what is allowed, creates a metadata record with status pending, and asks blob storage for a presigned URL: a link that grants one specific action (upload this object key) for a short time.
• 3. The API returns the URL. The client uploads the bytes straight to storage.
• 4. Storage confirms. The record moves to ready, either because the client tells the API, or because storage emits an event that your service listens to. The event is more reliable, because a client can crash before telling you.

Presigned URLs are signed with a secret held by your service, so storage can check them without calling you. Scope them tightly: one object key, one operation, a short expiry, and where the provider supports it, limits on size and content type.

Big files: multipart and resumable uploads

For large files, split into parts and upload them separately, in parallel, with the final step assembling them. If one part fails, retry only that part, and if the connection drops, resume from the last good part.

• Object stores offer multipart upload. Specific limits (minimum part size, maximum number of parts) differ by provider, so check the documentation rather than quoting numbers.
• Some protocols offer resumable uploads over plain HTTP. The idea is the same: the client asks "how much have you got?" and continues from there.
• Parts can carry a checksum so corruption is caught.

Downloads

• Public content: serve from storage through a CDN, so popular files are served from near the user and your origin is rarely hit.
• Private content: give out a signed URL for download with an expiry, or a signed CDN URL or cookie. The authorisation check happens in your API, and the transfer happens elsewhere.
• Large files: support range requests so video can seek and downloads can resume.

After the upload: processing

Work on the file (thumbnails, virus scan, video transcode) belongs to a long-running task. Storage emits an "object created" event, a queue picks it up, and workers write the results back as new objects. The metadata record shows processing until they finish.

A worked example: profile photo upload

Assumption: photos up to 10 MB, private until processed, shown at three sizes.

• 1. POST /v1/photos with { "content_type": "image/jpeg", "size": 3000000 }. The user ID comes from the token. The API creates photo_id, status pending, with a storage key such as raw/<photo_id>. It returns a presigned upload URL that expires in a few minutes.
• 2. The client PUTs the bytes to storage.
• 3. Storage emits "object created". A queue message starts a worker, which checks the file really is an image (do not trust the declared type), makes three sizes, writes them under thumb/<photo_id>/..., and sets the record ready.
• 4. The app shows images through the CDN. The page uses URLs built from the photo ID.
• 5. Cleanup: records stuck in pending for a day mean the upload never happened. A sweeper deletes them. Also set a storage lifecycle rule to remove stray objects under the raw prefix.

Choosing

• Small files (a few hundred KB, rare uploads): going through the API is fine and simpler. Do not build presigned flows for avatars on a hobby site.
• Files of many MB or more, or many uploads: presigned URLs.
• Very large or unreliable networks: multipart or resumable.
• Same file read by many: CDN.
• Private files: signed URLs, short expiry, authorisation in your API.

When to bring this up

• Signals in the prompt: photos, video, documents, backups, attachments, "upload", "share a file".
• Mention it briefly, draw the three-party picture (client, API, storage), and move on, unless the file path is the hard part of the problem (a video platform).

Deep dives interviewers use

The upload fails at 99 percent

With a single request, the whole upload is lost. With multipart or resumable uploads the client keeps a list of finished parts, asks storage which parts it has on reconnect, and sends only the rest. The final "complete" call assembles them. Also set a lifecycle rule that discards unfinished multipart uploads after a few days, since storage may bill for the parts already sent.

Preventing abuse

• Size and type limits in the signed request, and a re-check after upload by reading the real bytes.
• Quotas and rate limits per user on how many upload URLs you hand out. Each URL is a small cheque; limit how many you write.
• Scan for malware or banned content before a file becomes visible: keep it in a pending state, and publish only after the scan.
• Short expiry and one-object scope on every URL, so a leaked link is nearly useless.

Keeping metadata right

The record (owner, name, size, content type, state, storage key) lives in the database, and the bytes in storage. Create the record first as pending, so every object has an owner from its first second. Move to ready on a storage event, and run a periodic reconciliation that lists storage against the table to catch objects with no record and records with no object. Never let the object key reveal secrets, and never derive permission from the key.

Fast downloads

• Put a CDN in front, with long cache lifetimes and versioned keys, so changes are new URLs.
• Support range requests so players start before the file is complete.
• For video, store several encodings and use adaptive streaming: the file is cut into short segments in a few quality levels, and the player switches as bandwidth changes.
• Place storage near most users, or replicate to a second region if users are far apart.

Names differ, the idea does not

Every major cloud has the same pieces under different names: an object store, a way to sign a time-limited link for one action, a multipart mechanism, and an event on object creation. Say the concept first and the product name second, and check the provider's docs for limits.

Common mistakes

• Streaming big files through the app server.
• Storing blobs in the main database.
• Record and object out of step. The record says ready, but the object is not there, or the object exists with no record. Use a pending state, confirm from a storage event, and sweep for orphans.
• Trusting the client's declared file type or size. Validate after upload, and limit size in the signed request where possible.
• Presigned URLs with long lives or broad scope, or logged and leaked.
• No cleanup for abandoned uploads, so storage cost grows quietly.
• Serving user-uploaded files from your own domain without care. A malicious HTML or script file can then attack your users. Serve them from a separate domain or with safe headers.
• Forgetting to check the user may download a file because the URL looks unguessable. Unguessable is not authorisation.

Check yourself

• Why not upload through the app server?
• What does a presigned URL grant, and why is it short-lived?
• How do you resume an upload that failed at 80 percent?
• How do the metadata record and the stored object stay in step?
• How do you serve a private video to one user only?
• What happens to an upload that was started and never finished?