File Storage and Content Delivery

# CHAPTER 12

File Storage and Content Delivery #

1. Introduction #

2. Learning Objectives #

• Explain why storing large media files inside an SQL database is an architectural anti-pattern.

• Compare traditional Block Storage against modern Object Storage (AWS S3).

• Architect a highly scalable, secure File Upload pipeline using Pre-Signed URLs.

• Understand how Content Delivery Networks (CDNs) cache assets globally at the Edge.

• Design an asynchronous worker pipeline for heavy image/video transcoding.

3. Block Storage vs. Object Storage #

• Block Storage (The Hard Drive): This is the traditional hard drive attached directly to your server (e.g., AWS EBS). It is incredibly fast, but it is expensive and physically limited. You cannot easily share a block drive across 1,000 servers.

• Object Storage (The Data Lake): This is the modern cloud standard (e.g., AWS S3, Google Cloud Storage). Files are stored as "Objects" in a flat, infinitely scalable bucket. Each object has a unique URL.

• *The Magic:* Object storage scales infinitely. You never have to worry about "running out of disk space." It is also extremely cheap compared to Block Storage.

4. The Database Pointer Architecture #

• The Anti-Pattern: Storing the physical .jpg binary code inside the SQL database.

• The Standard Architecture: The physical profile.jpg file is uploaded directly to an Object Storage bucket (AWS S3). The bucket generates a secure URL: https://s3.aws.com/mybucket/profile.jpg. You then save *only that string URL* into your PostgreSQL database.

• *Result:* Your database remains incredibly small, fast, and lightweight.

5. Content Delivery Networks (CDNs) #

• The CDN (Edge Locations): A CDN (Cloudflare, AWS CloudFront) is a massive network of thousands of tiny proxy servers located in major cities worldwide (Tokyo, London, Sydney).

• The Workflow: When the Australian user requests the video, the request hits the Sydney CDN server first. If the CDN doesn't have it (Cache Miss), it fetches it once from Virginia, stores a copy in Sydney, and gives it to the user. The next 1 million Australian users will download it directly from Sydney, experiencing zero latency and completely protecting your Virginia bucket from the massive traffic load.

6. File Upload Pipelines (Pre-Signed URLs) #

• The Danger: If a user uploads a 2GB video through your Node.js API server, that connection must stay open for 10 minutes. If 1,000 users do this, your API servers will hit their maximum connection limits and crash, locking out all other traffic.

• The Pre-Signed URL Solution:

1. 1. Client asks API server: "I want to upload a video."

1. 2. API server secretly talks to AWS S3 and generates a temporary, secure "Pre-Signed URL" valid for exactly 15 minutes.

1. 3. API server sends URL to Client.

1. 4. Client uploads the 2GB video *directly* to AWS S3 using the URL, completely bypassing your API servers.

7. Diagrams/Visual Suggestions #

[ Mobile App ] -- (1. Request Upload Token) --> [ API Web Server ]
                                                  |
[ Mobile App ] <--------- (2. Return S3 Token) ---|
      |
      |-- (3. Upload 2GB Video DIRECTLY) -------> [ AWS S3 Object Storage ]

8. Best Practices #

• Media Optimization Workers: Never serve an 8MB 4K image directly to a mobile user on a 3G network. Architect an asynchronous event queue (as learned in Chapter 10). When an image lands in S3, S3 fires an event to a worker server. The worker compresses the image, creates 3 sizes (Thumbnail, Mobile, Desktop), saves them back to S3, and updates the database.

9. Common Mistakes #

• Public Buckets (The Data Leak): The most common security failure in the cloud is an engineer accidentally setting an AWS S3 bucket permission to "Public Read/Write." *The Failure:* Anyone on the internet can download your entire company's database backups, or worse, upload malicious malware into your application. *The Fix:* Buckets MUST be private by default, accessed only via strict IAM policies or CDN distributions.

10. Mini Project: Architect a Netflix-Clone Storage System #

1. 1. The Ingestion: Studios upload raw 50GB video masters directly into AWS S3 (US-East) via secure Pre-Signed URLs.

1. 2. The Transcoding: S3 triggers an event to a Kafka queue. Hundreds of Worker servers pull the video, slice it into 10-second chunks, compress it into multiple resolutions (480p, 1080p, 4K), and save the thousands of new files back to S3.

1. 3. The Global Distribution: We connect AWS CloudFront (CDN) to the S3 bucket.

1. 4. The User Experience: A user in Berlin clicks "Play." Their TV connects to the Berlin CDN edge server, instantly downloading the German-cached, 1080p compressed video chunks with 10ms latency.

11. Practice Exercises #

1. 1. Compare "Block Storage" with "Object Storage." Why do massive modern applications like Instagram rely exclusively on Object Storage for saving user media?

1. 2. Define the architectural danger of routing massive file uploads (e.g., 5GB videos) through your primary Node.js or Python API servers. How do Pre-Signed URLs solve this bottleneck?

12. MCQs with Answers #

13. Interview Questions #

• Q: Explain the mechanical flow of a "Pre-Signed URL" architecture for file uploads. Why is it structurally superior to having the client upload the file directly to your web server?

• Q: A client in London complains that your website's heavy header images take 10 seconds to load, while users in New York report the site is instantaneous. Walk me through exactly how you would configure a CDN to solve this geographic latency.

• Q: Walk me through the asynchronous architecture required to handle user video uploads. Once the raw video hits your S3 bucket, how do you queue, transcode, and compress the video without blocking the user's interface?

14. FAQs #

15. Summary #

16. Next Chapter Recommendation #