StorageX: Distributed Video Transcoding System
Scalable, high-performance video processing with S3-compatible storage.
- FastAPI
- Next.js
- Docker
- RabbitMQ
- FFmpeg
- PostgreSQL
- MinIO
Core Features
- Distributed video transcoding using FFmpeg across multiple background workers.
- S3-compatible object storage integration via MinIO for high-performance file handling.
- Real-time job tracking and status updates using RabbitMQ and FastAPI.
- Neo-brutalist UI/UX built with Next.js 16 and Tailwind CSS for a high-contrast, modern feel.
- Automated multi-container orchestration with a 7-service Docker Compose stack.
Architecture
- Built with a decoupled Frontend (Next.js) and Backend (FastAPI) for independent scaling.
- Message-driven architecture using RabbitMQ to handle asynchronous transcoding tasks.
- Relational data management with PostgreSQL for user metadata and video processing logs.
- Containerized deployment strategy ensuring environment parity from dev to production.
- Integrated FFmpeg processing pipelines optimized for Python-based worker nodes.
Technologies
- FastAPI serves as the high-performance asynchronous API gateway for all client requests.
- Next.js 16 provides a type-safe, server-side rendered frontend for the management dashboard.
- MinIO acts as the private cloud storage layer, providing S3-compatible APIs for media assets.
- RabbitMQ ensures reliable message delivery between the API and the transcoding workers.
- SQLAlchemy & Pydantic manage data integrity and seamless ORM mapping to PostgreSQL.
Key Highlights
- Scalable Worker Nodes — Easily add more transcoding power by scaling the worker container.
- Full-Stack Type Safety — End-to-end TypeScript and Pydantic integration.
- S3-Native Workflow — Ready for deployment on AWS, GCP, or self-hosted hardware.
- Complex Orchestration — Manages 7 distinct services (API, Workers, DB, Cache, Storage, UI).
- Developer-Centric — Designed with clean architecture for easy extension and auditing.