CloudDrop
A locally implemented file-processing pipeline built around tracked, retried, idempotent jobs.
← Back to WorkProblem
File processing pipelines are easy to get right on the happy path and quietly wrong everywhere else. Duplicate events, partial failures, job state nobody can see.
What I built
A locally implemented event-driven upload pipeline models a pre-signed upload handoff, durable queue, asynchronous processor, persisted job state, bounded retries, and a dead-letter path. Terraform defines the intended AWS infrastructure and least-privilege IAM, but the project has not been deployed or host-accepted on AWS.
Why it matters
It demonstrates real distributed-systems discipline: idempotency, retries, and observability on a path most projects leave to chance.
Result
Local validation covers duplicate delivery, idempotent processing, retry and dead-letter behavior, and deterministic job status transitions. No hosted AWS result is claimed.
Architecture
- Pre-signed upload API issues scoped, time-limited S3 write access
- S3 event notifications enqueue onto SQS
- Lambda consumers process asynchronously and write job state to DynamoDB
- Idempotency keys deduplicate re-delivered events
- Retry policy with backoff; dead-letter queue for exhausted retries
- Status API for polling job lifecycle (queued → processing → complete/failed)
- Terraform-managed infrastructure, least-privilege IAM per Lambda
Testing & reliability
Static Terraform validation and local duplicate-delivery and induced-failure scenarios check idempotent handling and correct dead-letter routing. Hosted AWS acceptance remains outstanding.
Stack
- Python
- AWS Lambda
- S3
- SQS
- DynamoDB
- Terraform