A distributed payment processing architecture focused on high performance, strict idempotency, and system resilience.
Hello! I'm Ezequiel Ranieri. I am a self-taught developer who discovered the world of programming through curiosity and a passion for building things. Everything I know—from architecture patterns to distributed systems—I've learned on my own through books, technical documentation, videos, and endless hours of practice.
I created this project to consolidate and demonstrate my understanding of high-performance backend systems. I don't claim to be a senior architect; I am a dedicated learner who enjoys solving complex technical challenges and building robust software that works under pressure.
Contact:
- Email: ez.ranieri@gmail.com
- GitHub: https://github.com/ezequielranieri
This service is designed to handle critical tasks (like payment processing) where data integrity and reliability are non-negotiable.
- Ingestion (FastAPI): A high-performance gateway receives requests and provides a "Cache-aside" status query endpoint.
- Queuing (RabbitMQ): Messages are routed through priority exchanges (
high_priority,default,low_priority). - Idempotency Filter (Redis): Before processing, workers check atomic locks in Redis to prevent duplicate executions (Strict Idempotency).
- Processing (Celery Workers): Workers consume tasks using a
prefetch-multiplier=1strategy for fair distribution. - Resilience: Tasks implement Exponential Backoff with Jitter to handle transient 5xx errors without overloading third-party APIs.
- Persistence (PostgreSQL): Every successful or terminal state is persisted for audit and long-term storage.
- Dead Letter Queue (DLQ): Tasks that exhaust all 5 retries are diverted to a dedicated queue for manual auditing.
- FastAPI: Asynchronous API for querying task status.
- Celery: Distributed task queue framework.
- RabbitMQ: Robust message broker for inter-service communication.
- Redis: High-speed idempotency lock and result backend.
- PostgreSQL: Relational database for persistent records.
- Docker & Compose: Containerization for consistent environments.
- Locust: Stress testing to validate the system under load.
- Docker and Docker Compose installed on your machine.
Clone the repository and set up your environment variables:
git clone https://github.com/ezequielranieri/high-performance-task-queue
cd high-performance-task-queue
cp .env.example .env(Note: You can use the default values in .env for local testing).
Spin up the entire infrastructure with a single command:
docker compose up -dThe system will automatically initialize the databases and start the workers.
- Query API: http://localhost:8000/payment/{payment_id}
- Monitoring (Flower): http://localhost:5555
- Management (RabbitMQ): http://localhost:15672
To validate the system's performance and reliability, I performed a stress test simulating a high-traffic scenario. The results demonstrate the architecture's ability to maintain high throughput while ensuring data integrity.
- Throughput: Sustained ~160 Requests Per Second (RPS) with 50 concurrent users.
- Volume: Successfully processed over 20,000 tasks in a single burst.
- Latency: Sub-millisecond response times from the API gateway (FastAPI), thanks to its asynchronous nature.
The following dashboards confirm the system's stability under pressure:
Visualizes the linear scaling of RPS and stable response times during the stress test.
Shows the three priority queues (high, default, low) handling the burst efficiently and the DLQ (Dead Letter Queue) ready to capture exhausted retries.
Critical validation point: The task list shows active retries with Exponential Backoff. This confirms that the worker identifies transient failures and waits incrementally longer between attempts, preventing cascading failures.
During the test, I injected duplicate payment_id values. The system correctly identified these using Redis SETNX, blocking the redundant task execution and returning the cached status, ensuring that no payment was processed twice.
Developing this project was a significant milestone in my self-taught journey. Some key takeaways include:
- Managing Distributed State: Using Redis for atomic operations and idempotency.
- Handling Failure Gracefully: Understanding the difference between retryable (5xx) and terminal (4xx) errors.
- Orchestration: Managing multiple services with Docker Compose and ensuring proper startup sequences with health checks.
- Performance Testing: Using Locust to find bottlenecks and observing how
prefetchsettings impact worker efficiency.
Thank you for checking out my work! I'm always open to feedback and looking for new opportunities to learn and grow.