Scalable Email Notification Service with Queue, Worker, and Real-Time Updates

Overview

This project demonstrates how to build a scalable email notification system that avoids blocking API requests by offloading email delivery to background workers. It uses Redis-based queues and real-time event streaming to provide visibility into job status.

Architecture

Client → API (Express) → Queue (Redis/BullMQ) → Worker → Email (SMTP)
                         ↘ Real-time events (Socket.IO) ↗

1. API Layer (Express)

• Receives email requests
• Validates payload
• Pushes job to queue
• Returns immediately (non-blocking)

2. Queue (Redis + BullMQ)

• Acts as a buffer between API and worker
• Ensures reliability and retry support
• Handles job states (waiting, active, completed, failed)

3. Worker

• Consumes jobs from queue
• Sends emails using Nodemailer
• Runs independently from API

4. Email Service

• Uses SMTP via Nodemailer
• Ethereal used for safe testing

5. Real-Time Updates (Socket.IO)

• Emits job lifecycle events
• Events: queued, processing, sent, failed
• Enables live monitoring

Key Features

• Asynchronous job processing
• Retry and failure handling
• Real-time status tracking
• Clean separation of API and worker
• Input validation and rate limiting
• Test coverage included

Why This Matters

Sending emails synchronously leads to slow APIs and poor user experience. This system introduces a queue-based architecture that:

• Improves performance
• Enables horizontal scaling
• Provides fault tolerance
• Decouples responsibilities

Trade-offs

• Increased system complexity
• Requires Redis and worker processes
• Eventual consistency (not instant delivery)

Improvements (Next Steps)

• Add database for job history (PostgreSQL)
• Implement dead-letter queue (DLQ)
• Add monitoring (Prometheus/Grafana)
• Secure API with authentication
• Deploy with container orchestration (Docker/Kubernetes)

Testing Strategy

• Functional: Verify job lifecycle events
• Failure: Simulate worker crashes
• Load: Send high-volume requests

Mental Model

API = Receptionist
Queue = Waiting room
Worker = Employee
Email = Task execution
Socket = Status display