# How We Solved the On-Time Notification Delivery Problem at Scale

Delivering notifications *exactly on time* sounds easy — until you have to do it for **thousands of users at the same second**.

Our medicine reminder app depends heavily on **precise dose reminders**. Even a 1–2 minute delay can cause users to miss their doses, so reliability was critical.

Initially, we used **BullMQ + Node.js workers** for scheduling and sending notifications. It worked fine for a small number of users, but at scale, the system started to break.

---

## **The Problem**

### **1\. Too Many Notifications at the Same Time**

* Thousands of notifications were scheduled for the same second.
    
* Workers pulled huge batches from Redis, causing **CPU & memory spikes**.
    
* Redis queues became congested and some jobs got delayed.
    

### **2\. Worker Overload**

* Even with multiple worker instances, the Node.js event loop struggled during peaks.
    
* Delays became more frequent as user count increased.
    

---

## **Step 1 — Moving to AWS Step Functions + Lambda**

We redesigned the scheduling process to **spread the load** more efficiently.

**New Flow:**

1. **Step Function** schedules notification batches at exact times.
    
2. Each execution **triggers a Lambda** dedicated to a subset of notifications.
    
3. We keep **2–3 hot Lambdas** ready during peak times to avoid cold starts.
    

**Why Step Functions?**

* Native CRON/Rate-based scheduling.
    
* Orchestration of multiple parallel Lambda executions.
    
* Fully managed — no manual queue management.
    

---

## **Step 2 — Horizontal Scaling in Lambda**

AWS Lambda scales automatically, so during high load, we got **dozens of Lambdas in parallel**.

This solved the compute bottleneck — but introduced a **new problem**…

---

## **Step 3 — The Database Connection Storm**

Each Lambda invocation created a new PostgreSQL connection.

At scale:

* RDS hit the **max\_connections** limit.
    
* Some Lambdas failed instantly due to connection errors.
    
* This caused missed or late notifications.
    

---

## **Step 4 — Introducing Amazon RDS Proxy**

**RDS Proxy** pools and shares DB connections across Lambdas.

**Benefits:**

* Reuses existing DB connections.
    
* Reduces connection churn and overhead.
    
* Eliminates `too many connections` errors.
    
* Lowers latency because connections are pre-warmed.
    

---

## **Step 5 — Putting Lambdas in a VPC**

Since **RDS Proxy** lives inside a VPC:

* All Lambdas were moved into **private subnets** within the same VPC.
    
* This allowed private, low-latency connections to RDS Proxy.
    

---

## **Step 6 — Adding Internet Access via NAT Gateway**

Once Lambdas were in the VPC, they **lost internet access** — which broke calls to FCM.

**Fix:**

* Created a **NAT Gateway** in the VPC.
    
* Updated route tables so Lambdas could:
    
    * Connect to RDS Proxy privately.
        
    * Still reach the internet for external APIs.
        

---

## **Final Architecture Diagram**

```plaintext
          ┌───────────────────┐
          │  Step Function    │
          │  (Scheduled CRON) │
          └─────────┬─────────┘
                    │
                    ▼
           ┌────────────────────────────────────┐
           │            AWS Lambda(s)           │
           │ (Horizontal Scaling, VPC-Enabled)  │
           │  1️⃣ Query DB via RDS Proxy         │
           │  2️⃣ Send Notifications via APIs    │
           └─────────┬────────────────┬─────────┘
                     │                │
        ┌────────────▼─────────┐   ┌──▼───────────────┐
        │   Amazon RDS Proxy   │   │ NAT Gateway      │
        │ (Connection Pooling) │   │ (Internet Access)│
        └───────────┬──────────┘   └──────────┬──────┘
                    │                        │
          ┌─────────▼──────────┐     ┌───────▼─────────────────┐
          │ PostgreSQL (RDS)   │     │ External APIs (FCM, SNS,│
          └────────────────────┘     │ Push Notification, etc.)│
                                     └─────────────────────────┘
```

---

## **Conclusion**

Building an on-time notification system at scale required more than just adding more workers — it demanded a complete architectural rethink. By moving from BullMQ workers to **AWS Step Functions** for scheduling, **Lambda** for scalable compute, and **RDS Proxy** for efficient database connectivity, we achieved a fully serverless, reliable, and low-maintenance solution.

Integrating **VPC networking** with a **NAT Gateway** ensured secure database access while still allowing internet connectivity for push and Firebase APIs. Today, our system delivers notifications **precisely on time**, even during heavy load, while remaining cost-efficient and easy to operate.