Asynchronous Batch Ingestion Engine

This is a simple batch ingestion system for large JSON payloads. It keeps the API fast by sending work to Redis/BullMQ and letting a background worker process the records while PostgreSQL tracks the job state.

How it works

flowchart LR
  A[Client] --> B[API]
  B --> C[(PostgreSQL)]
  B --> D[Redis Queue]
  D --> E[Worker]
  E --> C

Loading

1. Client sends raw JSON array text.
2. API creates a BatchJob.
3. API adds the job to BullMQ.
4. Worker reads the job and processes records in chunks.
5. Valid rows go into IngestedRecord.
6. Bad rows go into JobError.

Stack

• Node.js
• TypeScript
• Express
• PostgreSQL
• Prisma
• Redis
• BullMQ
• Docker

What is implemented

• API accepts raw JSON array text and returns quickly with a queued job id.
• Worker processes records in small chunks of 100 so memory use stays lower.
• PostgreSQL stores job state, valid rows, and row-level errors.
• Redis + BullMQ handles background processing.
• Docker runs API, worker, Postgres, and Redis together.
• npm run benchmark compares one-shot processing with chunked async processing.

Run locally

npm install
npx prisma generate
npm run prisma:deploy
npm run dev

Docker

docker compose up --build

Environment

PORT=3001
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/distributed_loader
REDIS_URL=redis://127.0.0.1:6379

Request

Send a raw JSON array:

[{"name":"A"},{"name":"B"}]

Response

{
  "success": true,
  "message": "Batch accepted and queued",
  "jobId": "uuid"
}

Check Job Status

Get one batch job by id:

GET /api/batch/:jobId

Response includes the job status, row counts, and record/error totals.