torque

A distributed job queue for Node & TypeScript — priorities, delays, retries with exponential backoff, a dead-letter queue, visibility-timeout leases with crash recovery, idempotency keys, and per-worker rate limiting. Runs in-memory out of the box; scales out on Redis.

npm install torque-queue

import { Queue, Worker, InMemoryBackend } from "torque-queue";

const backend = new InMemoryBackend();          // or: new RedisBackend(new Redis(url))
const queue = new Queue<{ email: string }>("welcome", backend);

await queue.add({ email: "a@x.com" });
await queue.add({ email: "vip@x.com" }, { priority: 10 });     // jumps the line
await queue.add({ email: "later@x.com" }, { delayMs: 60_000 }); // scheduled
await queue.add({ email: "a@x.com" }, { idempotencyKey: "a@x.com" }); // deduped

const worker = new Worker<{ email: string }>("welcome", backend, async (job) => {
  await sendEmail(job.data.email);              // throw to trigger retry/DLQ
}, { concurrency: 8, rateLimit: { limit: 100, intervalMs: 1000 } });

worker.on("failed", (job) => console.warn("dead-lettered", job.id));
worker.start();

Why

Most Node queue tutorials stop at "push a job, pop a job." The parts that make a queue safe to run in production are the semantics around failure: what happens when a worker crashes mid-job, when a handler throws, when the same event is delivered twice, when downstream is rate-limited. torque implements those explicitly, keeps the engine backend-agnostic, and proves the behavior with a test suite and a benchmark rather than asserting it.

Features

• Priorities & delays — higher priority runs first; delayMs schedules for later.
• Retries with exponential backoff + full jitter — no thundering herd.
• Dead-letter queue — jobs that exhaust maxAttempts are set aside for inspection/replay.
• Visibility timeouts + heartbeats — a leased job whose worker dies is automatically reclaimed (at-least-once delivery).
• Idempotency keys — a repeated add with the same key is a no-op.
• Per-worker rate limiting — token bucket, e.g. "100 jobs/sec".
• Concurrency — each worker runs N jobs in parallel.
• Pluggable backends — InMemoryBackend (zero infra) and RedisBackend (durable, multi-machine) behind one interface.
• Typed & tested — full TypeScript types, 18-test suite over both backends, 90%+ coverage.

Benchmark

Single-process, in-memory, 100k no-op jobs (Node 20, one dev machine). These measure torque's own overhead — not a cross-library comparison.

Path	Throughput
enqueue	~1.07M jobs/sec
drain — raw claim→complete loop (backend ceiling)	~1.2M jobs/sec
drain — through Worker (concurrency 50)	~930K jobs/sec

Reproduce: npm run bench (or npm run bench -- 200000 64). Claims are O(log n) via a priority heap. A Redis/BullMQ comparison needs a live Redis and is intentionally left to the reader — the harness supports it.

Reliability semantics

torque is at-least-once: a job runs one or more times, never zero. Design idempotent handlers (or use idempotencyKey). Full discussion — visibility timeouts, why exactly-once is a lie, the reclaim path — in docs/SEMANTICS.md.

Redis backend

import Redis from "ioredis";
import { Queue, Worker, RedisBackend } from "torque-queue";

const backend = new RedisBackend(new Redis(process.env.REDIS_URL!));
// ...same Queue / Worker API

ioredis is an optional peer dependency — only needed if you use RedisBackend. All queue state is hash-tagged so it lives in one Redis Cluster slot; the atomic claim and lease-reclaim run as Lua scripts. The Redis backend is exercised in CI against a real Redis service.

Try it

npm run demo        # producer + worker on the in-memory backend
npm run dashboard   # live stats at http://localhost:8080
npm run bench       # throughput benchmark
npm test            # the suite (add REDIS_URL to also test the Redis backend)

API

• new Queue<T>(name, backend, { defaultMaxAttempts? }) — add(data, opts), addBulk(items), stats(), getDead(), reclaim(), clear().
• new Worker<T>(name, backend, handler, opts) — start(), stop(); events completed / retry / failed / error. Options: concurrency, visibilityMs, heartbeatMs, pollMs, reclaimMs, rateLimit, backoff, timeoutMs.
• Backends: InMemoryBackend, RedisBackend(redis).

Architecture & design trade-offs: docs/ARCHITECTURE.md.

MIT © Lavya Tanotra