Background jobs and queues: retries, backoff and dead-letter without duplicates

A user clicks “sign up” and the page spins for two seconds while the request sends an email and resizes an avatar. Worse, SMTP goes down for five minutes and nobody can sign up at all. The fix is a background job: the request records the work and returns, and another process does it later. This article covers what makes a queue trustworthy and builds a working jobs table on Postgres.

Every SQL block was tested on PostgreSQL 18.6 (run through PGlite 0.5.11). SQS, BullMQ and Sidekiq defaults were checked against their official docs as of October 2026.

Why slow work leaves the request

Say a sign-up handler spends its time like this:

Step	Time
Save the user	40 ms
Send a confirmation email over SMTP	1,200 ms
Resize the avatar	800 ms
Total	2,040 ms

Turn the last two into jobs and the request is the user insert plus two job rows in the same transaction, about 40 ms and change: roughly 50 times faster (2,040 ÷ 40 ≈ 51). More important than speed, an SMTP outage no longer blocks sign-ups. The job waits and retries on its own.

Good candidates: notifications, image and video processing, slow AI API calls, PDFs, outgoing webhooks. Anything the user must see right now (a password check, reserving stock) stays in the request.

The parts of a trustworthy queue

Term	Meaning
producer	code that enqueues a job (usually the request handler)
worker	a process that takes jobs and runs them; run as many as you like
at-least-once	the queue promises a job runs at least once, possibly more
visibility timeout / lease	how long a job stays reserved by one worker; if it is not marked done in time, it reappears for another worker
retry + backoff	on failure, try again, waiting longer each time
dead-letter	where jobs go after their last failed attempt, waiting for a human

At-least-once means a job must survive running twice

A worker sends the email, then dies before recording success. The queue only sees an expired lease, so another worker runs the job again: two emails. No queue can fix this on its side; the SQS docs say even the visibility timeout does not guarantee a message is delivered only once.

The fix lives in the job: make it idempotent, so running it many times has the same effect as running it once.

• Image resizing: write to a path derived from the input, e.g. avatars/{user_id}/256.webp. A rerun overwrites the same file
• Charging money at a provider: use its idempotency key. Stripe’s Idempotency-Key header returns the saved result on a repeat instead of creating a second object (keys may be pruned after 24 hours). Use the job or order id
• Email has no such key, so pick your failure: mark “sent” first and you may never send; mark it after and you may send twice. For receipts, a duplicate beats a missing one

The queue promises “at least once”. “The effect happens once” is the job’s responsibility.

A jobs table on Postgres with SKIP LOCKED

If you already run Postgres, there is nothing to install. FOR UPDATE SKIP LOCKED makes each worker skip rows another worker has locked instead of waiting. The Postgres docs describe it as suited to queue-like tables with multiple consumers, and it is the mechanism behind pg-boss (Node) and Solid Queue (the default in new Rails 8 apps).

CREATE TABLE jobs (
  id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  kind         text        NOT NULL,
  payload      jsonb       NOT NULL,
  status       text        NOT NULL DEFAULT 'ready',  -- ready | running | done | dead
  attempts     int         NOT NULL DEFAULT 0,
  max_attempts int         NOT NULL DEFAULT 8,
  run_at       timestamptz NOT NULL DEFAULT now(),
  locked_until timestamptz,
  last_error   text,
  dedupe_key   text UNIQUE
);
CREATE INDEX jobs_due ON jobs (run_at) WHERE status IN ('ready', 'running');

Enqueue in the same transaction as the business data. This is the big advantage of a queue in the same database: the order commits, the job exists; the order rolls back, the job is gone too. There is no “order saved, but the job never reached Redis” case.

BEGIN;
INSERT INTO orders (id, total) VALUES (1042, 59000);
INSERT INTO jobs (kind, payload, dedupe_key)
VALUES ('send_receipt', '{"order_id": 1042}', 'receipt:1042')
ON CONFLICT (dedupe_key) DO NOTHING;
COMMIT;

dedupe_key prevents enqueuing a duplicate (a double click, say). It does not prevent running twice; that still needs idempotency.

A worker claims a job, reserving it with locked_until, which plays the role of the visibility timeout, and commits immediately, so no transaction stays open while the work runs. A job whose lease expired (its worker died) gets claimed again.

UPDATE jobs
SET status = 'running',
    attempts = attempts + 1,
    locked_until = now() + interval '5 minutes'
WHERE id = (
  SELECT id FROM jobs
  WHERE attempts < max_attempts
    AND ((status = 'ready'   AND run_at <= now())
      OR (status = 'running' AND locked_until < now()))
  ORDER BY run_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id, kind, payload, attempts;

Success and failure (backoff of 30 seconds doubling each time, capped at 1 hour, multiplied by a 0.5-1.0 jitter; after max_attempts the job is dead):

-- success
UPDATE jobs SET status = 'done', locked_until = NULL WHERE id = $1;

-- failure: $1 = error text, $2 = id
UPDATE jobs
SET last_error   = $1,
    locked_until = NULL,
    status = CASE WHEN attempts >= max_attempts THEN 'dead' ELSE 'ready' END,
    run_at = now() + least(interval '1 hour',
                           interval '30 seconds' * power(2, attempts - 1))
                     * (0.5 + random() / 2)
WHERE id = $2;

The worker in Python with psycopg 3 (swap $1, $2 for %s, in order) is this short:

import time, psycopg

conn = psycopg.connect("dbname=app", autocommit=True)  # each statement commits on its own

while True:
    job = conn.execute(CLAIM_SQL).fetchone()
    if job is None:
        time.sleep(1)
        continue
    job_id, kind, payload, attempts = job
    try:
        HANDLERS[kind](payload)          # one function per job kind; must be idempotent
        conn.execute(DONE_SQL, (job_id,))
    except Exception as e:
        conn.execute(FAIL_SQL, (repr(e)[:500], job_id))

A job that kills the worker every time (a corrupt image that crashes the library, say) never reaches FAIL. Sweep for those from cron every minute:

UPDATE jobs SET status = 'dead', last_error = coalesce(last_error, 'lease expired')
WHERE status = 'running' AND locked_until < now() AND attempts >= max_attempts;

Worked example: how many retries are enough?

Say your email provider’s SMTP is down for 20 minutes. With the backoff above (before jitter), the wait after failure n is 30 × 2^(n-1) seconds.

Failure #	Wait	Cumulative
1	30 s	30 s
2	60 s	1.5 min
3	120 s	3.5 min
4	240 s	7.5 min
5	480 s	15.5 min
6	960 s	31.5 min
7	1,920 s	63.5 min

• max_attempts = 5: attempt 5 runs at minute 7.5, the outage is still on, and every job goes dead. Someone has to retry them by hand
• max_attempts = 8: attempt 8 runs at 3,810 seconds ≈ 63.5 minutes. Even if jitter rolls the minimum (×0.5) every time, that is ≈ 31.75 minutes, safely past a 20-minute outage

Jitter matters: 500 emails that fail together would, without it, all retry at the same instant every round. The AWS Builders’ Library explains that randomising the delay is what stops retries becoming a fresh wave of overload.

How many workers: an AI summary job takes 8 seconds and 600 arrive an hour. That is 600 × 8 = 4,800 seconds of work per hour ÷ 3,600 = 1.33 workers, so at least 2. If 300 jobs arrive at once and 4 workers run, the queue clears in 300 × 8 ÷ 4 = 600 seconds = 10 minutes. Check that the API’s rate limit allows 4 concurrent calls.

The visibility timeout must outlast the work

Set the lease from the slowest acceptable run, not the average. A 6-minute job with a 5-minute lease gets picked up by a second worker while the first is still busy. For long, unpredictable jobs, extend the lease periodically (a heartbeat), as SQS recommends with ChangeMessageVisibility. Too long hurts too: when a worker dies, its job sits idle for the whole lease.

For reference: SQS defaults to 30 seconds, with a maximum of 12 hours from first receipt.

Dead-letter: the place a human must look

A job that fails every attempt is usually a bug or bad data, and more retries will not fix it. So it is parked (status = 'dead' here, the Dead set in Sidekiq, a DLQ in SQS once maxReceiveCount is exceeded). It must come with an alert: a dead-letter nobody watches is silent data loss. Feed SELECT count(*) FROM jobs WHERE status = 'dead' into a Telegram server alert, and always keep last_error.

Postgres or Redis?

	Postgres + SKIP LOCKED	Redis-based (BullMQ, Sidekiq)
Extra infrastructure	none, your existing database	a Redis instance
Enqueue with business data	one transaction	two systems; beware “committed, but the job never got sent”
Built-in retries	write them (above) or use pg-boss / Solid Queue	BullMQ: attempts + backoff 2^(attempts-1) × delay; Sidekiq: 25 retries over about 20 days
Dead-letter	status = 'dead'	Sidekiq Dead set (keeps 10,000 jobs or 6 months)
Must configure	index, purge old done jobs	AOF on, maxmemory-policy noeviction

Already on Postgres? Start with a jobs table. Already running Redis for a hot-path cache and writing Node? BullMQ is fine, but configure Redis as a data store, not a cache.

Common mistakes

• Holding a transaction open while the job runs: rows stay locked and a connection is tied up. Claim, commit, then work
• Retrying instantly with no backoff, hammering a service that is already down
• Retrying forever: a permanently broken job burns resources indefinitely
• Believing the queue delivers once, then writing a job that can charge twice
• Enqueuing to another system before the data commits: the worker cannot find the order yet
• Never deleting finished jobs: delete done rows older than 7 days nightly
• A lease shorter than the slowest run: jobs quietly run twice in parallel

Sources

• SELECT — The Locking Clause (SKIP LOCKED), PostgreSQL documentation
• Amazon SQS visibility timeout — AWS documentation
• Using dead-letter queues in Amazon SQS — AWS documentation
• Timeouts, retries, and backoff with jitter — AWS Builders’ Library
• Retrying failing jobs — BullMQ docs and Going to production — BullMQ docs
• Error Handling — Sidekiq wiki
• Idempotent requests — Stripe API reference
• pg-boss — GitHub and Solid Queue — GitHub