Skip to content

Add priority queue - #93

Open
donaldgray wants to merge 3 commits into
mainfrom
feature/priority
Open

Add priority queue#93
donaldgray wants to merge 3 commits into
mainfrom
feature/priority

Conversation

@donaldgray

Copy link
Copy Markdown
Member

Adds a VIP lane for PDF submissions, mirroring the DLCS API behaviour.

Adds a new endpoint POST /customers/{customer}/queue/priority (mimics POST /customers/{customer}/queue but skips queue and ends up in priority DLCS queue).

A priority submission is expedited at both in terms of processing and ingest:

  • Process: tasks are enqueued to a dedicated queue/cluster (named by PRIORITY_QUEUE_NAME) with its own small worker pool, so they are never blocked by a standard-queue backlog. Each engine container runs a second qcluster process for it (entrypoint-worker.sh);. This means no new instances or services running, the footprint is one mostly-idle process group per engine container.
  • Ingest: batches for priority collections are POSTed to the DLCS /customers/{customer}/queue/priority endpoint, so they also skip any DLCS-side backlog.

If PRIORITY_QUEUE_NAME is unset the behaviour is as before

Commits

  1. django-q==1.3.9django-q2==1.10.0, the maintained drop-in fork. Same django_q import path; ships extra django_q migrations applied by the normal migrate step. Prerequisite for multi-queue routing (ALT_CLUSTERS), and worth having regardless.
  2. Add priority queue for expedited submissions - the feature itself: new route, Collection.priority field + migration, cluster routing in the view, DLCS priority ingest, worker entrypoint, config/docs. Also repairs the integration test stack, which referenced a Dockerfile.CompositeHandler left over from the protagonist monorepo import and was unrunnable: it now builds the root Dockerfile, supplies required env vars, gates services on a postgres healthcheck, and bumps pytest-docker for Docker Compose v2.

Configuration

New configuration:

Variable Default Notes
PRIORITY_QUEUE_NAME unset (disabled) With the SQS broker, must equal the name of a real SQS queue
PRIORITY_WORKER_COUNT 1 Priority workers per engine instance

Infrastructure

New priority SQS queue (same settings as the standard one) and set the above env vars on the API and engine services.

Migrations

New migration to add priority field.

donaldgray and others added 2 commits July 31, 2026 16:13
django-q 1.3.9 has been unmaintained since 2021; django-q2 is the
maintained drop-in fork (same django_q import path and INSTALLED_APPS
entry). Ships additional django_q migrations, applied by the existing
'manage.py migrate' step. Prerequisite for routing tasks to multiple
queues via ALT_CLUSTERS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /customers/{customer}/queue/priority accepts the same payload as
the standard endpoint but routes processing through a VIP lane at both
levels, mirroring the DLCS API's own priority queue:

- locally, tasks are enqueued to a dedicated queue/cluster (named by
  PRIORITY_QUEUE_NAME) with its own worker pool, so they are not
  blocked by a standard-queue backlog. Each engine container runs a
  second qcluster process for it; no new instances are required.
- at ingest, batches for priority collections are POSTed to the DLCS
  /queue/priority endpoint.

If PRIORITY_QUEUE_NAME is unset the feature is inert: priority
submissions use the standard local queue but still ingest via the DLCS
priority endpoint. See docs/priority-queue.md for the full design.

Also repairs the integration test stack, which referenced a
Dockerfile.CompositeHandler left over from the protagonist monorepo
import and was unrunnable: builds the root Dockerfile instead, supplies
the required env vars, gates services on a postgres healthcheck, and
bumps pytest-docker for Docker Compose v2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@donaldgray
donaldgray requested a review from fmcc July 31, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant