Add priority queue - #93
Open
donaldgray wants to merge 3 commits into
Open
Conversation
django-q 1.3.9 has been unmaintained since 2021; django-q2 is the maintained drop-in fork (same django_q import path and INSTALLED_APPS entry). Ships additional django_q migrations, applied by the existing 'manage.py migrate' step. Prerequisite for routing tasks to multiple queues via ALT_CLUSTERS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /customers/{customer}/queue/priority accepts the same payload as
the standard endpoint but routes processing through a VIP lane at both
levels, mirroring the DLCS API's own priority queue:
- locally, tasks are enqueued to a dedicated queue/cluster (named by
PRIORITY_QUEUE_NAME) with its own worker pool, so they are not
blocked by a standard-queue backlog. Each engine container runs a
second qcluster process for it; no new instances are required.
- at ingest, batches for priority collections are POSTed to the DLCS
/queue/priority endpoint.
If PRIORITY_QUEUE_NAME is unset the feature is inert: priority
submissions use the standard local queue but still ingest via the DLCS
priority endpoint. See docs/priority-queue.md for the full design.
Also repairs the integration test stack, which referenced a
Dockerfile.CompositeHandler left over from the protagonist monorepo
import and was unrunnable: builds the root Dockerfile instead, supplies
the required env vars, gates services on a postgres healthcheck, and
bumps pytest-docker for Docker Compose v2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a VIP lane for PDF submissions, mirroring the DLCS API behaviour.
Adds a new endpoint
POST /customers/{customer}/queue/priority(mimicsPOST /customers/{customer}/queuebut skips queue and ends up in priority DLCS queue).A priority submission is expedited at both in terms of processing and ingest:
PRIORITY_QUEUE_NAME) with its own small worker pool, so they are never blocked by a standard-queue backlog. Each engine container runs a secondqclusterprocess for it (entrypoint-worker.sh);. This means no new instances or services running, the footprint is one mostly-idle process group per engine container./customers/{customer}/queue/priorityendpoint, so they also skip any DLCS-side backlog.If
PRIORITY_QUEUE_NAMEis unset the behaviour is as beforeCommits
django-q==1.3.9→django-q2==1.10.0, the maintained drop-in fork. Samedjango_qimport path; ships extradjango_qmigrations applied by the normalmigratestep. Prerequisite for multi-queue routing (ALT_CLUSTERS), and worth having regardless.Collection.priorityfield + migration, cluster routing in the view, DLCS priority ingest, worker entrypoint, config/docs. Also repairs the integration test stack, which referenced aDockerfile.CompositeHandlerleft over from the protagonist monorepo import and was unrunnable: it now builds the rootDockerfile, supplies required env vars, gates services on a postgres healthcheck, and bumps pytest-docker for Docker Compose v2.Configuration
New configuration:
PRIORITY_QUEUE_NAMEPRIORITY_WORKER_COUNT1Infrastructure
New priority SQS queue (same settings as the standard one) and set the above env vars on the API and engine services.
Migrations
New migration to add
priorityfield.