diff --git a/.gitignore b/.gitignore index 2d0969f9c..6df3ea309 100644 --- a/.gitignore +++ b/.gitignore @@ -1,21 +1,7 @@ .DS_Store __pycache__/ *.swp -config/maps.env -config/azure-api.env -config/chart.env -config/apis-and-selection.env -config/vision_api_key.json -config/ollama.env -config/healthcheck.env -config/image-summary.env -config/image_daily_cron.env -config/imagelogs.env -config/make_request.env -config/pp_daily_test.env -config/sendimagereq.env -config/slack-webhook-preprocessors.env -config/imageup.env -config/gemini.env *.sh docker-compose.override.yml +# Ignore all .env files in config/ +config/**/*.env \ No newline at end of file diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 4fe5e1600..b5cd4da3f 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -103,11 +103,11 @@ Docker Compose uses environment files to configure how services run. In IMAGE, y b) Component-specific envs: These configure runtime services (preprocessors, handlers, or standalone services) which are loaded by containers at runtime. They often hold API keys, model names, or URLs that you don't want to check into git. Filenames are specified in docker-compose. For example, ensure the following files exist in the config/ folder and are populated with appropriate credentials: - `apis-and-selection.env, azure-api.env, llm.env, maps.env` + `cloud-ocr.env, azure-api.env, llm.env, maps.env` If a service is not in use, simply create an empty file with the same name to avoid startup errors. Here is a command to create them all: - `touch config/{maps.env,express-common.env,llm.env,azure-api.env}` + `touch config/{maps.env,llm.env,azure-api.env}` TIP: once completed, these files include credentials that should not be committed in Git! @@ -199,9 +199,11 @@ networks: ``` # Connecting to, or running your own, visual LLM -Since multiple IMAGE preprocessors us a visual LLM (currently qwen), you need to either specify a cloud endpoint, or run your own LLM locally on your server. +Since multiple IMAGE preprocessors use a visual LLM (currently gemma), you need to either specify a cloud endpoint, or run your own LLM locally on your server. + Our production reference server uses vLLM as the runtime for LLM models. Like Traefik, it runs with its own docker-compose in its own directory on our server (e.g., `/var/docker/ollama`). + It’s not a preprocessor itself; instead, several preprocessors in IMAGE-server call out to whatever LLM endpoint you configure (Ollama/vLLM locally, or a remote API if you set one). For example, the following preprocessors connect to the LLM via env_file `/var/docker/image/config/llm.env`: content-categoriser, graphic-caption, text-followup, and multistage-diagram-segmentation. Key features: - Accepts multimodal graphic and text prompots @@ -217,8 +219,30 @@ You must specifiy services such as: - vllm: runs `vllm/vllm-openai:latest`, reserves a GPU, serves an OpenAI-style API at port 8000. - open-webui: optional WebUI that connects to Ollama or vLLM and is exposed via Traefik with a hostname like `ollama.unicorn.cim.mcgill.ca`. +## Using a different model per preprocessor +By default, `config/llm.env` sets the LLM endpoint and model used by every preprocessor that calls out to a vision LLM. If you want a specific preprocessor to use a different endpoint or model than this default, you can add a preprocessor-specific override. + +In `docker-compose.yml`, preprocessors that support this list two `env_file` entries: +```yaml +env_file: + - path: ./config/llm.env + required: true + - path: ./config/.env + required: false +``` +The global `config/llm.env` is always loaded first; the preprocessor-specific file is loaded second **only if it exists**, and any variable it sets overrides the global value for that preprocessor alone. If the file doesn't exist, the preprocessor simply falls back to the global configuration. No extra setup is required unless you actually want an override. + +To set one up, create a `.env` file within the existing `config/` directory using the naming convention `.env`. Set `LLM_API_KEY`, `LLM_URL`, and `LLM_MODEL`. Add this path to the `docker-compose.yml` file's `env_file` entry for that service (if it does not exist) and set the `required` flag to `False`. + +**If you override any one of these three, set all three**: they describe a single endpoint together, and partially overriding them will send incorrect credentials to the new endpoint and fail the request. + +Preprocessors currently supporting this override pattern (check +`docker-compose.yml` for the current list, as this may grow): +`content-categoriser`, `graphic-caption`, `text-followup`, `object-detection-llm`, `multistage-diagram-segmentation`. + +The same layered `env_file` override functionality can be used for other service configurations as well. -GPU Notes: +## GPU Notes: Some containers that require GPU (and don't use a cloud endpoint or the LLM) include espnet-tts, text-followup, semantic-segmentation,object-detection, action-recognition, and so on. You can see which ones need GPU directly in docker-compose.yml since they include a `deploy.resources.reservations.devices` stanza with `driver: nvidia`. TIP: if you run into `Cannot start service ...: could not select device driver "nvidia"`, you can use this checklist to guide you: diff --git a/config/Readme.md b/config/Readme.md index 97d7ef459..134f2b377 100644 --- a/config/Readme.md +++ b/config/Readme.md @@ -11,28 +11,23 @@ This file contains the API key used to call Google Places API. [Here](https://de * [autour preprocessor](https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/preprocessors/autour) * [openstreetmap preprocessor](https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/preprocessors/openstreetmap) -Following is the sample format of maps.env file: -``` -GOOGLE_PLACES_KEY = [INSERT KEY STRING] -``` +Check [`maps.env.example`](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/config/maps.env.example) for format specifications. -### apis-and-selection.env +### cloud-ocr.env Entries in this env file are required by the following preprocessors: * [ocr-clouds-preprocessor](https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/preprocessors/ocr) -Following is the sample format of apis-and-selection.env file: +Check [`cloud-ocr.env.example`](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/config/cloud-ocr.env.example) for format specifications. + +#### Note: +`CLOUD_SERVICE` is set within the [docker-compose](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/docker-compose.yml) file. -``` -AZURE_API_KEY = [INSERT KEY STRING] -FREEOCR_API_KEY = [INSERT KEY STRING] -GOOGLE_APPLICATION_CREDENTIALS = [INSERT KEY FILE PATH AS STRING] -CLOUD_SERVICE = [INSERT OPTION STRING (see options below)] -``` * `CLOUD_SERVICE` determines the desired cloud service to be used. Its possible values are: * `AZURE_OCR` (for [Microsoft Azure OCR API](https://westus.dev.cognitive.microsoft.com/docs/services/computer-vision-v3-2/operations/56f91f2e778daf14a499f20d)) * `AZURE_READ` (for [Microsoft Azure Read API](https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/how-to/call-read-api)) * `GOOGLE_VISION` (for [Google Cloud Vision API](https://cloud.google.com/vision/docs/ocr)) * `FREE_OCR` (for [Free OCR API](https://ocr.space/OCRAPI)) + * `AZURE_API_KEY` is found in your [Azure portal](https://portal.azure.com) * `FREEOCR_API_KEY` can be obtained at [OCR API portal](https://ocr.space/ocrapi) * `GOOGLE_APPLICATION_CREDENTIALS` contains path to credentials file. Refer [documentation](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) for details. @@ -42,7 +37,11 @@ CLOUD_SERVICE = [INSERT OPTION STRING (see options below)] This env file is required by the following preprocessors: * [graphic tagger](https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/preprocessors/graphic-tagger) -Following is the sample format of azure-api.env file: -``` -AZURE_API_KEY = [INSERT KEY STRING] -``` +Check [`azure-api.env.example`](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/config/azure-api.env.example) for format specifications. + +### llm.env +Entries in this env file are required by preprocessors that use an LLM via the shared LLM client ([`utils/llm/client.py`](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/utils/llm/client.py)). + +Check [`llm.env.example`](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/config/llm.env.example) for format specifications. + +**Using a different model per service:** `llm.env` sets the default LLM endpoint and model for *all* preprocessors that use the shared LLM client. If a specific preprocessor needs to use a different endpoint or model than this default, do not edit this file. Instead, create a separate `.env` file for that preprocessor override using the naming convention: `.env` within the existing `config/` directory. This applies to any service — preprocessor, handler, or otherwise — that may support its own override in the future, not just LLM-based preprocessors. \ No newline at end of file diff --git a/config/azure-api.env.example b/config/azure-api.env.example new file mode 100644 index 000000000..736fce7ae --- /dev/null +++ b/config/azure-api.env.example @@ -0,0 +1,2 @@ +# Following is the sample format of azure-api.env file: +AZURE_API_KEY = [INSERT KEY STRING] \ No newline at end of file diff --git a/config/cloud-ocr.env.example b/config/cloud-ocr.env.example new file mode 100644 index 000000000..e12d815d3 --- /dev/null +++ b/config/cloud-ocr.env.example @@ -0,0 +1,4 @@ +# Following is the sample format of cloud-ocr.env file: +AZURE_API_KEY = [INSERT KEY STRING] +FREEOCR_API_KEY = [INSERT KEY STRING] +GOOGLE_APPLICATION_CREDENTIALS = [INSERT KEY FILE PATH AS STRING] \ No newline at end of file diff --git a/config/express-common.env b/config/express-common.env deleted file mode 100644 index 2d18baedf..000000000 --- a/config/express-common.env +++ /dev/null @@ -1 +0,0 @@ -MAX_BODY=10mb diff --git a/config/llm.env.example b/config/llm.env.example new file mode 100644 index 000000000..88fa7eda1 --- /dev/null +++ b/config/llm.env.example @@ -0,0 +1,12 @@ +# Sample format of llm.env file: +LLM_API_KEY = [INSERT KEY STRING] +LLM_URL = [INSERT OPENAI-COMPATIBLE ENDPOINT URL] +LLM_MODEL = [INSERT MODEL NAME] + +# Note: + +# LLM_URL can point at a local model server (e.g. Ollama, vLLM) +# or a cloud provider that exposes an OpenAI-compatible API +# (e.g. OpenRouter). + +# LLM_MODEL must be a model the chosen endpoint supports. \ No newline at end of file diff --git a/config/maps.env.example b/config/maps.env.example new file mode 100644 index 000000000..b993fba79 --- /dev/null +++ b/config/maps.env.example @@ -0,0 +1,2 @@ +# Following is the sample format of maps.env file: +GOOGLE_PLACES_KEY = [INSERT KEY STRING] \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 4327e0c77..742482093 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,13 +5,12 @@ services: image: ghcr.io/shared-reality-lab/image-orchestrator:${REGISTRY_TAG} restart: "no" environment: + - MAX_BODY=10mb - PARALLEL_PREPROCESSORS=ON - STORE_IMAGE_DATA=ON - MEMCACHE_SERVERS=memcached:11211 - PREPROCESSOR_TIMEOUT=90000 - CACHE_OVERRIDE=/var/cache/IMAGE - env_file: - - ./config/express-common.env group_add: # This should be set in an environment variable or .env - ${DOCKER_GID} @@ -92,7 +91,10 @@ services: image: ghcr.io/shared-reality-lab/image-preprocessor-autour:${REGISTRY_TAG} restart: "no" env_file: - - ./config/maps.env + - path: ./config/maps.env + required: true + - path: ./config/autour.env + required: false labels: ca.mcgill.a11y.image.preprocessor: 1 ca.mcgill.a11y.image.port: 5000 @@ -113,7 +115,10 @@ services: ca.mcgill.a11y.image.required_dependencies: "" ca.mcgill.a11y.image.optional_dependencies: "" env_file: - ./config/llm.env + - path: ./config/llm.env + required: true + - path: ./config/content-categoriser.env + required: false environment: - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} - WARMUP_ENABLED=true @@ -129,7 +134,10 @@ services: ca.mcgill.a11y.image.required_dependencies: "" ca.mcgill.a11y.image.optional_dependencies: "" env_file: - ./config/llm.env + - path: ./config/llm.env + required: true + - path: ./config/graphic-caption.env + required: false environment: - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} - WARMUP_ENABLED=true @@ -152,7 +160,10 @@ services: ca.mcgill.a11y.image.required_dependencies: "" ca.mcgill.a11y.image.optional_dependencies: "" env_file: - ./config/llm.env + - path: ./config/llm.env + required: true + - path: ./config/text-followup.env + required: false object-detection: profiles: [default] @@ -191,7 +202,10 @@ services: ca.mcgill.a11y.image.required_dependencies: "content-categoriser" ca.mcgill.a11y.image.optional_dependencies: "" env_file: - ./config/llm.env + - path: ./config/llm.env + required: true + - path: ./config/object-detection-llm.env + required: false multistage-diagram-segmentation: profiles: [production, test, default] @@ -214,7 +228,10 @@ services: - SAM_MODEL_PATH=/usr/src/app/models/sam2.1_l.pt - WARMUP_ENABLED=true env_file: - ./config/llm.env + - path: ./config/llm.env + required: true + - path: ./config/multistage-diagram-segmentation.env + required: false object-grouping: profiles: [production, test, default] @@ -234,7 +251,10 @@ services: image: ghcr.io/shared-reality-lab/image-preprocessor-openstreetmap:${REGISTRY_TAG} restart: "no" env_file: - - ./config/maps.env + - path: ./config/maps.env + required: true + - path: ./config/openstreetmap.env + required: false labels: ca.mcgill.a11y.image.preprocessor: 3 ca.mcgill.a11y.image.port: 5000 @@ -294,8 +314,6 @@ services: profiles: [production, test, default] image: ghcr.io/shared-reality-lab/image-handler-photo-audio:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env labels: ca.mcgill.a11y.image.handler: enable ca.mcgill.a11y.image.required_dependencies: "semantic-segmentation,object-grouping,object-detection,graphic-caption" @@ -303,14 +321,13 @@ services: volumes: - sc-store:/tmp/sc-store environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} autour-handler: profiles: [production, test, default] image: ghcr.io/shared-reality-lab/image-handler-autour:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env labels: ca.mcgill.a11y.image.handler: enable ca.mcgill.a11y.image.required_dependencies: "autour-preprocessor" @@ -318,14 +335,13 @@ services: volumes: - sc-store:/tmp/sc-store environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} photo-audio-haptics-handler: profiles: [production, test, default] image: ghcr.io/shared-reality-lab/image-handler-photo-audio-haptics:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env labels: ca.mcgill.a11y.image.handler: enable ca.mcgill.a11y.image.required_dependencies: "semantic-segmentation,object-grouping,object-detection" @@ -333,14 +349,13 @@ services: volumes: - sc-store:/tmp/sc-store environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} high-charts-handler: profiles: [production, test, default] image: ghcr.io/shared-reality-lab/image-handler-high-charts:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env labels: ca.mcgill.a11y.image.handler: enable ca.mcgill.a11y.image.required_dependencies: "" @@ -348,6 +363,7 @@ services: volumes: - sc-store:/tmp/sc-store environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} svg-od-handler: @@ -407,14 +423,13 @@ services: profiles: [production, test, default] image: ghcr.io/shared-reality-lab/image-preprocessor-nominatim:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env labels: ca.mcgill.a11y.image.preprocessor: 2 ca.mcgill.a11y.image.cacheTimeout: 3600 ca.mcgill.a11y.image.required_dependencies: "" ca.mcgill.a11y.image.optional_dependencies: "" environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} - NOMINATIM_SERVER=https://pegasus.cim.mcgill.ca/nominatim # Optional fallback if Pegasus is unreachable @@ -473,9 +488,8 @@ services: profiles: [test, default] image: ghcr.io/shared-reality-lab/image-handler-motd:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env environment: + - MAX_BODY=10mb - MOTD=Hello, world! - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} labels: @@ -487,13 +501,12 @@ services: profiles: [test, default] image: ghcr.io/shared-reality-lab/image-handler-hello-haptics:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env labels: ca.mcgill.a11y.image.handler: enable ca.mcgill.a11y.image.required_dependencies: "object-detection-llm" ca.mcgill.a11y.image.optional_dependencies: "" environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} # For environment setup refer to https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/preprocessors/ocr @@ -502,7 +515,8 @@ services: image: ghcr.io/shared-reality-lab/image-preprocessor-ocr-clouds:${REGISTRY_TAG} restart: "no" env_file: - - ./config/apis-and-selection.env + - path: ./config/cloud-ocr.env + required: true labels: ca.mcgill.a11y.image.preprocessor: 4 ca.mcgill.a11y.image.port: 5000 @@ -510,6 +524,7 @@ services: ca.mcgill.a11y.image.required_dependencies: "" ca.mcgill.a11y.image.optional_dependencies: "object-detection-llm" environment: + - CLOUD_SERVICE=AZURE_READ - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} object-depth-calculator: @@ -553,8 +568,6 @@ services: profiles: [test, default] image: ghcr.io/shared-reality-lab/image-handler-osm-streets:${REGISTRY_TAG} restart: "no" - env_file: - - ./config/express-common.env depends_on: - supercollider - espnet-tts @@ -565,6 +578,7 @@ services: volumes: - sc-store:/tmp/sc-store environment: + - MAX_BODY=10mb - PII_LOGGING_ENABLED=${PII_LOGGING_ENABLED} svg-open-street-map-handler: diff --git a/orchestrator/README.md b/orchestrator/README.md index 321826e48..dc76da2d8 100644 --- a/orchestrator/README.md +++ b/orchestrator/README.md @@ -58,9 +58,8 @@ Here is a snippet of the service from our sample `docker-compose.yml`: ```yaml orchestrator: image: ghcr.io/shared-reality-lab/image-orchestrator:unstable - env_file: - - ./config/express-common.env environment: + - MAX_BODY=10mb - STORE_IMAGE_DATA=ON - PARALLEL_PREPROCESORS=ON group_add: @@ -70,12 +69,9 @@ orchestrator: - user-logs:/var/log/IMAGE ``` -The environment file loaded at `./config/express-common.env` increases the maximum body size allowed by Express. -If this is not included, larger requests (for example, those including larger graphics) will be rejected by the orchestrator -and not be handled by IMAGE. For consistency, this value should be used in all containers using Express. +The environment variable section sets the options for the orchestrator. The `MAX_BODY` variable increases the maximum body size allowed by Express. If this is not included, larger requests (for example, those including larger graphics) will be rejected by the orchestrator and not be handled by IMAGE. For consistency, this value should be used in all containers using Express. -The environment variable section sets the options for the orchestrator. These may be set another way, or the section can just -be omitted if the default behavior is desired. +Other variables may be set another way, or just omitted if the default behavior is desired. The orchestrator runs as a non-root user. As such, the container must be run with permissions of the `docker` group on the host in order to access the socket. The socket must also be mounted into the container (the bind mount `/var/run/docker.sock:/var/run/docker.sock:ro`). diff --git a/preprocessors/ocr/README.md b/preprocessors/ocr/README.md index a65fccdde..df2bec9df 100644 --- a/preprocessors/ocr/README.md +++ b/preprocessors/ocr/README.md @@ -22,17 +22,15 @@ The desired cloud service is determined by an environment variable called `CLOUD * `CLOUD_SERVICE="FREE_OCR"` The path to an environment file containing this variable should be provided in the `docker-compose.yml`, right in the `env_file` field of the `ocr-clouds-preprocessor` service. + ## Environment setup -The environment file (apis-and-selection.env) should contain the desired cloud service to be used, and the corresponding api keys. -Following is the sample format of apis-and-selection.env file: +The [docker-compose](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/docker-compose.yml) file should set the desired cloud service to be used under its environment variables. + +The environment file (cloud-ocr.env) should contain the corresponding api keys for the selected service. + +Check [`cloud-ocr.env.example`](https://github.com/Shared-Reality-Lab/IMAGE-server/blob/main/config/cloud-ocr.env.example) for format specifications. -``` -AZURE_API_KEY = [INSERT KEY STRING] -FREEOCR_API_KEY = [INSERT KEY STRING] -GOOGLE_APPLICATION_CREDENTIALS = [INSERT KEY FILE PATH AS STRING] -CLOUD_SERVICE = [INSERT OPTION STRING (see options below)] -``` * `CLOUD_SERVICE` determines the desired cloud service to be used. Its possible values are: * `AZURE_OCR` (for [Microsoft Azure OCR API](https://westus.dev.cognitive.microsoft.com/docs/services/computer-vision-v3-2/operations/56f91f2e778daf14a499f20d)) * `AZURE_READ` (for [Microsoft Azure Read API](https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/how-to/call-read-api))