Enable cloud LLMs to be used by preprocessors

[jeffbl]

the ollama.env file allows you to specify the local ollama / open-webui URL for making LLM queries. Goal of this work item is to test and document, and make code changes if necessary, to allow the use of cloud services as well, for a few reasons:

1. Can use larger models than we can fit on our local GPUs
2. Can run server without a GPU at all (or with wimpy GPUs like on our test server)
3. Have a backup in case our production server goes down
4. Have the ability to use local GPU until load increases, then start using cloud to keep query latency low

Note:

1. A single variable for all preprocessors (as currently implemented) probably doesn't make sense, since we would likely want to run smaller models locally, and larger ones in the cloud
2. This cannot go to production until our TOS is updated to allow data to be sent to cloud
3. Documenting what is tested, with examples for different cloud services, would be useful to others setting up IMAGE servers
4. Suggest that @shahdyousefak documentation for bringing up a new IMAGE server default to this, since it would mean a faster path to getting something working

[jeffbl]

To be considered as we move toward a hybrid server in #1061

[jeffbl]

This generally works, by changing the necessary .env files. But this changes it for all preprocessors. This can in theory be overriden within the docker-compose.yml on a per-preprocessor basis, but we have not tested that enough to close this item.

[MansiDhanania]

@jeffbl Since the repository is set up to use an OpenAI-compatible cloud VLM endpoint via client.py, and we wanted to move towards using only 1 VLM for all LLM-related tasks, do we want the provision of using each preprocessor with a different model?

[jeffbl]

@MansiDhanania The idea was that there would be a default one, with the ability to override. The only interesting case I can think of at the moment might be the Semantic Segmentation work to move to SAM (#1152). That is not an LLM (I think?), but we can imagine that an LLM could do that work eventually, but it might not be the same one as we have for everything else. Same with TTS, etc. The main thing we needed on this was making sure that a generic endpoint works for all the preprocessors. i think we've demonstrated that as we've converted and tested the ones that are already updated. However, it would also be a shame to one-off a bunch of preprocessors with custom variables/overrides, without making that a generic way to do it that makes sense for all preprocessors. Note also that how we run our server is not how others might choose to do it. Someone else may decide to run a server that instead of one by LLM, uses a bunch of smaller ones tailored to each specific preprocessor, since that is cheaper or better in their case. The idea of this work item is there should be a clean&consistent way of doing this.

I could also easily argue that unless there is another immediate use case for this, it is not a priority right now, but that is an @aidanbruneel call. First I'd verify what is already implemented, and then make a call on this.

[aidanbruneel]

I checked out what I could. The LLM preprocessors use the shared LLMClient, which reads LLM_API_KEY, LLM_URL, and LLM_MODEL from the container environment. The docker compose configuration supplies the same config/llm.env to each of them, while a deployment-specific compose override should be able to replace those variables.

We should not build a new preprocessor-by-preprocessor configuration abstraction unless testing shows the existing compose override mechanism is insufficient. We can continue using one default VLM, while other deployments retain the option to configure a different endpoint or model for a particular preprocessor.

So @MansiDhanania if you could:

1. Test one LLM preprocessor with a different endpoint/model than the default.
2. Document a short example of that override here.
3. If it works, we can close this issue as completed. If it does not, I'll open a follow-up issue describing the actual missing capability.

@jeffbl SAM and TTS do not seem to create an immediate use case here because they are not currently consumers of the shared LLM client, right?

[jeffbl]

@aidanbruneel no, but I'd argue the mechanism should be the same so it is consistent. i.e., every preprocessor by default uses a "main/default" llm. If it has more specific needs or something that the LLM cannot do (maybe TTS, SAM, etc.) then it should override it. That probably means a separate API endpoint URL, token/auth, and model name.