Evaluate caption adherence for LLM captions?

[schrum2]

Not sure how to do this. Current caption adherence calculation depends on the fact that scenes can be given deterministic captions, which does not work for the wide variety of LLM captions. Still, need to evaluate how well images match the captions.

• Before figuring out how to evaluate, I have to make the process use the right captions. Using evaluate_caption_adherence.py currently just uses the caption field, so I need to add options to make it actually use the LLM captions to generate scenes. In fact, I probably want to generate a scene for each LLM caption, to see if all of the different captions of the same scene all generate the same (or similar) scene.

However, once I have the generated scenes, there is still a question of how to evaluate performance. Probably the only reasonable way to do it is with either an LLM and/or a VLM. Here are ideas:

• Take generated scene and the caption that generated it. Use the ASCII version of the scene, and ask an LLM (would have to be a good one like Claude) to what extent the generated scene fits the caption (since we are grading the diffusion model).
• Instead of sending ASCII version to an LLM, could send the generated scene to a VLM (though Claude can also handle image input) and ask how well the scene fits the caption.
• This idea only works for specific captions from training data. Compare the original scene that the caption(s) were derived from to the generated scene in terms of similarity. This is actually pretty easy to do and does not require an LLM/VLM.
• However, I can also ask VLMs to distinguish between two images: the original data and the generated version. There is already an issue related to this Use Vision Models to Assess Level Similarity (and other metrics) #222
• It is probably also important to take all of the captions for one scene (since each scene has multiple captions now) and see how similar the variety of scenes from the different captions are.