Add LLM-as-a-judge to eval suite and bump NDD version by sarahyurick · Pull Request #2324 · NVIDIA-NeMo/Curator

sarahyurick · 2026-08-20T03:51:32Z

Things to consider:

Agent skills file for how to write the YAML and Jinja files
How to handle empty examples
How to truncate according to token limits (opened a Linear issue to track it as a follow-up: https://linear.app/nvidia/issue/NMCUR-402/enhance-llm-judge-eval-to-filtermodify-by-token-count-limits)
Multiple LLMs doing the same judging task
Instructions about LLMs to use as judges
Instructions for Slurm array partitioning
Instructions for resumability and retry support. Like with my small test run, there were 192 input records but 189 output records. Likely an NDD refusal or error for some reason. Do we care about getting every single judgement?
Test NDD's LLMJudgeColumnConfig.multi_modal_context (e.g., ImageContext)
vLLM shutdown issue when I Ctrl+C a run (not in scope for this PR)

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

copy-pr-bot · 2026-08-20T03:51:36Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

sarahyurick · 2026-08-20T03:57:01Z

+    files_per_partition: int | None,
+) -> Pipeline:
+    """Build one streaming Curator reader → optional language gate → NDD stages → filters → writer pipeline."""
+    # TODO: Add an optional TokenLengthFilter stage before NDD stages so prompts


Big TODO. Right now I just do string truncation in the jinja file. But we should decide the best course of action.

Also, since some extractors might return null while others return content, we should decide whether those examples are evaluated or not? Right now I force the empty string and still give it to the judge.

Opened a Linear issue, since this PR is already fairly large: https://linear.app/nvidia/issue/NMCUR-402/enhance-llm-judge-eval-to-filtermodify-by-token-count-limits.

sarahyurick · 2026-08-20T04:02:06Z

+          extract_reasoning_content: false
+          scores:
+            - name: best_extraction
+              description: Select the candidate that is most useful as clean document text.


I had Codex generate all the Jinja, scoring, etc. prompts. Just a POC for now.

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

ayushdg

Minor comments around dependencies. Thanks a lot for the bump!

ayushdg · 2026-08-26T18:15:31Z

    "kaldiio;  sys_platform == 'never'",
    "levenshtein;  sys_platform == 'never'",
    "numba==0.65.0", # Override RAPIDS/legacy caps for the inference image; vLLM 0.22 requires numba 0.65.0
+    "packaging>=25", # Required by data-designer==0.9.1; overrides Lightning's packaging<25 cap from nemo-toolkit[asr]


Let's add a followup ticket to bump asr. I think some of these overrides can be avoided.

Opened a GitHub issue: NVIDIA-NeMo/Speech#16144 and a Linear issue: https://linear.app/nvidia/issue/NMCUR-404/relax-the-lightning240-constraint-for-nemo-speech-asr.

ayushdg · 2026-08-26T18:17:27Z

    "xgrammar>=0.1.32", # Override vllm's ==0.1.29 pin to address CVE GHSA-7rgv-gqhr-fxg3 (DoS via multi-layer nesting)
-    "sqlfluff>=4.2.0", # Address CVE-2026-46373/46374 (parser DoS); overrides data-designer-engine==0.5.5 sqlfluff<4 cap
-    "pandas>=3.0.0, <= 3.0.4" # Override data-designers constraint of <3
+    "sqlfluff>=4.2.0", # Address CVE-2026-46373/46374 (parser DoS); data-designer-engine==0.9.1 supports sqlfluff>=4.1,<5


We can probably get rid of this and do uv lock --upgrade-package sqlfluff to have uv pick up a newer version rather than needing an override now.

ayushdg · 2026-08-26T18:17:56Z

 [[package]]
 name = "cryptography"
-version = "50.0.0"
+version = "49.0.0"


Can we also override cryptography. Looks like NDD adds a pin but should work with newer versions. It's just following older conventions of keeping an upper pin: https://github.com/NVIDIA-NeMo/DataDesigner/pull/821/changes.
>=50 is recommended for https://github.com/NVIDIA-NeMo/Curator/security/dependabot/217.

We can also open an issue in NDD to remove the upper pin on cryptography

ayushdg · 2026-08-26T18:30:25Z

 [[package]]
 name = "gitpython"
-version = "3.1.58"
+version = "3.1.57"


Do we know if a specific package is downgrading this?

Not sure. Looks like the latest is 3.1.60. I can do uv lock --upgrade-package GitPython.

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

VibhuJawa

Still understanding the workflow. Left an initial review . I wonder if we should do pairwise comparisons as a first pass vs score based judges ?

VibhuJawa · 2026-08-27T05:25:09Z

+}
+```
+
+NDD adds one top-level column for each judge. A judge named `extraction_quality` with a `quality` score produces a result shaped like this:


Quick question: Is each judge 1 LLM call or multiple LLM calls ?

Each judges entry (under execution.stages.name) in the YAML file is 1 LLM call per row (no matter how many scores it has).

cc_extract_example/text_extraction_qwen_judge.yaml: 2 judges = 2 calls/row

cc_extract_example/text_extraction_qwen_gemma_judges.yaml: 4 judges = 4 calls/row

VibhuJawa · 2026-08-27T05:55:58Z

+
+
+def _place_filters(config: dict[str, object], stages: list[dict[str, object]]) -> list[list[dict[str, object]]]:
+    """Place top-level filters after the NDD stage that produces their judge column."""


For my understanding: What is the use of place filters produces their judge column

The idea is to place Filter stages immediately after the judge stage that is producing the relevant column. For example, if judge 1 produces extraction_quality and you filter on it, placing the filter right after stage 1 means rows that fail the filter get dropped before judge 2 ever runs on them, saving that judge's LLM calls on rows you were going to discard anyway.

Although I am not so sure I want to keep all the filtering logic anyway. Like realistically, the user will probably want to inspect the "bad" columns themselves and keep the judging information, rather than just throwing it away. WDYT?

VibhuJawa · 2026-08-27T05:58:25Z

+def _build_language_filter_stage(
+    *,
+    language: str | None,
+    model_path: str | None,
+    min_score: float,
+    text_field: str,
+) -> ScoreFilter | None:
+    """Build an optional FastText language gate without retaining its score column."""
+    if not language:
+        return None
+    if not model_path:
+        msg = "--fasttext-langid-model-path is required when --language is provided."
+        raise ValueError(msg)
+    if not 0.0 <= min_score <= 1.0:
+        msg = "--min-langid-score must be between 0 and 1."
+        raise ValueError(msg)
+
+    # FastText is optional, so import it only for jobs that enable this stage.
+    from nemo_curator.stages.text.filters.fasttext import FastTextLangId
+
+    return ScoreFilter(
+        filter_obj=FastTextLangId(
+            model_path=model_path,
+            min_langid_score=min_score,
+            lang=language,
+        ),
+        text_field=text_field,
+        verbose=True,
+    ).with_(name="fasttext_language_filter")
+


I get the motivation for this but i wonder if we want it as a generic feature here ?

Can you explain more what your suggestion is? IMO it is nice for doing language-specific testing, although I can understand it isn't applicable to every use case. It is mostly supposed to be a "nice-to-have."

VibhuJawa · 2026-08-27T06:01:29Z

+            judge_stages = []
+            for stage, filters_after_stage in zip(configured_stages, stage_filters, strict=True):


What is the intuition b/w using 1 stage vs multiple stages , do we expect perf to be better b/w them ?

Initially I was seeing better perf by using 1 stage, but the advantage of using multiple stages is that it enables resumability between each Data Designer stage.

I can try to do more rigorous benchmarks though. Like if multi stage can get the same perf as single stage, then we should only support multi stage.

~~Hmm I ran another comparison benchmark and still found single stage to be faster. So I am thinking for now to just keep both, so that we can still enable resumability with multi stage.~~

Actually, the single_stage vs multi_stage comparison I was looking at isn't totally valid. single_stage took 67m but only wrote 411 rows; multi_stage took 80m and wrote 1553 rows from the same eligible pool (409 of the 411 overlap). So single_stage "won" by doing ~1/4 the work.

Root cause (from Codex): both judges use the same model, so their calls share one concurrency pool and one 60-second per-request timeout (inference_parameters.timeout defaults to 60s in NDD's factory.py). In single_stage, both judges' tasks (judge1: 3 scored fields with reasoning, much slower; judge2: 1 field, fast) get dispatched concurrently into the same actor's queue since they're independent columns with no dependency between them. Mixing a heavy and light workload in one bounded-concurrency queue inflates tail latency - enough of judge1's calls blew past 60s that the run spiraled into timeouts and never recovered (confirmed: "degraded performance" warnings hit 90–100% and then the run went silent for its last ~6 minutes). multi_stage splits the two judges into separate actor pools, so each queue is homogeneous and never hit this.

So actually, I am thinking I will just support multi_stage.

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

Add LLM-as-a-judge to eval suite

357df49

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

sarahyurick commented Aug 20, 2026

View reviewed changes

VibhuJawa self-requested a review August 20, 2026 04:11

sarahyurick commented Aug 20, 2026

View reviewed changes

Comment thread eval/llm_judge/generic_text_judge.py Outdated

Comment thread eval/llm_judge/generic_text_judge.py Outdated

Comment thread eval/llm_judge/generic_text_judge.py Outdated

sarahyurick and others added 11 commits August 20, 2026 10:38

code and readme cleanups

38c7009

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

filter logging

dad191a

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

reorganize example files

7c6e14f

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

add skill file and some script enhancements

2c5c5d7

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

ruff

642663f

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

add advice about how to pick models

1fdbffc

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

Merge branch 'main' into llm_judge

2553e83

add multi model example

db178e2

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

update name

b078b37

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

add pytests

b3b6596

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

update deps

bf9eff7

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

sarahyurick marked this pull request as ready for review August 26, 2026 17:08

sarahyurick requested review from a team and huvunvidia as code owners August 26, 2026 17:08

copy-pr-bot Bot deployed to public August 26, 2026 17:10 Active

copy-pr-bot Bot deployed to test August 26, 2026 17:10 Active

copy-pr-bot Bot temporarily deployed to nemo-ci August 26, 2026 17:11 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci August 26, 2026 17:11 Failure

copy-pr-bot Bot temporarily deployed to nemo-ci August 26, 2026 17:11 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci August 26, 2026 17:32 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci August 26, 2026 17:32 Failure

copy-pr-bot Bot had a problem deploying to nemo-ci August 26, 2026 17:32 Error

copy-pr-bot Bot temporarily deployed to nemo-ci August 26, 2026 17:32 Inactive

copy-pr-bot Bot deployed to public August 26, 2026 17:36 Active

fix sdg cpu tests

633752e

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

copy-pr-bot Bot deployed to public August 26, 2026 17:50 Active

copy-pr-bot Bot deployed to test August 26, 2026 17:51 Active

copy-pr-bot Bot temporarily deployed to nemo-ci August 26, 2026 17:51 Inactive

greptile-apps Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread eval/llm_judge/run_llm_judge.py

ayushdg reviewed Aug 26, 2026

View reviewed changes

sarahyurick added 2 commits August 26, 2026 12:03

greptile and readme

2ad3558

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

dep updates from Ayush

fc5c355

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

VibhuJawa reviewed Aug 27, 2026

View reviewed changes

sarahyurick and others added 5 commits August 27, 2026 11:34

add Vibhu's feedback

c896c7b

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

Merge branch 'main' into llm_judge

d29825f

Merge branch 'main' into llm_judge

14fe970

remove single_stage arg

6cf00cd

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

Merge branch 'main' into llm_judge

f6cc422



		def _place_filters(config: dict[str, object], stages: list[dict[str, object]]) -> list[list[dict[str, object]]]:
		"""Place top-level filters after the NDD stage that produces their judge column."""

		judge_stages = []
		for stage, filters_after_stage in zip(configured_stages, stage_filters, strict=True):

Uh oh!

Conversation

sarahyurick commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Aug 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayushdg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayushdg Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VibhuJawa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarahyurick Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarahyurick Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

sarahyurick commented Aug 20, 2026 •

edited

Loading

ayushdg Aug 26, 2026 •

edited

Loading

sarahyurick Aug 27, 2026 •

edited

Loading

sarahyurick Aug 28, 2026 •

edited

Loading