Integrate NL2AL harms testing with current red-team infrastructure - #786

Draft

Sun Haoran (haoranpb) with Copilot wants to merge 21 commits into

mainfrom

copilot/continue-previous-plan-in-cloud

Commits on Aug 11, 2026

Initial plan
Copilot
authored
f60f8d8
Add harms testing for BCAL via --harms-fixture (direct + indirect vectors)

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
a302881
Harms: add code_vulnerability evaluator, harms evaluate re-score command, fix query mapping

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
b34d30d
Harms: add injection-landing validation from tool-call logs

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
e085f89
harms: target Caption in smoke suite placement (reliably surfaced by inspect_page)

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
7cc970a
harms: integrate AI Red Teaming Agent + comprehensive multi-vector suite

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
e0eac72
harms: land red-team XPIA injections + resilient per-evaluator scoring

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
a4ee66b
harms: pin bcal via BCAL_EXECUTABLE + log resolved build/version

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
0619787
harms: record per-line harm delivery + validity, add 'harms annotate'

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
a70ada1
chore: gitignore AI red-teaming scan scratch dirs (.scan_*/)

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
1cbc9d2
harms: add rule/detector/expected to HarmsCase + vendor BCQuality security knowledge

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
4c3f636
harms: AL-aware code-vulnerability scoring (detector + BCQuality judge) + dataset

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
9bd3aa4
docs(harms): document code-vulnerability scoring (detector + BCQuality judge) + dataset

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
d16e9cf
harms: add --num-objectives to harvest (plumb through run_scan)

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
534f941
harms: add instruction-injection suite (benign unauthorized-instruction XPIA)

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
6d04cf3
harms: merge instruction-injection cases into the manual code-vulnerability suite

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
4c5c895
harms: short-circuit scoring for no-code responses (refusals/content-filter)

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
4dcff95
harms: narrow scoring short-circuit to genuine non-responses only

Thaddeus Loke (thloke)
and
Sun Haoran (haoranpb)
authored
448538c
Integrate harms testing onto current main

Copilot
and
Sun Haoran (haoranpb)
authored
2441252
Address harms integration review

Copilot
and
Sun Haoran (haoranpb)
authored
4bb5697
Clean up harms integration API

Copilot
and
Sun Haoran (haoranpb)
authored
771c366

Navigation Menu

Uh oh!

Integrate NL2AL harms testing with current red-team infrastructure - #786

Integrate NL2AL harms testing with current red-team infrastructure#786
Sun Haoran (haoranpb) with Copilot wants to merge 21 commits into
mainfrom
copilot/continue-previous-plan-in-cloud

Commits on Aug 11, 2026

Initial plan

Add harms testing for BCAL via --harms-fixture (direct + indirect vectors)

Harms: add code_vulnerability evaluator, harms evaluate re-score command, fix query mapping

Harms: add injection-landing validation from tool-call logs

harms: target Caption in smoke suite placement (reliably surfaced by inspect_page)

harms: integrate AI Red Teaming Agent + comprehensive multi-vector suite

harms: land red-team XPIA injections + resilient per-evaluator scoring

harms: pin bcal via BCAL_EXECUTABLE + log resolved build/version

harms: record per-line harm delivery + validity, add 'harms annotate'

chore: gitignore AI red-teaming scan scratch dirs (.scan_*/)

harms: add rule/detector/expected to HarmsCase + vendor BCQuality security knowledge

harms: AL-aware code-vulnerability scoring (detector + BCQuality judge) + dataset

docs(harms): document code-vulnerability scoring (detector + BCQuality judge) + dataset

harms: add --num-objectives to harvest (plumb through run_scan)

harms: add instruction-injection suite (benign unauthorized-instruction XPIA)

harms: merge instruction-injection cases into the manual code-vulnerability suite

harms: short-circuit scoring for no-code responses (refusals/content-filter)

harms: narrow scoring short-circuit to genuine non-responses only

Integrate harms testing onto current main

Address harms integration review

Clean up harms integration API