Add data-query category: AL query-generation benchmark - #740

Open

Onat Buyukakkus (onbuyuka) wants to merge 48 commits into

mainfrom

onbuyuka/data-query-category

Commits on Jul 12, 2026

Add data-query category: AL query-generation benchmark

Onat Buyukakkus
and
Copilot
committed
e3dc7fa

Commits on Jul 13, 2026

Merge remote-tracking branch 'origin/main' into onbuyuka/data-query-category
Onat Buyukakkus
committed
0d46002
Fix pre-commit: ruff-format + ty (Sequence[Mapping] for result_sets_match)

Onat Buyukakkus
and
Copilot
committed
8d08fbe
Register data-query in Get-BCBenchDatasetPath (container setup)

Onat Buyukakkus
and
Copilot
committed
f62ace6
data-query: clear workspace contents instead of rmtree (dir is mounted into container)

Onat Buyukakkus
and
Copilot
committed
a06dec4
data-query: drop invalid 'Extensible' property from wrapped API query (AL0124)

Onat Buyukakkus
and
Copilot
committed
ec5dfee
data-query: don't crash the job on a gold-query compile failure

Onat Buyukakkus
and
Copilot
committed
614dea7
data-query: normalize query object name and use per-object API entity sets

Onat Buyukakkus
and
Copilot
committed
5761e0e
data-query: use proven Invoke-AppBuildAndPublish + in-container OData fetch

Onat Buyukakkus
and
Copilot
committed
237c930
data-query: build Basic auth header by hand for the in-container OData fetch

Onat Buyukakkus
and
Copilot
committed
d7ad1dd
data-query: calibrate prompts + fix the intersection gold query

Onat Buyukakkus
and
Copilot
committed
8268f07

Commits on Jul 28, 2026

data-query: add 5 more deterministically-scorable tasks (6 -> 11)

Onat Buyukakkus
and
Copilot
committed
2a41a6c
data-query: address PR review feedback (scoring integrity + robustness)

Onat Buyukakkus
and
Copilot
committed
129c5ef
data-query: fix invalid Count columns in gold queries and skill

Onat Buyukakkus
and
Copilot
committed
afde215
data-query: exclude unscorable results from the bceval export too

Onat Buyukakkus
and
Copilot
committed
56f2335
data-query: validate the gold query before evaluating the agent's

Onat Buyukakkus
and
Copilot
committed
faf22ba
Withhold gold queries from the agent's filesystem during generation

Onat Buyukakkus
and
Copilot
committed
45715a3
Merge remote-tracking branch 'origin/main' into onbuyuka/data-query-category
Onat Buyukakkus
committed
adfca7c
Harden gold withholding: also hide the .git object database from the agent

Onat Buyukakkus
and
Copilot
committed
1c9aa72
Add --skills dispatch flag to toggle agent skills per run

Onat Buyukakkus
and
Copilot
committed
d24eb18
Normalize only numeric result values, preserve Code strings verbatim

Onat Buyukakkus
and
Copilot
committed
cad29ec

Commits on Jul 29, 2026

Merge branch 'main' into onbuyuka/data-query-category
Sun Haoran (haoranpb)
authored
979a388
Simplify data-query harness per owner review

Onat Buyukakkus
and
Copilot
committed
9820ae7

Commits on Jul 30, 2026

Commits on Aug 11, 2026

Commits on Aug 21, 2026

Merge branch 'main' of https://github.com/microsoft/BC-Bench into onbuyuka/data-query-category
Onat Buyukakkus
committed
7517c75
Add BC MCP + Microsoft Learn MCP as agent capabilities for data-query

Onat Buyukakkus
and
Copilot
committed
9732dcb
TEMP: BC MCP runner diagnostics (revert before merge)

Onat Buyukakkus
and
Copilot
committed
eb47c22
TEMP: pull BC 29 sandbox artifact from insider feed (revert before merge)

Onat Buyukakkus
and
Copilot
committed
63a62f9
Build the MCP config app inside the shared repo folder

Onat Buyukakkus
and
Copilot
committed
c718146
Pivot data-query to answer DATA, not just write a query (+ contamination canary)

Onat Buyukakkus
and
Copilot
committed
5693f08
Bake gold data, bump build timeout, and remove BC MCP diagnostics

Onat Buyukakkus
and
Copilot
committed
1774a75
Capture Copilot tool usage from the JSON event stream

Onat Buyukakkus
and
Copilot
committed
27c570f

Commits on Aug 22, 2026

Scrub BC connection vars from the agent env; temp-raise gold timeout

Onat Buyukakkus
and
Copilot
committed
eb4abc8
TEMP: re-add agent *.log upload to diagnose BC MCP tool use (revert before merge)

Onat Buyukakkus
and
Copilot
committed
1f8515f

Commits on Aug 24, 2026

Add localhost MCP gateway to isolate agent from BC /api (Phase A)

Onat Buyukakkus
and
Copilot
committed
50dcda3
Use Copilot-licensed user PAT for MCP registry policy in Actions

Onat Buyukakkus
and
Copilot
committed
62cc621
Note server-prefixed MCP tool names in the bc-al-query-mcp skill

Onat Buyukakkus
and
Copilot
committed
f3bfd71
Warm up + probe BC MCP tools through the gateway at startup

Onat Buyukakkus
and
Copilot
committed
7cefd9f
Fix warm-up probe: read SSE incrementally + log timing, raise timeout

Onat Buyukakkus
and
Copilot
committed
c3f9d23
Add NST log capture to diagnose BC MCP exposing zero tools

Onat Buyukakkus
and
Copilot
committed
bb9664a
Fix MCP Configuration DB query flags + add app install-status capture

Onat Buyukakkus
and
Copilot
committed
f3b2ee9
Remove accidentally-committed diagnostic download folder; ignore _*/

Onat Buyukakkus
and
Copilot
committed
8b6ee1a
Log raw HTTP status/body of an empty tools/list in the warm-up probe

Onat Buyukakkus
and
Copilot
committed
833d361