Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"version": "3.3"
}
},
"postCreateCommand": "uv sync --all-groups && pre-commit install && cd docs && printf 'source \"https://rubygems.org\"\\ngem \"github-pages\", group: :jekyll_plugins\\ngem \"webrick\"\\n' > Gemfile && bundle install",
"postCreateCommand": "uv sync --all-groups && pre-commit install && cd docs && printf 'source \"https://rubygems.org\"\\ngem \"github-pages\", \"~> 232\", group: :jekyll_plugins\\ngem \"webrick\"\\n' > Gemfile && bundle install",
"customizations": {
"vscode": {
"extensions": [
Expand Down
11 changes: 11 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,17 @@ BC-Bench is category-based and designed to grow over time. It currently has two
- Prefer high-order functions like map, filter, reduce over loops
- Prefer immutable data structures where possible

### Architecture design conventions

Follow this dependency direction when evolving the codebase:

- `bcbench.dataset` provides data models independently of runtime packages such as `agent`, `collection`, `commands`, `contamination`, `evaluate`, `operations`, and `results`.
- Implementation packages (`agent`, `collection`, `contamination`, `dataset`, `evaluate`, `operations`, and `results`) sit below `bcbench.cli` and `bcbench.commands`.
- `bcbench.evaluate` receives agent implementations through `AgentRunner` rather than selecting them.
- `bcbench.results` provides result models independently of execution packages such as `agent`, `evaluate`, and `operations`.

`bcbench.types` currently remains the central category registry through `EvaluationCategory`. Preserve the dependency direction during implementation and review.

### Readable code over documentation or comments
Function names should be self-explanatory. Do NOT add docstrings to functions unless absolutely necessary.
When a docstring is necessary, keep it short and use Google style. Include only useful sections such as `Args:` and `Returns:`; skip details that are obvious from names and type hints.
Expand Down
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,20 @@ BC-Bench is open source, and you're welcome to fork and adapt it for your own us
- **[CONTRIBUTING.md](CONTRIBUTING.md)** — fork setup, repo layout, versioning, day-to-day maintainer ops
- **[EXPERIMENT.md](EXPERIMENT.md)** — run an experiment (toggle instructions / skills / agents / MCP / model) against an existing category
- **[CATEGORIES.md](CATEGORIES.md)** — add a new evaluation category alongside the existing `bug-fix` / `test-generation` / `code-review` / `nl2al`

## Citation

The [paper](https://arxiv.org/abs/2608.20851) and its [LaTeX source](paper/) describe BC-Bench's design and evaluation. If you use BC-Bench in your research, please cite:

```bibtex
@misc{sun2026bcbench,
title={{BC-Bench}: Evaluating Agentic Engineering in a Domain-Specific Language for ERP},
author={Sun, Haoran and Hansen, Klaus Marius},
year={2026},
eprint={2608.20851},
archivePrefix={arXiv},
primaryClass={cs.SE},
doi={10.48550/arXiv.2608.20851},
url={https://arxiv.org/abs/2608.20851}
}
```
47 changes: 47 additions & 0 deletions docs/_layouts/default.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
<!--
# Override of jekyll-theme-cayman's default layout to add an arXiv button.
# Copied verbatim from the cayman gem with one extra <a class="btn"> added
-->

<!DOCTYPE html>
<html lang="{{ site.lang | default: "en-US" }}">
<head>
<meta charset="UTF-8">

{% seo %}
<link rel="preconnect" href="https://fonts.gstatic.com">
<link rel="preload" href="https://fonts.googleapis.com/css?family=Open+Sans:400,700&display=swap" as="style" type="text/css" crossorigin>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#157878">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
<link rel="stylesheet" href="{{ '/assets/css/style.css?v=' | append: site.github.build_revision | relative_url }}">
{% include head-custom.html %}
</head>
<body>
<a id="skip-to-content" href="#content">Skip to the content.</a>

<header class="page-header" role="banner">
<h1 class="project-name">{{ page.title | default: site.title | default: site.github.repository_name }}</h1>
<h2 class="project-tagline">{{ page.description | default: site.description | default: site.github.project_tagline }}</h2>
{% if site.github.is_project_page %}
<a href="{{ site.github.repository_url }}" class="btn">View on GitHub</a>
{% endif %}
<a href="https://arxiv.org/abs/2608.20851" class="btn" target="_blank" rel="noopener">Read the Paper (arXiv)</a>
{% if site.show_downloads %}
<a href="{{ site.github.zip_url }}" class="btn">Download .zip</a>
<a href="{{ site.github.tar_url }}" class="btn">Download .tar.gz</a>
{% endif %}
</header>

<main id="content" class="main-content" role="main">
{{ content }}

<footer class="site-footer">
{% if site.github.is_project_page %}
<span class="site-footer-owner"><a href="{{ site.github.repository_url }}">{{ site.github.repository_name }}</a> is maintained by <a href="{{ site.github.owner_url }}">{{ site.github.owner_name }}</a>.</span>
{% endif %}
<span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a>.</span>
</footer>
</main>
</body>
</html>
439 changes: 439 additions & 0 deletions paper/bcbench.bib

Large diffs are not rendered by default.

Loading
Loading