Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,8 +264,9 @@ cache is safe to use by comparing the current build input hash and the cached
`BK_BUILD_CACHE_DIR` が設定されていればそれを cache root として使います。未設定の場合、custom runner が `CUSTOM_DIR` を渡していれば `$CUSTOM_DIR/build_cache/$CUSTOM_RUNNER_PROJECT_SLUG` を使います。どちらもなければ build cache は無効です。
復元してよいかは script 側が現在の build input hash と cache 内の `results/source_info.env` を比較して判断します。

Git sources are rechecked with `git ls-remote`; file/archive sources are
rechecked by SHA-256. If a container image hash is recorded, that image hash is
Git sources are rechecked with `git ls-remote` using the recorded `ref_name`
and `resolved_commit`; file/archive sources are rechecked by SHA-256. If a
container image hash is recorded, that image hash is
also verified. Host builds that go through the common `make` / `cmake` /
`ninja` wrappers are matched by a build-environment fingerprint collected just
before the build tool runs. The fingerprint includes loaded modules, selected
Expand All @@ -276,7 +277,7 @@ does not declare a GitLab `cache:` stanza; the cache directory must be a
site-managed persistent path such as the custom runner's `CUSTOM_DIR`, not a
per-job cleanup directory.

Git source `git ls-remote` で再確認し、file/archive source は SHA-256 を再計算します。
Git source は記録済みの `ref_name` と `resolved_commit` を使って `git ls-remote` で再確認し、file/archive source は SHA-256 を再計算します。
container image hash が記録されている場合は image hash も確認します。
common の `make` / `cmake` / `ninja` wrapper を通る host build は、build tool 実行直前に収集した build environment fingerprint で照合します。この fingerprint には loaded modules、選択された build 環境変数、tool の real path、version、binary SHA-256 hash が含まれます。
cache miss の場合は通常の `programs/<code>/build.sh` 経路に戻り、成功後に新しい cache を保存します。
Expand Down
10 changes: 6 additions & 4 deletions docs/cx/BENCHKIT_SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -378,13 +378,14 @@ tag や公式 version が付与されない段階でも性能評価・推定・
Benchkit が最低限追跡すべき項目は以下である。

- 最上位アプリケーションの source repository
- 最上位アプリケーションの branch
- 最上位アプリケーションの branch または tag/ref 名
- 最上位アプリケーションの commit hash
- ref の種類と、実際に解決された commit hash
- 必要に応じて source URL
- 補助情報としての version や tag

例:
最上位アプリケーションが GitHub 上の `qws` であれば、`main` ブランチのどの commit hash から得られた結果かを追跡できなければならない。
最上位アプリケーションが GitHub 上の `qws` であれば、`main` ブランチや指定 tag/ref のどの commit hash から得られた結果かを追跡できなければならない。

また、ポータル上では `/results/usage` を通じて、各 app / system の最新 result を基準に source tracking の current-state を確認できることが望ましい。
ここでは `source_status`、`source_type`、`source_reference`、不足している source field を軽く見られる形が自然である。
Expand All @@ -405,13 +406,14 @@ even before tags or official versions are created, while benchmarking, estimatio
At minimum, Benchkit should track:

- the source repository of the top-level application
- the branch of the top-level application
- the branch or tag/ref name of the top-level application
- the commit hash of the top-level application
- the ref kind and the resolved commit hash actually used
- the source URL when needed
- version or tag information as supporting metadata

Example:
If the top-level application is `qws` on GitHub, Benchkit must be able to trace which commit hash on the `main` branch produced the result.
If the top-level application is `qws` on GitHub, Benchkit must be able to trace which commit hash on the `main` branch or specified tag/ref produced the result.

It is also desirable for the portal to expose source-tracking current state through `/results/usage`, based on the latest result for each application/system pair.
A lightweight view of `source_status`, `source_type`, `source_reference`, and missing source fields is a natural form for that visibility.
Expand Down
4 changes: 2 additions & 2 deletions docs/cx/CX_FRAMEWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ It applies to:
- `拠点接続`:
実システム、runner、Jacamar CI、scheduler、module 環境、共有ストレージ、実行アカウント、結果回収条件などを、CX 基盤や Benchkit から接続可能な形で定義・管理すること。
- `ソース出自情報`:
実行結果や推定結果が、どのソースコード状態に由来するかを追跡するための情報。少なくとも最上位アプリケーションの source repository、branchcommit hash を含みうる。
実行結果や推定結果が、どのソースコード状態に由来するかを追跡するための情報。少なくとも最上位アプリケーションの source repository、branch または tag/ref 名、実際に解決された commit hash を含みうる。
- `最上位アプリケーション`:
CX 基盤が直接対象とするベンチマーク対象アプリケーション本体。依存パッケージや依存ライブラリではなく、性能結果や最適化結果を第一義的に結び付ける主体を指す。

Expand All @@ -134,7 +134,7 @@ In this specification and its lower-level specifications, the following terms ar
- `site integration`:
The definition and management of real systems, runners, Jacamar CI, scheduler behavior, module environments, shared storage, execution accounts, and result-collection conditions in a form that can be integrated with the CX Platform and Benchkit.
- `source provenance`:
Information used to trace which source-code state produced a benchmark or estimation result. At minimum, it may include the source repository, branch, and commit hash of the top-level application.
Information used to trace which source-code state produced a benchmark or estimation result. At minimum, it may include the source repository, branch or tag/ref name, and resolved commit hash of the top-level application.
- `top-level application`:
The benchmark target application directly handled by the CX Platform. It does not mean dependency packages or libraries, but the primary subject to which performance and optimization results are attached.

Expand Down
9 changes: 6 additions & 3 deletions docs/guides/add-app.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,9 @@ Benchkit では、実行条件とシステム運用設定を明確に分けま

Benchkit では、まず **top-level application の source provenance** を追えることを優先します。
具体的には、Git 管理のアプリであれば `repo_url`、`branch`、`commit_hash` を `source_info` として入れられる形が望ましいです。
通常のアプリ build は、対象 repo の対象 branch の最新 commit を使います。
`branch` は表示用の ref 名であり、branch だけでなく tag 名が入る場合もあります。
新しい result では、追加で `ref_name`、`ref_kind`、`resolved_commit` も記録します。
通常のアプリ build は、対象 repo の対象 ref の最新 commit を使います。
再現実行や監査で commit を固定したい場合だけ、`bk_fetch_source` の第4引数に expected commit を渡して、意図した commit を build してください。
tar archive を使う場合も、必要に応じて第4引数に expected SHA-256 を渡せます。

Expand Down Expand Up @@ -163,7 +165,8 @@ cache miss の場合は通常どおり `build.sh` が実行され、`artifacts/`
cache hit の場合は保存済みの `artifacts/` と build provenance が復元され、`build.sh` は実行されません。

cache hit は、少なくとも現在の app build input hash と source provenance が一致するときだけ許可されます。
Git source では cache 内の `repo_url` / `branch` / `commit_hash` に対し、現在の branch commit を `git ls-remote` で再解決します。
Git source では cache 内の `repo_url` / `ref_name` / `resolved_commit` に対し、現在の ref commit を `git ls-remote` で再解決します。
新 metadata がない既存 cache entry は miss になり、通常の build 後に新しい cache として保存されます。
file/archive source では SHA-256 を再計算します。
container image SHA-256 が source_info に入っている場合は container image も再検証します。
container ではない host build でも、common の `make` / `cmake` / `ninja` wrapper を通る場合は build tool 実行直前の build environment fingerprint で照合します。
Expand Down Expand Up @@ -599,7 +602,7 @@ git push origin add-<code>
- ビルド・実行ファイルの衝突は基本的に発生しない

### Git リポジトリの取り扱い
- ソース取得は原則 `scripts/bk_functions.sh` の `bk_fetch_source <source> <dest_dir> [branch] [expected_commit_or_sha256]` を使う
- ソース取得は原則 `scripts/bk_functions.sh` の `bk_fetch_source <source> <dest_dir> [branch_or_tag] [expected_commit_or_sha256]` を使う
- `bk_fetch_source` は Git URL または tar archive を取得・展開し、`results/source_info.env` に source provenance を書く
- Git source では expected commit を指定すると、その commit に checkout して一致しなければ失敗する
- tar archive では expected SHA-256 を指定すると、一致しなければ失敗し、`source_info` に `sha256sum` も記録する
Expand Down
7 changes: 6 additions & 1 deletion result_server/tests/test_result_metadata_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ def test_extract_result_index_record_from_benchmark_result():
"repo_url": "https://example.org/repo.git",
"branch": "develop",
"commit_hash": "abcdef123456",
"ref_name": "v1.0",
"ref_kind": "tag",
"resolved_commit": "0123456789abcdef0123456789abcdef01234567",
},
}

Expand All @@ -46,10 +49,12 @@ def test_extract_result_index_record_from_benchmark_result():
assert record["exp"] == "case0"
assert record["pipeline_id"] == "3152"
assert record["source_type"] == "git"
assert record["source_ref"] == "abcdef123456"
assert record["source_ref"] == "0123456789abcdef0123456789abcdef01234567"
metadata = json.loads(record["metadata_json"])
assert metadata["fom"] == 42.5
assert metadata["source_info"]["branch"] == "develop"
assert metadata["source_info"]["ref_kind"] == "tag"
assert metadata["source_info"]["resolved_commit"] == "0123456789abcdef0123456789abcdef01234567"


def test_index_result_metadata_upserts_rows(tmp_path):
Expand Down
28 changes: 27 additions & 1 deletion result_server/tests/test_result_quality_rollup.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,35 @@ def test_build_result_quality_rollup(tmp_path):
assert qws["source_status"] == "not tracked"
assert qws["source_type"] == "git"
assert qws["source_reference"] == "main"
assert qws["source_missing_fields"] == ["commit_hash"]
assert qws["source_missing_fields"] == ["resolved_commit"]
assert qws["breakdown_present"] is False
assert qws["estimation_ready"] is False
assert qws["rich"] is False
assert qws["next_action"] == "fill the missing top-level source_info fields"
assert "complete source_info fields" in qws["validator_candidates"]


def test_build_result_quality_rollup_uses_git_ref_name(tmp_path):
_write_result(
tmp_path / "result_20260401_010101_aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.json",
{
"code": "salmon",
"system": "Fugaku",
"FOM": 1.0,
"source_info": {
"source_type": "git",
"repo_url": "https://example.com/repo.git",
"branch": "FugakuNEXT-v4",
"commit_hash": "abcdef1234567890",
"ref_name": "v1.0",
"ref_kind": "tag",
"resolved_commit": "0123456789abcdef",
},
},
)

rollup = build_result_quality_rollup(str(tmp_path))

row = rollup["rows"][0]
assert row["source_tracked"] is True
assert row["source_reference"] == "v1.0@0123456"
2 changes: 1 addition & 1 deletion result_server/tests/test_results_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ def test_existing_columns_unchanged(self, flask_app, tmp_dir):
expected_columns = [
{"label": "Timestamp", "key": "timestamp", "tooltip": "Date and time when benchmark execution completed and results were automatically submitted to server", "tooltip_class": "tooltip-left"},
{"label": "CODE", "key": "code"},
{"label": "Branch/Hash", "key": "source_hash", "tooltip": "Source code branch name and short commit hash (git) or short sha256/md5 hash (file archive)"},
{"label": "Ref/Hash", "key": "source_hash", "tooltip": "Source code ref name and short resolved commit hash (git) or short sha256/md5 hash (file archive)"},
{"label": "Exp", "key": "exp", "tooltip": "Experimental conditions (filtered by CODE)"},
{"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value with its unit when available"},
{"label": "FOM version", "key": "fom_version", "tooltip": "Version identifier for the FOM measurement section - helps identify which code region was measured when users modify the timing boundaries"},
Expand Down
3 changes: 3 additions & 0 deletions result_server/tests/test_script_source_info_security.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,8 @@ def test_bk_fetch_source_writes_encoded_source_info_values():
assert "BK_FILE_PATH_B64" in bk_functions
assert "BK_SHA256SUM_B64" in bk_functions
assert "BK_CONTAINER_IMAGE_SHA256SUM_B64" in bk_functions
assert "BK_SOURCE_REF_NAME_B64" in bk_functions
assert "BK_SOURCE_REF_KIND_B64" in bk_functions
assert "BK_SOURCE_RESOLVED_COMMIT_B64" in bk_functions
assert 'export BK_REPO_URL="$BK_REPO_URL"' not in bk_functions
assert 'export BK_FILE_PATH="$BK_FILE_PATH"' not in bk_functions
21 changes: 20 additions & 1 deletion result_server/tests/test_source_info_properties.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))

from test_support import install_portal_test_stubs
from utils.result_table_rows import _build_source_link
from utils.result_table_rows import _build_source_link, _format_source_hash

install_portal_test_stubs()

Expand Down Expand Up @@ -69,13 +69,18 @@ def test_git_source_info_structure_valid_with_commit_hash(self, commit_hash):
"repo_url": "https://github.com/example/repo.git",
"branch": "main",
"commit_hash": commit_hash,
"ref_name": "main",
"ref_kind": "branch",
"resolved_commit": commit_hash,
}

assert source_info["source_type"] == "git"
assert "repo_url" in source_info
assert "branch" in source_info
assert "commit_hash" in source_info
assert source_info["ref_kind"] in {"branch", "tag", "commit", "unknown"}
assert COMMIT_HASH_PATTERN.match(source_info["commit_hash"]) is not None
assert COMMIT_HASH_PATTERN.match(source_info["resolved_commit"]) is not None

@given(md5sum=md5sum_strategy, sha256sum=sha256sum_strategy)
@settings(max_examples=100)
Expand Down Expand Up @@ -108,6 +113,20 @@ def test_git_source_link_allows_http_urls_only():
assert link["title"] == "https://github.com/example/repo.git"


def test_git_source_hash_prefers_resolved_ref_fields():
source_info = {
"source_type": "git",
"repo_url": "https://github.com/example/repo.git",
"branch": "main",
"commit_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"ref_name": "v1.0",
"ref_kind": "tag",
"resolved_commit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
}

assert _format_source_hash(source_info) == "v1.0@bbbbbbb"


def test_git_source_link_rejects_javascript_urls():
source_info = {
"source_type": "git",
Expand Down
14 changes: 13 additions & 1 deletion result_server/utils/result_metadata_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,16 @@ def _nested_text(data: dict[str, Any], *keys: str) -> str:
def _source_ref(source_info: dict[str, Any]) -> str:
if not isinstance(source_info, dict):
return ""
for key in ("commit_hash", "sha256sum", "md5sum", "branch", "file_path", "repo_url"):
for key in (
"resolved_commit",
"commit_hash",
"sha256sum",
"md5sum",
"ref_name",
"branch",
"file_path",
"repo_url",
):
value = _as_text(source_info.get(key))
if value:
return value
Expand All @@ -59,6 +68,9 @@ def _metadata_for_result(payload: dict[str, Any]) -> dict[str, Any]:
"repo_url",
"branch",
"commit_hash",
"ref_name",
"ref_kind",
"resolved_commit",
"file_path",
"md5sum",
"sha256sum",
Expand Down
15 changes: 11 additions & 4 deletions result_server/utils/result_quality_rollup.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,15 @@ def _summarize_source_info(data: Dict[str, Any]) -> Dict[str, Any]:

source_type = source_info.get("source_type") or "unknown"
if source_type == "git":
required_fields = ("repo_url", "branch", "commit_hash")
branch = source_info.get("branch") or ""
commit_hash = source_info.get("commit_hash") or ""
missing_fields = []
if not source_info.get("repo_url"):
missing_fields.append("repo_url")
if not (source_info.get("ref_name") or source_info.get("branch")):
missing_fields.append("ref_name")
if not (source_info.get("resolved_commit") or source_info.get("commit_hash")):
missing_fields.append("resolved_commit")
branch = source_info.get("ref_name") or source_info.get("branch") or ""
commit_hash = source_info.get("resolved_commit") or source_info.get("commit_hash") or ""
short_hash = commit_hash[:7] if commit_hash else ""
reference = (
f"{branch}@{short_hash}"
Expand All @@ -58,7 +64,8 @@ def _summarize_source_info(data: Dict[str, Any]) -> Dict[str, Any]:
required_fields = ()
reference = "unknown source type"

missing_fields = [field for field in required_fields if not source_info.get(field)]
if source_type != "git":
missing_fields = [field for field in required_fields if not source_info.get(field)]
if source_type == "file" and not source_info.get("sha256sum") and not source_info.get("md5sum"):
missing_fields.append("sha256sum")
if source_type == "unknown":
Expand Down
8 changes: 7 additions & 1 deletion result_server/utils/result_records.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,13 @@ def summarize_result_quality(data):
if has_source_info:
source_type = source_info.get("source_type")
if source_type == "git":
source_missing_fields = [key for key in ("repo_url", "branch", "commit_hash") if not source_info.get(key)]
source_missing_fields = []
if not source_info.get("repo_url"):
source_missing_fields.append("repo_url")
if not (source_info.get("ref_name") or source_info.get("branch")):
source_missing_fields.append("ref_name")
if not (source_info.get("resolved_commit") or source_info.get("commit_hash")):
source_missing_fields.append("resolved_commit")
source_info_complete = not source_missing_fields
elif source_type == "file":
source_missing_fields = [key for key in ("file_path",) if not source_info.get(key)]
Expand Down
4 changes: 2 additions & 2 deletions result_server/utils/result_table_rows.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,8 @@ def _format_source_hash(source_info):

source_type = source_info.get("source_type")
if source_type == "git":
branch = source_info.get("branch", "")
commit_hash = source_info.get("commit_hash", "")
branch = source_info.get("ref_name") or source_info.get("branch", "")
commit_hash = source_info.get("resolved_commit") or source_info.get("commit_hash", "")
short_hash = commit_hash[:7] if commit_hash else ""
return f"{branch}@{short_hash}" if branch and short_hash else short_hash or branch or "-"

Expand Down
Loading
Loading