diff --git a/docs/ci.md b/docs/ci.md index 00417a2..9eaa425 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -264,8 +264,9 @@ cache is safe to use by comparing the current build input hash and the cached `BK_BUILD_CACHE_DIR` が設定されていればそれを cache root として使います。未設定の場合、custom runner が `CUSTOM_DIR` を渡していれば `$CUSTOM_DIR/build_cache/$CUSTOM_RUNNER_PROJECT_SLUG` を使います。どちらもなければ build cache は無効です。 復元してよいかは script 側が現在の build input hash と cache 内の `results/source_info.env` を比較して判断します。 -Git sources are rechecked with `git ls-remote`; file/archive sources are -rechecked by SHA-256. If a container image hash is recorded, that image hash is +Git sources are rechecked with `git ls-remote` using the recorded `ref_name` +and `resolved_commit`; file/archive sources are rechecked by SHA-256. If a +container image hash is recorded, that image hash is also verified. Host builds that go through the common `make` / `cmake` / `ninja` wrappers are matched by a build-environment fingerprint collected just before the build tool runs. The fingerprint includes loaded modules, selected @@ -276,7 +277,7 @@ does not declare a GitLab `cache:` stanza; the cache directory must be a site-managed persistent path such as the custom runner's `CUSTOM_DIR`, not a per-job cleanup directory. -Git source は `git ls-remote` で再確認し、file/archive source は SHA-256 を再計算します。 +Git source は記録済みの `ref_name` と `resolved_commit` を使って `git ls-remote` で再確認し、file/archive source は SHA-256 を再計算します。 container image hash が記録されている場合は image hash も確認します。 common の `make` / `cmake` / `ninja` wrapper を通る host build は、build tool 実行直前に収集した build environment fingerprint で照合します。この fingerprint には loaded modules、選択された build 環境変数、tool の real path、version、binary SHA-256 hash が含まれます。 cache miss の場合は通常の `programs//build.sh` 経路に戻り、成功後に新しい cache を保存します。 diff --git a/docs/cx/BENCHKIT_SPEC.md b/docs/cx/BENCHKIT_SPEC.md index 82930b8..d2d32eb 100644 --- a/docs/cx/BENCHKIT_SPEC.md +++ b/docs/cx/BENCHKIT_SPEC.md @@ -378,13 +378,14 @@ tag や公式 version が付与されない段階でも性能評価・推定・ Benchkit が最低限追跡すべき項目は以下である。 - 最上位アプリケーションの source repository -- 最上位アプリケーションの branch +- 最上位アプリケーションの branch または tag/ref 名 - 最上位アプリケーションの commit hash +- ref の種類と、実際に解決された commit hash - 必要に応じて source URL - 補助情報としての version や tag 例: -最上位アプリケーションが GitHub 上の `qws` であれば、`main` ブランチのどの commit hash から得られた結果かを追跡できなければならない。 +最上位アプリケーションが GitHub 上の `qws` であれば、`main` ブランチや指定 tag/ref のどの commit hash から得られた結果かを追跡できなければならない。 また、ポータル上では `/results/usage` を通じて、各 app / system の最新 result を基準に source tracking の current-state を確認できることが望ましい。 ここでは `source_status`、`source_type`、`source_reference`、不足している source field を軽く見られる形が自然である。 @@ -405,13 +406,14 @@ even before tags or official versions are created, while benchmarking, estimatio At minimum, Benchkit should track: - the source repository of the top-level application -- the branch of the top-level application +- the branch or tag/ref name of the top-level application - the commit hash of the top-level application +- the ref kind and the resolved commit hash actually used - the source URL when needed - version or tag information as supporting metadata Example: -If the top-level application is `qws` on GitHub, Benchkit must be able to trace which commit hash on the `main` branch produced the result. +If the top-level application is `qws` on GitHub, Benchkit must be able to trace which commit hash on the `main` branch or specified tag/ref produced the result. It is also desirable for the portal to expose source-tracking current state through `/results/usage`, based on the latest result for each application/system pair. A lightweight view of `source_status`, `source_type`, `source_reference`, and missing source fields is a natural form for that visibility. diff --git a/docs/cx/CX_FRAMEWORK.md b/docs/cx/CX_FRAMEWORK.md index bead254..258a26a 100644 --- a/docs/cx/CX_FRAMEWORK.md +++ b/docs/cx/CX_FRAMEWORK.md @@ -115,7 +115,7 @@ It applies to: - `拠点接続`: 実システム、runner、Jacamar CI、scheduler、module 環境、共有ストレージ、実行アカウント、結果回収条件などを、CX 基盤や Benchkit から接続可能な形で定義・管理すること。 - `ソース出自情報`: - 実行結果や推定結果が、どのソースコード状態に由来するかを追跡するための情報。少なくとも最上位アプリケーションの source repository、branch、commit hash を含みうる。 + 実行結果や推定結果が、どのソースコード状態に由来するかを追跡するための情報。少なくとも最上位アプリケーションの source repository、branch または tag/ref 名、実際に解決された commit hash を含みうる。 - `最上位アプリケーション`: CX 基盤が直接対象とするベンチマーク対象アプリケーション本体。依存パッケージや依存ライブラリではなく、性能結果や最適化結果を第一義的に結び付ける主体を指す。 @@ -134,7 +134,7 @@ In this specification and its lower-level specifications, the following terms ar - `site integration`: The definition and management of real systems, runners, Jacamar CI, scheduler behavior, module environments, shared storage, execution accounts, and result-collection conditions in a form that can be integrated with the CX Platform and Benchkit. - `source provenance`: - Information used to trace which source-code state produced a benchmark or estimation result. At minimum, it may include the source repository, branch, and commit hash of the top-level application. + Information used to trace which source-code state produced a benchmark or estimation result. At minimum, it may include the source repository, branch or tag/ref name, and resolved commit hash of the top-level application. - `top-level application`: The benchmark target application directly handled by the CX Platform. It does not mean dependency packages or libraries, but the primary subject to which performance and optimization results are attached. diff --git a/docs/guides/add-app.md b/docs/guides/add-app.md index 71f99c2..5d0d62c 100644 --- a/docs/guides/add-app.md +++ b/docs/guides/add-app.md @@ -116,7 +116,9 @@ Benchkit では、実行条件とシステム運用設定を明確に分けま Benchkit では、まず **top-level application の source provenance** を追えることを優先します。 具体的には、Git 管理のアプリであれば `repo_url`、`branch`、`commit_hash` を `source_info` として入れられる形が望ましいです。 -通常のアプリ build は、対象 repo の対象 branch の最新 commit を使います。 +`branch` は表示用の ref 名であり、branch だけでなく tag 名が入る場合もあります。 +新しい result では、追加で `ref_name`、`ref_kind`、`resolved_commit` も記録します。 +通常のアプリ build は、対象 repo の対象 ref の最新 commit を使います。 再現実行や監査で commit を固定したい場合だけ、`bk_fetch_source` の第4引数に expected commit を渡して、意図した commit を build してください。 tar archive を使う場合も、必要に応じて第4引数に expected SHA-256 を渡せます。 @@ -163,7 +165,8 @@ cache miss の場合は通常どおり `build.sh` が実行され、`artifacts/` cache hit の場合は保存済みの `artifacts/` と build provenance が復元され、`build.sh` は実行されません。 cache hit は、少なくとも現在の app build input hash と source provenance が一致するときだけ許可されます。 -Git source では cache 内の `repo_url` / `branch` / `commit_hash` に対し、現在の branch commit を `git ls-remote` で再解決します。 +Git source では cache 内の `repo_url` / `ref_name` / `resolved_commit` に対し、現在の ref commit を `git ls-remote` で再解決します。 +新 metadata がない既存 cache entry は miss になり、通常の build 後に新しい cache として保存されます。 file/archive source では SHA-256 を再計算します。 container image SHA-256 が source_info に入っている場合は container image も再検証します。 container ではない host build でも、common の `make` / `cmake` / `ninja` wrapper を通る場合は build tool 実行直前の build environment fingerprint で照合します。 @@ -599,7 +602,7 @@ git push origin add- - ビルド・実行ファイルの衝突は基本的に発生しない ### Git リポジトリの取り扱い -- ソース取得は原則 `scripts/bk_functions.sh` の `bk_fetch_source [branch] [expected_commit_or_sha256]` を使う +- ソース取得は原則 `scripts/bk_functions.sh` の `bk_fetch_source [branch_or_tag] [expected_commit_or_sha256]` を使う - `bk_fetch_source` は Git URL または tar archive を取得・展開し、`results/source_info.env` に source provenance を書く - Git source では expected commit を指定すると、その commit に checkout して一致しなければ失敗する - tar archive では expected SHA-256 を指定すると、一致しなければ失敗し、`source_info` に `sha256sum` も記録する diff --git a/result_server/tests/test_result_metadata_index.py b/result_server/tests/test_result_metadata_index.py index abe5306..dca7dd0 100644 --- a/result_server/tests/test_result_metadata_index.py +++ b/result_server/tests/test_result_metadata_index.py @@ -30,6 +30,9 @@ def test_extract_result_index_record_from_benchmark_result(): "repo_url": "https://example.org/repo.git", "branch": "develop", "commit_hash": "abcdef123456", + "ref_name": "v1.0", + "ref_kind": "tag", + "resolved_commit": "0123456789abcdef0123456789abcdef01234567", }, } @@ -46,10 +49,12 @@ def test_extract_result_index_record_from_benchmark_result(): assert record["exp"] == "case0" assert record["pipeline_id"] == "3152" assert record["source_type"] == "git" - assert record["source_ref"] == "abcdef123456" + assert record["source_ref"] == "0123456789abcdef0123456789abcdef01234567" metadata = json.loads(record["metadata_json"]) assert metadata["fom"] == 42.5 assert metadata["source_info"]["branch"] == "develop" + assert metadata["source_info"]["ref_kind"] == "tag" + assert metadata["source_info"]["resolved_commit"] == "0123456789abcdef0123456789abcdef01234567" def test_index_result_metadata_upserts_rows(tmp_path): diff --git a/result_server/tests/test_result_quality_rollup.py b/result_server/tests/test_result_quality_rollup.py index aa23520..6b55df2 100644 --- a/result_server/tests/test_result_quality_rollup.py +++ b/result_server/tests/test_result_quality_rollup.py @@ -86,9 +86,35 @@ def test_build_result_quality_rollup(tmp_path): assert qws["source_status"] == "not tracked" assert qws["source_type"] == "git" assert qws["source_reference"] == "main" - assert qws["source_missing_fields"] == ["commit_hash"] + assert qws["source_missing_fields"] == ["resolved_commit"] assert qws["breakdown_present"] is False assert qws["estimation_ready"] is False assert qws["rich"] is False assert qws["next_action"] == "fill the missing top-level source_info fields" assert "complete source_info fields" in qws["validator_candidates"] + + +def test_build_result_quality_rollup_uses_git_ref_name(tmp_path): + _write_result( + tmp_path / "result_20260401_010101_aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.json", + { + "code": "salmon", + "system": "Fugaku", + "FOM": 1.0, + "source_info": { + "source_type": "git", + "repo_url": "https://example.com/repo.git", + "branch": "FugakuNEXT-v4", + "commit_hash": "abcdef1234567890", + "ref_name": "v1.0", + "ref_kind": "tag", + "resolved_commit": "0123456789abcdef", + }, + }, + ) + + rollup = build_result_quality_rollup(str(tmp_path)) + + row = rollup["rows"][0] + assert row["source_tracked"] is True + assert row["source_reference"] == "v1.0@0123456" diff --git a/result_server/tests/test_results_loader.py b/result_server/tests/test_results_loader.py index c16bccc..22b9fd9 100644 --- a/result_server/tests/test_results_loader.py +++ b/result_server/tests/test_results_loader.py @@ -270,7 +270,7 @@ def test_existing_columns_unchanged(self, flask_app, tmp_dir): expected_columns = [ {"label": "Timestamp", "key": "timestamp", "tooltip": "Date and time when benchmark execution completed and results were automatically submitted to server", "tooltip_class": "tooltip-left"}, {"label": "CODE", "key": "code"}, - {"label": "Branch/Hash", "key": "source_hash", "tooltip": "Source code branch name and short commit hash (git) or short sha256/md5 hash (file archive)"}, + {"label": "Ref/Hash", "key": "source_hash", "tooltip": "Source code ref name and short resolved commit hash (git) or short sha256/md5 hash (file archive)"}, {"label": "Exp", "key": "exp", "tooltip": "Experimental conditions (filtered by CODE)"}, {"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value with its unit when available"}, {"label": "FOM version", "key": "fom_version", "tooltip": "Version identifier for the FOM measurement section - helps identify which code region was measured when users modify the timing boundaries"}, diff --git a/result_server/tests/test_script_source_info_security.py b/result_server/tests/test_script_source_info_security.py index 773d90b..f878dfb 100644 --- a/result_server/tests/test_script_source_info_security.py +++ b/result_server/tests/test_script_source_info_security.py @@ -31,5 +31,8 @@ def test_bk_fetch_source_writes_encoded_source_info_values(): assert "BK_FILE_PATH_B64" in bk_functions assert "BK_SHA256SUM_B64" in bk_functions assert "BK_CONTAINER_IMAGE_SHA256SUM_B64" in bk_functions + assert "BK_SOURCE_REF_NAME_B64" in bk_functions + assert "BK_SOURCE_REF_KIND_B64" in bk_functions + assert "BK_SOURCE_RESOLVED_COMMIT_B64" in bk_functions assert 'export BK_REPO_URL="$BK_REPO_URL"' not in bk_functions assert 'export BK_FILE_PATH="$BK_FILE_PATH"' not in bk_functions diff --git a/result_server/tests/test_source_info_properties.py b/result_server/tests/test_source_info_properties.py index dfda194..e37f387 100644 --- a/result_server/tests/test_source_info_properties.py +++ b/result_server/tests/test_source_info_properties.py @@ -11,7 +11,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) from test_support import install_portal_test_stubs -from utils.result_table_rows import _build_source_link +from utils.result_table_rows import _build_source_link, _format_source_hash install_portal_test_stubs() @@ -69,13 +69,18 @@ def test_git_source_info_structure_valid_with_commit_hash(self, commit_hash): "repo_url": "https://github.com/example/repo.git", "branch": "main", "commit_hash": commit_hash, + "ref_name": "main", + "ref_kind": "branch", + "resolved_commit": commit_hash, } assert source_info["source_type"] == "git" assert "repo_url" in source_info assert "branch" in source_info assert "commit_hash" in source_info + assert source_info["ref_kind"] in {"branch", "tag", "commit", "unknown"} assert COMMIT_HASH_PATTERN.match(source_info["commit_hash"]) is not None + assert COMMIT_HASH_PATTERN.match(source_info["resolved_commit"]) is not None @given(md5sum=md5sum_strategy, sha256sum=sha256sum_strategy) @settings(max_examples=100) @@ -108,6 +113,20 @@ def test_git_source_link_allows_http_urls_only(): assert link["title"] == "https://github.com/example/repo.git" +def test_git_source_hash_prefers_resolved_ref_fields(): + source_info = { + "source_type": "git", + "repo_url": "https://github.com/example/repo.git", + "branch": "main", + "commit_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "ref_name": "v1.0", + "ref_kind": "tag", + "resolved_commit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + } + + assert _format_source_hash(source_info) == "v1.0@bbbbbbb" + + def test_git_source_link_rejects_javascript_urls(): source_info = { "source_type": "git", diff --git a/result_server/utils/result_metadata_index.py b/result_server/utils/result_metadata_index.py index f807651..e851d66 100644 --- a/result_server/utils/result_metadata_index.py +++ b/result_server/utils/result_metadata_index.py @@ -35,7 +35,16 @@ def _nested_text(data: dict[str, Any], *keys: str) -> str: def _source_ref(source_info: dict[str, Any]) -> str: if not isinstance(source_info, dict): return "" - for key in ("commit_hash", "sha256sum", "md5sum", "branch", "file_path", "repo_url"): + for key in ( + "resolved_commit", + "commit_hash", + "sha256sum", + "md5sum", + "ref_name", + "branch", + "file_path", + "repo_url", + ): value = _as_text(source_info.get(key)) if value: return value @@ -59,6 +68,9 @@ def _metadata_for_result(payload: dict[str, Any]) -> dict[str, Any]: "repo_url", "branch", "commit_hash", + "ref_name", + "ref_kind", + "resolved_commit", "file_path", "md5sum", "sha256sum", diff --git a/result_server/utils/result_quality_rollup.py b/result_server/utils/result_quality_rollup.py index 2459b65..0b986a5 100644 --- a/result_server/utils/result_quality_rollup.py +++ b/result_server/utils/result_quality_rollup.py @@ -31,9 +31,15 @@ def _summarize_source_info(data: Dict[str, Any]) -> Dict[str, Any]: source_type = source_info.get("source_type") or "unknown" if source_type == "git": - required_fields = ("repo_url", "branch", "commit_hash") - branch = source_info.get("branch") or "" - commit_hash = source_info.get("commit_hash") or "" + missing_fields = [] + if not source_info.get("repo_url"): + missing_fields.append("repo_url") + if not (source_info.get("ref_name") or source_info.get("branch")): + missing_fields.append("ref_name") + if not (source_info.get("resolved_commit") or source_info.get("commit_hash")): + missing_fields.append("resolved_commit") + branch = source_info.get("ref_name") or source_info.get("branch") or "" + commit_hash = source_info.get("resolved_commit") or source_info.get("commit_hash") or "" short_hash = commit_hash[:7] if commit_hash else "" reference = ( f"{branch}@{short_hash}" @@ -58,7 +64,8 @@ def _summarize_source_info(data: Dict[str, Any]) -> Dict[str, Any]: required_fields = () reference = "unknown source type" - missing_fields = [field for field in required_fields if not source_info.get(field)] + if source_type != "git": + missing_fields = [field for field in required_fields if not source_info.get(field)] if source_type == "file" and not source_info.get("sha256sum") and not source_info.get("md5sum"): missing_fields.append("sha256sum") if source_type == "unknown": diff --git a/result_server/utils/result_records.py b/result_server/utils/result_records.py index 790a494..fa05a88 100644 --- a/result_server/utils/result_records.py +++ b/result_server/utils/result_records.py @@ -79,7 +79,13 @@ def summarize_result_quality(data): if has_source_info: source_type = source_info.get("source_type") if source_type == "git": - source_missing_fields = [key for key in ("repo_url", "branch", "commit_hash") if not source_info.get(key)] + source_missing_fields = [] + if not source_info.get("repo_url"): + source_missing_fields.append("repo_url") + if not (source_info.get("ref_name") or source_info.get("branch")): + source_missing_fields.append("ref_name") + if not (source_info.get("resolved_commit") or source_info.get("commit_hash")): + source_missing_fields.append("resolved_commit") source_info_complete = not source_missing_fields elif source_type == "file": source_missing_fields = [key for key in ("file_path",) if not source_info.get(key)] diff --git a/result_server/utils/result_table_rows.py b/result_server/utils/result_table_rows.py index f4ddc99..6e3f0dc 100644 --- a/result_server/utils/result_table_rows.py +++ b/result_server/utils/result_table_rows.py @@ -123,8 +123,8 @@ def _format_source_hash(source_info): source_type = source_info.get("source_type") if source_type == "git": - branch = source_info.get("branch", "") - commit_hash = source_info.get("commit_hash", "") + branch = source_info.get("ref_name") or source_info.get("branch", "") + commit_hash = source_info.get("resolved_commit") or source_info.get("commit_hash", "") short_hash = commit_hash[:7] if commit_hash else "" return f"{branch}@{short_hash}" if branch and short_hash else short_hash or branch or "-" diff --git a/result_server/utils/results_loader.py b/result_server/utils/results_loader.py index 9775f2a..05f3918 100644 --- a/result_server/utils/results_loader.py +++ b/result_server/utils/results_loader.py @@ -19,7 +19,7 @@ RESULT_TABLE_COLUMNS = [ {"label": "Timestamp", "key": "timestamp", "tooltip": "Date and time when benchmark execution completed and results were automatically submitted to server", "tooltip_class": "tooltip-left"}, {"label": "CODE", "key": "code"}, - {"label": "Branch/Hash", "key": "source_hash", "tooltip": "Source code branch name and short commit hash (git) or short sha256/md5 hash (file archive)"}, + {"label": "Ref/Hash", "key": "source_hash", "tooltip": "Source code ref name and short resolved commit hash (git) or short sha256/md5 hash (file archive)"}, {"label": "Exp", "key": "exp", "tooltip": "Experimental conditions (filtered by CODE)"}, {"label": "FOM", "key": "fom", "tooltip": "Figure of Merit - Benchmark performance metric value with its unit when available"}, {"label": "FOM version", "key": "fom_version", "tooltip": "Version identifier for the FOM measurement section - helps identify which code region was measured when users modify the timing boundaries"}, diff --git a/scripts/bk_functions.sh b/scripts/bk_functions.sh index d7db074..cbeaa74 100644 --- a/scripts/bk_functions.sh +++ b/scripts/bk_functions.sh @@ -1859,6 +1859,9 @@ bk_write_source_info_env() { _bk_sha256sum="${7:-}" _bk_container_path="${8:-${BK_SOURCE_CONTAINER_PATH:-}}" _bk_container_sha256sum="${9:-${BK_SOURCE_CONTAINER_SHA256:-}}" + _bk_ref_name="${10:-}" + _bk_ref_kind="${11:-}" + _bk_resolved_commit="${12:-}" if ! command -v base64 >/dev/null 2>&1 && ! command -v openssl >/dev/null 2>&1; then echo "bk_write_source_info_env: neither base64 nor openssl found" >&2 @@ -1876,6 +1879,9 @@ bk_write_source_info_env() { printf 'BK_SHA256SUM_B64=%s\n' "$(bk_base64_encode_value "$_bk_sha256sum")" printf 'BK_CONTAINER_IMAGE_PATH_B64=%s\n' "$(bk_base64_encode_value "$_bk_container_path")" printf 'BK_CONTAINER_IMAGE_SHA256SUM_B64=%s\n' "$(bk_base64_encode_value "$_bk_container_sha256sum")" + printf 'BK_SOURCE_REF_NAME_B64=%s\n' "$(bk_base64_encode_value "$_bk_ref_name")" + printf 'BK_SOURCE_REF_KIND_B64=%s\n' "$(bk_base64_encode_value "$_bk_ref_kind")" + printf 'BK_SOURCE_RESOLVED_COMMIT_B64=%s\n' "$(bk_base64_encode_value "$_bk_resolved_commit")" } > results/source_info.env } @@ -1918,15 +1924,46 @@ bk_record_file_source_info() { bk_write_source_info_env "file" "" "" "" "$BK_FILE_PATH" "$BK_MD5SUM" "$BK_SHA256SUM" } +bk_git_ref_kind() { + _bk_git_dir="$1" + _bk_ref_name="${2:-}" + + case "$_bk_ref_name" in + ""|HEAD|detached) + printf '%s\n' "commit" + return 0 + ;; + esac + + if git -C "$_bk_git_dir" show-ref --verify --quiet "refs/heads/${_bk_ref_name}"; then + printf '%s\n' "branch" + return 0 + fi + if git -C "$_bk_git_dir" show-ref --verify --quiet "refs/remotes/origin/${_bk_ref_name}"; then + printf '%s\n' "branch" + return 0 + fi + if git -C "$_bk_git_dir" show-ref --verify --quiet "refs/tags/${_bk_ref_name}"; then + printf '%s\n' "tag" + return 0 + fi + if git -C "$_bk_git_dir" rev-parse --verify --quiet "${_bk_ref_name}^{commit}" >/dev/null 2>&1; then + printf '%s\n' "commit" + return 0 + fi + + printf '%s\n' "unknown" +} + # bk_fetch_source - Fetch source code and collect metadata. # # Usage: -# bk_fetch_source [branch] [expected_commit_or_sha256] +# bk_fetch_source [branch_or_tag] [expected_commit_or_sha256] # # Arguments: # $1 - source: Repository URL or archive file path # $2 - dest_dir: Destination directory name -# $3 - branch: (optional) Git branch to clone +# $3 - branch_or_tag: (optional) Git branch or tag to clone # $4 - expected commit for git, expected sha256 for archive (optional) # # Auto-detection: @@ -1936,8 +1973,11 @@ bk_record_file_source_info() { # Environment variables set: # BK_SOURCE_TYPE - "git" or "file" # BK_REPO_URL - (git) Repository URL -# BK_BRANCH - (git) Branch name +# BK_BRANCH - (git) Source ref display name # BK_COMMIT_HASH - (git) Full 40-char commit hash +# BK_SOURCE_REF_NAME - (git) Source ref name used for provenance/cache checks +# BK_SOURCE_REF_KIND - (git) branch, tag, commit, or unknown +# BK_SOURCE_RESOLVED_COMMIT - (git) Full 40-char commit resolved from the source ref # BK_FILE_PATH - (file) Absolute path to archive # BK_MD5SUM - (file) Full 32-char md5sum # BK_SHA256SUM - (file) Full 64-char sha256sum @@ -2024,9 +2064,14 @@ bk_fetch_source() { fi fi - export BK_BRANCH BK_COMMIT_HASH + BK_SOURCE_REF_NAME="${BK_BRANCH:-}" + BK_SOURCE_REF_KIND=$(bk_git_ref_kind "$_bk_dest" "$BK_SOURCE_REF_NAME") + BK_SOURCE_RESOLVED_COMMIT="$BK_COMMIT_HASH" + export BK_BRANCH BK_COMMIT_HASH BK_SOURCE_REF_NAME BK_SOURCE_REF_KIND BK_SOURCE_RESOLVED_COMMIT - bk_write_source_info_env "git" "$BK_REPO_URL" "$BK_BRANCH" "$BK_COMMIT_HASH" + bk_write_source_info_env \ + "git" "$BK_REPO_URL" "$BK_BRANCH" "$BK_COMMIT_HASH" "" "" "" "" "" \ + "$BK_SOURCE_REF_NAME" "$BK_SOURCE_REF_KIND" "$BK_SOURCE_RESOLVED_COMMIT" else # --- File archive path --- diff --git a/scripts/build_with_cache.sh b/scripts/build_with_cache.sh index eaa44d0..bd5b2cf 100755 --- a/scripts/build_with_cache.sh +++ b/scripts/build_with_cache.sh @@ -362,7 +362,7 @@ validate_cached_source() { local source_info_file="$1" local source_type local repo_url - local branch + local ref_name local cached_commit local current_commit local file_path @@ -378,10 +378,14 @@ validate_cached_source() { case "$source_type" in git) repo_url=$(env_file_value "$source_info_file" BK_REPO_URL) - branch=$(env_file_value "$source_info_file" BK_BRANCH) - cached_commit=$(env_file_value "$source_info_file" BK_COMMIT_HASH) - if ! current_commit=$(resolve_git_ref_commit "$repo_url" "$branch"); then - echo "cannot verify current git ref for ${repo_url} ${branch}" + ref_name=$(env_file_value "$source_info_file" BK_SOURCE_REF_NAME) + cached_commit=$(env_file_value "$source_info_file" BK_SOURCE_RESOLVED_COMMIT) + if [ -z "$repo_url" ] || [ -z "$ref_name" ] || [ -z "$cached_commit" ]; then + echo "cached git source metadata is incomplete" + return 1 + fi + if ! current_commit=$(resolve_git_ref_commit "$repo_url" "$ref_name"); then + echo "cannot verify current git ref for ${repo_url} ${ref_name}" return 1 fi if [ "$current_commit" != "$cached_commit" ]; then diff --git a/scripts/result.sh b/scripts/result.sh index 7dba2b7..5999ca1 100644 --- a/scripts/result.sh +++ b/scripts/result.sh @@ -132,10 +132,16 @@ build_source_info_block() { --arg repo_url "$(source_info_env_value BK_REPO_URL)" \ --arg branch "$(source_info_env_value BK_BRANCH)" \ --arg commit_hash "$(source_info_env_value BK_COMMIT_HASH)" \ + --arg ref_name "$(source_info_env_value BK_SOURCE_REF_NAME)" \ + --arg ref_kind "$(source_info_env_value BK_SOURCE_REF_KIND)" \ + --arg resolved_commit "$(source_info_env_value BK_SOURCE_RESOLVED_COMMIT)" \ --arg container_path "$(source_info_env_value BK_CONTAINER_IMAGE_PATH)" \ --arg container_sha256sum "$(source_info_env_value BK_CONTAINER_IMAGE_SHA256SUM)" \ ' {source_type: $source_type, repo_url: $repo_url, branch: $branch, commit_hash: $commit_hash} + + (if $ref_name != "" then {ref_name: $ref_name} else {} end) + + (if $ref_kind != "" then {ref_kind: $ref_kind} else {} end) + + (if $resolved_commit != "" then {resolved_commit: $resolved_commit} else {} end) + (if $container_path != "" or $container_sha256sum != "" then { container_image: { file_path: $container_path, diff --git a/scripts/tests/test_bk_fetch_source.sh b/scripts/tests/test_bk_fetch_source.sh index 43e7986..70fe81c 100644 --- a/scripts/tests/test_bk_fetch_source.sh +++ b/scripts/tests/test_bk_fetch_source.sh @@ -38,6 +38,7 @@ printf 'first\n' > "${TMP_DIR}/src/value.txt" git -C "${TMP_DIR}/src" add value.txt git -C "${TMP_DIR}/src" commit -q -m "first" commit_one=$(git -C "${TMP_DIR}/src" rev-parse HEAD) +git -C "${TMP_DIR}/src" tag -a v1.0 -m "version one" "$commit_one" printf 'second\n' > "${TMP_DIR}/src/value.txt" git -C "${TMP_DIR}/src" commit -q -am "second" git -C "${TMP_DIR}/src" clone -q --bare "${TMP_DIR}/src" "${TMP_DIR}/origin.git" @@ -51,7 +52,26 @@ bash "${REPO_DIR}/scripts/result.sh" app TestSystem native build run 123 >/dev/n jq -e --arg commit "$commit_one" ' .source_info.source_type == "git" and .source_info.branch == "main" and - .source_info.commit_hash == $commit + .source_info.commit_hash == $commit and + .source_info.ref_name == "main" and + .source_info.ref_kind == "branch" and + .source_info.resolved_commit == $commit +' results/result0.json >/dev/null +popd >/dev/null + +mkdir -p "${TMP_DIR}/git-tag-work" +pushd "${TMP_DIR}/git-tag-work" >/dev/null +write_minimal_result +bk_fetch_source "${TMP_DIR}/origin.git" checkout v1.0 +test "$(git -C checkout rev-parse HEAD)" = "$commit_one" +bash "${REPO_DIR}/scripts/result.sh" app TestSystem native build run 123 >/dev/null +jq -e --arg commit "$commit_one" ' + .source_info.source_type == "git" and + .source_info.branch == "v1.0" and + .source_info.commit_hash == $commit and + .source_info.ref_name == "v1.0" and + .source_info.ref_kind == "tag" and + .source_info.resolved_commit == $commit ' results/result0.json >/dev/null popd >/dev/null