Add support for UC secrets

[andrewnester]

Changes

Add support for Unity Catalog (UC) secrets as a first-class DAB resource type under the direct deployment engine.

Notes:

• Secret values are never persisted to deployment state — the remote plaintext is not returned by the API and is intentionally omitted from state to avoid leaking secrets
• New validation mutator (ValidateSecretValueIsVariable) that rejects plain-text value fields at bundle validation time — the value must be a variable reference (e.g. ${var.my_secret}) so secrets are never committed to databricks.yml

Fixes #3689

Also requires #5896

Example

bundle:
  name: my-bundle

variables:
  my_secret_value:
    description: "Secret value passed at deploy time"

resources:
  secrets:
    my_secret:
      catalog_name: main
      schema_name: default
      name: my_api_key
      value: ${var.my_secret_value}
      comment: "API key for external service"

Tests

Acceptance tests covering create/update/delete, update-value, direct-only enforcement, and the plain-text validation error

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

32 files changed
Suggested: @pietern
Also eligible: @janniklasrose, @denik, @shreyas-goenka, @anton-107, @lennartkats-db

/bundle/ - needs approval

21 files changed
Suggested: @pietern
Also eligible: @janniklasrose, @denik, @shreyas-goenka, @anton-107, @lennartkats-db

General files (require maintainer)

6 files changed
Based on git history:

• @pietern -- recent work in libs/testserver/, bundle/direct/dresources/, bundle/internal/validation/generated/

_{Any maintainer (@anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: 3e8298d

Run: 29344645940

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	4	4	230	1092	5:26
💚​	aws windows	4	4	232	1090	7:21
💚​	aws-ucws linux	4	4	316	1009	6:25
💚​	aws-ucws windows	4	4	318	1007	7:20
💚​	azure linux	4	4	230	1091	5:15
💚​	azure windows	4	4	232	1089	7:05
💚​	azure-ucws linux	4	4	318	1006	6:07
💚​	azure-ucws windows	4	4	320	1004	7:28
💚​	gcp linux	4	4	229	1093	5:08
💚​	gcp windows	4	4	231	1091	7:03

8 interesting tests: 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R

Top 10 slowest tests (at least 2 minutes):

duration	env	testname
6:25	azure-ucws windows	TestAccept
6:15	aws windows	TestAccept
6:10	gcp windows	TestAccept
6:09	aws-ucws windows	TestAccept
6:06	azure windows	TestAccept
2:58	azure linux	TestAccept
2:55	aws linux	TestAccept
2:53	gcp linux	TestAccept
2:48	aws-ucws linux	TestAccept
2:47	azure-ucws linux	TestAccept

[shreyas-goenka]

Should we disallow this pattern? Error if we detect this? Secrets should not be stored in plain text in YAML.