Docs: redact the Spark credential config in catalog examples - #5468
Conversation
flyrain
left a comment
flyrain
left a comment
There was a problem hiding this comment.
LGTM. Thanks @huaxingao ! Do we also need to change the config in the regtests?
github-project-automation
Bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
This PR updates Spark configuration examples to reduce the risk of leaking Polaris OAuth2 client credentials via Spark UI/event logs by enabling explicit redaction of credential-named configs.
Changes:
- Adds
spark.redaction.regexto multiple Spark CLI examples that passspark.sql.catalog.*.credential. - Adds a short explanatory note about why the redaction config is needed (especially on older Spark versions).
- Adds the same redaction configuration to the SparkSession builder example.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| site/content/in-dev/unreleased/polaris-spark-client.md | Adds spark.redaction.regex to Spark shell and SparkSession examples plus an explanation of credential redaction. |
| site/content/in-dev/unreleased/getting-started/using-polaris/_index.md | Adds redaction config and a note to the getting-started Spark SQL example. |
| site/content/in-dev/unreleased/configuration/configuring-polaris-for-production/configuring-azure-blob-cloud-storage-specific.md | Adds redaction config and a note to the Azure Spark SQL example. |
| site/content/in-dev/unreleased/configuration/configuring-polaris-for-production/configuring-aws-s3-cloud-storage-specific.md | Adds redaction config and a note to the S3 Spark SQL example. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| --conf spark.sql.catalog.<spark-catalog-name>=org.apache.polaris.spark.SparkCatalog \ | ||
| --conf spark.sql.catalog.<spark-catalog-name>.uri=<polaris-service-uri> \ | ||
| --conf spark.sql.catalog.<spark-catalog-name>.credential='<client-id>:<client-secret>' \ | ||
| --conf spark.redaction.regex='(?i)secret|password|token|credential' \ |
| .config("spark.sql.catalog.<spark-catalog-name>.uri", <polaris-service-uri>) | ||
| .config("spark.sql.catalog.<spark-catalog-name>.token-refresh-enabled", "true") | ||
| .config("spark.sql.catalog.<spark-catalog-name>.credential", "<client-id>:<client_secret>") | ||
| .config("spark.redaction.regex", "(?i)secret|password|token|credential") |
| The `spark.redaction.regex` line redacts the `credential` secret from the Spark UI and logs, since | ||
| Spark's default redaction pattern does not cover `credential`. Newer Spark releases redact this key | ||
| by default; the line keeps it redacted on earlier versions. |
| Spark's default redaction pattern does not cover `credential`. Newer Spark releases redact this key | ||
| by default; the line keeps it redacted on earlier versions. |
d6e2bef to
fc63b62
Compare
|
@flyrain We don't need to change the config in the regtests because the regtests are an internal test harness, not examples users copy, so the redaction guidance belongs in the docs (this PR) rather than the harness. |
flyingImer
left a comment
flyingImer
left a comment
There was a problem hiding this comment.
LGTM. No blocking concerns from my side.
github-project-automation
Bot
moved this from Ready to merge
to Done
in Basic Kanban Board
The Spark examples pass the OAuth2 client credentials via spark.sql.catalog..credential. Spark surfaces its full configuration in the Spark UI, event logs, and diagnostic output, and Spark's default spark.redaction.regex does not match credential, so the secret can be written in cleartext.
Add spark.redaction.regex='(?i)secret|password|token|credential' to each Spark example, with a short note explaining it, so users copying the configuration redact the credential by default. Newer Spark releases redact this key on their own; the config keeps it redacted on earlier versions.
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)