[CELEBORN-2388] Decouple inter-master Ratis TLS into its own SSL module#3767
Open
shellfish007 wants to merge 2 commits into
Open
[CELEBORN-2388] Decouple inter-master Ratis TLS into its own SSL module#3767shellfish007 wants to merge 2 commits into
shellfish007 wants to merge 2 commits into
Conversation
Master<->master Ratis TLS is currently derived from the rpc_service
module (MasterClusterInfo.scala -> HARaftServer.configureSsl). But
rpc_service is also the client-facing control cert, so Ratis and
external clients are forced to share one presented cert. That cert
must simultaneously satisfy:
- clients: signed by a CA they trust (e.g. the corp CA) + the
gateway/SNI hostname, and
- Ratis peers: gRPC hostname verification against the internal
master pod FQDNs (*.<svc>-master-svc.<ns>.svc.cluster.local).
In a cross-cluster / TLS-passthrough-gateway deployment these
requirements conflict (the client cert has no internal SANs), so HA
masters can't complete the Ratis TLS handshake (UNAVAILABLE: io
exception during leader election). Decoupling lets operators give
Ratis its own cert (internal SANs) while leaving the client-facing
cert untouched.
- TransportModuleConstants: add RATIS_MODULE = "ratis".
- MasterClusterInfo.scala: compute Ratis SSL from the ratis module
with fallback to rpc_service:
ratisSslEnabled = sslEnabled(ratis) || sslEnabled(rpc_service)
ratisSslModule = ratis if sslEnabled(ratis) else rpc_service
- HARaftServer.configureSsl: build the SSLFactory from the selected
module (same fallback).
Config keys (celeborn.ssl.ratis.enabled/.keyStore/.trustStore/...)
come from the existing celeborn.ssl.<module>.* machinery - no
enumerated module list needed.
Default behavior is unchanged: with only rpc_service SSL configured,
Ratis uses the rpc_service cert exactly as today. The ratis module
only takes effect when celeborn.ssl.ratis.enabled=true is explicitly
set.
Member
|
@shellfish007, please create new issues in JIRA which follows #1053. |
There was a problem hiding this comment.
Pull request overview
This PR introduces a dedicated SSL/TLS “ratis” transport module so inter-master Ratis (Raft) gRPC TLS can use its own keystore/certificate, while preserving the current behavior by falling back to the existing rpc_service SSL module when ratis SSL is not explicitly enabled.
Changes:
- Add
TransportModuleConstants.RATIS_MODULE = "ratis". - Update
MasterClusterInfoto enable Ratis TLS when eitherratisorrpc_serviceSSL is enabled (backward-compatible fallback). - Update
HARaftServer.configureSslto build theSSLFactoryfrom the selected SSL module (ratisif enabled, elserpc_service).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| master/src/main/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfo.scala | Adds helper logic for determining whether Ratis TLS is enabled and which SSL module to use (with rpc_service fallback). |
| master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java | Selects the SSL module used to create the Ratis SSLFactory (preferring ratis when enabled). |
| common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java | Introduces the new ratis transport module constant for per-module SSL config resolution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+453
to
+457
| // MasterClusterInfo.ratisSslModule (kept in sync as the single source of truth for the policy). | ||
| String sslModule = | ||
| conf.sslEnabled(TransportModuleConstants.RATIS_MODULE) | ||
| ? TransportModuleConstants.RATIS_MODULE | ||
| : TransportModuleConstants.RPC_SERVICE_MODULE; |
Author
|
JIRA ticket filed: https://issues.apache.org/jira/browse/CELEBORN-2388. PR title updated accordingly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a dedicated
ratisSSL module so inter-master Ratis (Raft consensus) gRPC TLS can use its own certificate/keystore, independent of the client-facingrpc_servicemodule — with a fully backward-compatible fallback.TransportModuleConstants: addRATIS_MODULE = "ratis".MasterClusterInfo.scala: compute Ratis SSL from theratismodule with fallback torpc_service:ratisSslEnabled = sslEnabled(ratis) || sslEnabled(rpc_service)ratisSslModule = ratis if sslEnabled(ratis) else rpc_serviceHARaftServer.configureSsl: build theSSLFactoryfrom the selected module (same fallback).Config keys (
celeborn.ssl.ratis.enabled/.keyStore/.trustStore/...) come from the existingceleborn.ssl.<module>.*machinery — no enumerated module list needed.Why are the changes needed?
Master↔master Ratis TLS is currently derived from the
rpc_servicemodule (MasterClusterInfo.scala→HARaftServer.configureSsl). Butrpc_serviceis also the client-facing control cert, so Ratis and external clients are forced to share one presented cert. That cert must simultaneously satisfy:*.<svc>-master-svc.<ns>.svc.cluster.local).In a cross-cluster / TLS-passthrough-gateway deployment these requirements conflict (the client cert has no internal SANs), so HA masters can't complete the Ratis TLS handshake ("UNAVAILABLE: io exception" during leader election). Decoupling lets operators give Ratis its own cert (internal SANs) while leaving the client-facing cert untouched.
Note: CELEBORN-1356 already split the unified
rpcmodule intorpc_app/rpc_servicefor client-vs-server separation, but never separated Ratis inter-master traffic fromrpc_servicespecifically — this PR closes that remaining gap.Does this PR introduce any user-facing change?
Default behavior is unchanged: with only
rpc_serviceSSL configured, Ratis uses therpc_servicecert exactly as today. The newratismodule only takes effect whenceleborn.ssl.ratis.enabled=trueis explicitly set.How was this patch tested?
Follows existing SSL-module patterns; no new APIs beyond the additive
ratismodule constant and the fallback resolution helpers.