Skip to content

[CELEBORN-2388] Decouple inter-master Ratis TLS into its own SSL module#3767

Open
shellfish007 wants to merge 2 commits into
apache:mainfrom
shellfish007:upstream-pr/ratis-tls-module
Open

[CELEBORN-2388] Decouple inter-master Ratis TLS into its own SSL module#3767
shellfish007 wants to merge 2 commits into
apache:mainfrom
shellfish007:upstream-pr/ratis-tls-module

Conversation

@shellfish007

@shellfish007 shellfish007 commented Jul 23, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

Adds a dedicated ratis SSL module so inter-master Ratis (Raft consensus) gRPC TLS can use its own certificate/keystore, independent of the client-facing rpc_service module — with a fully backward-compatible fallback.

  • TransportModuleConstants: add RATIS_MODULE = "ratis".
  • MasterClusterInfo.scala: compute Ratis SSL from the ratis module with fallback to rpc_service:
    • ratisSslEnabled = sslEnabled(ratis) || sslEnabled(rpc_service)
    • ratisSslModule = ratis if sslEnabled(ratis) else rpc_service
  • HARaftServer.configureSsl: build the SSLFactory from the selected module (same fallback).

Config keys (celeborn.ssl.ratis.enabled/.keyStore/.trustStore/...) come from the existing celeborn.ssl.<module>.* machinery — no enumerated module list needed.

Why are the changes needed?

Master↔master Ratis TLS is currently derived from the rpc_service module (MasterClusterInfo.scalaHARaftServer.configureSsl). But rpc_service is also the client-facing control cert, so Ratis and external clients are forced to share one presented cert. That cert must simultaneously satisfy:

  • clients: signed by a CA they trust (e.g. a corporate CA) + the gateway/SNI hostname, and
  • Ratis peers: gRPC hostname verification against the internal master pod FQDNs (*.<svc>-master-svc.<ns>.svc.cluster.local).

In a cross-cluster / TLS-passthrough-gateway deployment these requirements conflict (the client cert has no internal SANs), so HA masters can't complete the Ratis TLS handshake ("UNAVAILABLE: io exception" during leader election). Decoupling lets operators give Ratis its own cert (internal SANs) while leaving the client-facing cert untouched.

Note: CELEBORN-1356 already split the unified rpc module into rpc_app/rpc_service for client-vs-server separation, but never separated Ratis inter-master traffic from rpc_service specifically — this PR closes that remaining gap.

Does this PR introduce any user-facing change?

Default behavior is unchanged: with only rpc_service SSL configured, Ratis uses the rpc_service cert exactly as today. The new ratis module only takes effect when celeborn.ssl.ratis.enabled=true is explicitly set.

How was this patch tested?

Follows existing SSL-module patterns; no new APIs beyond the additive ratis module constant and the fallback resolution helpers.

Master<->master Ratis TLS is currently derived from the rpc_service
module (MasterClusterInfo.scala -> HARaftServer.configureSsl). But
rpc_service is also the client-facing control cert, so Ratis and
external clients are forced to share one presented cert. That cert
must simultaneously satisfy:

  - clients: signed by a CA they trust (e.g. the corp CA) + the
    gateway/SNI hostname, and
  - Ratis peers: gRPC hostname verification against the internal
    master pod FQDNs (*.<svc>-master-svc.<ns>.svc.cluster.local).

In a cross-cluster / TLS-passthrough-gateway deployment these
requirements conflict (the client cert has no internal SANs), so HA
masters can't complete the Ratis TLS handshake (UNAVAILABLE: io
exception during leader election). Decoupling lets operators give
Ratis its own cert (internal SANs) while leaving the client-facing
cert untouched.

- TransportModuleConstants: add RATIS_MODULE = "ratis".
- MasterClusterInfo.scala: compute Ratis SSL from the ratis module
  with fallback to rpc_service:
    ratisSslEnabled = sslEnabled(ratis) || sslEnabled(rpc_service)
    ratisSslModule = ratis if sslEnabled(ratis) else rpc_service
- HARaftServer.configureSsl: build the SSLFactory from the selected
  module (same fallback).

Config keys (celeborn.ssl.ratis.enabled/.keyStore/.trustStore/...)
come from the existing celeborn.ssl.<module>.* machinery - no
enumerated module list needed.

Default behavior is unchanged: with only rpc_service SSL configured,
Ratis uses the rpc_service cert exactly as today. The ratis module
only takes effect when celeborn.ssl.ratis.enabled=true is explicitly
set.
@SteNicholas

Copy link
Copy Markdown
Member

@shellfish007, please create new issues in JIRA which follows #1053.

@shellfish007 shellfish007 changed the title Decouple inter-master Ratis TLS into its own SSL module [CELEBORN-2388] Decouple inter-master Ratis TLS into its own SSL module Jul 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a dedicated SSL/TLS “ratis” transport module so inter-master Ratis (Raft) gRPC TLS can use its own keystore/certificate, while preserving the current behavior by falling back to the existing rpc_service SSL module when ratis SSL is not explicitly enabled.

Changes:

  • Add TransportModuleConstants.RATIS_MODULE = "ratis".
  • Update MasterClusterInfo to enable Ratis TLS when either ratis or rpc_service SSL is enabled (backward-compatible fallback).
  • Update HARaftServer.configureSsl to build the SSLFactory from the selected SSL module (ratis if enabled, else rpc_service).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
master/src/main/scala/org/apache/celeborn/service/deploy/master/clustermeta/ha/MasterClusterInfo.scala Adds helper logic for determining whether Ratis TLS is enabled and which SSL module to use (with rpc_service fallback).
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java Selects the SSL module used to create the Ratis SSLFactory (preferring ratis when enabled).
common/src/main/java/org/apache/celeborn/common/protocol/TransportModuleConstants.java Introduces the new ratis transport module constant for per-module SSL config resolution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +453 to +457
// MasterClusterInfo.ratisSslModule (kept in sync as the single source of truth for the policy).
String sslModule =
conf.sslEnabled(TransportModuleConstants.RATIS_MODULE)
? TransportModuleConstants.RATIS_MODULE
: TransportModuleConstants.RPC_SERVICE_MODULE;
@shellfish007

Copy link
Copy Markdown
Author

JIRA ticket filed: https://issues.apache.org/jira/browse/CELEBORN-2388. PR title updated accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants