Skip to content

[ISSUE #54 #50] Operator multiple mq cluster && start sequence - #56

Open
linjiemiao wants to merge 2 commits into
apache:masterfrom
silotrd:issue-54-codereview
Open

[ISSUE #54 #50] Operator multiple mq cluster && start sequence#56
linjiemiao wants to merge 2 commits into
apache:masterfrom
silotrd:issue-54-codereview

Conversation

@linjiemiao

Copy link
Copy Markdown
  1. ensure rocketmq-operator can operator more than one rocketmq cluster;
  2. make sure nameserver must ready before broker cluster.

@liuruiyiyang liuruiyiyang changed the title 【ISSUE #54 #50】Operator multiple mq cluster && start sequence [ISSUE #54 #50] Operator multiple mq cluster && start sequence Sep 22, 2020
@liuruiyiyang liuruiyiyang added the enhancement New feature or request label Sep 22, 2020
size: 1
# nameServers is the [ip:port] list of name service
nameServers: ""
# rocketMQName is the rocketmq name, must equal to nameserver.spec.rocketMQName and topictransfer.spec.rocketMQName

@liuruiyiyang liuruiyiyang Sep 24, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can not ensure users set insistent name correctly, is there a better way to do this?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this because we don't have a rocketmq resource as the parent resource of the broker and nameserver. Without a common parent controller, we can only specify the connection of the child resource in the spec. So I hope to add a rocketmq api.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case should we add rocketmq higher level api before this PR?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so

@drivebyer

Copy link
Copy Markdown
Contributor

still need this

@RockteMQ-AI

Copy link
Copy Markdown

⚠️ Merge conflict detected

This PR has conflicts with the base branch and cannot be merged. Please rebase or merge the base branch into your branch and resolve the conflicts:

git fetch origin
git checkout issue-54-codereview
git rebase origin/main
# resolve conflicts, then:
git push --force-with-lease

This is a one-time reminder. Feel free to @mention me for a re-review after conflicts are resolved.


Automated notification by github-manager-bot

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR modifies 19 file(s) with 702 lines of diff. No test changes detected — consider adding test coverage.


Automated review by github-manager-bot

Additional notes (not anchored to a changed line)

  • [INFO] README.md:1 — Large diff (702 lines). Consider breaking into smaller, focused PRs for easier review. (line outside diff)

Comment thread cmd/manager/main.go
@@ -27,7 +27,6 @@ import (
"github.com/apache/rocketmq-operator/pkg/apis"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test changes detected alongside source modifications. Consider adding tests to cover the changes.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

PR received and logged for review. This PR requires detailed code review by a maintainer.

Diff size: 702 lines
Author: linjiemiao (NONE)


Automated review by RockteMQ-AI

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Review of PR #56: [ISSUE #54 #50] Operator multiple mq cluster && start sequence

Findings: 15 issue(s) identified (4 critical).
CLA: unknown

Please address the inline comments above.


Automated review by github-manager-bot

}

func (sMap *ItemSyncMap) LoadOrStore(key string, value ShareItem) (actual ShareItem, loaded bool) {
a, loaded := sMap.m.LoadOrStore(key, value)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoadOrStore will panic if the key exists and the stored value is nil, because a.(ShareItem) is an unconditional type assertion. Although ShareItem is a struct (not a pointer), any future refactor to a pointer type would cause a nil panic. More critically, if the underlying sync.Map somehow stores a non-ShareItem value (e.g., due to a bug), this will panic at runtime. Use a safe assertion: actual, _ = a.(ShareItem) to avoid panic.

return reconcile.Result{}, err
}

actualKey := broker.Namespace + "-" + broker.Spec.RocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The broker reconcile loop spins in a tight busy-wait (for { if actual.IsNameServersStrInitialized { break } else { time.Sleep(...) } }) on the controller goroutine. This blocks the entire reconcile goroutine indefinitely, preventing any other reconcile requests from being processed for this controller. This should be replaced with a requeue-based approach (return reconcile.Result{Requeue: true, RequeueAfter: ...}) so the controller manager can continue handling other events.

return reconcile.Result{}, err
}

actualKey := broker.Namespace + "-" + broker.Spec.RocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Race condition: actual is loaded from the sync map into a local variable at line ~134, then the busy-wait loop re-loads actual from the map inside the loop body, but the outer defer at line ~139 always stores the local actual variable back on function exit. If actual.IsNameServersStrInitialized becomes true during the wait loop (set by the nameservice controller), the defer will overwrite the map with the snapshot captured at loop-entry time, potentially clobbering fields like NameServersStr that the nameservice controller wrote.

@@ -189,8 +201,8 @@ func (r *ReconcileBroker) Reconcile(request reconcile.Request) (reconcile.Result
// Check for name server scaling
if broker.Spec.AllowRestart {
// The following code will restart all brokers to update NAMESRV_ADDR env

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation in the if actual.IsNameServersStrUpdated block: the inner for loop is indented with extra tabs compared to surrounding code. This is a minor formatting issue but indicates the code may not have been run through gofmt, which can cause CI lint failures.

@@ -253,10 +265,17 @@ func (r *ReconcileBroker) Reconcile(request reconcile.Request) (reconcile.Result
podNames := getPodNames(podList.Items)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the early return when len(podNames) == 0, the reconcile returns without storing the updated actual back to the sync map (the defer will still run, but actual.GroupNum and actual.BrokerClusterName will have been set just before this point). However, actual.NameServersStr may not be populated yet if nameServers is empty and the wait loop hasn't run. The defer stores a potentially incomplete actual, which could overwrite a valid previously-stored value if the nameservice controller already populated it.

sourceCluster := topicTransfer.Spec.SourceCluster

nameServer := strings.Split(share.NameServersStr, ";")[0]
actualKey := topicTransfer.Namespace + "-" + topicTransfer.Spec.RocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strings.Split(actual.NameServersStr, ";")[0] will return an empty string (not panic) if actual.NameServersStr is empty, and the subsequent len(nameServer) < cons.MinIpListLength check handles that. However, if actual was just default-initialized by LoadOrStore (i.e., the nameservice for this rocketMQName has not yet reconciled), the TopicTransfer will silently terminate rather than requeue with an informative error. Consider returning a requeue result instead of terminating.

SourceCluster string `json:"sourceCluster,omitempty"`
// The cluster where the topic will be transferred to
TargetCluster string `json:"targetCluster,omitempty"`
// // RocketMQ Name, the broker and nameserver in the same cluster must be filled with the same name

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double comment marker on line 40: // // RocketMQ Name, ... — there is a tab and an extra // before the actual comment text. This is a copy-paste artifact and should be // RocketMQ Name, ....

@@ -93,6 +96,7 @@ spec:
- volumes

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rocketMQName is now a required field in the CRD. This is a breaking change for any existing Broker CR that does not have this field set. Existing clusters that upgrade to this operator version will have their Broker CRs fail validation. A defaulting webhook or a non-required field with a documented migration path is needed for safe upgrades.

@@ -73,6 +76,7 @@ spec:
- storageMode

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same breaking change as the Broker CRD: making rocketMQName required on NameService CRD will invalidate all existing NameService CRs on upgrade. Needs a migration strategy.

Comment thread cmd/manager/main.go
@@ -100,7 +93,6 @@ func main() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing Namespace from manager.Options changes the operator from namespace-scoped to cluster-scoped watching. This is intentional (given the new ClusterRole), but it is a significant behavioral change: the operator will now watch all namespaces, increasing API server load and requiring the new ClusterRole/ClusterRoleBinding to be applied. This should be explicitly documented in the PR and migration notes, and the old namespace-scoped Role/RoleBinding files should be deprecated or removed to avoid confusion.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The multi-cluster isolation approach via rocketMQName keyed sync.Map is a reasonable direction, but the implementation has a critical blocking infinite loop, a value-type race condition in the shared state, and backward-incompatible CRD changes that will break existing deployments.

Findings

  • [CRITICAL] pkg/controller/broker/broker_controller.go:147 — Infinite blocking for loop with time.Sleep has no timeout, context cancellation, or requeue. If the NameServer never becomes ready, this permanently blocks a controller worker goroutine, starving all other reconcile requests. Replace with return reconcile.Result{RequeueAfter: ...}, nil so the work is re-enqueued without blocking.
  • [CRITICAL] pkg/share/share.go:23ShareItem is a value type (struct), not a pointer. LoadOrStore and Load return a copy; mutations to fields like actual.NameServersStr are local until Store is explicitly called. When broker and nameservice controllers reconcile concurrently for the same key, they each get independent copies, mutate them, and the last Store wins — silently dropping the other's changes (e.g., IsNameServersStrInitialized could be overwritten back to false). Use *ShareItem (pointer) or a mutex-guarded struct to ensure atomic read-modify-write.
  • [CRITICAL] deploy/crds/rocketmq_v1alpha1_broker_crd.yaml:99rocketMQName is added to the required list. Any existing Broker CR that does not include this field will fail validation after CRD upgrade, breaking backward compatibility. Either provide a default value via an admission webhook / mutating webhook, remove it from required, or use a conversion/migration step for existing resources.
  • [CRITICAL] deploy/crds/rocketmq_v1alpha1_nameservice_crd.yaml:79 — Same backward-compatibility break as the Broker CRD: rocketMQName is added to required, which will reject any existing NameService CR that lacks the field upon upgrade.
  • [WARNING] deploy/clusterrole_binding.yaml:22 — The ServiceAccount namespace is hardcoded to default. If the operator is deployed to any other namespace, the binding will reference a non-existent ServiceAccount and RBAC will deny all API calls. Make this configurable (e.g., via Kustomize or Helm templating).
  • [WARNING] pkg/controller/broker/broker_controller.go:204 — Indentation regression: the for loop inside the if actual.IsNameServersStrUpdated block has an extra tab, misaligning it from the enclosing if. This is likely a merge artifact and should be corrected to maintain readability.
  • [WARNING] pkg/controller/broker/broker_controller.go:478getENV calls LoadOrStore(actualKey, share.ShareItem{}) which will insert an empty ShareItem (all zero values) if the key does not yet exist. If the broker reconciles before the nameservice controller has stored real values, NameServersStr will be empty and broker pods will be created with an invalid NAMESRV_ADDR. Consider using Load instead and returning an error / requeue when the key is absent.
  • [WARNING] pkg/controller/nameservice/nameservice_controller.go:177 — Early return when len(hostIps) != instance.Spec.Size happens before the deferred Store, so no ShareItem is created for this key. This is correct behavior, but if the nameservice never reaches full size (e.g., a node shortage), brokers waiting on IsNameServersStrInitialized will block indefinitely (see broker_controller.go infinite loop issue). Consider setting a partial-readiness indicator or adding a timeout.
  • [INFO] pkg/apis/rocketmq/v1alpha1/topictransfer_types.go:40 — Malformed comment: // // RocketMQ Name... contains a stray tab and duplicate // prefix, likely a copy-paste artifact. Clean up to // RocketMQ Name, the broker and nameserver in the same cluster must be filled with the same name.
  • [INFO] deploy/cluster_role.yaml:66pods/exec is listed under the rocketmq.apache.org API group, which is incorrect — pods/exec belongs to the core ("") API group and is already listed there (line 30). The duplicate under the custom API group is meaningless and should be removed.
  • [WARNING] pkg/controller/topictransfer/topictransfer_controller.go:130LoadOrStore with an empty ShareItem means if the TopicTransfer reconciles before any NameService has populated the map, an empty entry is persisted. strings.Split("", ";")[0] returns "" which is caught by the length check, but the stale empty entry remains in the map. Use Load and requeue if the key is not found.
  • [INFO] pkg/controller/broker/broker_controller.go:298 — The new len(podNames) == 0 guard returns early with a requeue, but the subsequent status update at line 302 adds a len(broker.Status.Nodes) != 0 guard that prevents clearing Status.Nodes when all pods are gone. This asymmetry means the status can never reflect a fully-scaled-down state; if that is intentional (safety), document it.

Automated review by github-manager-bot

} else {
share.GroupNum = broker.Status.Size
groupNum = broker.Status.Size
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Infinite blocking for loop with time.Sleep has no timeout, context cancellation, or requeue. If the NameServer never becomes ready, this permanently blocks a controller worker goroutine, starving all other reconcile requests. Replace with return reconcile.Result{RequeueAfter: ...}, nil so the work is re-enqueued without blocking.

Comment thread pkg/share/share.go

import "sync"

var (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShareItem is a value type (struct), not a pointer. LoadOrStore and Load return a copy; mutations to fields like actual.NameServersStr are local until Store is explicitly called. When broker and nameservice controllers reconcile concurrently for the same key, they each get independent copies, mutate them, and the last Store wins — silently dropping the other's changes (e.g., IsNameServersStrInitialized could be overwritten back to false). Use *ShareItem (pointer) or a mutex-guarded struct to ensure atomic read-modify-write.

- volumes
- volumeClaimTemplates
- scalePodName
- rocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rocketMQName is added to the required list. Any existing Broker CR that does not include this field will fail validation after CRD upgrade, breaking backward compatibility. Either provide a default value via an admission webhook / mutating webhook, remove it from required, or use a conversion/migration step for existing resources.

- storageMode
- hostPath
- volumeClaimTemplates
- rocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same backward-compatibility break as the Broker CRD: rocketMQName is added to required, which will reject any existing NameService CR that lacks the field upon upgrade.

name: rocketmq-operator
subjects:
- kind: ServiceAccount
name: rocketmq-operator

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ServiceAccount namespace is hardcoded to default. If the operator is deployed to any other namespace, the binding will reference a non-existent ServiceAccount and RBAC will deny all API calls. Make this configurable (e.g., via Kustomize or Helm templating).

}
hostIps := getNameServers(podList.Items)

if len(hostIps) != int(instance.Spec.Size) || len(hostIps) == 0 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Early return when len(hostIps) != instance.Spec.Size happens before the deferred Store, so no ShareItem is created for this key. This is correct behavior, but if the nameservice never reaches full size (e.g., a node shortage), brokers waiting on IsNameServersStrInitialized will block indefinitely (see broker_controller.go infinite loop issue). Consider setting a partial-readiness indicator or adding a timeout.

SourceCluster string `json:"sourceCluster,omitempty"`
// The cluster where the topic will be transferred to
TargetCluster string `json:"targetCluster,omitempty"`
// // RocketMQ Name, the broker and nameserver in the same cluster must be filled with the same name

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Malformed comment: // // RocketMQ Name... contains a stray tab and duplicate // prefix, likely a copy-paste artifact. Clean up to // RocketMQ Name, the broker and nameserver in the same cluster must be filled with the same name.

Comment thread deploy/cluster_role.yaml
- update
- apiGroups:
- rocketmq.apache.org
resources:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pods/exec is listed under the rocketmq.apache.org API group, which is incorrect — pods/exec belongs to the core ("") API group and is already listed there (line 30). The duplicate under the custom API group is meaningless and should be removed.

sourceCluster := topicTransfer.Spec.SourceCluster

nameServer := strings.Split(share.NameServersStr, ";")[0]
actualKey := topicTransfer.Namespace + "-" + topicTransfer.Spec.RocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoadOrStore with an empty ShareItem means if the TopicTransfer reconciles before any NameService has populated the map, an empty entry is persisted. strings.Split("", ";")[0] returns "" which is caught by the length check, but the stale empty entry remains in the map. Use Load and requeue if the key is not found.

cmd = []string{"/bin/bash", "-c", MakeConfigDirCommand + " && " + ChmodDirCommand + " && " + topicsCommand + " && " + subscriptionGroupCommand}
}

// Update status.Nodes if needed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new len(podNames) == 0 guard returns early with a requeue, but the subsequent status update at line 302 adds a len(broker.Status.Nodes) != 0 guard that prevents clearing Status.Nodes when all pods are gone. This asymmetry means the status can never reflect a fully-scaled-down state; if that is intentional (safety), document it.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

10 finding(s) to address.

Findings

  • [CRITICAL] pkg/controller/broker/broker_controller.go:162 — actual.NameServersStr is only assigned on the local copy fetched at the top of Reconcile; it is written back to the sync.Map in the deferred Store (line 139), i.e. after getBrokerStatefulSet() has already run. getENV() (line 484) re-loads the item from the map, so on the first reconcile of a Broker with spec.nameServers set explicitly, the StatefulSet is created with an empty NAMESRV_ADDR env var and the brokers cannot start. The old global-variable code made the mutation immediately visible to getENV; the copy-in/copy-out semantics break that. Store the item immediately after mutating it, or pass the name-server string down into getBrokerStatefulSet/getENV as a parameter.
  • [WARNING] pkg/controller/broker/broker_controller.go:139 — ShareItem is copied out, mutated, and stored back as a whole by different controllers (broker, nameservice, topictransfer) with no per-key locking. Concurrent reconciles for the same key perform read-modify-write on independent struct copies, so the last Store wins and the other controller's changes are silently lost — e.g. a broker reconcile finishing here can revert a NameServersStr / IsNameServersStrUpdated value just written by the nameservice controller. sync.Map only protects the map itself, not lost updates on the values. Guard each key with a mutex, or have each controller store only the fields it owns.
  • [WARNING] pkg/controller/broker/broker_controller.go:150 — This busy-wait loop blocks the reconcile worker indefinitely. With per-key state, a mismatch between broker.spec.rocketMQName and the NameService CR's rocketMQName (a plain string with no cross-CRD validation — a single typo) means IsNameServersStrInitialized is never set and this goroutine sleeps forever. With the default MaxConcurrentReconciles=1, one misconfigured cluster starves every broker CR in every other cluster, defeating the multi-cluster goal of this PR. Prefer returning reconcile.Result{RequeueAfter: ...} like the pod-not-ready handling already added further down in this function.
  • [WARNING] deploy/crds/rocketmq_v1alpha1_broker_crd.yaml:99 — rocketMQName is added to required (also in the nameservice and topictransfer CRDs). Existing Broker/NameService/TopicTransfer CRs created by older operator versions lack this field, so after the CRD update any subsequent write to those objects (kubectl apply/edit, and potentially the operator's own status updates) fails OpenAPI validation — there is no migration path documented. Also note that CRs patched with an empty value silently collide on the key "-". Consider making the field optional with a documented default, or document a mandatory pre-upgrade migration for all existing CRs.
  • [WARNING] deploy/clusterrole_binding.yaml:21 — The ClusterRoleBinding subject hardcodes namespace: default. If the operator is installed in any other namespace, its ServiceAccount never receives the ClusterRole and — since the manager now watches all namespaces — informer cache sync fails and the operator cannot start. The binding should be parameterized by the install namespace, and deploy/operator.yaml needs to be updated to reference this cluster-scoped RBAC (the diff does not touch it, so existing deployments still ship the namespaced Role and WATCH_NAMESPACE).
  • [WARNING] cmd/manager/main.go:94 — Removing GetWatchNamespace()/the Namespace option silently ignores the WATCH_NAMESPACE env var that existing operator deployments set. After upgrading, the operator expands from namespaced to cluster-wide watching with no notice; deployments that deliberately scoped it for least privilege will break at startup because the caches now need list/watch across all namespaces (and old namespaced Role manifests no longer suffice). Keep honoring WATCH_NAMESPACE, or at minimum log a loud warning when it is set.
  • [INFO] pkg/controller/nameservice/nameservice_controller.go:233 — clusterName (and GroupNum) come from in-memory cross-controller state that is only populated after a Broker reconcile for the same rocketMQName has run. After an operator restart the map is empty, so a name-server list change triggers the mqAdmin updateBrokerConfig command with -c "", which fails and silently leaves brokers configured with stale name-server addresses. Consider looking the broker cluster name up from the Broker CR (label/owner lookup) instead of relying on shared memory. This weakness is inherited from the old globals, but the restart-loss scenario is worth fixing while this state is being reworked.
  • [INFO] pkg/share/share.go:28 — No unit tests accompany the new share package or the reworked reconcile logic, although both are easily testable (LoadOrStore/Store round-trips, concurrent updates to one key, and the Reconcile path where spec.nameServers is set explicitly). A test asserting that a newly created broker StatefulSet carries the expected NAMESRV_ADDR would have caught the stale-copy bug in getENV.
  • [INFO] pkg/controller/broker/broker_controller.go:205 — The for loop under if actual.IsNameServersStrUpdated is indented one level too deep — a gofmt violation introduced by this change; please run gofmt/goimports on the touched files.
  • [INFO] pkg/apis/rocketmq/v1alpha1/topictransfer_types.go:40 — The comment is doubled (// \t// RocketMQ Name...) — leftover from editing. Similar "and and" duplications appear in the new README and example YAML comments; worth a quick copy pass.

Automated review by github-manager-bot

}
} else {
share.NameServersStr = broker.Spec.NameServers
actual.NameServersStr = broker.Spec.NameServers

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actual.NameServersStr is only assigned on the local copy fetched at the top of Reconcile; it is written back to the sync.Map in the deferred Store (line 139), i.e. after getBrokerStatefulSet() has already run. getENV() (line 484) re-loads the item from the map, so on the first reconcile of a Broker with spec.nameServers set explicitly, the StatefulSet is created with an empty NAMESRV_ADDR env var and the brokers cannot start. The old global-variable code made the mutation immediately visible to getENV; the copy-in/copy-out semantics break that. Store the item immediately after mutating it, or pass the name-server string down into getBrokerStatefulSet/getENV as a parameter.

defer func() {
reqLogger.Info("Broker actualKey:" + actualKey + " actual.NameServerStr:" + actual.NameServersStr +
" actual.BrokerClusterName:" + actual.BrokerClusterName + " IsNameServersStrInitialized:" + strconv.FormatBool(actual.IsNameServersStrInitialized))
share.GetInstance().Store(actualKey, actual)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShareItem is copied out, mutated, and stored back as a whole by different controllers (broker, nameservice, topictransfer) with no per-key locking. Concurrent reconciles for the same key perform read-modify-write on independent struct copies, so the last Store wins and the other controller's changes are silently lost — e.g. a broker reconcile finishing here can revert a NameServersStr / IsNameServersStrUpdated value just written by the nameservice controller. sync.Map only protects the map itself, not lost updates on the values. Guard each key with a mutex, or have each controller store only the fields it owns.


if broker.Spec.NameServers == "" {
// wait for name server ready when create broker cluster if nameServers is omitted
for {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This busy-wait loop blocks the reconcile worker indefinitely. With per-key state, a mismatch between broker.spec.rocketMQName and the NameService CR's rocketMQName (a plain string with no cross-CRD validation — a single typo) means IsNameServersStrInitialized is never set and this goroutine sleeps forever. With the default MaxConcurrentReconciles=1, one misconfigured cluster starves every broker CR in every other cluster, defeating the multi-cluster goal of this PR. Prefer returning reconcile.Result{RequeueAfter: ...} like the pod-not-ready handling already added further down in this function.

- volumes
- volumeClaimTemplates
- scalePodName
- rocketMQName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rocketMQName is added to required (also in the nameservice and topictransfer CRDs). Existing Broker/NameService/TopicTransfer CRs created by older operator versions lack this field, so after the CRD update any subsequent write to those objects (kubectl apply/edit, and potentially the operator's own status updates) fails OpenAPI validation — there is no migration path documented. Also note that CRs patched with an empty value silently collide on the key "-". Consider making the field optional with a documented default, or document a mandatory pre-upgrade migration for all existing CRs.

metadata:
name: rocketmq-operator
subjects:
- kind: ServiceAccount

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ClusterRoleBinding subject hardcodes namespace: default. If the operator is installed in any other namespace, its ServiceAccount never receives the ClusterRole and — since the manager now watches all namespaces — informer cache sync fails and the operator cannot start. The binding should be parameterized by the install namespace, and deploy/operator.yaml needs to be updated to reference this cluster-scoped RBAC (the diff does not touch it, so existing deployments still ship the namespaced Role and WATCH_NAMESPACE).

Comment thread cmd/manager/main.go
@@ -100,7 +93,6 @@ func main() {

// Create a new Cmd to provide shared dependencies and start components

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing GetWatchNamespace()/the Namespace option silently ignores the WATCH_NAMESPACE env var that existing operator deployments set. After upgrading, the operator expands from namespaced to cluster-wide watching with no notice; deployments that deliberately scoped it for least privilege will break at startup because the caches now need list/watch across all namespaces (and old namespaced Role manifests no longer suffice). Keep honoring WATCH_NAMESPACE, or at minimum log a loud warning when it is set.



clusterName := share.BrokerClusterName
clusterName := actual.BrokerClusterName

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clusterName (and GroupNum) come from in-memory cross-controller state that is only populated after a Broker reconcile for the same rocketMQName has run. After an operator restart the map is empty, so a name-server list change triggers the mqAdmin updateBrokerConfig command with -c "", which fails and silently leaves brokers configured with stale name-server addresses. Consider looking the broker cluster name up from the Broker CR (label/owner lookup) instead of relying on shared memory. This weakness is inherited from the old globals, but the restart-loss scenario is worth fixing while this state is being reworked.

Comment thread pkg/share/share.go
instance *Singleton
)

type ShareItem struct {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No unit tests accompany the new share package or the reworked reconcile logic, although both are easily testable (LoadOrStore/Store round-trips, concurrent updates to one key, and the Reconcile path where spec.nameServers is set explicitly). A test asserting that a newly created broker StatefulSet carries the expected NAMESRV_ADDR would have caught the stale-copy bug in getENV.

if share.IsNameServersStrUpdated {
for brokerGroupIndex := 0; brokerGroupIndex < broker.Spec.Size; brokerGroupIndex++ {
if actual.IsNameServersStrUpdated {
for brokerGroupIndex := 0; brokerGroupIndex < broker.Spec.Size; brokerGroupIndex++ {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The for loop under if actual.IsNameServersStrUpdated is indented one level too deep — a gofmt violation introduced by this change; please run gofmt/goimports on the touched files.

SourceCluster string `json:"sourceCluster,omitempty"`
// The cluster where the topic will be transferred to
TargetCluster string `json:"targetCluster,omitempty"`
// // RocketMQ Name, the broker and nameserver in the same cluster must be filled with the same name

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is doubled (// \t// RocketMQ Name...) — leftover from editing. Similar "and and" duplications appear in the new README and example YAML comments; worth a quick copy pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants