Feature/user resource quota#18202
Open
hongzhi-gao wants to merge 4 commits into
Open
Conversation
Introduce ConfigNode persistence and DataNode acquire/release enforcement for CPU/MEMORY quotas, keep throttle quota backward compatible, and cover the path with UT/IT.
… scheduling fixes. Add DELETE USER QUOTA, concrete acquire-reject reasons, temp-disk default min(100GiB, disk/10), unlimited users in minGap scheduling, and CN SHOW rows for all Running DataNodes via heartbeat usage snapshots.
Cover quota_enable/root/invalid attrs, partial merge, cross THROTTLE SHOW, temp-disk isolation, and read_cpu release path from the requirements checklist.
Awaitility now ignores transient InconsistentDataException while space quota usage catches up across DataNodes via heartbeat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a unified user-level resource quota framework so multi-tenant workloads can set per-user CPU / memory / temp-disk / disk-IO limits with min (guarantee) + max (cap), while staying compatible with existing
THROTTLE QUOTA.SQL & model
SET / SHOW / DELETE USER QUOTACPU,MEMORY,TEMP_DISK(capacity, Byte),DISK_IO(rate, reuse throttle)read_*/write_*); keepSET/SHOW THROTTLE QUOTAand mapcpu/mem→read_cpu_max/read_memory_maxquota_enable(enabled semantics; compatible with THROTTLE/SPACE)Runtime enforcement (DataNode-local)
UserResourceQuotaManager.acquire/releaseon query (DriverScheduler) and write pathsDELETE USER QUOTAclears config only and keeps in-flightinUseuntil tokens releaseCluster SHOW & defaults
SHOW USER QUOTA: NodeID = all Running DataNodes; DN reports inUse via heartbeat; CN aggregates; lost nodes omitteddn_quota_temp_disk_bytesdefault =min(100GiB, ΣdataDirs.totalSpace/10)at startup; explicit config overridesClusterCachePropagatorwhen availableTests
IoTDBUserResourceQuotaIT(SET/SHOW/DELETE, throttle mapping, lower max, NodeID)This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb-core/datanode/.../quotas/UserResourceQuotaManageriotdb-core/datanode/.../quotas/CapacityResourceLimiter/LimiterAcquireResultiotdb-core/datanode/.../quota/ShowUserResourceQuotaTask/SetUserResourceQuotaTask/DeleteUserResourceQuotaTaskiotdb-core/confignode/.../ClusterQuotaManageriotdb-core/confignode/.../quota/SetUserResourceQuotaPlan/DeleteUserResourceQuotaPlaniotdb-core/node-commons/.../quota/*iotdb-protocol/thrift-{commons,datanode,confignode}(quota + heartbeat usage snapshot)integration-test/.../IoTDBUserResourceQuotaIT