Skip to content

[Cosmos] Clean up test databases left behind by CI runs - #50002

Open
tvaron3 wants to merge 11 commits into
Azure:mainfrom
tvaron3:tomasvaron-microsoft-cosmos-ci-resource-cleanup
Open

[Cosmos] Clean up test databases left behind by CI runs#50002
tvaron3 wants to merge 11 commits into
Azure:mainfrom
tvaron3:tomasvaron-microsoft-cosmos-ci-resource-cleanup

Conversation

@tvaron3

@tvaron3 tvaron3 commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

Cosmos live CI tests leak databases on the long lived shared accounts and never reclaim them. Six stages in sdk/cosmos/tests.yml run against accounts that are never torn down — thin client ($(thinclient-test-endpoint)), the two thin client canaries, and GSI ($(gsi-pipeline-uri)) — unlike the other stages, whose ARM provisioned accounts are deleted with their resource group. Anything a test forgets to delete on those accounts stays there permanently.

Two root causes:

  1. Cleanup only recognized one naming convention. CosmosDatabaseForTest.cleanupStaleTestDatabases only queried STARTSWITH(c.id, "RxJava.SDKTest.SharedDatabase"). Tests naming databases with a raw UUID.randomUUID() or a fixed literal (HubRegionProcessingOnlyTests used "testDatabase" in five places) were invisible to cleanup forever.
  2. Cleanup only ran at @AfterSuite. A cancelled job, or one hitting its 210 minute timeout, killed the JVM before anything ran.

The complication is that several matrix legs and concurrent pipeline runs share these accounts, so cleanup must never delete a resource belonging to another in-flight run.

What this does

Every test created database id now carries a run id (CosmosTestRunId, derived from System.JobId with the build id carried along for traceability):

RxJava.SDKTest.SharedDatabase_<timestamp>_<runId>_<label><random>

That lets cleanup delete exactly what this run created. Databases from other runs are only ever removed by the age based sweep (8h, comfortably above the 210 minute stage timeout), and ids that do not parse — hand created fixtures — are never touched.

Cleanup then runs in four layers, widest last:

Layer Where Catches
CosmosTestResourceRegistry + CosmosTestResourceJanitor test JVM, end of run normal completion; deletes leftovers and fails the run, naming the test that leaked
JVM shutdown hook test JVM crashes and hard aborts
cleanup-test-resources.yml post step pipeline, condition: always() failed jobs, and jobs whose JVM died
janitor.yml scheduled every 6h cancelled and timed out jobs, where nothing in the job ever ran

Preventing regressions

  • TestSuiteBase.createTestDatabase(client, "label") names and registers automatically. The arbitrary id creators are @Deprecated but still register, so nothing escapes tracking during migration.
  • TestResourceHygieneTest runs in the unit group (every PR, no account needed) and ratchets against direct database creation using a checked in baseline. It has a self test so the scanner cannot silently rot.
  • Convention documented in a new sdk/cosmos/AGENTS.md so AI generated tests follow it by default.

Also fixes a latent NPE and an unsafe 2 hour cleanup threshold in the azure-cosmos copy of DatabaseForTest — the new four segment ids would otherwise have made it throw, and its threshold sat below the stage timeout, so it could have deleted in-flight databases.

How to verify

  • mvn -f sdk/cosmos/azure-cosmos-tests/pom.xml test-compile — green with checkstyle on.
  • mvn -f sdk/cosmos/azure-cosmos-tests/pom.xml -P unit failsafe:integration-test failsafe:verify — includes TestResourceHygieneTest, CosmosDatabaseForTestTest and CosmosTestResourceJanitorTest (20 tests).
  • Live tests are the real verification — see /azp run below. Expect databases created during the run to be gone from the shared accounts afterwards, and the run to fail if any test leaks.

Notes for reviewers

  • The safety invariant is the thing to scrutinize: cleanup must never delete another run's in-flight database. It is pinned by CosmosDatabaseForTestTest, which I mutation tested — swapping exact run id matching for startsWith, dropping the .block(), and removing the parse failure skip each fail the suite.
  • CosmosTestResourceJanitorTest covers the delete ordering and leak classification rules; mutation tested against the specific bugs found in review.
  • FAIL_ON_LEAK was verified to be a real gate, not decoration: an AssertionError thrown from IExecutionListener.onExecutionFinish fails failsafe:verify (MAVEN_EXIT=1), with a clean control run for comparison.
  • One consequence worth knowing: TestNG writes its reports before execution listeners run, so a leak-failed run publishes Tests run: 0 rather than the leak. The leak is in the build log and as an ADO issue annotation — documented in AGENTS.md.
  • The always() post step runs on cancellation only within cancelTimeoutInMinutes (5 by default), so it is a bonus layer; janitor.yml is the actual backstop for cancelled jobs. Wording in the templates reflects this.
  • janitor.yml needs its pipeline definition linked to the same variable group as the Cosmos live test pipeline before it will resolve the account variables.
  • Escape hatches: -DCOSMOS.TEST_RESOURCE_JANITOR_ENABLED=false disables the janitor; -DCOSMOS.TEST_RESOURCE_JANITOR_FAIL_ON_LEAK=false keeps cleanup but stops it failing the run.

Scoped to azure-cosmos-tests. azure-cosmos-encryption has a near identical copy of this logic and is a follow up.

tvaron3 and others added 2 commits July 30, 2026 11:09
@tvaron3
Clean up Cosmos test databases left behind by CI runs
e6c6941 
Six live test stages in sdk/cosmos/tests.yml run against long lived shared
accounts (thin client, thin client canaries, GSI) that are never torn down and
are used by several matrix legs and pipeline runs concurrently. Databases a test
forgot to delete stayed on those accounts permanently:

- cleanupStaleTestDatabases only recognized ids starting with
  "RxJava.SDKTest.SharedDatabase", so tests naming databases with raw UUIDs or
  fixed literals leaked forever.
- Cleanup only ran at @AfterSuite, so a cancelled job or one hitting its
  210 minute timeout killed the JVM and leaked everything it had created.

Every test created database id now carries a run id derived from System.JobId,
which lets cleanup delete exactly what the current run created without touching
resources a concurrently executing leg is still using. Cleanup runs in four
layers: an in-JVM registry plus a TestNG listener that deletes leftovers and
fails the run naming the offending test, a JVM shutdown hook, an always() post
step per stage, and a new six-hourly janitor pipeline for jobs that died before
anything in them could run.

To stop new tests reintroducing the problem, TestSuiteBase gains sanctioned
createTestDatabase helpers that name and register automatically, the arbitrary
id creators are deprecated, and TestResourceHygieneTest ratchets against direct
database creation using a checked in baseline.

Also fixes a latent NPE and an unsafe 2 hour cleanup threshold in the
azure-cosmos copy of DatabaseForTest, which the new four segment ids would
otherwise have triggered.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Declare the Cosmos test secrets variable group in janitor.yml
67828b8 
Linking a variable group on the pipeline definition is how sdk/cosmos/tests.yml
works, but that linkage is applied by the engineering system's pipeline
generator. janitor.yml needs a hand created definition, so declare the group in
YAML instead - creating the pipeline then only requires authorizing the group
for it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 30, 2026 23:29
@tvaron3
tvaron3 requested review from a team and kirankumarkolli as code owners July 30, 2026 23:29
@tvaron3

tvaron3 commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Jul 30, 2026

Copy link
Copy Markdown
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

Copilot AI reviewed Jul 30, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses Cosmos live CI tests leaking databases on long-lived shared Cosmos accounts by introducing run-scoped database naming, multi-layer cleanup (in-process, shutdown hook, pipeline post-step, and scheduled janitor), and guardrails to prevent regressions in azure-cosmos-tests.

Changes:

  • Add run-id tagged database IDs and cleanup logic that can safely delete only the current job’s resources on shared accounts, plus an age-based sweep for older orphans.
  • Add pipeline-level cleanup: per-stage always-run post steps and a scheduled janitor.yml sweep for timeouts/cancellations.
  • Add test hygiene enforcement (TestResourceHygieneTest + baseline) and wire a CosmosTestResourceJanitor TestNG listener across suites to fail runs that leak.

Reviewed changes

Copilot reviewed 64 out of 64 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
sdk/cosmos/tests.yml Adds PostSteps to run Cosmos cleanup after stages that use long-lived shared accounts.
sdk/cosmos/janitor.yml Introduces a scheduled pipeline to sweep stale test databases on shared accounts.
sdk/cosmos/dev.md Documents requirement to create databases via TestSuiteBase.createTestDatabase(...) for CI cleanup compatibility.
sdk/cosmos/cleanup-test-resources.yml Adds a reusable always-run Maven exec step to clean up the current job’s Cosmos test resources.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/DatabaseForTest.java Makes cleanup UTC-safe, raises stale threshold to 8h, tolerates 3/4-part IDs, and avoids NPE on non-test IDs.
sdk/cosmos/azure-cosmos-tests/src/test/resources/thinclient-testng.xml Registers CosmosTestResourceJanitor listener for thinclient suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/thinclient-endpoint-probe-testng.xml Registers CosmosTestResourceJanitor listener for endpoint probe suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/split-testng.xml Registers CosmosTestResourceJanitor listener for split suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/query-testng.xml Registers CosmosTestResourceJanitor listener for query suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/multi-region-testng.xml Registers CosmosTestResourceJanitor listener for multi-region suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/multi-region-strong.xml Registers CosmosTestResourceJanitor listener for multi-region-strong suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/multi-master-testng.xml Registers CosmosTestResourceJanitor listener for multi-master suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/manual-http-network-fault-testng.xml Registers CosmosTestResourceJanitor listener for manual fault suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/long-testng.xml Registers CosmosTestResourceJanitor listener for long suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/long-emulator-testng.xml Registers CosmosTestResourceJanitor listener for long-emulator suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/gsi-testng.xml Registers CosmosTestResourceJanitor listener for GSI suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/flaky-multi-master-testng.xml Registers CosmosTestResourceJanitor listener for flaky suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-thinclient-multi-region-testng.xml Registers CosmosTestResourceJanitor listener for FI thinclient multi-region suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-thinclient-multi-master-testng.xml Registers CosmosTestResourceJanitor listener for FI thinclient multi-master suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-sm-customer-workflows-testng.xml Registers CosmosTestResourceJanitor listener for FI SM customer workflows suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-multi-master-testng.xml Registers CosmosTestResourceJanitor listener for FI multi-master suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fi-customer-workflows-testng.xml Registers CosmosTestResourceJanitor listener for FI customer workflows suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/fast-testng.xml Registers CosmosTestResourceJanitor listener for fast suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/examples-testng.xml Registers CosmosTestResourceJanitor listener for examples suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/emulator-vnext-testng.xml Registers CosmosTestResourceJanitor listener for emulator-vnext suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/emulator-testng.xml Registers CosmosTestResourceJanitor listener for emulator suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/e2e-testng.xml Registers CosmosTestResourceJanitor listener for e2e suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/direct-testng.xml Registers CosmosTestResourceJanitor listener for direct suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/consistency-overrides-testng.xml Registers CosmosTestResourceJanitor listener for consistency-overrides suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/circuit-breaker-read-all-read-many-testng.xml Registers CosmosTestResourceJanitor listener for circuit-breaker suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/circuit-breaker-misc-gateway-testng.xml Registers CosmosTestResourceJanitor listener for circuit-breaker misc gateway suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/circuit-breaker-misc-direct-testng.xml Registers CosmosTestResourceJanitor listener for circuit-breaker misc direct suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/cfp-split-testng.xml Registers CosmosTestResourceJanitor listener for CFP split suite cleanup/leak-fail behavior.
sdk/cosmos/azure-cosmos-tests/src/test/resources/test-resource-hygiene-baseline.properties Adds the initial baseline for the hygiene ratchet detecting direct database creation.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/TestResourceHygieneTest.java Adds a unit-test ratchet to prevent introducing new direct database creation patterns.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/WorkloadIdE2ETests.java Switches DB ID creation to CosmosDatabaseForTest.generateId(...) for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/WorkloadIdDirectInterceptorTests.java Switches DB ID creation to CosmosDatabaseForTest.generateId(...) for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java Registers/unregisters created DBs/containers; introduces createTestDatabase(...) helpers; deprecates arbitrary ID creators.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ResourceTokenTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ReadFeedPermissionsTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ReadFeedOffersTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/PermissionQueryTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/OfferReadReplaceTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/OfferQueryTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/GatewayReadConsistencyStrategySpyWireTest.java Updates DB naming to run-scoped helper for cleanup safety.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/GatewayReadConsistencyStrategyE2ETest.java Updates DB naming to run-scoped helper for cleanup safety.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/DocumentCrudTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/changefeed/pkversion/IncrementalChangeFeedProcessorTest.java Replaces UUID-based DB IDs with run-scoped generated IDs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/changefeed/epkversion/IncrementalChangeFeedProcessorTest.java Replaces UUID-based DB IDs with run-scoped generated IDs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/ResourceTokenTestForV4.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/MaxRetryCountTests.java Replaces UUID-based DB IDs with run-scoped generated IDs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/directconnectivity/MetadataRequestRetryPolicyTests.java Replaces UUID-based DB IDs with run-scoped generated IDs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/FaultInjectionWithAvailabilityStrategyTestsBase.java Replaces UUID-based DB IDs with run-scoped generated IDs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/EndToEndTimeOutValidationTests.java Replaces UUID-based DB IDs with run-scoped generated IDs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestRunId.java Adds a stable per-job/per-run identifier for tagging DB IDs used by cleanup.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestResourceRegistry.java Adds a JVM-global registry of created DBs/containers for leak reporting and cleanup.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestResourceJanitorTest.java Adds unit tests to pin janitor delete ordering and leak classification behavior.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestResourceJanitor.java Adds in-process cleanup + leak-fail listener and shutdown hook for best-effort cleanup.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestAccountJanitor.java Adds CLI entry point invoked by pipelines to sweep current-run or old test DBs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosDiagnosticsTest.java Moves off DatabaseForTest.generateId() to CosmosDatabaseForTest.generateId() for cleanup attribution.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosDatabaseForTestTest.java Adds unit tests validating run-scoped vs age-based cleanup safety invariants.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosDatabaseForTest.java Updates ID format to include run id, adds run-scoped + age-based cleanup results, and makes timestamps UTC-safe.
sdk/cosmos/azure-cosmos-tests/pom.xml Configures exec-maven-plugin for running the CLI janitor main class from pipelines.
sdk/cosmos/AGENTS.md Documents Cosmos-specific agent guidance and the required test resource hygiene conventions.

Comment thread sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestResourceRegistry.java
Comment thread sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestResourceRegistry.java
Comment thread sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosTestRunId.java Outdated
Comment thread sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/TestResourceHygieneTest.java
tvaron3 and others added 7 commits July 30, 2026 16:41
@tvaron3
Merge upstream/main into cosmos-ci-resource-cleanup
27fb664 
Reconciles this change with Azure#49735, which moved the main and Http2 live-test
stages onto fixed self-owned accounts and reworked the shared-database
lifecycle.

- CosmosDatabaseForTest: upstream reduced this to a bare id generator (UTC
  timestamp + UUID) and dropped create()/DatabaseManager. Keep that shape and
  layer the run id back on, so ids stay four-segment and cleanup can still
  attribute a database to the run that created it. Legacy three-segment ids are
  still parsed.
- azure-cosmos DatabaseForTest: upstream deleted cleanupStaleTestDatabases and
  moved to UTC, which supersedes the NPE and threshold fixes here. Take
  upstream's version wholesale; this branch no longer touches the file.
- TestSuiteBase: adopt upstream's executeCreateWithRetry /
  createDatabaseWithRetry / createLegacyCollectionWithRetry wrappers and keep
  resource registration, moving registration inside the legacy wrappers so every
  caller is tracked rather than only the overloads touched here.
- Drop the in-run age-based sweep. Upstream removed it from afterSuite, and a
  test run should only ever delete its own resources; reclaiming other runs'
  orphans is the scheduled janitor's job, where the threshold cannot race an
  in-flight job.
- tests.yml: add the always-run cleanup post step to the two stages that
  Azure#49735 moved onto fixed accounts. Those accounts are never torn down, so they
  leak exactly like the thin-client and GSI ones - 8 stages now, not 6.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Validate test database ids at creation, and fix cspell
ced4ee3 
The static ratchet in TestResourceHygieneTest counts creation call sites per
file, so it cannot see three ways a database can still end up with a name CI
cleanup will never find:

- an id built at runtime rather than written literally,
- an id swapped inside one of the 56 files that already carry a baseline
  allowance, with the call count unchanged,
- creation through an API the scanner does not match on.

All three matter specifically when the JVM is killed - a cancelled or timed out
job - because the pipeline post step and the scheduled janitor locate databases
by name, so a wrongly named one is invisible to both and leaks permanently on a
shared account.

CosmosTestResourceRegistry now rejects, at registration time, any database id
that does not parse as a test id, failing the test immediately and naming the
offender. Containers are validated on their parent database, since deleting a
database reclaims them. Verified by reproducing all three vectors: the swapped
id inside a baselined file passes the static ratchet and is caught here.

Also add a cspell override for the two changed markdown files. The flagged
tokens are JVM system property flags (-DACCOUNT_HOST, -Dcodesnippet.skip,
-DCOSMOS...); those in dev.md are pre-existing and only surfaced because the
file is now part of the diff. Verified with the same command CI runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Address review feedback on the test resource registry and guard
7e7f2a3 
- registerDatabase / registerContainer used put, so re-registering an existing
  resource (createDatabaseIfNotExists, or a container recreated with the same
  id) reattributed it to whichever test touched it last, defeating the point of
  naming the creator in a leak report. Use putIfAbsent; a genuine
  delete-then-recreate still records the new owner because unregister removes
  the entry first.

- The run id hash was CRC32, and the comment claimed collisions were
  "structurally impossible". CRC32 is 32 bits, so that was wrong. Use a
  truncated SHA-256 (56 bits) and state the real guarantee: the job id makes the
  hash input unique per leg, and a collision is negligible rather than
  impossible. Id shape is unchanged - 20 chars, build id still readable.

- METHOD_DECLARATION required a leading modifier, so package-private
  declarations were counted as violations. Key the match on the
  <returnType> <name>(...) { shape instead, which a call never has. Regenerated
  violation counts are identical to the baseline, so nothing is newly skipped.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Isolate registry state between unit test classes
4266d9b 
CosmosDatabaseForTestTest asserted on the size of the JVM-global registry
snapshot, so it failed whenever CosmosTestResourceJanitorTest ran first and left
an entry behind. That is ordering dependent, which is why it passed locally and
on the macOS leg but failed on ubuntu2404_18.

Clear the registry before and after every method in both classes, and assert
that the specific database id is present rather than counting global entries.
Verified in both class orderings and against the full unit suite (2659 tests).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Name the test, not the helper, in leak reports
bff5282 
The first CI run with the janitor enabled reported leaks as "created by
com.azure.cosmos.rx.TestSuiteBase.createDatabaseInternal", which identifies the
shared helper rather than the test that leaked - useless for acting on the
report, and the report naming the offender is the whole point.

Skip the shared test infrastructure when walking the stack for an owner, and
fall back to an infrastructure frame (marked as such) only when no test frame is
on the stack at all.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Use an absolute link to the Cosmos AGENTS.md
fce3289 
Verify Links rejects relative links, and the anchored form was additionally
flagged as invalid format:

  DO NOT use relative link AGENTS.md#test-resource-hygiene-...
  'sdk/cosmos/dev.md' has 1 broken link(s)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3
Delete the databases three CosmosDiagnosticsTest tests created
6b26417 
The janitor's first CI run found these, which is what it is for:

  negativeE2ETimeoutWithPointOperation
  negativeE2ETimeoutWithQueryOperation
  responseStatisticRequestStartTimeUTCForDirectCall

Each creates a database and its finally block only closes the client, so the
database survives the run. Harmless on the emulator, but these tests also run
against the shared fixed accounts, where it is a permanent leak.

Delete the database before closing the client that owns it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Jul 31, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tvaron3
Shorten generated test database ids
a5bcca3 
ThroughputControlTests fails on this branch with

  ThroughputControlInitializationException
  caused by CosmosException 400, body "<h2>Bad Request - Invalid URL</h2>"

and the failure count tracked the length of the shared database id: 1 failure at
a 103 character id, 12 at 108. The same tests pass on main, where the id is 82
characters (build 6635933, 124 successful invocations), so this is caused by
this branch, not pre-existing.

The database id is not used only as a database name. Throughput control derives
a group id of <database>/<container>/<group>/<suffix>, base64 encodes it and
appends a UUID to form a control item id, so every character added here is
amplified. I could not pin the exact limit from the logs - the arithmetic says
main should already exceed 255 characters, yet main passes - so rather than
guess at a threshold, bring the id back well under the length that is known to
work.

  shared database id: 109 -> 71 characters
  worst case with a label:  81 characters

Achieved by replacing the UUID random suffix with 8 alphanumerics, capping the
run id at 16 characters and labels at 10, and dropping the label from the shared
database. Uniqueness is unaffected: the timestamp and run id already scope the
id, and 36^8 random values sit underneath. Both limits are pinned by assertions
so this cannot silently regrow.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tvaron3

tvaron3 commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

/azp run java - cosmos - tests

@azure-pipelines

azure-pipelines Bot commented Jul 31, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tvaron3
Fix the cleanup post step, and the leak it found on the live accounts
5b5a0f3 
First live run of this change (build 6646150): 68 of 76 legs green, including
every thin client, GSI, Spring and Spark leg. Two real problems in the other 8.

1. The always-run cleanup post step never deleted anything. It failed with

     Could not find artifact com.azure:azure-cosmos:jar:4.82.0-beta.1

   because it did not pass DefaultOptions, so Maven resolved against the agent's
   default ~/.m2 instead of $(MAVEN_CACHE_FOLDER) where the built jar lives.
   continueOnError masked it as SucceededWithIssues. Pass DefaultOptions in both
   cleanup-test-resources.yml and janitor.yml, and align janitor.yml's build step
   so install and exec:java share one local repository.

   This is the layer that covers a dead JVM, so it was the least visible and the
   most important to fix.

2. PermissionCrudTest created a database in @BeforeClass and only closed the
   client in @afterclass, leaking it on every "fast" leg - 6 of the 8 failures,
   all attributed to before_PermissionCrudTest by the janitor. Delete it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@kirankumarkolli kirankumarkolli Awaiting requested review from kirankumarkolli kirankumarkolli is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Cosmos

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tvaron3