Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/cli/changelog/cmd-bundle.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@ Supply filter flags directly when you don't have a profile configured or need a

Exactly one of the following filter flags is required:

- `--all` — include every changelog in the directory
- `--input-products` — match by product, target date, and lifecycle (e.g. `"elasticsearch * *"`)
- `--prs` — filter by PR URLs or a newline-delimited file of PR URLs
- `--issues` — filter by issue URLs or a newline-delimited file of issue URLs
- `--all` — include every changelog in the directory (local sourcing only; on the CDN this errors and asks for `--force-local`)
- `--input-products` — match by product, target date, and lifecycle (e.g. `"elasticsearch * *"`). On the CDN this also requires a PR, issue, or file identity; product-only CDN runs error and ask for `--force-local`.
- `--prs` — filter by PR URLs or a newline-delimited file of PR URLs. An entry matches when **leading filename digits** equal the PR number **or** YAML `prs:` contains it (the same join git-ref uses). Empty `prs:` is fine when the filename carries the PR (for example `12345.yaml` after public scrubbing).
- `--issues` — filter by issue URLs or a newline-delimited file of issue URLs (YAML `issues:` on the downloaded pool)
- `--release-version` — fetch PR references from a GitHub release tag (e.g. `v9.2.0` or `latest`)
- `--report` — filter by PRs referenced in a promotion report (URL or local file)
- `--files` — include specific changelog YAML paths, or a newline-delimited path list file
- `--files` — include specific changelog YAML paths, or a newline-delimited path list file. On the CDN this GETs those basenames only and does not read `registry.json`.
- `--start-git-ref` + `--end-git-ref` — derive the PR list from a git commit range (see [Commit-range mode](#git-ref-mode))

`--force-local` is not a filter. It forces local entry sourcing for the run (equivalent to `bundle.use_local_changelogs: true` without editing config) and is allowed in both option-based and profile-based modes.
Expand Down Expand Up @@ -408,7 +408,7 @@ In profile mode, pass the same path list as a positional argument:
docs-builder changelog bundle serverless-release 2026-07-07 ./docs/temp/changelog_files.txt
```

`--files` / path-list selection follows the standard entry-sourcing rules. When entries are sourced from the CDN (the default when `bundle.repo` resolves), the listed paths are matched to CDN pool entries by file name and do not need to exist locally — useful for private repositories whose entries exist only in S3 and whose public copies have PR/issue references scrubbed, so PR-based filters cannot match. With local sourcing (`--force-local`, `--directory`, or `bundle.use_local_changelogs`), the listed files are read from disk and must exist. In either mode, a listed entry that cannot be found fails the run, and `rules.bundle` still applies after selection.
`--files` / path-list selection follows the standard entry-sourcing rules. When entries are sourced from the CDN (the default when `bundle.repo` resolves), the listed paths are **GET by file name** and do not need to exist locally — the pool `registry.json` is not read. This is useful for private repositories whose entries exist only in S3. With local sourcing (`--force-local`, `--directory`, or `bundle.use_local_changelogs`), the listed files are read from disk and must exist. In either mode, a listed entry that cannot be found fails the run, and `rules.bundle` still applies after selection.

### Force local entry sourcing [changelog-bundle-force-local]

Expand Down
7 changes: 3 additions & 4 deletions docs/cli/changelog/cmd-upload.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,9 @@ s3://{bucket}/bundle/{product}/{filename} # --artifact-type bun
Changelog entries are written once under the authoring org/repo/branch. A bundle that applies to multiple products is uploaded to multiple keys — one per product.

The command writes YAML objects only — it never writes a `registry.json`. The public
`bundle/{product}/registry.json` manifests are produced exclusively by the scrubber Lambda,
reconciled from public bucket state on the S3 events each upload emits; the
`changelog/{org}/{repo}/{branch}/registry.json` pool manifests are legacy client-authored
objects that only older CLI versions still write. See
`bundle/{product}/registry.json` and `changelog/{org}/{repo}/{branch}/registry.json`
manifests are produced exclusively by the scrubber Lambda, reconciled from public
bucket state on the S3 events each upload emits. See
[Changelog bundle registry](/development/changelog-bundle-registry.md).

When several repositories publish bundles for the same shared product (for example `cloud-serverless`), use a `{repo}-{dateOrVersion}.yaml` bundle filename convention so they don't overwrite each other under `bundle/{product}/`.
Expand Down
2 changes: 1 addition & 1 deletion docs/data/release-notes/configure-ref.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ The authoring repo is resolved with the same precedence as `changelog upload`: `
Sourcing is decided per run:

- **Local folder.** Used when `bundle.use_local_changelogs: true`, when `--force-local` is passed, when `--directory` is passed, or when the authoring repo cannot be resolved. The folder must contain the changelog files.
- **CDN (default when a repo resolves).** Used when the authoring repo resolves, local sourcing is not forced, and a CDN base URL is configured (`DOCS_BUILDER_CHANGELOG_CDN`, defaulting to the public distribution). The command fetches `changelog/{org}/{repo}/{branch}/registry.json` and the entries it lists, then applies the bundle's own product/PR/issue/file filters to the downloaded set. Path-list / `--files` filters match pool entries by file name, so the listed paths do not need to exist locally.
- **CDN (default when a repo resolves).** Used when the authoring repo resolves, local sourcing is not forced, and a CDN base URL is configured (`DOCS_BUILDER_CHANGELOG_CDN`, defaulting to the public distribution). `--prs`, `--issues`, `--report`, `--release-version`, a URL list, and git-ref fetch `changelog/{org}/{repo}/{branch}/registry.json` and the entries it lists, then apply the same PR join locally and on the CDN: **filename-derived PR numbers or YAML `prs:`**. Path-list / `--files` GETs those pool objects by file name and does not read the registry. CDN `--all` and product-only filters (no PR, issue, or file identity) error; pass `--force-local` to read the local folder. Local `--all` is unchanged.

Use `--force-local` for uncommon ad hoc runs that need the local folder without editing `changelog.yml` — including path-list / `--files` runs that should read freshly authored files from disk instead of the CDN pool.

Expand Down
46 changes: 23 additions & 23 deletions docs/development/changelog-bundle-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ copies, no cross-repo file syncing.
flowchart LR
CI["Client CI<br/>(docs-actions)"] -->|"changelog upload<br/>(YAML objects only)"| Private["Private S3 bucket<br/>bundle/{product}/*.yaml<br/>changelog/{org}/{repo}/{branch}/*.yaml"]
Private -->|"s3:ObjectCreated / ObjectRemoved<br/>→ SQS"| Scrubber["Changelog scrubber<br/>Lambda"]
Scrubber -->|"scrub + copy/delete,<br/>then reconcile bundle registry.json<br/>+ shallow maps from public listing"| Public["Public S3 bucket<br/>+ CloudFront CDN<br/>(incl. registry.json)"]
Scrubber -->|"scrub + copy/delete,<br/>then reconcile bundle and pool registry.json<br/>+ shallow maps from public listing"| Public["Public S3 bucket<br/>+ CloudFront CDN<br/>(incl. registry.json)"]
Public -->|"reads via CDN"| Directive["{changelog} directive<br/>(cdn: mode)"]
```

Expand Down Expand Up @@ -60,21 +60,18 @@ Both indexes share this schema, serialized with `snake_case` keys.

### Ownership per tree [ownership-per-tree]

The two trees part ways on who writes the manifest
(the [2026-08-10 update on elastic/docs-eng-team#688](https://github.com/elastic/docs-eng-team/issues/688)
narrowed reconciliation to the bundle tree):
The two trees share a producer and differ only in what each listing records:

- **Bundle index** — `bundle/{product}/registry.json`, **public bucket only**, produced
exclusively by the scrubber Lambda's `BundleRegistryReconciler`. This is the manifest the
`{changelog}` directive and external CDN consumers enumerate, and the subject of the rest of
this page.
- **Changelog-entry index** — `changelog/{org}/{repo}/{branch}/registry.json`, a **legacy
client-authored pass-through**: the current `changelog upload` never writes one, but manifests
written by older CLI versions are still mirrored verbatim from the private bucket, because
[`changelog bundle` entry sourcing](#entry-sourcing) still enumerates a pool through its
manifest. It is *not* reconciled — its `producer` is null and its recorded `etag` is the old
pre-scrub private-object hash (consumers ignore it). It goes away entirely once release-note
discovery starts from PR lists (RFC [elastic/docs-eng-team#698](https://github.com/elastic/docs-eng-team/issues/698)).
exclusively by the scrubber Lambda's `BundleRegistryReconciler`. Each entry records a
`target` (version or date) so the `{changelog}` directive and external CDN consumers can
enumerate bundles. This is the subject of most of the rest of this page.
- **Changelog-entry index** — `changelog/{org}/{repo}/{branch}/registry.json`, **public
bucket only**, produced by the same reconciler from the public YAML listing. It is
**listing-only** (`target` is always null). `changelog bundle --prs` and git-ref still
download this listing first (the public CDN cannot `ListObjects`); `--files` / a path
list GETs named objects and does not read it. Upload never writes this file. Stale
leftover client JSON is healed on the next YAML or registry-key event.

```json
{
Expand Down Expand Up @@ -149,19 +146,19 @@ event's *type* — an event only means "this key may have changed":
*current* content and PUT to public; 404 → conditionally delete the public copy. After the
write, a HEAD re-validates that the private object still matches the snapshot the write was
derived from, redoing the reconcile if a concurrent invocation raced it.
2. **Group reconcile** (`bundle/{product}/` keys only — the pool tree has none) — list the
group's public prefix (paginated), reuse entries whose recorded ETag still matches the
listing, GET and recompute the rest (amends always recomputed), and write the manifest back.
2. **Group reconcile** (`bundle/{product}/` and `changelog/{org}/{repo}/{branch}/` keys) —
list the group's public prefix (paginated), reuse entries whose recorded ETag still
matches the listing, GET and recompute bundle targets (pool listings skip the YAML GET
and record a null `target`), and write the manifest back.
3. **Shallow-map reconcile** — patch the touched tree's
[folder→token map](#shallow-maps) from the same public listings.

Within an SQS batch this work is coalesced: one object reconcile per distinct key, one group
reconcile per distinct group, one shallow-map reconcile per touched tree.

Registry-key events split by tree. A **bundle** manifest is never copied or deleted — the event
only schedules the group reconcile, so client-authored JSON never reaches the tree consumers
enumerate. A **pool** manifest is mirrored verbatim (the
[legacy pass-through](#ownership-per-tree)). Any other `.json` key is skipped with a warning.
Registry-key events never copy or delete the JSON object — the event only schedules the
group reconcile, so client-authored JSON never reaches the tree consumers enumerate. Any
other `.json` key is skipped with a warning.

### Concurrency: optimistic, conditional writes

Expand Down Expand Up @@ -235,8 +232,11 @@ The `changelog bundle` command aggregates individual changelog **entries**. It c
entries from the local folder or fetch the **authoring pool's** published entries from the CDN
(`changelog/{org}/{repo}/{branch}/registry.json` → `changelog/{org}/{repo}/{branch}/{file}`, via
`CdnChangelogEntryFetcher`). The pool manifest it enumerates is the
[legacy client-authored index](#ownership-per-tree); this enumeration is what keeps the
pass-through alive until PR-list-driven discovery (RFC elastic/docs-eng-team#698) replaces it.
[Lambda-owned listing](#ownership-per-tree). `--prs` and git-ref download that listing, then
match by **filename-derived PR numbers or YAML `prs:`** (the same join git-ref uses). `--files`
/ a path list GETs those basenames only and does not read the registry. CDN `--all` and
product-only filters (no PR, issue, or file identity) are not supported yet — pass
`--force-local`. Local `--all` is unchanged.

Under the artifact-root layout, entries are org/repo/branch-scoped — not product-scoped — so CDN
entry sourcing keys off the resolvable authoring pool (repo with the same precedence as upload:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@ namespace Elastic.Documentation.Configuration.ReleaseNotes;
/// <summary>
/// Fetches the individual (scrubbed) changelog entries for a single authoring org/repo/branch pool from
/// the public CDN, for the <c>changelog bundle</c> command when sourcing entries from S3 rather than a
/// local folder. It reads <c>{base}/changelog/{org}/{repo}/{branch}/registry.json</c> to enumerate entries
/// and downloads each <c>{base}/changelog/{org}/{repo}/{branch}/{file}</c> as raw YAML; the bundle command
/// then applies its usual filter (products / prs / issues) to the downloaded set.
/// local folder. <see cref="FetchAsync"/> enumerates via the pool <c>registry.json</c> then downloads
/// each listed YAML. <see cref="FetchNamedAsync"/> GETs requested basenames only (used by
/// <c>--files</c> / a path list) and does not read the registry. The bundle command then applies its
/// usual filter (products / prs / issues) to the downloaded set, except <c>--files</c> which includes
/// every fetched name.
/// </summary>
/// <remarks>
/// <para>
Expand All @@ -39,6 +41,7 @@ public sealed class CdnChangelogEntryFetcher : IDisposable
private const int DefaultMaxAttempts = 4;
private const int BaseRetryDelayMs = 500;
private const int MaxRetryDelayMs = 2000;
private const int MaxParallelNamedReads = 4;

/// <summary>
/// Bounds an individual registry/entry HTTP request so a stalled CDN connection cannot hang a bundle run.
Expand Down Expand Up @@ -178,6 +181,79 @@ public async Task<IReadOnlyList<CdnChangelogEntry>> FetchAsync(
return entries;
}

/// <summary>
/// Downloads only the named changelog entries from the authoring pool, without reading
/// <c>registry.json</c>. Used by <c>changelog bundle --files</c> / a path list so a stale or
/// missing pool listing cannot hide a requested object. A requested name that 404s after the
/// retry budget is a hard error (fail-fast, same as a missing local file).
/// </summary>
public async Task<IReadOnlyList<CdnChangelogEntry>> FetchNamedAsync(
Uri baseUri,
string org,
string repo,
string branch,
IReadOnlyList<string> fileNames,
Action<string> emitError,
Action<string> emitWarning,
Cancel ctx)
{
_ = emitWarning;
var poolLabel = $"{org}/{repo}/{branch}";

if (!ChangelogKeys.IsValidOrg(org) || !ChangelogKeys.IsValidRepo(repo) || !ChangelogKeys.IsValidBranch(branch))
{
emitError(
$"Invalid changelog pool '{poolLabel}': the org, repo, and each '/'-delimited branch segment must be non-empty ASCII letters, digits, '.', '_' or '-' (org allows only letters, digits and '-') and must not be '.' or '..'.");
return [];
}

var poolSegments = ChangelogKeys.PoolSegments(org, repo, branch);
var built = new CdnChangelogEntry?[fileNames.Count];
var errors = new string?[fileNames.Count];

await Parallel.ForEachAsync(
Enumerable.Range(0, fileNames.Count),
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelNamedReads, CancellationToken = ctx },
async (i, ct) =>
{
var fileName = fileNames[i];
if (!ChangelogKeys.IsSafeFileName(fileName))
{
errors[i] =
$"Changelog entry '{fileName}' for '{poolLabel}' is not a valid pool file name.";
return;
}

var entryUri = CombineSegments(baseUri, [.. poolSegments, fileName]);
var (fetched, content, lastError) = await TryFetchEntryAsync(entryUri, fileName, poolLabel, ct).ConfigureAwait(false);
if (fetched)
{
built[i] = new CdnChangelogEntry(fileName, content);
return;
}

errors[i] =
$"Changelog entry '{fileName}' for '{poolLabel}' could not be fetched from {entryUri} after {_maxAttempts} attempt(s): {lastError}. " +
"Ensure the entry was uploaded (changelog upload), or pass --force-local / --directory to bundle local files instead.";
}).ConfigureAwait(false);

var failed = false;
for (var i = 0; i < errors.Length; i++)
{
if (errors[i] is not { } message)
continue;
emitError(message);
failed = true;
}

if (failed)
return [];

var entries = built.Where(e => e is not null).Select(e => e!.Value).ToList();
_logger.LogInformation("Fetched {Count} named changelog entry(ies) for {Pool} from {BaseUri}", entries.Count, poolLabel, baseUri);
return entries;
}

/// <summary>
/// Fetches a single entry, retrying transient failures (most importantly a not-yet-propagated 404)
/// up to <see cref="_maxAttempts"/> times with exponential backoff. Retry requests are cache-busted
Expand Down
Loading
Loading