elastic-cloud-alternative

An index migration tool for Elasticsearch, and a fair reading of what you give up by leaving Elastic Cloud.

The tool does three things in order: export mappings and settings, copy documents by scroll and bulk with retries, then compare the two clusters — document counts, field mappings, sampled _source bodies, and the ranked output of a set of relevance probes you write yourself.

The target used in every example here is a free Elasticsearch 8.12 instance, but nothing in the tooling assumes it — any 8.x endpoint works.

That last part is the reason this repo exists. Document counts matching is a weak signal. An index can hold every one of your documents and still return them in a different order, because an analyzer did not survive the trip.

---

The checklist

Work down it. Nothing later is meaningful if something earlier failed.

• Baseline the source. Run the relevance probes against the cluster you are leaving and keep the output. You cannot detect a regression without a pre-migration number.
• plan — see which indices exist, how many documents each holds, and whether the target already has an index by that name.
• mappings — export mappings, settings and aliases to disk, then read the settings file. This is where custom analyzers, ILM policy references and plugin-dependent tokenizers are visible.
• create — build the indices on the target cluster from those files, with cluster-specific settings stripped.
• copy — scroll the source, bulk into the target, retrying on 429 and 5xx.
• es_verify.py — counts, mapping diff, sampled documents, and the probes again. Compare against the baseline.
• Only then, repoint an application. Read traffic first, writes after.
• Keep migration-state.json until the source is decommissioned. It is what rollback reads.

pip install requests

python3 tools/es_migrate.py plan
python3 tools/es_migrate.py mappings --index jobs-2026 --out mappings/
python3 tools/es_migrate.py create   --index jobs-2026 --from mappings/ --shards 1 --replicas 0
python3 tools/es_migrate.py copy     --index jobs-2026
python3 tools/es_verify.py --index jobs-2026 --queries examples/queries.json

Rollback, at any point:

python3 tools/es_migrate.py rollback --yes    # deletes only indices this tool created

Details that decide whether this works

Documents keep their source _id. Bulk actions are index with an explicit id, so re-running copy after a failure overwrites rather than duplicates. If the tool generated ids instead, every retry would inflate the index and every count comparison afterwards would be meaningless.

Settings are sanitised, not replayed. index.uuid, index.creation_date, index.provided_name, index.version.*, index.history_uuid and the index.resize.* family are read-only or cluster-specific; posting them back gets you a 400. Shard and replica counts are set from flags, because the right answer on the target is rarely the answer that was right on the source. index.lifecycle.* is dropped unless you pass --keep-ilm, since a policy reference that points at a policy the target does not have is a slow-motion failure rather than an immediate one.

_reindex from a remote source is not used. It is the obvious tool and it requires reindex.remote.whitelist on the target cluster, which you cannot set on most managed services. Scroll-and-bulk runs entirely from your machine and needs nothing but HTTP access to both ends — which is all a hosted endpoint gives you.

Retries are bounded and specific. 429, 502, 503 and 504 back off exponentially with jitter. A 400 does not retry — a mapping conflict is not a transient condition, and hammering it just delays the moment you read the error. Per-document rejections from _bulk are collected and printed with the document id and reason.

What the verifier actually checks

python3 tools/es_verify.py --index jobs-2026 --queries examples/queries.json --k 10 --sample 50

Pass	Question it answers
_count on both sides	did every document arrive
flattened mapping diff	is any field missing, extra, or a different type — including .keyword sub-fields
random _source sample	did the documents arrive intact, or did a field get dropped at index time
relevance probes	do the same queries return the same documents, in the same order

The probe file is JSON: a name, a note about what failure it catches, and a search body. Eight are supplied in examples/queries.json covering match, match_phrase, boosted multi_match, a filtered bool, terms on a keyword field, fuzzy, an aggregation, and geo_distance.

On scores

BM25 statistics are computed per shard. The same documents in a 3-shard index and a single-shard instance will score differently, and deleted documents still contribute until the segments merge. So the verifier asserts on ranking, reports score deltas as information, and offers --dfs to add search_type=dfs_query_then_fetch on both sides when you want the numbers themselves to be comparable.

The failure worth staring at is same document set, different order. That is almost always an analyzer that did not come across, or a text field that arrived as a keyword.

---

What Elastic Cloud does that a single node does not

Last verified: 2026-08-18. Elastic's tiering changes; follow the links.

	Elastic Cloud	A single free Elasticsearch 8.12 instance
Free tier	None. A 14-day trial, then a subscription	n/a
Kibana	Included and operated	Point your own Kibana at the endpoint
ILM / hot-warm-cold	Available across Cloud tiers, per the Cloud feature matrix	ILM exists in the free Basic distribution, but with one node there is nowhere to roll data to
Cross-cluster search and replication	Gold tier and above in the Cloud matrix	Needs a second cluster you also operate
Machine learning / anomaly detection	Part of the Cloud subscription tiers	Present in the stack, but you supply the nodes to run it
Multi-node clusters, replicas, failover	Managed, resizable	One node, number_of_replicas: 0
Frozen tier / searchable snapshots	Platinum and Enterprise per the matrix	No

Said plainly:

• Multi-node is the product. Elasticsearch's design assumptions — replication, shard rebalancing, hot-warm-cold movement — only mean anything with more than one node. A free single instance is a fine search index and a poor cluster, and pretending otherwise helps nobody.
• ILM is genuinely load-bearing for logs. If you are rolling time-series indices through tiers and out to a snapshot repository, that machinery is operational value you are paying for, and re-creating it is a project.
• Kibana is more than a UI. Discover, Lens, alerting rules and the ML jobs built on top of them are how a lot of teams actually use Elasticsearch. Nothing here replaces them.
• The self-managed stack is free. If cost is the driver and you have somewhere to run it, downloading the Basic distribution and operating it yourself is a legitimate answer that this repo's tooling also serves.

Do not migrate if any of these are true

• Your retention story is ILM. Rollover, shrink, force-merge, freeze, delete — that pipeline needs tiers, and tiers need nodes. A single instance has no cold tier for data to move to.
• You run anomaly detection or the relevance engine features. Those are subscription features. There is no export in this repo that recreates a trained ML job somewhere else.
• Kibana dashboards are a deliverable. If people outside the engineering team open Kibana daily, migrating the index is the small half of the job.
• Your index is large and hot. Scroll-and-bulk over the public internet is fine for tens of millions of documents and unpleasant beyond that. Snapshot and restore between clusters that share a repository is the grown-up path, and it is not what this tool does.

A search index is usually rebuildable from a system of record, which makes search migrations less frightening than database migrations. If that is true for you, reindexing from source may beat copying — and the verifier in this repo is still the right way to prove the result matches.

Elasticsearch on freebase.cloud

Create a session at freebase.cloud and pick the Elasticsearch engine; see the free Elasticsearch instance page.

Elasticsearch 8.12.0 over the standard REST API, so the official clients for JavaScript, Python, Java, Go and .NET connect without changes, and the tooling here is plain requests against documented endpoints.

curl "$TARGET_ES_URL/_cluster/health?pretty"

Query DSL, BM25 scoring, custom analyzers and token filters, the aggregations framework, geo_point with geo_distance, and kNN vector search over dense_vector fields all behave as documented. Access is over HTTP rather than a raw 9200 TCP socket you administer.

The free tier is intended for development, prototyping and small production search workloads — a product catalogue, a docs site, an internal search box. It is not a log-analytics cluster.

The same index over MCP

Settings → MCP → New Token in the freebase.cloud console, choose the connection, copy the URL. For VS Code and GitHub Copilot Chat, .vscode/mcp.json — note the top-level key is servers, not mcpServers:

{
  "inputs": [
    { "type": "promptString", "id": "freebase-token", "description": "freebase.cloud MCP token", "password": true }
  ],
  "servers": {
    "search": { "type": "http", "url": "https://freebase.cloud/api/mcp/${input:freebase-token}" }
  }
}

MCP: Add Server in the Command Palette does the same thing interactively. Using an inputs entry keeps the token out of the committed file.

The Elasticsearch connection exposes four tools:

Tool	What you would use it for
search_query	run a query DSL body and read the hits
search_store	index documents
search_list_tables	list indices
search_annotate_table	describe an index so a model knows jobs-2026 is postings, not applicants

Claude-specific setup: how to connect Claude to Elasticsearch. Transport is Streamable HTTP; the older HTTP+SSE transport is deprecated in the current MCP specification.

When something goes wrong

copy reports rejections with mapper_parsing_exception. The target mapping disagrees with the document. Usually a field that was dynamically mapped on the source as long and arrives as a string, or a date format that was inferred rather than declared. Fix the mapping, re-run copy — ids are stable, so it overwrites.

create fails with unknown setting [index.xyz]. A setting from a plugin or a newer version. Delete it from the exported JSON and note what you removed; that note belongs in your migration record.

Counts match but a probe returns nothing. Check the analyzer. GET /index/_analyze with the same text on both clusters, side by side, is the fastest way to see the difference. Custom analyzers and token filters run the same way on a hosted 8.12 node, so a mismatch here is normally something the mappings export did not carry.

total differs by a handful and nothing else does. Documents were written to the source after the scroll started. Scroll gives you a point-in-time view; new writes are not in it. Either freeze writes or plan a delta pass keyed on a timestamp field.

---

tools/es_migrate.py    plan · mappings · create · copy · rollback
tools/es_verify.py     counts · mapping diff · sampled documents · relevance probes
examples/queries.json          eight probes, each labelled with what it catches
examples/seed_jobs_index.sh    curl-only index + eight documents to test against

MIT licensed. Pull requests welcome — especially additional relevance probes.

freebase.cloud is an independent service and is not affiliated with Elasticsearch B.V., Anthropic, PBC or Microsoft Corporation.