Elasticsearch and search databases

Search is a different problem from lookup. People type a few words, spell some wrong, and expect the best matches first. A relational LIKE '%word%' cannot do that at scale. A search database can.

Crux: a search database is an index built for text, fed from your real database. Use it for relevance and filters, and keep the source of truth elsewhere.

What it is and why it exists

Elasticsearch (and its fork OpenSearch) is a distributed search engine built on Apache Lucene. You store documents (JSON) and query them by words, filters and relevance. Typical needs: product search, log search, "search posts", autocomplete.

A LIKE '%term%' query cannot use a normal index, so it scans every row. That is slow and gives no ranking.

How it works, at the level you must explain

The inverted index

At write time the text is cut into tokens (an analyzer lowercases, splits on word boundaries, drops very common words, and can reduce words to a stem, so "running" matches "run"). The engine then stores a map from each term to the list of documents that contain it.

At query time the terms are looked up, the lists are combined, and the matches are scored by relevance. The classic scoring idea is that a term counts more when it is rare across all documents and frequent in this one. The standard formula is BM25. You need the idea, not the formula.

Clusters, shards and refresh

• An index (the Elasticsearch word for a collection) is split into shards. Each shard is a self-contained Lucene index. A query goes to every shard and the results are merged.
• Each shard has replicas for failure and for extra read capacity.
• Writes are first held in memory and made searchable by a refresh, every second by default. So search is near real-time: a new document is not visible for about a second. It is not a read-your-own-write system.
• The number of primary shards is fixed when the index is created, so plan it. Changing it later means reindexing.
• Documents are immutable inside Lucene. An update writes a new version, and old ones are cleaned up by background merging.

Keeping it in step with the source of truth

Do not write to two stores from the app and hope. Pick one flow.

• Change data capture (CDC) or the database's change stream feeds a pipeline into the search index. DynamoDB streams and a Postgres log reader are examples.
• Publish events to Kafka and have an indexer consume them.
• Make indexing idempotent, keyed by the document id, so a replayed event is harmless.
• Keep a way to rebuild the index from the source of truth. That is your safety net, and it is the reason the index is allowed to be disposable.

Mappings and the main query types

A mapping says how each field is indexed. Choose the field type deliberately: text is analysed for full-text search, keyword is stored as one exact value (for filters, sorting and aggregations), and numbers, dates and geographic points have their own types. The same field can be indexed both ways. Index only the fields you will search or filter on. Every indexed field costs memory and disk, and leaving the rest out is the cheapest tuning there is.

• Match query: full-text search on a text field.
• Bool query: combine clauses with must, should, must-not and filter. Filter clauses skip scoring and are cached, so put exact conditions (category, price range, in stock) there.
• Range query: numbers and dates.
• Geo-distance query: "within 3 km of this point". Geographic points are held in a tree structure built for ranges over coordinates.
• Nested objects: when a list of sub-objects (reviews, each with an author and a rating) must match on fields together, you need the nested type. A plain object list flattens, so "author A and rating 5" can match across two different reviews. Often the simpler answer is to denormalise: store one document per thing you search for, so a query needs one or two calls and no join.

Sorting and pagination

By default results are sorted by relevance score. You can sort on a field instead (price, date). Sorting on a text field is not allowed, so use its keyword form.

Pagination has three options.

• from and size: easy, but each shard must build from + size results and the coordinator must merge them. Deep pages become slow and expensive, and the engine caps the window by default at 10,000 results.
• search_after: pass the sort values of the last hit you saw to get the next page. Cost stays flat. Pages can shift if documents change in between.
• Point in time (PIT): freeze a view of the index and combine it with search_after, so pagination sees one stable snapshot. Needed for exports and stable scrolling.

For a user-facing "load more", search_after is the right default. Do not offer "jump to page 4,000".

Inside a shard

• A shard is built from immutable segments. A refresh creates a new small segment, and background merges combine them into larger ones. A delete only marks a document, and an update is a delete plus an insert, which is why heavy updating is costly.
• A search runs in two phases. In the query phase, every shard finds its best matches and returns ids and scores. In the fetch phase, the coordinator asks the owning shards for the document bodies of the final page only.

• For sorting and aggregation the engine reads doc values, a column-style copy of each field, instead of walking the inverted index backwards.
• A request can go to any node. Dedicated master nodes manage cluster state, data nodes hold shards, and coordinating nodes route and merge. In a design you only say "a cluster with replicated shards".

When not to bring it in

For a few thousand or tens of thousands of rows that change rarely, a database index or Postgres full-text search is simpler. As a rough guide, I would not add a separate cluster below roughly a hundred thousand documents without another reason. Treat that as a heuristic, not a rule.

When to use it, trade-offs, numbers

• What: a distributed full-text search and filter engine.
• Use when: full-text search, typo tolerance, ranking, faceted filters, autocomplete, log search, and geo queries at moderate scale.
• Know: inverted index, analyzer, shards and replicas, near real-time refresh, and keeping it fed from another store.
• Careful: it is not a transactional database. Writes are eventually visible, updates are expensive, and a badly chosen shard count hurts for a long time.

Elasticsearch or something else?

• Postgres full-text search (GIN indexes): simpler, no second system, and often enough for a few million rows. Start here when search is not the product.
• Redis structures: exact lookups and leaderboards, not text relevance.
• A dedicated vector database or a vector feature in your search engine: for semantic similarity search. Name it only if the prompt needs it.
• Managed search services: the same idea with less to run.

Numbers: I am omitting shard-size recommendations and node sizing, since guidance changes by version and workload. A safe habit is to estimate the data size and queries per second first, then say you would size shards by testing.

Common mistakes

• Making Elasticsearch the primary database.
• Writing to both stores from the app, with no way to repair drift.
• Expecting a new document to be searchable instantly.
• Choosing a shard count by guess, with no thought for growth.
• Reaching for it when a database index or Postgres full-text would do.

• Indexing every field instead of the few that are searched.
• Using deep from and size paging.
• Using nested objects when a flat, denormalised document would do, or the reverse, flat lists where fields must match together.
• Updating documents constantly, which multiplies segment merges.

Check yourself

• What is an inverted index, and why is it faster than a scan?
• What does an analyzer do to the text before it is indexed?
• Why is Elasticsearch called near real-time?
• How would you keep a search index in step with your main database?
• What would you do if the search index were lost?
• When would you stay with Postgres full-text search?
• What is the difference between a text field and a keyword field?
• Why is deep paging with from and size expensive, and what replaces it?
• Why is an update more costly than an insert?
• Describe the two phases of a distributed search.