Elasticsearch Essentials for Backend Developers

Elasticsearch is a search engine that stores JSON documents and answers two kinds of questions fast: which documents match a search, and what the matching data adds up to. The first is a query. The second is an aggregation, Elasticsearch's version of GROUP BY, COUNT and AVG. Both are fast because Elasticsearch builds an inverted index of every word ahead of time, so a search looks words up instead of scanning rows. WHERE title LIKE '%query%' works fine until users expect ranked results in under a second across millions of rows. This post covers the architecture and shard sizing, the queries you'll write most, aggregations in depth, and why Postgres should still hold the real data.

The architecture: cluster, node, index, shard, replica

A cluster is the whole collection of Elasticsearch servers working together: one node in development, many in production for fault tolerance. A node is a single server instance inside that cluster. Starting Elasticsearch starts a node. Nodes on the same machine can find each other on their own in development, but in production you tell each node where the others are (discovery.seed_hosts). An index is the top-level container for your data, roughly like a table in PostgreSQL. A blog_posts index holds all blog post documents. Because an index can grow too large for one machine, it's split into shards: smaller, independent pieces spread across nodes so indexing and searching run in parallel. Each shard can also have a replica, a copy stored on a different node, which protects against a node failing and lets searches be served from any copy.

How many shards should an index have?

Since Elasticsearch 7.0, a new index gets one primary shard and one replica by default, and for most small apps that's the right answer. The number of primary shards is fixed when the index is created. Changing it later means the split or shrink API, or a full reindex, so it's worth a minute of thought up front. Elastic's shard sizing guidance comes down to three points:

• –Aim for shards between 10GB and 50GB, and keep each shard under 200 million documents.
• –Don't over-shard. Every shard costs memory and cluster bookkeeping, and each shard runs a search on a single CPU thread. Searching a thousand 50MB shards is much slower than searching one 50GB shard holding the same data.
• –A replica is never placed on the same node as its primary. On a one-node dev cluster the replicas stay unassigned and cluster health shows yellow. That's expected, not broken.

To see how big your shards actually are, run GET _cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb in Kibana Dev Tools.

A document is the basic unit of data, stored as JSON: one blog post, one product. The mapping is Elasticsearch's version of a schema. It defines how each field is stored and searched. Elasticsearch can guess a mapping from the first document you insert (dynamic mapping), but relying on that in production is a classic mistake. Every string becomes text with a keyword sub-field, whole numbers become long, a date in an unusual format stays a string, and a number sent as a string, like "42", is mapped as text. Changing a field's type once data exists means reindexing.

The inverted index: why search is fast

A normal database index on a title column helps you find rows where the title equals something. An inverted index flips the question around: for every term, it stores the list of documents that contain it. Index three short posts and you get something like this:

• –authentication → posts 1, 3
• –nestjs → posts 1, 2
• –redis → posts 2, 3

A search for "nestjs authentication" reads two short lists and combines them instead of reading every document. The terms in those lists are whatever the analyzer produced (more on that below), which is why match and term behave so differently. Aggregations and sorting use a second structure: doc values, a column-oriented copy of each field's values stored on disk. Doc values are on by default for keyword, numeric, date and boolean fields. Text fields don't have them, which is why you aggregate on author.keyword and not on author.

CRUD, but as a REST API

Every Elasticsearch operation is a plain HTTP request. PUT /blog_posts/_doc/1 with a JSON body creates or replaces a document with ID 1. POST /blog_posts/_doc without an ID lets Elasticsearch generate one. GET /blog_posts/_doc/1 retrieves it. POST /blog_posts/_update/1 with { "doc": { "views": 5300 } } patches just that one field, leaving everything else untouched. DELETE /blog_posts/_doc/1 removes it. If you've used any REST API before, this will feel familiar. There's no separate query language for basic CRUD. One catch on a fresh install: since 8.0, security is on by default, so outside Kibana these calls need HTTPS and a username and password (for example the elastic user), or they fail with an auth or certificate error.

// Create with a specific ID (replaces it if one already exists)
PUT /blog_posts/_doc/1
{
  "title": "Getting Started with NestJS Authentication",
  "views": 5200,
  "status": "published"
}

// Let Elasticsearch generate the ID
POST /blog_posts/_doc
{ "title": "Another Post" }

// Read
GET /blog_posts/_doc/1

// Partial update - only touches the "views" field
POST /blog_posts/_update/1
{ "doc": { "views": 5300 } }

// Delete
DELETE /blog_posts/_doc/1

Searching: Query DSL

Query DSL is the JSON language for expressing searches. match is the standard full-text query: it analyzes your search term and finds documents containing those tokens, ranking better matches higher. multi_match runs the same search across several fields at once (title, content, tags). term is the opposite of match. It does no analysis and looks for the exact stored value, which is what you want for IDs, status codes, or enums. Mixing these up (using match on a keyword field, or term on a text field) is one of the easiest Elasticsearch mistakes to make. range handles numeric and date comparisons, like views between 1000 and 10000.

To combine conditions, the bool query has four clauses: must (has to match, adds to the relevance score), should (adds to the score if it matches), filter (has to match, no scoring, can be cached) and must_not (must not match, no scoring, can be cached). One rule catches people out: if a bool query has should clauses but no must or filter, at least one should clause has to match. Add a must or filter, and should becomes optional and only nudges the ranking, unless you set minimum_should_match. A query for "published NestJS posts about authentication with 500+ views" puts the authentication match in must and the tags, status and views conditions in filter. That split matters for both correctness and speed.

GET /blog_posts/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "content": "authentication" } }
      ],
      "filter": [
        { "term": { "tags.keyword": "nestjs" } },
        { "term": { "status.keyword": "published" } },
        { "range": { "views": { "gte": 500 } } }
      ]
    }
  }
}

All four bool clauses in one query

A fuller example: posts about authentication, ranked higher if the title mentions JWT or sessions, published in the last year, and never tagged deprecated.

GET /blog_posts/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "content": "authentication" } }
      ],
      "should": [
        { "match": { "title": "jwt" } },
        { "match": { "title": "session" } }
      ],
      "filter": [
        { "term": { "status.keyword": "published" } },
        { "range": { "publishedAt": { "gte": "now-1y/d" } } }
      ],
      "must_not": [
        { "term": { "tags.keyword": "deprecated" } }
      ]
    }
  }
}

Only must and should change the order of results here. filter and must_not just decide what's in or out, so Elasticsearch skips scoring them. If every hit comes back with a score of 0, everything ended up in filter or must_not. That's fine for a plain listing page, but not for a search box.

Analyzers — what actually happens to your text

When you index a text field, Elasticsearch keeps the original JSON in _source (that's what comes back in results), but what it actually searches is a list of tokens. It runs the text through an analyzer, which breaks it into tokens and transforms them. The same process runs on your search query, so the two sides can match. With the default standard analyzer, "Getting Started with NestJS Authentication" is split into words and lowercased: ["getting", "started", "with", "nestjs", "authentication"]. Add a stop filter, like the one in the custom analyzer below, and "with" is dropped too. That's why searching "NestJS", "nestjs", or "NESTJS" all return the same document. The english analyzer goes further and stems words ("running" becomes "run"), so a search for "run" also matches documents that only contain "running".

PUT /blog_posts
{
  "settings": {
    "analysis": {
      "analyzer": {
        "post_title_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "english_stemmer"]
        }
      },
      "filter": {
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "post_title_analyzer" }
    }
  }
}

Filters vs. queries — the most important performance habit

A query context asks "how well does this document match?" and contributes to the relevance score. A filter context asks a plain yes/no question (does this match at all?) and never touches the score. The practical rule: use filter inside a bool query for exact-value conditions like status, tags, and date or numeric ranges. Save must and should for the parts that need relevance scoring, like a free-text search term. Filters skip scoring, and Elasticsearch caches the ones you use often, so a status: "published" filter reused across thousands of searches stays cheap. Putting everything into must throws that away for no benefit.

Elasticsearch aggregations: GROUP BY for search data

Aggregations compute summary statistics over your data, the same job GROUP BY, COUNT, and AVG do in SQL. A terms aggregation on author.keyword with size: 0 (meaning "don't return documents, just the aggregation") returns a count of posts per author in one request. By default you get the top 10 authors, and counts can be approximate across shards. Set size inside terms for more, or use a composite aggregation to page through all of them. This is what powers faceted search UIs, like the sidebar showing "Category (42), Brand (17)" next to a search box, computed live from the same index you're searching.

GET /blog_posts/_search
{
  "size": 0,
  "aggs": {
    "posts_per_author": {
      "terms": { "field": "author.keyword" }
    }
  }
}

Bucket, metric and pipeline aggregations

Elastic sorts aggregations into three categories, and most real requests nest one inside another:

• –Bucket aggregations group documents: terms (one bucket per distinct value), date_histogram (one bucket per day, week or month), range and filters. They're the GROUP BY.
• –Metric aggregations calculate a number from field values: avg, sum, min, max, stats and cardinality (an approximate distinct count). They're the COUNT, SUM and AVG. Put one inside a bucket aggregation and you get that number per bucket.
• –Pipeline aggregations work on the output of other aggregations instead of on documents: avg_bucket, max_bucket, derivative, cumulative_sum and bucket_sort. They point at their input with buckets_path.

A working example: posts per month, average views, top authors

This request answers three questions about published posts from August to October in one round trip: how many posts went out each month and their average views (a metric inside a bucket), who the top three authors are, and the average number of posts per month (a pipeline over the monthly buckets). The query runs first, and the aggregations only see the documents it matched.

GET /blog_posts/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "term": { "status.keyword": "published" } },
        { "range": { "publishedAt": { "gte": "2026-08-01", "lt": "2026-11-01" } } }
      ]
    }
  },
  "aggs": {
    "posts_per_month": {
      "date_histogram": {
        "field": "publishedAt",
        "calendar_interval": "month"
      },
      "aggs": {
        "avg_views": { "avg": { "field": "views" } }
      }
    },
    "top_authors": {
      "terms": { "field": "author.keyword", "size": 3 }
    },
    "avg_posts_per_month": {
      "avg_bucket": { "buckets_path": "posts_per_month>_count" }
    }
  }
}

The response, trimmed to the parts that matter (your numbers will differ):

{
  "took": 9,
  "timed_out": false,
  "hits": {
    "total": { "value": 53, "relation": "eq" },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "posts_per_month": {
      "buckets": [
        {
          "key_as_string": "2026-08-01T00:00:00.000Z",
          "key": 1785542400000,
          "doc_count": 18,
          "avg_views": { "value": 2311.5 }
        },
        {
          "key_as_string": "2026-09-01T00:00:00.000Z",
          "key": 1788220800000,
          "doc_count": 21,
          "avg_views": { "value": 1984.2 }
        },
        {
          "key_as_string": "2026-10-01T00:00:00.000Z",
          "key": 1790812800000,
          "doc_count": 14,
          "avg_views": { "value": 1207.0 }
        }
      ]
    },
    "top_authors": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 4,
      "buckets": [
        { "key": "prashant", "doc_count": 30 },
        { "key": "asha", "doc_count": 12 },
        { "key": "ravi", "doc_count": 7 }
      ]
    },
    "avg_posts_per_month": { "value": 17.666666666666668 }
  }
}

A few things to notice. hits is empty because the request set size: 0. Each date_histogram bucket has a key (the bucket's start time in epoch milliseconds), a readable key_as_string, a doc_count and the nested avg_views result. sum_other_doc_count: 4 means four matching posts belong to authors outside the top three. calendar_interval takes calendar units like month, week or day. fixed_interval takes fixed lengths like 30d or 12h, and can't express a month because months aren't all the same length.

Size and accuracy: the caveats that bite

An aggregation request has two different sizes. The top-level size: 0 controls how many documents come back. The size inside terms controls how many buckets come back, and it defaults to 10. Then there's accuracy:

• –Each shard picks its own top terms, and the node coordinating the search merges them. To catch terms that are just below the cut on some shards, each shard returns shard_size candidates, which defaults to size × 1.5 + 10. Counts can still be approximate on an index with more than one shard. Raising shard_size is much cheaper than raising size.
• –sum_other_doc_count is the number of matching documents in buckets that didn't make the top N. Set show_term_doc_count_error: true to get doc_count_error_upper_bound on each bucket, the worst-case number of documents missing from that count.
• –If you need every bucket rather than the top N, use a composite aggregation and page through it by passing each response's after_key back as after, instead of setting size to a huge number.
• –cardinality is approximate by design. Fine for "about 12,000 unique visitors", not for billing.

GET /blog_posts/_search
{
  "size": 0,
  "aggs": {
    "all_authors": {
      "composite": {
        "size": 100,
        "sources": [
          { "author": { "terms": { "field": "author.keyword" } } }
        ]
      }
    }
  }
}

Why an aggregation on a text field fails

Run a terms aggregation on a plain text field and Elasticsearch returns an error saying fielddata is disabled on that field. Text fields are split into tokens and have no doc values, so there's nothing sensible to group by. The fix is almost always the .keyword sub-field (author.keyword instead of author) or an explicit keyword mapping. Setting fielddata: true on the text field makes the error go away, but it loads the tokens into heap memory and groups by individual words, so "Jane Doe" becomes two buckets, jane and doe.

Relevance scoring and boosting

Every hit gets a relevance score based on how well it matched. You can tune this: multi_match with fields: ["title^3", "tags^2", "content"] means a match in the title counts three times as much as a plain content match, and a tag match counts twice as much — which is why a document with your search term in the title correctly outranks one where the term only appears once, buried in the body.

Pagination: why from/size breaks down at scale

from and size work fine for the first few pages — from: 0, size: 10 for page one, from: 10 for page two. But at a high from value, Elasticsearch still has to fetch and discard every earlier result internally before returning your page, which gets expensive fast. For deep pagination, search_after is the efficient alternative: each response includes sort values on its last hit, and you pass those values into the next request to fetch the following page directly, with no wasted work regardless of how deep you go.

Index design habits worth adopting early

• –Define explicit mappings before inserting real data — don't rely on dynamic inference.
• –Use keyword type for anything you'll sort, filter exactly, or aggregate on — status, tags, IDs.
• –For fields that need both full-text search and exact filtering, define text with a .keyword sub-field, and use whichever side fits each query.
• –Don't store large content blobs in Elasticsearch — keep only searchable fields there, and fetch the full record from PostgreSQL by ID once you have a match.

Where Elasticsearch fits alongside PostgreSQL and Redis

Elasticsearch doesn't support ACID transactions (the all-or-nothing guarantees a normal database gives you) and isn't built for complex relational joins, so it should never be your only copy of important data. The pattern that works in production: PostgreSQL stores the canonical record, Elasticsearch indexes just the searchable fields whenever a record is created or updated (usually via a background job), and Redis caches popular or repeated search results. A search request hits Elasticsearch, gets back a ranked list of IDs, and the full records come from PostgreSQL or Redis. Each tool does the part it's good at.

A note on licensing: in 2021 Elastic moved Elasticsearch off Apache 2.0, and AWS forked it as OpenSearch. In 2024 Elastic added AGPLv3 as a license option, so Elasticsearch is open source again. The basics in this post work on both, but the two have drifted apart in newer features.

Common questions

What is the difference between bucket and metric aggregations?

A bucket aggregation groups documents (per author, per month). A metric aggregation calculates a number (an average, a sum, a count of distinct values) from field values. You usually nest a metric inside a bucket to get something like average views per month.

Why are terms aggregation counts approximate?

Each shard sends back only its own top terms, so a term that falls just below the cut on some shards loses part of its count. A single-shard index gives exact counts. With more shards, raise shard_size or switch to a composite aggregation.

What's the difference between must and filter in a bool query?

Both have to match. must adds to the relevance score, while filter skips scoring and can be cached. Use filter for yes/no conditions like status, tags and date ranges.

How many shards should an Elasticsearch index have?

Start with the default of one primary shard, and add more only when a shard is heading past roughly 50GB or 200 million documents. Too many small shards is a more common problem than too few.

Should I use Elasticsearch or PostgreSQL full-text search?

If your data already lives in PostgreSQL and search is a side feature, Postgres full-text search with a GIN index is often enough, and it's one less system to run. Reach for Elasticsearch when you need fine relevance tuning, custom analyzers, fuzzy matching, facets over large datasets, or log-sized volumes. The PostgreSQL essentials post covers the indexes on the Postgres side, and a BullMQ background job is how I keep the two in sync.

Where to go from here

Start with match, term and a simple bool query with filter. Aggregations and custom analyzers can wait until you have a real search feature that needs them. This is the last post in the series, and the overview shows how all five tools fit together. I'm a frontend engineer who builds the UI on top of APIs like this. If you need that, see my work or get in touch.