Elasticsearch Search & Relevance Tuning: Mapping, Queries, Aggregations, and Performance

Why Elasticsearch Deserves Real Tuning

Elasticsearch's "works out of the box" experience is usually fine for a demo. But once you cross hundreds of millions of documents and thousands of QPS, an untuned mapping and queries will quickly sink the cluster: write stalls, query spikes, and heap OOM. Tuning is fundamentally about doing the expensive work at index time and only the necessary work at query time.

Dimension	Untuned	Tuned
Field type	all `text` or all `keyword`	split precisely by retrieve/aggregate need
Filter conditions	inside `must`	inside `filter` context
Shard count	arbitrary 5/10	derived from data size and nodes
Bulk writes	single `_doc`	`bulk` with sensible batch
Relevance	default BM25	combined with boost / business weight

Tokenization and Analyzer: The Root of Retrieval Quality

The analyzer decides whether "what the user searches" matches "what is in the document." Chinese especially needs the right tokenizer.

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_cn": {
          "tokenizer": "ik_max_word",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "my_cn" },
      "status": { "type": "keyword" }
    }
  }
}

When debugging mappings and query DSL, run the request body through the JSON Formatter tool first to avoid losing track of deeply nested objects.

text vs keyword: Retrieval vs Aggregation

text: analyzed (tokenized), good for "full-text search," but bad for exact match and aggregation.
keyword: not analyzed, good for "exact match, sorting, terms aggregation," but cannot do full-text search.

The common approach is multi-fields to keep both:

{
  "mappings": {
    "properties": {
      "city": {
        "type": "text",
        "fields": {
          "raw": { "type": "keyword" }
        }
      }
    }
  }
}

Search with city (tokenized), aggregate/sort with city.raw (exact).

bool Query: The filter vs query Chasm

This is the most commonly misused part. The filter context does not compute relevance scores and its results are cacheable; the query context does compute scores. Any "yes/no" condition (status, time range, category) belongs in filter.

{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "phone" } }
      ],
      "filter": [
        { "term": { "status": "on" } },
        { "range": { "price": { "gte": 100, "lte": 5000 } } }
      ]
    }
  }
}

The match in must contributes to scoring, while the term/range in filter are fast and reused via node-level cache.

Aggregations: Push Analytics Down to the Engine

Instead of pulling data back to the app layer to compute stats, use ES aggregations directly:

{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category.raw", "size": 10 }
    },
    "price_stats": {
      "stats": { "field": "price" }
    }
  }
}

size: 0 means no hit documents are returned, only aggregation results, which saves significant bandwidth.

Index Settings: Shards and Replicas

More shards is not better. Rules of thumb:

Keep a single shard between 10GB and 50GB.
Shard count ≈ total data / target shard size.
Replica count = read-throughput need / single-node capacity, at least 1 for HA.

{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "30s"
  }
}

Raising refresh_interval to 30s significantly reduces write amplification from near-real-time refresh (good for log use cases).

Bulk Writes and Caching

bulk, not single docs

curl -XPOST "localhost:9200/orders/_bulk" -H 'Content-Type: application/json' --data-binary @bulk.jsonl

Keep each batch around 5MB ~ 15MB; too large triggers queue buildup.

Use caching wisely

Node query cache: filter results cached per node.
Shard request cache: results of aggregations / zero-size requests cached.
Fielddata / doc_values: columnar storage for sort/aggregate; avoid aggregating text directly.

Relevance Tuning (BM25)

Default BM25 is decent, but you can layer business weight:

{
  "query": {
    "bool": {
      "should": [
        { "match": { "title": { "query": "phone", "boost": 3 } } },
        { "match": { "description": { "query": "phone", "boost": 1 } } }
      ]
    }
  }
}

A title hit weighs 3x a description hit. function_score can also introduce time decay, sales volume, etc.

For API auth (e.g. the Service Token writing to ES), use the JWT Decoder tool to quickly inspect expiry and permission claims in the payload.

FAQ

Q1: Why does terms aggregation error on a text field?

text fields are tokenized and cannot be aggregated exactly; use the .raw (keyword) sub-field.

Q2: Which is faster, filter or must?

filter is faster, does not score, and is cacheable — put any "hard condition" in filter first.

Q3: How many shards should I set?

Derive from total data; keep single shards 10–50GB to avoid runaway cluster metadata overhead.

Q4: How do I debug slow writes?

Check single-doc writes (switch to bulk), whether refresh_interval is too small, and whether replicas are excessive.

Q5: Chinese search quality is poor — what now?

Use a Chinese tokenizer like IK / pinyin and specify the matching analyzer for search fields in the mapping.

Recommended Tools

In Elasticsearch development, these tools help:

JSON Formatter — organize mapping and DSL request bodies
SQL Formatter — design ES queries by analogy with SQL
Base64 Encoder — handle doc IDs / binary payloads
JWT Decoder — inspect write-auth token claims

Elasticsearch performance is seven parts mapping and analyzer design before indexing, three parts filter/aggregation trade-offs at query time. Compute what should be computed early and cache what should be cached, and the cluster stays millisecond-fast even at hundreds of millions of documents.