Vietnamese search in Elasticsearch: diacritics, n-grams and filter order

Accent-insensitive Vietnamese search: asciifolding, why stopwords must come before it, n-grams for partial codes, search_after paging and rebuildable indexes.

· Published · 7 min read · Versions: Elasticsearch 8.19

Vietnamese users type searches every which way: with diacritics, without them, half a word, part of a device code. A search box has to understand that "may sieu am" means "Máy siêu âm" (ultrasound machine), and that "SAMS" is part of the code "SA-SAMSUNG".

This post walks through the Elasticsearch setup for the equipment inventory in my hospital procurement system: analyzers, the query, deep pagination, rebuilding the index, and a filter-order mistake I only caught while reviewing the config for this post.

Removing diacritics with asciifolding

The asciifolding filter turns accented characters into ASCII: "Máy siêu âm" becomes may, sieu, am. It handles đ too, which becomes d. Use the same analysis when indexing and searching, and accented and unaccented input both match:

vietnamese_analyzer: {
  type: 'custom',
  tokenizer: 'standard',
  filter: ['lowercase', 'asciifolding', 'vietnamese_stop'],
},

The cost is losing the difference between words that differ only by their marks: "lá" (leaf), "là" (is) and "la" (to shout) become one. For equipment names that's almost harmless. If you want exact-accent matches ranked higher, keep an extra subfield without folding and boost matches on it.

Another limit of the standard tokenizer for Vietnamese: it splits on syllables, not words. "Máy siêu âm" is three tokens, not the two words "máy" and "siêu âm". For an equipment list, adding phrase-style matching (below) is enough. For real word segmentation there are community Vietnamese analysis plugins built on Cốc Cốc's tokenizer, at the cost of installing a plugin into the cluster.

Filter order: stopwords go before asciifolding

The config above also has a stopword filter with words like "và" (and), "của" (of), "cho" (for), "có" (have), "là" (is). It looks reasonable, but it never matches anything.

Filters run in order. By the time a token reaches vietnamese_stop, asciifolding has already turned "và" into va and "của" into cua. The stopword list is still accented, so every comparison misses. Nothing errors, search keeps working, the filter just does nothing.

The naive fix is to rewrite the list without accents. But once folded, co is both "có" (have) and "cơ" (muscle) or "cổ" (neck); la is both "là" and "lá"; cho is both "cho" (for) and "chó" (dog). Dropping co deletes real words. The right order is to remove stopwords while the accents are still there, then fold:

filter: ['lowercase', 'vietnamese_stop', 'asciifolding'],
The token Của through two filter orders. Old order: lowercase gives của, asciifolding gives cua, and vietnamese_stop keeps cua because the list has của, not cua. Fixed order: lowercase gives của, and vietnamese_stop removes it before asciifolding runs.Old orderthe list never matchesCủalowercasecủaasciifoldingcuavietnamese_stopcuakept: the list has“của”, not “cua”Fixed orderstopword removed firstCủalowercasecủavietnamese_stopremovedasciifolding seesonly real words
Filters run in order: an accented stopword list has to see the token before asciifolding strips the marks.

Two general lessons:

  • For any list-based filter (stopwords, synonyms), ask: what does the token look like by the time it reaches this filter?
  • Check analyzers with the _analyze API instead of guessing. One call shows which tokens survive:
POST /equipments/_analyze
{ "analyzer": "vietnamese_analyzer", "text": "Máy thở và monitor của ICU" }

Partial codes with n-grams

Device codes like "SA-SAMSUNG" or "ECG-FUKUDA" are often typed partially: "sams", "fuku". A regular analyzer only matches whole tokens, so add an n-gram subfield: at index time, each token is cut into every substring of 2 to 10 characters.

ngram_tokenizer: { type: 'ngram', min_gram: 2, max_gram: 10, token_chars: ['letter', 'digit'] },

modelNumber: {
  type: 'text',
  analyzer: 'vietnamese_analyzer',
  fields: {
    keyword: { type: 'keyword' },
    ngram: { type: 'text', analyzer: 'ngram_analyzer', search_analyzer: 'ngram_search' },
  },
},

The key detail is that search_analyzer differs from analyzer: indexing cuts n-grams, searching does not, it only lowercases and folds. Cut the query into n-grams too and "sams" becomes sa, am, ms... and matches a lot of unrelated things.

Three things to know about n-grams:

  • The index grows. A 10-character token produces 45 n-grams. Use them on a few short fields that need partial matching (names, codes), not on long descriptions.
  • A query token longer than max_gram won't match the n-gram field, since no n-gram is that long. That's one reason n-grams shouldn't be the only way to match.
  • By default Elasticsearch allows max_gram - min_gram of at most 1; a wider range needs a higher index.max_ngram_diff.

One query, three ways to match

The query combines three ways to match in one bool.should, and any one of them is enough:

should: [
  // 1. Every word, spread across fields: "siêu âm samsung"
  { multi_match: { query, type: 'cross_fields', operator: 'and',
      fields: ['equipmentName^3', 'modelNumber^2', 'manufacturer^2', 'unit', 'origin'] } },
  // 2. Last word half typed: "máy siêu â"
  { multi_match: { query, type: 'phrase_prefix',
      fields: ['equipmentName^3', 'modelNumber^2', 'manufacturer^2'] } },
  // 3. Part of a name or code: "sams"
  { multi_match: { query, operator: 'and',
      fields: ['equipmentName.ngram', 'modelNumber.ngram'] } },
],
minimum_should_match: 1,
  • cross_fields treats the fields as one big field, so "siêu âm samsung" matches even though "siêu âm" is in the name and "samsung" in the manufacturer.
  • phrase_prefix covers input that's still being typed, which suits search-as-you-type boxes.
  • The ^3, ^2 boosts rank matches on the name above matches on origin or unit.

Department and user-scope filtering goes in filter, not must: filter doesn't affect scoring and can be cached by Elasticsearch, which suits yes-or-no conditions.

Deep pagination with search_after

Paging with from + size is capped by index.max_result_window, 10,000 results by default, because each shard has to fetch and sort from + size documents. Past that, Elasticsearch returns an error.

The alternative is search_after: each page returns the sort values of its last hit, and the next page starts from there.

sort: [
  { [sortKey]: { order } },
  { id: { order: 'asc' } },   // tiebreaker, so the order is always deterministic
],
search_after: lastSort,       // from the previous page

Two details that are easy to miss:

  • Always end sort with a unique tiebreaker such as id. When several documents share a sort value, without one some can repeat or be skipped between pages.
  • search_after only moves forward: to reach page 3 you need page 2's cursor. In this project the UI remembers the cursor of each page it has visited, uses from for the early pages and switches to cursors further in. If users need to jump to a far page, the answer is usually a filter, not deeper pagination.

The database is the source of truth, the index is rebuildable

Equipment is written to MySQL first and pushed to Elasticsearch after the transaction commits. Dual writes like this are simple, but if the push fails the index falls behind. So you need a reindex script that rebuilds the whole index from the database. A few techniques from that script worth reusing:

  • Keyset pagination (WHERE id > lastId ORDER BY id LIMIT n) instead of OFFSET: each batch stays fast however deep into the table you are.
  • Preload small tables (departments, activities) into a Map to add their names to documents, instead of joining per batch.
  • Turn off refresh during bulk loading (refresh_interval: -1) and turn it back on at the end, so Elasticsearch doesn't refresh after every batch.

The current script deletes the old index and creates a new one, so search is empty while it runs. For a system that has to stay up, the standard approach is to build the new index alongside and switch an alias to it in one operation: users always search through the alias and never see a half-built index. And if the index must never drift from the database, the next steps are the outbox pattern or CDC (reading the binlog), at the cost of one more component to run.

Summary

  • Fold diacritics with asciifolding, but put list-based filters before it.
  • Use n-grams for partial matching, with a separate search_analyzer, on a few short fields only.
  • Combine cross_fields, phrase_prefix and n-grams to cover different ways of typing.
  • Page deeply with search_after and a tiebreaker.
  • Treat the index as a rebuildable copy, and check analyzers with _analyze instead of guessing.

There are many other approaches to Vietnamese search, especially for word segmentation. If you've found one that works well, I'd love to compare notes. My email is on the home page.