Querylight TS Demo

Analyzers, Tokenizers, and Filters

Normalization is explicit, so you can tailor indexing and querying behavior per field.

Back to docs search

Reference Entry

Analyzers, Tokenizers, and Filters

Analysis · foundation · order 10

Normalization is explicit, so you can tailor indexing and querying behavior per field.

Relevant APIs

Analyzer KeywordTokenizer NgramTokenFilter EdgeNgramsTokenFilter

Tags

analysis analyzer tokenizer ngrams normalization

Analyzers, Tokenizers, and Filters

The analysis pipeline lets you choose how text is normalized before it is indexed or queried.

Default behavior

The default Analyzer applies:

  • LowerCaseTextFilter
  • ElisionTextFilter
  • InterpunctionTextFilter
  • SplittingTokenizer

Exact keyword fields

Use KeywordTokenizer when a field should stay as a single token, for example tags, categories, and IDs.

import { Analyzer, KeywordTokenizer, TextFieldIndex } from "@tryformation/querylight-ts";

const keywordAnalyzer = new Analyzer([], new KeywordTokenizer());
const tags = new TextFieldIndex(keywordAnalyzer, keywordAnalyzer);

Fuzzy and autocomplete-oriented analyzers

Use NgramTokenFilter for typo-tolerant matching and EdgeNgramsTokenFilter for suggestion-style prefix indexing.

import { Analyzer, EdgeNgramsTokenFilter, NgramTokenFilter } from "@tryformation/querylight-ts";

const fuzzyAnalyzer = new Analyzer(undefined, undefined, [new NgramTokenFilter(3)]);
const suggestAnalyzer = new Analyzer(undefined, undefined, [new EdgeNgramsTokenFilter(2, 6)]);

Learn more