Tokenization¶

The indexing and search quality of a text field depends on the tokenizer and dictionary you choose. AkasicDB text search lets you set an analyzer per field, and Korean uses morphological analysis based on lindera and ko-dic. Tokenizer and dictionary configuration is part of the index definition (field mapping); there is no separate registration SQL.

The examples on this page reuse the same index name as Indexing, docs_texton_idx, so each one is preceded by DROP INDEX IF EXISTS — with one exception, explained under User Dictionary below.

Standard Analyzer¶

A text field with no analyzer in its mapping is indexed with the language-neutral standard analyzer. standard splits tokens on whitespace and punctuation and normalizes to lowercase. It suits English text, code, and identifier fields that do not need morphological analysis.

Korean Morphological Analysis (lindera ko-dic)¶

Korean fields are indexed and searched with the lindera tokenizer and the ko-dic dictionary. For example, 국제공항에서 splits into 국제 / 공항 / 에서, so it matches regardless of word boundaries.

DROP INDEX IF EXISTS docs_texton_idx;

CREATE INDEX docs_texton_idx
  ON docs
  USING texton ((docs.*))
  WITH (mapping='{"fields":{
    "body":{"type":"text","analyzer":{"tokenizer":{"lindera":{"dictionary":"ko-dic"}}}}
  }}');

The ko-dic main dictionary is embedded in the extension binary, so no separate dictionary deployment is required.

Per-Field Dictionaries¶

Within a single index, each field can use a different tokenizer and language dictionary. Specify an analyzer per text field.

DROP INDEX IF EXISTS docs_texton_idx;

CREATE INDEX docs_texton_idx
  ON docs
  USING texton ((docs.*))
  WITH (mapping='{"fields":{
    "title":{"type":"text","analyzer":{"tokenizer":{"lindera":{"dictionary":"ko-dic"}}}},
    "body_en":{"type":"text"}
  }}');

Note

Opt-in CJK dictionaries such as ipadic / unidic / cc-cedict / jieba are available only in an extension built (via build feature) to include them. Which dictionaries are included depends on the deployment build.

User Dictionaries¶

To make domain terms or new words recognized as index units, specify a user dictionary. A user dictionary is a binary compiled with lindera-cli, declared per field via the user_dictionary path. AkasicDB text search references only the binary path, not the dictionary contents.

This example alone uses a separate index name. CREATE INDEX fails if no dictionary binary exists at the declared path, and reusing the shared name would take docs_texton_idx — built by the earlier examples — down with it.

CREATE INDEX docs_userdict_texton_idx
  ON docs
  USING texton ((docs.*))
  WITH (mapping='{"fields":{
    "body":{"type":"text","analyzer":{"tokenizer":{"lindera":{
      "dictionary":"ko-dic","user_dictionary":"/var/lib/texton/dict/ko.bin"}}}}
  }}');

Dictionary Path Allowlist¶

To restrict reads from arbitrary paths, set the dictionary path allowlist GUC. Once set, only paths under this directory are allowed as user dictionaries, and only a superuser can change the value.

SET texton.dictionary_base_dir = '/var/lib/texton/dict';

Applying Dictionary Updates¶

A field mapping cannot be changed in place after the index is created, so changing a dictionary does not trigger an automatic reindex. Recompile the dictionary binary at the same path and run REINDEX to apply the new dictionary.

-- After recompiling the user dictionary binary at the same location:
REINDEX INDEX docs_texton_idx;

To change the dictionary path itself, edit user_dictionary in the mapping and recreate the index.