Querylight TS Demo

Tokenizer

Tokenizer that turns raw text into token strings.

Back to docs search

Reference Entry

Tokenizer

API Reference · foundation · order 5

Tokenizer that turns raw text into token strings.

Relevant APIs

Tokenizer

Tags

api-reference interfaces

Tokenizer

Tokenizer that turns raw text into token strings.

Declaration

interface Tokenizer

Members

tokenize

```ts tokenize(text: string): string[] ``` Parameters: - `text: string`