TokenizationDictionary

DictionaryRepository.TokenizationDictionary property

Gets the dictionary of characters that defines whether a token is considered as a valid word or a separator when using external tokenization tools.

public Alphabet TokenizationDictionary { get; }

Property Value

The dictionary of characters.

See Also