Language Identification

Because the multilanguage locale supports many languages, it generally must be able to identify the language of each document to be indexed, so that the proper language-specific processing is applied.

The Verity language-identification filter is used to detect the language of incoming documents before they are indexed. The filter makes use of the document’s encoding and language features to make the identification.