IndexMode
 
Type

String

Default

REFERENCE

Allowed range

Minimum:  

Maximum:  

Recommended range

Minimum:  

Maximum:  

Required

no

Configuration section

[Default] and [SpiderJob]

Description

Prevents documents or document content from being stored in IDOL server more than once.

Enter one of the following options to determine how IDOL server handles duplicate content:

NONE
Documents in IDOL server are never replaced with new documents.

REFERENCE
If the connector downloads a document with the same DREREFERENCE field value as a document already stored in IDOL server, the document contained in IDOL server is replaced with the new document.

REFERENCEMATCH<N>
If the connector downloads a document in which <N> percent or more of the content is similar to the content of a document already stored in IDOL server, the document contained in IDOL server is replaced with the new document. This parameter only replaces the document stored in the IDOL server into which the connector is indexing.

For example, if you set IndexMode to REFERENCEMATCH80, and the connector downloads a document in which 80 percent or more of the content is similar to a document already stored in IDOL server, the document in IDOL server is replaced by the new document.

<FieldName>
If the connector downloads a document containing a <FieldName> Reference field with the same content as the <FieldName> Reference field in a document already stored in IDOL server, the document contained in IDOL server is deleted and replaced with the new document.

You can specify multiple Reference fields, in which case, IDOL server deletes documents containing any of the specified fields with identical content. To specify multiple Reference fields, separate them with a plus symbol, a space, or an underscore symbol.

Note: Fields are identified as Reference fields through field processes in the IDOL server configuration file. If you use a <FieldName> Reference field to eliminate duplicate documents, IDOL server automatically reads any fields listed alongside this field for the PropertyFieldCSVs parameter in the field process, and also uses these fields to eliminate duplicate documents. If you want to define multiple Reference fields but do not want them all to be used for document elimination, you must set up multiple field processes.

If you postfix any of these options with =2, the KillDuplicates process is applied to all IDOL server databases (rather than just the database into which the current IDX or XML file is indexed).

If you do not set IndexMode, it defaults to the value you specify in KillDuplicates in the IDOL server configuration file's [Server] section.

Example

IndexMode=REFERENCEMATCH80

See also