KillDuplicates

This parameter determines how IDOL Server handles duplicate documents. It allows you to prevent the same document or document content from being stored in IDOL Server more than once.

Use one of the following options:

NONE Duplicate documents are allowed in IDOL Server and are not replaced or deleted.
REFERENCE If the document being indexed has the same DREREFERENCE field value as a document that already exists in the IDOL Server, IDOL Server deletes the existing document and replaces it with the new document.
REFERENCEMATCHN

If the content of the document being indexed is more than N percent similar to the content of a document that already exists in the IDOL Server database, IDOL Server deletes the existing document and replaces it with the new document. IDOL Server determines the similarity by comparing the content of the SourceType fields in the document, or the Index fields if no SourceType fields are configured.

NOTE:

This method can deduplicate only on documents that are already synced in the IDOL Server index. It cannot deduplicate similar documents in the same index job.

FieldName

If the document being indexed contains a FieldName reference field with the same value as the FieldName reference field in a document that already exists in IDOL Server, IDOL Server deletes the existing document and replaces it with the new document.

To specify multiple Reference fields, separate the fields with a plus sign (+) or a space. IDOL Server deletes documents that contain any of the specified fields with identical content. You must percent-encode any punctuation characters in the field name.

NOTE:

Fields are identified as Reference fields by field processes in the IDOL Server configuration file. If you use a FieldName Reference field to eliminate duplicate documents, IDOL Server automatically reads any fields listed alongside this field for the PropertyFieldCSVs parameter in the field process, and also uses these fields to eliminate duplicate documents. If you want to define multiple Reference fields but do not want them all to be used for document elimination, you must set up multiple field processes (refer to the IDOL Server Administration Guide).

ReferenceField,GREATER:VersionField

If the document being indexed contains a ReferenceField reference field with the same value as the ReferenceField reference field in a document that already exists in IDOL Server, IDOL Server checks the VersionField for the documents, and keeps the copy of the document with the highest value in the VersionField. For XML documents, you must fully qualify the path of the XML field that you want to use as the version field (you cannot use wildcard values).

VersionField must contain a positive integer value, but you do not need to configure it as a numeric field. If only one of the incoming and current documents has a valid value in the VersionField, IDOL Server keeps the version with a valid VersionField. When both documents have the same VersionField, IDOL Server keeps the existing document.

NOTE:

When you index IDX documents, for the version comparison to work correctly, the value in the field that you use as the VersionField must be listed in quotation marks (""). That is, the field must have the following format in the IDX:

#DREFIELD MyField="N"

IDOL Server treats existing documents with a missing or non-numeric value in the VersionField as having a version number of negative infinity. It treats a new document with a missing or non-numeric value in the VersionField as having a version number of 0.

You can postfix any of these options with =2, to apply the KillDuplicates process to all IDOL Server databases (rather than only to the database into which the current IDX or XML file is being indexed).

You can override KillDuplicates for individual DREADD or DREADDDATA index actions by using the KillDuplicates index action parameter.

Type: String
Default: REFERENCE
Required: No
Configuration Section: Server
Example: KillDuplicates=REFERENCE
In this example, if the document being indexed has the same DREREFERENCE as a document that already exists in IDOL Content Component, IDOL Content Component deletes the existing document and replaces it with the new document.
See Also: KillDuplicatesChecksumField

_HP_HTML5_bannerTitle.htm