Parameter and Command Reference > Connector Framework Server Parameters > DRERemoveDuplicates 

DRERemoveDuplicates 

Topics in this Section

Prevents documents or document content from being stored in IDOL server more than once. If you do not set DRERemoveDuplicates, it defaults to the value specified in KillDuplicates in the IDOL server configuration file's [Server] section.
 
DRERemoveDuplicates=REFERENCEMATCH80
Enter one of the following options to determine how IDOL server handles duplicate content:
 
If the connector downloads a document containing a FieldName Reference field with the same content as the FieldName Reference field in a document already stored in IDOL server, the document contained in IDOL server is deleted and replaced with the new document.
You can specify multiple Reference fields, in which case, IDOL server deletes documents containing any of the specified fields with identical content. To specify multiple Reference fields, separate them with a plus symbol, a space, or an underscore symbol.
Fields are identified as Reference fields through field processes in the IDOL server configuration file. If you use a FieldName Reference field to eliminate duplicate documents, IDOL server automatically reads any fields listed alongside this field for the PropertyFieldCSVs parameter in the field process, and also uses these fields to eliminate duplicate documents. If you want to define multiple Reference fields but do not want them all to be used for document elimination, you must set up multiple field processes.
If the connector downloads a document with the same DREREFERENCE field value as a document already stored in IDOL server, the document contained in IDOL server is replaced with the new document.
If the connector downloads a document in which N percent or more of the content is similar to the content of a document already stored in IDOL server, the document contained in IDOL server is replaced by the new document. This parameter only replaces the document stored in the IDOL server into which the connector is indexing.
For example, if you set DRERemoveDuplicates to REFERENCEMATCH80, and the connector downloads a document in which 80 percent or more of the content is similar to a document already stored in IDOL server, the document in IDOL server is replaced by the new document.
If the connector downloads a document in which N percent or more of the content is similar to the content of a document already stored in IDOL server, the document contained in IDOL server is replaced with the new document. Unlike ReferenceMatchN, Reference2MatchN checks across all available IDOL servers.
For example, if you set DRERemoveDuplicates to Reference2Match80, and the connector downloads a document in which 80 percent or more of the content is similar to any document stored in any IDOL server databases, all instances of the document in any IDOL server database are replaced by the new document.
When you specify an DRERemoveDuplicates option, note the following:
*
If you postfix any of these options with =2, the KillDuplicates process is applied to all IDOL server databases (rather than just the database into which the current IDX or XML file is indexed).
*
If you are using the connector in a system that contains a DIH, and the DIH’s DistributeByReference option is set to true, then the only valid settings for DRERemoveDuplicates are None or Reference.