FollowRobotProtocol
 
Type

Boolean

Default

true

Allowed range

Minimum:  

Maximum:  

Recommended range

Minimum:  

Maximum:  

Required

no

Configuration section

[Default] and [SpiderJob]

Description

If enabled, the connector follows Robot protocol. Use the parameter FollowRobotMeta to determine whether the connector follows the Robot META tags.

If the robots protocol disallows your spider access to the Web site or to certain areas of the Web site, disable this parameter so your spider ignores the robots protocol.

Example

FollowRobotProtocol=false

See also

FollowRobotMeta