If enabled, the connector follows Robot protocol. Use the parameter FollowRobotMeta to determine whether the connector follows the Robot META tags.

If the robot’s protocol disallows your spider access to the Web site or to certain areas of the Web Site, disable this parameter so your spider ignores the robot’s protocol.

Type: Boolean
Default: True
Required: No
Configuration Section: TaskName or Default
Example: FollowRobotProtocol=False
See Also: