FollowRobotMeta
 
Type

Boolean

Default

false

Allowed range

Minimum:  

Maximum:  

Recommended range

Minimum:  

Maximum:  

Required

no

Configuration section

[Default] and [SpiderJob]

Description

If enabled, the connector follows the Robot META tags. If FollowRobotProtocol is enabled, use FollowRobotMeta to determine whether the connector follows the Robot META tags.

The Robot META tag, placed in the HTML <HEAD> section of a page, can specify whether a page should be indexed or links on the page should be followed.

Example

FollowRobotMeta=true

See also

FollowRobotProtocol