The MaxSpiderThreads parameter specifies the maximum number of threads used by a single fetch task to perform page requests. During spidering, the connector builds a list of URLs to retrieve. When a thread becomes available, it is assigned a URL to process and downloads the page. The thread becomes available for the next URL when the download and processing are complete.

If the server’s response time is longer than the time spent by the connector processing the page (that is, if threads are frequently waiting for pages to be returned), it may be advisable to increase the value of this parameter.

You should also consider the load on the HTTP server - more threads will result in requests being sent more frequently.

Type: Integer
Default: 1
Required: No
Configuration Section: TaskName or Default
Example: MaxSpiderThreads=2
See Also: