MaxSpiderThreads
 
Type

Integer

Default

1

Allowed range

Minimum: 1

Maximum: ActiveSpiderJobs x (1 + MaxSpiderThreads)

Recommended range

Minimum: 1

Maximum: (1 + CPUs)

Required

no

Configuration section

[Default] and [JobName]

Description

This parameter specifies the maximum number of threads used by an individual job to perform page requests. During spidering, it builds up a list of URLs to retrieve. When a thread becomes available, it is assigned a URL to process and performs the download. The thread becomes available for the next URL once the download and processing are complete.

It may be advisable to increase the MaxSpiderThreads value if the server response time for requests is longer than the time spent by the connector processing the page (that is, if the threads are frequently waiting around for the pages to be returned).

Before choosing values, the load on the HTTP server should also be considered - more threads will result in requests being sent more frequently.

Also, the two settings (ActiveSpiderJobs and MaxSpiderThreads) should probably be balanced - if one is large, the other should be small.

Example

MaxSpiderThreads=1

See also

ActiveSpiderJobs