Spider job configuration section

You must create an [SpiderJob] section for each spider job listed in the [Spider] section. A spider job’s section contains parameters that only apply to that job. The parameters configured in a spider job section override the parameters configured in the [Default] section for that job. In addition to the connector configuration parameters, you can specify Import Module parameters in the [Default] or [Spider] sections.

For example:

[MyFirstSpiderJob]
URL=http://my.site.com/
Directory=mysite
Logfile=mysite.log

[MySecondSpiderJob]
URL=http://my.site2.com/
Directory=mysite2
Logfile=mysite2.log
FollowRedirect=false
StayOnSite=false
MinPageSize=4090
MaxPageSize=163840
MaxLinksPerPage=50
PageTimeout=75