Forum Moderators: open
It does request robots.txt again before fetching each page, but various spiders/bots have that bad habbit.
So far it only seems to be requesting pages that are externally linked to, it hasn't done a deep crawl of my site as yet.
Apart from requesting robots.txt was it well behaved?
Only grabbed robots and main page of my smaller site.
Don