Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
ArchitextSpider going mad? robots.txt spidered 200 times in 2 hours starec msg:467946 12:46 pm on Jun 6, 2001 (gmt 0) Yes, I just checked the logs. I thought it was dead, but suddenly the excite spider is back, but focusing on the robots.txt file exclusively (only 5 other url's spidered). If it continues like this, I will just bann him for bad behavior:)
littleman msg:467947 1:49 am on Jun 10, 2001 (gmt 0)
It usually will come in several days later and do a crawl of your site. At least that is what I've seen. gekoviola msg:467948 5:22 am on Jun 13, 2001 (gmt 0)
After a quick look at my logs, everyday I see that the ArchiText Spider IS going mad, he make thousands of requests for non-existant urls. I've noticed this behavior happened also 2 years ago.