Welcome to WebmasterWorld Guest from

Forum Moderators: bakedjake

Message Too Old, No Replies

ArchitextSpider going mad?

robots.txt spidered 200 times in 2 hours

12:46 pm on Jun 6, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 17, 2001
votes: 0

Yes, I just checked the logs. I thought it was dead, but suddenly the excite spider is back, but focusing on the robots.txt file exclusively (only 5 other url's spidered). If it continues like this, I will just bann him for bad behavior:)
1:49 am on June 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
votes: 0

It usually will come in several days later and do a crawl of your site. At least that is what I've seen.


5:22 am on June 13, 2001 (gmt 0)

Inactive Member
Account Expired


After a quick look at my logs, everyday I see that the ArchiText Spider IS going mad, he make thousands of requests for non-existant urls.
I've noticed this behavior happened also 2 years ago.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members