homepage Welcome to WebmasterWorld Guest from 54.211.95.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

    
ArchitextSpider going mad?
robots.txt spidered 200 times in 2 hours
starec




msg:467946
 12:46 pm on Jun 6, 2001 (gmt 0)

Yes, I just checked the logs. I thought it was dead, but suddenly the excite spider is back, but focusing on the robots.txt file exclusively (only 5 other url's spidered). If it continues like this, I will just bann him for bad behavior:)

 

littleman




msg:467947
 1:49 am on Jun 10, 2001 (gmt 0)

It usually will come in several days later and do a crawl of your site. At least that is what I've seen.

gekoviola




msg:467948
 5:22 am on Jun 13, 2001 (gmt 0)

After a quick look at my logs, everyday I see that the ArchiText Spider IS going mad, he make thousands of requests for non-existant urls.
I've noticed this behavior happened also 2 years ago.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved