Welcome to WebmasterWorld Guest from 54.162.211.233

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

216.32.64.10 ?

     

mark roach

2:25 pm on Sep 21, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


Any ideas where 216.32.64.10 is from ?

It took about 400 pages from 200 domains yesterday.

20 hours prior to that Architext took my robots.txt (and nothing else) on those domains, Coincidence ?

mark roach

3:42 pm on Sept 21, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


Just found it on another domain of mine. Some of the pages it is taking have question marks in the urls.

Last month lycos.co.uk indexed some pages containing question marks in the URLs. Direct hit grabber came to this site for the 1st time ever yesterday and also took pages with question marks in the URLs. I think I need a robots.txt on my cgi directory.

8:09 pm on Sept 21, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:July 28, 2000
posts:134
votes: 0


216.32.64.10 is [cyveillance.com...]
waste of bandwidth
8:23 pm on Sept 21, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member nffc is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 22, 2000
posts:3604
votes: 0


>We do honor robots.txt files by following rules designated for all user agents (the wildcard character '*').

Does this mean that they cannot be excluded, except by excluding all spiders?

8:23 pm on Sept 21, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


deja vu
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members