Welcome to WebmasterWorld Guest from 54.144.79.200

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

216.32.64.10 ?

     

mark roach

2:25 pm on Sep 21, 2000 (gmt 0)

10+ Year Member



Any ideas where 216.32.64.10 is from ?

It took about 400 pages from 200 domains yesterday.

20 hours prior to that Architext took my robots.txt (and nothing else) on those domains, Coincidence ?

mark roach

3:42 pm on Sep 21, 2000 (gmt 0)

10+ Year Member



Just found it on another domain of mine. Some of the pages it is taking have question marks in the urls.

Last month lycos.co.uk indexed some pages containing question marks in the URLs. Direct hit grabber came to this site for the 1st time ever yesterday and also took pages with question marks in the URLs. I think I need a robots.txt on my cgi directory.

PeteU

8:09 pm on Sep 21, 2000 (gmt 0)

10+ Year Member



216.32.64.10 is [cyveillance.com...]
waste of bandwidth

NFFC

8:23 pm on Sep 21, 2000 (gmt 0)

WebmasterWorld Senior Member nffc is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>We do honor robots.txt files by following rules designated for all user agents (the wildcard character '*').

Does this mean that they cannot be excluded, except by excluding all spiders?

littleman

8:23 pm on Sep 21, 2000 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



deja vu
 

Featured Threads

Hot Threads This Week

Hot Threads This Month