Welcome to WebmasterWorld Guest from 50.19.34.234

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Unknown robot eating up bandwidth

     

seomasters

8:57 am on Sep 1, 2009 (gmt 0)

5+ Year Member



Hello,

I am watching my AWStats data and following are eating up my bandwidth heavily. What are those? Can I control them eating up unnecessary bandwidth?

Unknown robot (identified by empty user agent string)
Unknown robot (identified by 'robot')
Unknown robot (identified by 'spider')

Thanks in advance for any help.

tangor

4:57 am on Sep 4, 2009 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Do you have a robots.txt file? Whitelist the bots you want. Nuke (.htaccess) the rest. Cleans up your logs very nicely!

bigcat1967

2:17 am on Sep 8, 2009 (gmt 0)

5+ Year Member



Tangor is right. In my logs - I saw one of Amazon's bots on my site all the time and I "disallowed" this certain bot in my robots.txt file.

theantagonizers

7:34 pm on Sep 13, 2009 (gmt 0)

5+ Year Member



There is a possibility that the bots are not robots.txt friendly. AFTER you try robots.txt restrictions if the problem persists you may try to identify and restrict the IPs. Tangors solution is a nice balance of the two and white-listing is the ideal way of controlling crawler access.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month