Welcome to WebmasterWorld Guest from 54.145.104.199

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Unknown robot eating up bandwidth

     
8:57 am on Sep 1, 2009 (gmt 0)

5+ Year Member



Hello,

I am watching my AWStats data and following are eating up my bandwidth heavily. What are those? Can I control them eating up unnecessary bandwidth?

Unknown robot (identified by empty user agent string)
Unknown robot (identified by 'robot')
Unknown robot (identified by 'spider')

Thanks in advance for any help.

4:57 am on Sep 4, 2009 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Do you have a robots.txt file? Whitelist the bots you want. Nuke (.htaccess) the rest. Cleans up your logs very nicely!
2:17 am on Sep 8, 2009 (gmt 0)

5+ Year Member



Tangor is right. In my logs - I saw one of Amazon's bots on my site all the time and I "disallowed" this certain bot in my robots.txt file.
7:34 pm on Sep 13, 2009 (gmt 0)

5+ Year Member



There is a possibility that the bots are not robots.txt friendly. AFTER you try robots.txt restrictions if the problem persists you may try to identify and restrict the IPs. Tangors solution is a nice balance of the two and white-listing is the ideal way of controlling crawler access.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month