Welcome to WebmasterWorld Guest from 54.161.99.20

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Unknown robot eating up bandwidth

     
8:57 am on Sep 1, 2009 (gmt 0)

New User

5+ Year Member

joined:Sept 1, 2009
posts: 1
votes: 0


Hello,

I am watching my AWStats data and following are eating up my bandwidth heavily. What are those? Can I control them eating up unnecessary bandwidth?

Unknown robot (identified by empty user agent string)
Unknown robot (identified by 'robot')
Unknown robot (identified by 'spider')

Thanks in advance for any help.

4:57 am on Sept 4, 2009 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6897
votes: 377


Do you have a robots.txt file? Whitelist the bots you want. Nuke (.htaccess) the rest. Cleans up your logs very nicely!
2:17 am on Sept 8, 2009 (gmt 0)

New User

5+ Year Member

joined:Sept 7, 2009
posts:15
votes: 0


Tangor is right. In my logs - I saw one of Amazon's bots on my site all the time and I "disallowed" this certain bot in my robots.txt file.
7:34 pm on Sept 13, 2009 (gmt 0)

New User

5+ Year Member

joined:Sept 13, 2009
posts:3
votes: 0


There is a possibility that the bots are not robots.txt friendly. AFTER you try robots.txt restrictions if the problem persists you may try to identify and restrict the IPs. Tangors solution is a nice balance of the two and white-listing is the ideal way of controlling crawler access.