Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Unknown robot eating up bandwidth



8:57 am on Sep 1, 2009 (gmt 0)

5+ Year Member


I am watching my AWStats data and following are eating up my bandwidth heavily. What are those? Can I control them eating up unnecessary bandwidth?

Unknown robot (identified by empty user agent string)
Unknown robot (identified by 'robot')
Unknown robot (identified by 'spider')

Thanks in advance for any help.


4:57 am on Sep 4, 2009 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Do you have a robots.txt file? Whitelist the bots you want. Nuke (.htaccess) the rest. Cleans up your logs very nicely!


2:17 am on Sep 8, 2009 (gmt 0)

5+ Year Member

Tangor is right. In my logs - I saw one of Amazon's bots on my site all the time and I "disallowed" this certain bot in my robots.txt file.


7:34 pm on Sep 13, 2009 (gmt 0)

5+ Year Member

There is a possibility that the bots are not robots.txt friendly. AFTER you try robots.txt restrictions if the problem persists you may try to identify and restrict the IPs. Tangors solution is a nice balance of the two and white-listing is the ideal way of controlling crawler access.

Featured Threads

Hot Threads This Week

Hot Threads This Month