| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Chinanet bot not playing nice|
ignores the robots.text file, needs to be whipped
| 4:41 am on Sep 8, 2006 (gmt 0)|
Am I the only one having trouble with the CHINANET bot crawler?
I have updated my robots file to cut out the non text pages, yet this bot doesn't obey it. Is there something wrong with my file? Or is there some way to beat the bot into submission?
| 10:38 pm on Sep 21, 2006 (gmt 0)|
I put an IP block on all of china...
| 10:44 pm on Sep 21, 2006 (gmt 0)|
I wondered about that, I also wonder what sort of legitmate traffic would be lost doing that.
BTW I fixed the robots text above, got rid of the google entries and the blank line.
Makes no difference to the bots though
| 10:49 pm on Sep 21, 2006 (gmt 0)|
Block by IP address or (better) by behaviour.
robots.txt is voluntary, and is ignored by bad bots, in the same way as a "please do not burgle this house" notice would be treated by a thief...
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved