Welcome to WebmasterWorld Guest from 54.197.66.254

Forum Moderators: bakedjake

Message Too Old, No Replies

Chinanet bot not playing nice

ignores the robots.text file, needs to be whipped

     
4:41 am on Sep 8, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:May 27, 2005
posts:614
votes: 0


Am I the only one having trouble with the CHINANET bot crawler?

I have updated my robots file to cut out the non text pages, yet this bot doesn't obey it. Is there something wrong with my file? Or is there some way to beat the bot into submission?

User-agent: *
User-agent: Mediapartners-Google*
User-Agent: Googlebot

Disallow:/post.php?
Disallow:/member.php?
Disallow:/misc.php?

10:38 pm on Sept 21, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 26, 2004
posts:1392
votes: 0


I put an IP block on all of china...
10:44 pm on Sept 21, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:May 27, 2005
posts:614
votes: 0


I wondered about that, I also wonder what sort of legitmate traffic would be lost doing that.

BTW I fixed the robots text above, got rid of the google entries and the blank line.

Makes no difference to the bots though

10:49 pm on Sept 21, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Hi,

Block by IP address or (better) by behaviour.

robots.txt is voluntary, and is ignored by bad bots, in the same way as a "please do not burgle this house" notice would be treated by a thief...

Rgds

Damon

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members