homepage Welcome to WebmasterWorld Guest from 50.17.167.146
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

    
Chinanet bot not playing nice
ignores the robots.text file, needs to be whipped
netchicken1

5+ Year Member



 
Msg#: 3075758 posted 4:41 am on Sep 8, 2006 (gmt 0)

Am I the only one having trouble with the CHINANET bot crawler?

I have updated my robots file to cut out the non text pages, yet this bot doesn't obey it. Is there something wrong with my file? Or is there some way to beat the bot into submission?

User-agent: *
User-agent: Mediapartners-Google*
User-Agent: Googlebot

Disallow:/post.php?
Disallow:/member.php?
Disallow:/misc.php?

 

Wlauzon

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3075758 posted 10:38 pm on Sep 21, 2006 (gmt 0)

I put an IP block on all of china...

netchicken1

5+ Year Member



 
Msg#: 3075758 posted 10:44 pm on Sep 21, 2006 (gmt 0)

I wondered about that, I also wonder what sort of legitmate traffic would be lost doing that.

BTW I fixed the robots text above, got rid of the google entries and the blank line.

Makes no difference to the bots though

DamonHD

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3075758 posted 10:49 pm on Sep 21, 2006 (gmt 0)

Hi,

Block by IP address or (better) by behaviour.

robots.txt is voluntary, and is ignored by bad bots, in the same way as a "please do not burgle this house" notice would be treated by a thief...

Rgds

Damon

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved